It's also possible to just make NPCs behave rationally. Players, however....
Disengaging, withdrawing, ought to be a decision made when it's clear you can't achieve your goals without unacceptable losses. The problem in an RPG context is that losses are meaningless and the basic game paradigm is "kill the monster to get the loot." There is no reward in withdrawing: no win, no loot. And if your piece of paper dies, you make a new piece of paper who says, "whose is all this cool gear?" and then takes it over.
I think the only solution is to subject players to electric shock when their characters are hit, with a baseball bat to the kneecap if your character dies. But I find this discourages participation for some reason, not sure why.
It's tough to make dice checks model that kind of decision, because it's not really a matter of CUF. This is just an RPG problem, which like metagaming is really about norms around your table.