Three kinds of preference
The decision-theoretic concept of preference is linked to the concepts of subjective probability and utility by the expected utility principle:
(EUP) A rational agent prefers X to Y iff the expected utility of X exceeds the expected utility of Y.
Economists usually take preference to be the more basic concept and interpret the EUP as an implicit definition of the agent's utilities (and sometimes also her probabilities).
But there two kinds of expected utility, one causal, one evidential. Which is involved in EUP?
There's no need to choose. If we read 'expected utility' in EUP causally, we get a causal notion of preference; if we read it evidentially, we get an evidential notion. Both are useful and intuitive.
Informally, you causally prefer X over Y if you judge that it would be better if X were the case than if Y were the case; you evidentially prefer X over Y if you judge that it is better if X is the case than if Y is the case.
The reason why economists like to take preference as basic is that it is supposedly revealed by choices. Since (from the perspective of Causal Decision Theory) it is causal preference that tracks choiceworthiness, this suggests that causal preference is basic; evidential preference should be somehow derived. Indeed, if causal preference satisfies the usual axioms, then it is possible to derive the corresponding evidential preference relation. (The two concepts of preference are not completely independent. In particular, they coincide for maximially specific propositions, and for propositions that are specific enough to settle everything that matters to an agent.)
Now here's a complication. Having a closer look at the connection between (causal) preference and choice suggests that there is a third kind of preference, one that's usually neglected. If anything, it's this third kind of preference that's revealed by choice.
Consider the following matrix.
| PA | PB | PC | |
|---|---|---|---|
| A | 5 | 11 | 0 |
| B | 4 | 10 | 10 |
| C | 5 | 10 | 9 |
Here, PA (PB, PC) is the proposition that a highly reliable predictor predicts A (B, C). Assume there is no causal influence from the rows (A, B, C) to the columns (PA, PB, PC). Then the causal expected utility of B is greater than the causal expected utility of A iff the probability of PC is greater than 1/11. And arguably a rational agent could assign probability greater than 1/11 to PC.
On the other hand, a rational agent who is presented with a choice represented by the above matrix would have to choose A over B and C. For A is the only stable choice. If the agent starts in a state of uncertainty in which PC has probability greater than 1/11 and so the agent is inclined towards option B or C, then the dynamics of rational deliberation should change her beliefs, eventually making her certain of PA, at which point A has greatest causal expected utility.
So there is a sense in which any rational agent prefers A to B and C: that's what she would choose. This is true even for agents who assign greater (causal) expected utility to B than to A and thus causally prefer B to A.
So choices don't directly reveal causal preference. They at most reveal the third kind of preference.
How is the third kind of preference related to the other two? Are probabilities and utilities determined by the third kind of preference? If so, how? I don't know. As Eells and Harper 1991 show (in effect), the third kind of preference does not satisfy some popular axioms of decision theory. That's not surprising, for the third kind of preference tracks expected utility after a change to the probability function, and the relevant change varies with the set of options we consider available.