(A simple bandit model) Suppose there are two projects available for selection in each of three periods. Project 1 yields a reward of one unit and always occupies state s and the other, project 2, occupies either state f or state u. When project 2 is selected, and it occupies state u, it yields a reward of 2 and moves to state f at the next decision epoch with probability 0.5. When selected in state t, it yields a reward of 0 and moves to state u at the next decision epoch with probability 1. Assume a terminal reward of 0, and that project 2 does not change state when it is not selected.

Using backward induction determine a strategy that maximizes the expected total reward.