@Bma It is indeed true that every polytope can be represented as a convex hull and equivalently as the intersection of a finite number of closed halfspaces. Perhaps this is a good way to prove the result. Let me think about it. Thanks for your inputs.

@MichaelGreinecker There is a separate operator $A$, which is defined as $Av:=\sup_{\delta \in \Delta}H_\delta v$. This theorem, however, does not talk about an optimal policy or $A$. It talks about any policy even if it is suboptimal. You can see the precise statement of the theorem at imgur.com/XGL8OAv

@MichaelGreinecker Since you focus on game theory, I can add that Porteus specifies in his book that this theorem was stated/proven by Shapley in his 1953 paper on "Stochastic Games". He further states that he himself proved this in Porteus' 1982 paper "Conditions for characterizing the structure of optimal strategies in infinite horizon dynamic programs" that appearead in JOTA. I have both these paper, but this theorem does not seem to be stated / proven exactly in these works in my reading.

@MichaelGreinecker I described operator $H$ in greater clarify to specify the discount rate and the payoffs.