This question has already been posed in Cross Validated without receiving a correct formal answer, so I reformulate it here to gain attention of mathematicians. I am referring to chapter 3 of Sutton and barto book "Reinforcement learning. An introduction" available here: 
[1]

Let us assume that we have three finite sets, $S$ the set of *states*, $A$ the set of *actions*, and $R\subset \mathbb{R}$ the set of *rewards*.
Let $p : S \times R  \times S \times A \rightarrow [0,1]$ be a function such that

$$\sum_{s'\in S} \sum_{r \in R} p(s',r,s,a)=1$$ for each $s\in S, a \in a$, so that it defines a joint discrete probability distribution on $S\times R$ for every choice of  $s\in S, a \in a$, and we denote it then, with a slight abuse of notation, as $p(s',r|s,a)$. From a probabilistic point of view, we can think that if at time $t$ we are in state $S_t=s$ and we perform an action $A_t=a \in A$ then the probability of the two random variables $S_{t+1},R_{t+1}$ representing respectively next state and the reward obtained is given exactly by $p$:

$$\mathbb{P}\{S_{t+1}=s',R_{t+1}=r|S_{t}=s,A_t=a\} = p(s',r|s,a)$$

and so that, for example, given present state $s$ and action $a$,  the expected value of immediate reward is $r(s,a)=\sum_{r \in R} r\sum_{s'\in S}p(s',r|s,a)$, and the state transition probability (again with a slight abuse of notation) is $p(s'|s,a)=\sum_{r\in R}p(s',r|s,a)$.

Let us define $G_t$ as the random variable representing the sum of discounted future rewards obtainable from time $t$
$$G_t=\sum_{k=0}^{+\infty} \gamma^k R_{t+1+k}$$
that it is immediate to prove having this representation
$$G_t= R_{t+1}+\gamma G_{t+1}$$

We define a *policy* $\pi$ as a function $\pi: A \times S \rightarrow [0,1]$ such that for every $s \in S$ $\sum_{a \in A}\pi(a,s)=1$, so that it defines, for every choice of $s \in S$ a probability distribution over $A$, and we denote it with $\pi(a|s)$. We also define the *state-value function* $v_{\pi}$ for a policy $\pi$ as

$$v_{\pi}(s)=\mathbb{E}_{\pi}[G_t|S_t=s]$$

that is the expected value cof $G_t$ conditioned on being in state $S_t=s$ and using policy $\pi$ to select (randomly) actions at present time and also in future time steps, and analogously the *action-value* function $q_{\pi}$ as

$$q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t |S_t=s,A_t=a]$$.

It is quite simple to prove that these two functions satisfy these two mutual relations

\begin{equation}
v_{\pi}(s)=\sum_{a \in A} \pi(a|s)q_{\pi}(s,a)\label{eq1}\\
q_\pi(s,a)=\sum_{r \in R}r\sum_{s'\in S} p(s',r|s,a)+\gamma \sum_{s'\in S}v_{\pi}(s')\sum_{r\in R}p(s',r|s,a)
\end{equation}

and the two recursive *Bellman* equations
\begin{equation}
v_{\pi}(s)=\sum_{a\in A}\pi(a|s)\sum_{s'\in S, r\in R}p(s',r|s,a)[r+\gamma v_{\pi}(s')]\\
q_{\pi}(s,a)=\sum_{s'\in S, r\in R}p(s',r|s,a)\sum_{a' \in A}\pi(a'|s')[r+\gamma q_{\pi}(s',a')]
\end{equation}

A policy $\pi$ is said to be better of another one $\pi'$ if and only if $\pi(s) \geq \pi'(s)$ for all $s\in S$; an *optimal* policy $\pi_*$ is a policy that is better than all aother ones, that is $v_{\pi_*}(s) \geq v_{\pi}(s)$ for each $s \in S$ and for each policy $\pi$.
Let us assume that there exists at least one optimal policy (this also should be proved, but let's skip it in this question). Then we can define the *optimal state-value function* 

$$v_*(s) := max_\pi v_{\pi}(s)$$ 

for each $s \in S$, and it is clear that, for each optimal policy $\pi_*$, we have $v_{\pi_*}(s)=v_*(s)$ for each $s \in S$. The same apply to the *optimal action-value function* 
$$q_*(s,a) \doteq max_{\pi} q_\pi(s,a)$$

that is for each optimal policy $\pi_*$ it is $q_{\pi_*}(s,a)=q_*(s,a)$.

Finally, here it is my question:

**why is it true that $$v_*(s) = max_{a \in A} q_*(s,a)$$ for each $s \in S$?**

It is obvious that, given an optimal policy $\pi_*$, if for each $s \in S$ we take $a_s \in argmax_{a \in A}q_{\pi_*}(s,a)$, then we have
$$v_*(s)=v_{\pi_*}(s)=\sum_{a \in A} \pi_*(a|s)q_{\pi_*}(s,a)\leq \sum_{a \in A} \pi_*(a|s)q_{\pi_*}(s,a_s)=q_{\pi_*}(s,a_s)=q_*(s,a)$$

but I couldn't find a simple way to prove the reverse inequality, neither directly nor by contradiction.
Can anybody help?


  [1]: http://incompleteideas.net/book/the-book-2nd.html