Existence of stationary Nash equilibrium of discounted stochastic game

Question

$N$-player discounted stochastic games with finite state and action spaces possess a Nash equilibrium in stationary strategies. This has been proved by Fink (1964) and a closely related result by Takahashi (1964). Various generalizations have appeared since then.

I have read Fink's proof and understood it (at least I think so). To prove equilibrium, he must compare his proposed strategy to all other possible strategies. However it seems to me that he only considers the set of stationary strategies.

I can see one reason that looking only at stationary strategies is sufficient. Namely, if every other player uses a stationary strategy (this is the candidate Nash equilibrium), then the $n$-th player must solve a Markov Decision problem and it has been proved (e.g., look at Puterman's or Derman's books) that for the MDP nothing is lost by using a stationary strategy.

However, I have looked and have not been able to find some work in which the above or some other argument is used to justify considering only stationary strategies for the $N$-player case.

So my question is this: do you know of any work that addresses the above concerns? Any help will be greatly appreciated.

Answer 1

The point that stationary best responses to stationary strategies are best responses without any further restrictions was already made in the very first paper on stochastic games in the context of zero-sum games in

Shapley, Lloyd S. "Stochastic games." Proceedings of the National Academy of Sciences 39.10 (1953): 1095-1100.

For a more explicit statement of the argument, which is exactly the one you gave, you can look at

Sobel, Matthew J. "Noncooperative stochastic games." The Annals of Mathematical Statistics 42.6 (1971): 1930-1935.

Answer 2

You can consult Chapter 15 in the second edition of the book "Game Theory" by Maschler, Solan, and Zamir (2020), where Shapley's Theorem is proved with an explicit treatment of general strategies (rather than stationary strategies).