What proportion of chess positions that one can set up on the board, using a legal collection of pieces, can actually arise in a legal chess game?

Question

Many chess positions that one may easily set up on a chess board are impossible to achieve in a game of legal moves. For example, among the impossible situations would be:

• A position in which both kings are in check.
• A position in which there are pawns on the first or on the last rank.
• A position with two white pawns on the same file, but black still has all his pieces.
• A position with a white bishop on the first rank, trapped by two white pawns on the second rank, but the bishop is not on c1 or f1.
• A position with two black-square white bishops and eight white pawns.

The logician Raymond Smullyan wrote a delightful book The Chess Mysteries of Sherlock Holmes: Fifty Tantalizing Problems of Chess Detection, containing many interesting chess detective stories, some involving positions that were impossible for sometimes very subtle reasons.

My question is:

Question. What proportion of the chess positions that one can set up on the board, using a legal collection of chess pieces, can actually arise in a legal chess game?

What I mean is that collection of pieces is legal, if it occurs in a position of a legal chess game, a game played according to the rules. This collection is somewhat broader than one might naively expect, since it is legally possible, for example, to have a king of each color with nine white queens, as white may have promoted all the pawns while all other pieces were captured. And other similarly strange collections of pieces are possible. So the collection of positions I am considering are those that can be obtained by messing up the pieces on the board from an actual legal game.

Of course it will be too difficult to get an exact answer, and I shall be satisfied merely with good bounds. The Wikipedia page on chess and mathematics mentions some numbers, including estimates on the number of legal positions, but the information there doesn't seem to answer this question. Perhaps those who are more familiar with that work can point to where this question is answered there.

I guess the answer must be a rather small proportion, because it seems that many legal chess positions can be easily transformed into many illegal ones, by placing both kings in check, by adding a pawn to the first rank (unless all pawns are already used), etc. Is this right, and can such an argument be used to make tight bounds?

I am here at the Mountain Lake Chess Camp, where we've been discussing the question, when one of the instructors mentioned the numerical bounds on the total number of chess positions, and the question arose whether this included impossible-to-achieve positions or not.

Answer 1

The overwhelming majority of positions using the full original set of 32 pieces are illegal and cannot arise in a legal game of chess. Specifically, the proportion of legal positions among all those using the 32 piece set is strictly less than $4.0763\cdot 10^{-10}$.

To see this, consider a legal position using the 32 piece set. Since it has 32 pieces, there can have been no captures yet. In particular, each pawn must still be on its original file, not in the first or last rank, and furthermore, still opposed by the opposite-color pawn still facing it on that file. Within each file, therefore, you can easily count precisely 15 arrangements of one black pawn and one white pawn that exhibit this feature. Thus, there are precisely $15^8$ many ways to arrange the pawns overall in such a way that the position is not immediately seen as illegal. But there are are ${64 \choose 8}\cdot{56\choose 8}$ many ways to arrange the $16$ black and white pawns on an empty board. For each arrangement of the pawns, there are exactly the same number of ways to arrange the remaining pieces. The proportion of legal positions using the full 32 piece set is therefore at most $${15^8 \over {64\choose 8}\cdot{56\choose 8}}= { 2562890625\over 4426165368\cdot 1420494075}\approx 4.0762706\cdot 10^{-10},$$ and so we get the upper bound as claimed.

Answer 2

I count $173558929221226891302430767615551561533417485504 = 1.7356 \times 10^{47}$ ways to place a legal collection of chessmen on the board. Of these, $24639467089379915386890365075260915223928803040 = 2.4639 \times 10^{46}$ have no pawns on the first or last ranks, so at most $14.2\%$ of ways to place a legal collection of chessmen on the board produce a legal position. It is likely that improvements would find more significant restrictions on the set of legal positions.

It takes some effort to determine the possible collections of legal chessmen. ("Chessmen" includes pawns. "Piece" technically refers to a non-pawn.) For example, if White has 8 pawns, 2 bishops, 2 knights, and a king, then it is not possible for Black to have 4 pawns, 3 queens, 4 rooks, 2 bishops, 2 knights, and a king. However, Black could have 4 pawns, 3 queens, 4 rooks, 2 bishops, 1 knight, and a king. If White still has 8 pawns, then each promotion of a Black pawn can be paired with a capture for one side or the other, and in the first collection there were at least 4 promotions and at most 3 captures.

I started by finding the possible vectors of opposed pawns, unopposed pawns, and original pieces not counting kings for each side. Call a pawn opposed if it has not moved from its original column, and neither has the opponent's pawn in that column. Each side starts with $8$ opposed pawns, $0$ unopposed pawns, and $7$ original pieces. Moves include $(-2,+2,0,-2,+1,0)$, which occurs when a White opposed pawn captures a Black opposed pawn, which converts $2$ White opposed pawns into unopposed pawns, and converts $1$ Black pawn into an unopposed pawn. There are $17932$ possible vectors of opposed pawns, unopposed pawns, and original pieces.

Next, convert each vector into the possible vectors of pawns, promotions, and captures. Each unopposed pawn can promote or stay a pawn. Convert each vector of pawns, promotions, and captures into the possible vectors of pawns, knights, bishops, rooks, queens, and king for each side. I used a hash table to avoid duplicates, since the same vector of counts may occur from different numbers of promoted pawns. Count how many ways there were to place these chessmen on the board, and how many ways there were under the restriction that the pawns have to be placed within the $48$ squares in the second through seventh ranks. I used C# with a big integer package. The computation took $2$ hours $34$ minutes on a $2$ GHz processor. Here are the counts filtered by the total number of pieces other than kings:

$$\begin{matrix} 4032 & 4032\newline 2499840 & 2374848\newline 762451200 & 688021440\newline 152490240000 & 130689523200\newline 22492310400000 & 18305596012800\newline 2609108006400000 & 2016206442390528\newline 247865260608000000 & 181841738563525632\newline 19829220848640000000 & 13808933884008775680\newline 1363258933344000000000 & 901054038699836313600\newline 81795535837048927998720 & 51305183887862557540800\newline 4335162450829839506110464 & 2580103899370081990144800\newline 204934580948509440712459776 & 115714657023418620663724800\newline 8709625161842041324664929152 & 4665050555573374751447039520\newline 334969016041319033638193529600 & 170173825673382454767520248000\newline 11721777898941512268159714220800 & 5647729008729626602595924112000\newline 374885578524998042949781990656000 & 171301515200792754380618149854720\newline 10995798990911566467975707768400000 & 4765529886571048034261032746780960\newline 296499744424712483942201077363200000 & 121910613390103852405088592091960320\newline 7359259246674152754334338180912672000 & 2871962662751699192209092269498673600\newline 168140268992852540807133006561199334400 & 62319197518162641915225476408569670400\newline 3532159391229553383833527063326170154240 & 1244357903407170974320709795637765066240\newline 68072251892766786279733898221633351480320 & 22815946937203267917191715682760288217600\newline 1199612470722563512991607127423133252405760 & 382949338893972841238132883627848055957120\newline 19244922194683190664837287286758917707571200 & 5858794635470333659860790877285971462732800\newline 278995789496829967243384718498911484120064000 & 81113217425471696961324265298226557473881600\newline 3512763516196652188175623254752242110008524800 & 967860625456447714742338884049918383560140800\newline 30032513167097689765684275178492499448148377600 & 7180859208049984902733460962414426023904281600\newline 90835621190174574287987518323083620781868441600 & 13544187413659480412286575232707703090739353600\newline 46469725378042907543299835558295158351228160000 & 2814661208855794476422354880818644910233920000\newline 2404159138638549255694710397739697094118400000 & 44498154253195662774573947559960251923200000\newline 4634726695587809641192045982323285670400000 & 21392082322155637297725861387221535360000\newline \end{matrix}$$

The $14.2\%$ value from restricting pawns to $3/4$ of the board indicates that typical positions of legal sets of chessmen have many pawns, about $\log .142/\log .75 \sim 7$, which is not obvious because there are many ways a pawn could underpromote or could be captured. Perhaps one could get a better restriction by keeping track of pairs of opposed pawns, which have to be in their original columns in order. This would add a lot of complexity to the bookkeeping, but it would produce a severe restriction for collections of chessmen which can only occur with a pair of opposed pawns still in their original column, perhaps trimming a few percent off of the $14.2\%$.

Answer 3

Here I try to describe the legal collection of pieces, as defined by Joel, so this is not an answer to the original question. I think that a collection of pieces is legal if it can be obtained from the original collection using the following steps:

(1) Delete any (non-king) piece and promote at most one white and at most one black pawn.

(2) Delete a pawn and promote at most one pawn of the same color and at most two pawns of the opposite color.

I don't have a rigorous proof that this is correct, so let me know if I've missed anything.

Answer 4

A series of moves from the starting position to position X (therefore showing X can arrive in a legal game) is called a "proof game" and finding them is a branch of "retrograde analysis".

• http://en.wikipedia.org/wiki/Proof_game

has some info and links, particularly to a program called Natch, which purports to find the shortest proof game for an arbitrary position (I don't offhand see how to do that in a reasonable amount of time, hmm).

Anyway one thing to try is generating a bunch of random positions and running them through Natch, to see what proportion are reachable. If not Natch, then maybe some other heuristics or algorithm can be figured out.

Answer 5

In order to make some concrete progress, let me make a definite upper bound, based on the idea of my earlier comment.

What I claim is that at most $3612/4032\approx 89.6\%$ of the positions using a legal collection of pieces are legal.

My reason is that I claim proportion $420/4032$ of the positions using a legal collection of pieces have adjacent kings, and this arrangement is illegal in any chess game. To see this, consider a fixed legal collection of pieces. Thus, this collection has one white and one black king. Let us partition the possible positions using this collection of pieces into groups, where two positions are in the same group if the kings have the same locations in the two positions. Each group is exactly the same size, since the other pieces occupy all the other $62$ squares in all possible ways that they can. There are $4032=64\cdot 63$ many groups, since there are this many ways to place the two kings. Among those, there are $420=36\cdot 8+24\cdot 5+4\cdot 3$ many ways to place the kings on adjacent squares, since if we place the white king on any of the 36 center squares, there are 8 choices for an adjacent black king, but on the 24 edge (non-corner) squares, there are 5 choices each for the black king, and for each of the 4 corner squares for the white king, there are 3 adjacent squares each for the black king. So proportion $420/4032$ of the groups are invalid based solely on the adjacency of the kings. Since with any fixed collection of pieces, the groups are all the same size, it follows that at most proportion $3612/4032\approx 89.6\%$ of the positions using this fixed collection of pieces can legally occur in a chess game. And since this same proportion arises for each of the legal collecton of pieces, it follows that at most this proportion of all positions using a legal collection of pieces can be legal.

I expect that further analysis will greatly improve this upper bound.

Answer 6

EDIT 3

These things always come to me after I post.

Let's build one cage for both kings. Allocate a corner and a 2-rank by 4-file space for the cage. (One can do a vertical 4 by 2 cage also, but the analysis with pawns is harder. Let's stick to horizontal for now.) Allocate 5 pieces to form the walls of the cage. The nice thing here is that pawns of both colors can be involved, and you can put either king in the corner office. There will be three forbidden squares for one color knight and only one for the other color in the remaining 56 squares. We only need at most two promotions to build a cage, but there are 7x5x7x7x6=10290 different cages to build. Swapping kings or using a different corner gives another factor of 2x4. Without even considering differing multisets beyond what is needed to build a cage, we can use much of the analysis below to get over 10000P many legal games. With more work, we can push the provable lower bound for the ratio to above 10^-12, and that is assuming the weak upper bound for all positions of roughly Qx10^5. I now expect the actual ratio to be between 10^-4 and 10^-7.

3 TIDE

EDIT 2

Here is a lower bound which hinges on several parts.  We will build two cages to isolate the kings, promote 9 pawns, scattershoot the remaining pieces, and remove the illegal and some legal combinations, and then indicate some variations to pump up the lower bound.

The basic setup involves castling both kings on King side, and moving them to the corners and using two stationary pawns and a preselected piece to guard the king.  To simplify things, we will choose one of twenty combinations for guarding: each king gets a nonpawn of their color, or each gets an opposing knight or bishop.  Both kings are on the h file corners and the guard pieces are adjacent to them on the g file.

Once guard pieces are chosen, advance King and Queen pawns, advance material to castle, and then maneuver the guard pieces into position, moving a king out of check if needed (although it should not be necessary).  Now that the kings are caged, advance pawns and move nonpawns in a nonthreatening manner to the g and h files.

Now plan for three pawn captures, especially if bishop color matters.  Place the remaining nonpawns (two of each color) in the file behind pawns of the same color that will promote on that file. To make the analysis simpler, promote no more than four pawns to queens of the same color, and no more than three of any other type.

Once all of the promotions have been made, there will be 56 squares for 21 pieces, and I assert that all the illegal placements are covered by monochromatic bishop placement and by knights putting opposing kings in check.  Before dealing with that assertion, let's count how many potentially legal positions that could be.  As a starter calculation, assume among the 21 pieces outside the cages that there are three or two of each of the 8 allowed types of nonpawns, and call this multiset R  There are P=(56_R)=$(56!)/((35!)(3!)^5(2!)^3)$ such positions.  When we compare this to the standard Q defined below, we get P=Q[((8!)^2(2!)^3 (56!)(32!))/((3!)^5(64!)(35!)) ] This means P/Q is roughly about 1/2x10^-12 . But wait! There's more!

Let us get some hand-waving out of the way. I assert that any legal arrangement of 21 pieces on the reduced board can be reached from any other legal arrangement. Here legal is more strict and includes no knights placing kings in check. The basic idea is to move all 21 pieces into files a-e, and then argue that you can move enough pieces into a restricted portion of files f-h so that you can swap any two pieces, sort of like a complex 14-15 puzzle with the only parity issue involving bishops. I handwave that from any arrangement of 21 pieces in files a-e, you can legally place 8 or more pieces in files f-h without checking either king, leaving 13 or fewer pieces in files a-e. If it helps, I invoke the fact that we can promote so that all of the 8 types of pieces have at least 2 representatives in the 21. (One may need to prove that there is room enough to permute arbitrary piece types, but there are 12 squares to work with in files f-h, so I am asserting with confidence. One also needs to show that there is no locked configuration of 13 pieces on a 5 by 8 board, but by tiling the board by p pentominoes, you can likely show that there will be more than one piece that can move usefully.)

Now let us handle the illegal combinations. They are monochromatic bishops and checking knights. In case there is only one white bishop among the 21, it needs to be on a color square that is different from the square that the guarding white bishop is on. Potentially half of those P many positions may thus be illegal, and less than half if there are two or more bishops among the 21. Similarly the black knights are forbidden from simultaneously occupying f2 and g3, so let them occupy the other 54 squares instead. With at most five knights of one color, this reduces the number of positions by a factor of at most (51x50)/(56x55), so by a factor of more than 80%. So I claim that at least 64/4=16% of the P many combinations are legal, by factoring in contributions from both colors of bishops and knights.

Now to pump up the numbers. First note that we can use a queen side arrangement to double the number of legal positions, counteracting the white bishops complication. Also, we can move the pawns in the h file (or a file for Queen side) and reduce the number of squares from 56 to 53 or 52, depending on if you want a piece between black and white pawns. This (more handwaving using 10*35*34*33/56*55*54) restores another factor of close to 2. Also, by considering capturing one or two pieces of the 21 gives us a factor of about (1 + 21/35(1 + something positive)), so we build back up to P many positions which are legal, using queen side and using slightly fewer pieces.

Now to bump things up. The above was based on a fixed multiset R, and using just that multiset we got at least 0.64P legal positions, building up to P by capturing one or two of the 21 pieces not involved in the cages. However, there are 56 different multisets allowed for the 21 pieces, if there are only two or three of each type. If we allow one color to have a 4 3 3 2 distribution and the other a 3 2 2 2 among the types, we get 2x12x4 additional multisets at a cost of reducing P by 3/4. So we actually have at least a factor of (56 + 96x3/4)=128 to use in our lower bound. Finally, we have the factor of 20 different guard combinations. Even if we don't consider multisets of size smaller than 29, we get a lower bound of 1600P many legal combinations, which puts us within striking distance of Qx10^-9.

Although there is much work left, I now expect the ratio lies between 10^-15 and 10^-7. The feeling I get is that games involving 26 pieces or less will make no significant contribution to the ratio, and even 28 piece games will not shift the order of magnitude of approximations.

2 TIDE

EDIT

Unfortunately, the details will not be ready until after the bounty expires. Someone else may be able to use the sketch below.

Let S be the multiset of pieces used at the start of a chess game. Let Q=(64_S) be the number of possible arrangements, one piece per square, using S on a chessboard. Q is (the value of) a multinomial coefficient using data from S, and is about 4x10^42.

If one looks at the number of possible arrangements using a legal submultiset S' of S (in particular the collection of pieces arising from a legal game of chess involving no promotions), one encounters an expression like Q(1 + 30/33*(1 + 29/34*(1 + 28/35...))), which evaluates to something less than 5Q. Further, considering legal collections T, one needs only those submultisets of T which are legal and have not been considered earlier in a sequential enumeration of legal submultisets. Thus, a good approximation to the total number of all positions considered should be the sum over T a maximal legal multiset of terms (64_T).

The T I have found so far that gives a maximum value for (64_T) is slighly less than 8Q, involves one pawn capture and 3 queen promotions of different colors. After two pieces being captured, it is hard for me to see any optimal T occurring.

There are 165 possible multisets arising from promoting 8 pawns of one color. I suspect an upper bound for the number of maximal legal multisets is 165*165, given that 2k promotions usually require k captures. This suggests an upper bound of 27225*8Q for the number of all possible positions, or about 217800Q.

I was hoping to find a T such that I could beat Joel Hamkins's ratio of legal game positions using T/(64_T) while also satisfying (64_T) > Q, but time has run out for that. It looks like 2^-32 remains unverified at this writing.

TIDE

If the idea is to come up with a rough figure, the following approach using labeled pieces might help.

Consider the white king's pawn. It cannot inhabit 24 of the squares on the chessboard (I assume promotion is strict and monotone. I let others argue whether to add 8 to the number 24.) . For the other king side pawns 24 grows to 26, 30, and 36. Starting with these ratios, a set of four pawns labeled with their starting squares can occupy about 9% of all possible positions available. Raise this to the fourth power, and one gets a ratio of 6x10^-5 as an upper bound on positions of labeled pawns, where as a rough count I include pawns sharing the same square.

Now the bishops provide another factor of about 6x10^-2, the two kings together a factor of about 7/8, and the rest provide a factor that is likely closer to 1. This gives a weak upper bound for all 32 labeled pieces of 3 x 10^-7 fraction of legal positions to all positions.

I imagine with refinements one can probably subtract one more from the exponent. Hopefully the fraction involving unlabeled pieces will not be much different.

I just noticed domotorp's comment regarding positions having just 28 pieces. While this suggests the fraction of legal positions may be higher, for the labeled case, the ratio (all possible for 28) to (all for 32) should still be small, and should have a small effect on the rough calculation.

And now I just noticed more recent comments from Joel David Hamkins, who made some similar and more refined observations. Perhaps he can place the above musings on a more rigorous footing, and maybe someone can convert the labeled analysis to an unlabeled one.

Answer 7

I think some positive proportion, like 50% of all positions with legal pieces may arise in a legal game. I also think that the upper bound of $10^{47}$ in wikipedia might be wrong. Consider only the 28-piece, no pawn configurations. If all pieces were different, this would give $(64!)/(36!)\approx 3.4\cdot 10^{47}$ positions. Also, there are ${19 \choose 7}\approx 5\cdot 10^4$ ways to promote the 12 pawns into 8 possible non-king pieces (not counting those with too many white/black pieces). Of course now we have to divide with the figures that are the same, but that should be around (and here is the only place where I make an estimate) $4^8=6.5 \cdot 10^4$ usually, as there are 8 non-king pieces, on average 3 of each, and I rounded up because factorials are like that... So multiplying and dividing these numbers we get about $10^{47}$ positions. Of course because of the kings in chess (few because of too many bishops on same color or when the one is in check, the other could not make the last move etc) not all of these are possible, but we were quite generous when not counting positions with pawns, so I believe that there are at least this many legal positions that can arise during a game and the same order of magnitude when counting positions that can be set up with legal pieces, as almost all with 28 pieces are from a legal game.

Answer 8

This answer was originally a specific argument that the problem might be intractable due to dominance by positions that are unreachable for a specific reason. I've rewritten the answer to be more general.

Joel David Hamkins' answer has put an upper bound on the result. The bound comes from a certain mechanism, a specific constraint involving the arrangement of the pawns. Let's call call this mechanism $M_1$.

Let $x$ be the fraction of reachable positions. Suppose our goal is to put bounds on it, $a<\log_{10} x < b$, with a relatively small value of $\Delta=|a-b|$. Mechanism $M_1$ gives $b=-9.4$. Douglas Zare's answer estimates $10^{47}$ positions, and if, say, at least $10^6$ distinct positions have been reached in real games, we have $a=-41$. That gives us $\Delta\approx 32$, which is pretty wide. I would consider the problem intractable if this can't be improved to something more like $\Delta=4$.

Here is a second mechanism, $M_2$, which may also make many positions unreachable. As an illustration, consider two sets of positions. A is the set of all positions in which white has 8 pawns, 2 bishops, and no queens, and black has the same. B is the set of all positions in which white has no pawns, 5 bishops and 5 queens, and the same for black. B is about 30 times bigger than A. We should expect that most positions have this character: boards crowded with powerful pieces as a result of many pawn promotions, including a lot of underpromotions.

A given position in B may or may not be reachable. It's pretty difficult to get that many powerful pieces on the board without causing a checkmate. If such a position is reachable, then watching it be developed on the board would probably resemble a chessboard history in which two amicable superpowers cooperate very carefully to allow one another the utmost possible peaceful development of their respective civilizations. Every time they approach the brink of a Cuban missile crisis, they unexpectedly find a clever way to avoid a premature end to the game.

I could imagine that no positions in B are reachable or that some significant fraction of them are. Getting the answer would require developing an entire theory for positions of type B, which would probably be as much work as developing a topic of practical chess theory such as bishop versus knight endings with pawns.

Some folks have expressed skepticism in comments that $M_2$ really makes very many positions unreachable. I don't know -- all I've offered is a plausibility argument. The question arises of how one would ever establish the answer reliably and verifiably. I don't think it helps much to construct and analyze sample positions as suggested in Douglas Zare's comment, because this proves nothing about the probability in general that a position is unreachable due to $M_2$. Possibly some kind of random sampling would work.

The answers so far seem to have focused on looking for insight into mechanisms $M_i$ that prevent a position from being reachable, and then trying to estimate the probability $P_i$ that a randomly chosen position is unreachable due to that mechanism. We could then guess $\log x=\Sigma \log (1-P_i)$, assuming that the probabilities are independent. But there are some real problems with this approach.

First and most importantly, we can't necessarily enumerate all the mechanisms $M_i$ or convince other people that we've enumerated them all.

Some of the $P_i$ may be impossible to estimate by hand as Joel David Hamkins did for $P_1$, which leaves us with the possibility of estimating them by random sampling on a computer. But the definition of $M_i$ may not be specific enough to allow software to determine whether it is "the" reason that a certain position is unreachable. Also, $1-P_i$ may be too small to make it possible to find any reachable positions in a random sample. Or even if $1-P_i$ is 0.5, we may be unable to demonstrate that by sampling, because determining the reachability of a single position may be an intractable problem in many cases.

Answer 9

To JDH: Double checks can be legal. OF course, one can't double check with the following pairs: QQ, RR, BB, NN, NP, PP, BP. I believe that's it. But other than these combinations, double checks are allowed in certain circumstances. However, triple checks can never occur.