The definition may be surprising at first look, but it turns out to be natural.
Let $K$ be a simplicial complex equipped with an action of a group $\Gamma$. Clearly, there is an induced action on the geometric realization $|K|$. The set $|K|^\Gamma$ of fixed points of the induced action is a subspace of $|K|$. We would like to find a simplicial complex that triangulates the space $|K|^\Gamma$.
Let $\mathcal{F}(K,\Gamma):=\{\sigma\in K:\Gamma\sigma=\sigma\}$ be the family of simplices of $K$ invariant under the action of $\Gamma$. In general the simplices in $\mathcal{F}(K,\Gamma)$ do not form a subcomplex of $K$. However, if the action of $\Gamma$ is admissible, i.e., for every $\gamma\in\Gamma$ and every simplex $\sigma\in K$ such that $\gamma\sigma=\sigma$ we have $\gamma v=v$ for all vertices $v\in\sigma$, then the simplices in $\mathcal{F}(K,\Gamma)$ form a subcomplex of $K$, which we denote by $K^\Gamma$. (It is the full subcomplex of $K$ induced on the set of vertices fixed by the action of $\Gamma$.)
An action of $\Gamma$ on $K$ induces an admissible action on the barycentric subdivision $\operatorname{sd}(K)$. Moreover, $|K|^\Gamma$ is naturally homeomorphic to $|\operatorname{sd}(K)|^\Gamma$. Thus, on the topological side, we do not really lose anything by passing to the barycentric subdivision, while on the combinatorial side we get admissibility. Because of that many authors simply ignore non-admissible actions; they do not lose much generality and have an intuitive definition of the fixed point complex $K^\Gamma$ of an admissible action.
How is that related to the simplicial complex of fixed points $\operatorname{Fix}(K,\Gamma)$ as defined in the question (and in the paper by Brehm and Kühnel)? If we start with a non-admissible action, then the triangulation of $|K|^\Gamma$ that we get by passing to the subdivision is unnecessarily subdivided - one can find a simpler triangulation.
Let us denote by $\mathcal{P}(K)$ the face poset of $K$ (i.e., the set of simplicies of $K$ ordered by inclusion). For a poset $P$ let $\mathcal{K}(P)$ be the order complex of $P$, whose vertices are the elements of $P$ and whose simplices are the finite, non-empty chains in $P$. Note that $\mathcal{K}(\mathcal{P}(K))=\operatorname{sd}(K)$.
An action of a group $\Gamma$ on $K$ induces an action on $\mathcal{P}(K)$. Denote by $\mathcal{P}(K)^\Gamma$ the sets of fixed points of this action. The elements of $\mathcal{P}(K)^\Gamma$ are just the invariant simplices $\mathcal{F}(K,\Gamma)$. It is easy to see that $\mathcal{K}(\mathcal{P}(K)^\Gamma) = \mathcal{K}(\mathcal{P}(K))^\Gamma = \operatorname{sd}(K)^\Gamma$.
Now, the poset $\mathcal{P}(K)^\Gamma$ is itself a face poset. As it turns out, $\mathcal{P}(K)^\Gamma$ is isomorphic to $\mathcal{P}(\operatorname{Fix}(K,\Gamma))$. Indeed, simplex orbits, i.e. those simplices of $K$ that are $\Gamma$-orbits of some vertex, are the minimal elements of $\mathcal{P}(K)^\Gamma$. Any invariant simplex is an union of a number of simplex orbits.
Note that for an admissible action $\operatorname{Fix}(K,\Gamma)$ is isomorphic to $K^\Gamma$. Thus, we may change the notation a bit and define for any, not necessarily admissible, action $K^\Gamma:=\operatorname{Fix}(K,\Gamma)$. We then have the following isomorphisms: $$\operatorname{sd}(K^\Gamma) = \mathcal{K}(\mathcal{P}(K^\Gamma)) = \mathcal{K}(\mathcal{P}(K)^\Gamma) = \mathcal{K}(\mathcal{P}(K))^\Gamma = \operatorname{sd}(K)^\Gamma$$ and $$|K^\Gamma|=|K|^\Gamma.$$
On a side note, one could also consider another definition of a fixed point complex. Let $K_{\mathcal{F}(K,\Gamma)}$ be the smallest subcomplex of $K$ containing the family $\mathcal{F}(K,\Gamma)$. This is the `fat' fixed point complex. One may show that $|K_{\mathcal{F}(K,\Gamma)}|$ is homotopy equivalent to $|K|^\Gamma$. In fact, there is a stronger, combinatorial correspondence between these two complexes. However, this is beyond the scope of this answer.
As for the second question, I am not aware of any standard references. It may just be a part of the folklore.
