In Barry Mazur's paper introducing Galois deformations, he hints at having a general theory for representations which are not residually Schur, but with more complicated statements. Does anyone know where/if this got written up with a similar level of detail as in Mazur's paper? I'm particularly interested in applications to Faltings-Serre.
I don't know where this got written, but it's certainly well-known. Here's how it works. There are usually problems with deforming objects that have automorphisms because in many cases the corresponding functors are "obviously" not representable (Gabber produces a fine counterexample to an overoptimistic attempt to make this statement precise in Katz-Mazur, in a situation where an object and all of its deformations have precisely two automorphisms, but let's not get into this).
So the way to fix this is to rigidify everything. You have your object (for example a mod p Galois representation) and then you add some extra structure to make it rigid (for example you don't just think of it as an abstract map to some $Aut(V)$, you choose a basis for $V$ and think of it as a map to $GL(n,k)$ and you lift the actual matrices to elements of $GL(n,A)$ -- a "framing"). Once you've killed the automorphisms, the general machinery kicks in and you get representable functors. All the dimension calculations in Mazur's paper are now off by some factor because you've changed the problem slightly, so you go through the proofs and find that your tangent spaces etc are all bigger by a factor of something like $n^2-1$.
Because you have fixed this extra data, and you're deforming it, your universal object comes with this extra data too (in this case an action of the formal group $PGL(n)$ I guess, or maybe $GL(n)$, it all depends on how rigid you made everything) and so in particular your answer comes with an action of a group on it. The "correct" deformation ring, the deformation of the object you wanted to deform, is just the quotient of your rigidified deformation by this group -- well -- in the category of affine schemes it is, in the category of rings it's something like the invariants. But somehow that's the problem. The thing is, quotients by random group actions don't exist (or more precisely don't give the right answer) in the category of affine schemes, and that's because the actual quotient representing the answer isn't a ring, it's a stack.
So now you either go and read the stacks project, or you take a more pragmatic approach and just carry around the universal "framed" deformation ring and the group action, and you make no attempt to form the scheme-theoretic quotient. In the category of rings this corresponds to taking invariants by the group action -- but the invariants are in some sense the wrong answer -- the right answer is the framed ring with its group action up to some notion of equivalence which boils down to "gives the same stacky quotient".
If the original problem was rigid enough, e.g. irreducible mod $p$ rep, then the group acts freely on the spec of the ring and the quotient is just spec of the invariants and you recover Mazur's deformation ring.
So why didn't Mazur set things up in this generality in the first place? Because, even before Wiles, Mazur knew that his deformation rings should be related to Hecke algebras. If you deform the way Mazur does, with his conditions, then Mazur could compute the Krull dimensions (at least conjecturally) of the deformation rings, and in the 2-d case he realised that there should be a map to a Hida Hecke algebra (in the ordinary case where the det isn't fixed) and already in Mazur and Tilouine (historically before Wiles) they conjectured that this should be an isomorphism -- a result now famously known as an "$R=T$ theorem" (Mazur and Tilouine had no idea how to prove their conjecture of course, this was Wiles' insight). If Mazur had rigidified everything then his deformation rings would have had too large a Krull dimension and the conjecture would have looked far uglier -- indeed it's not immediately clear to me how to make it -- one would have to tensor the Hecke algebra over the base with the affine coordinate ring of formal $PGL(2)$ I guess, and then there would be some fiddling around. Mazur avoided all this with his approach but irreduciblility/Schur got built in. So to a certain extent it was a historical coincidence that Mazur set things up as they ended up. But I'm sure that even in the 80s Mazur would have known everything I've written here.
Nowadays people do frame things as a matter of course, but because other aspects of the theory have moved on in leaps and bounds someone who just wants to know about this part of the story is faced with the prospect of unravelling the framing stuff from all the other new ideas. It would probably make a good Masters project to unravel this stuff!