There are many nonequivalent generalities in which one can define a formal scheme, for example the definitions in Hartshorne and in EGA are not quite the same. (I use in this answer some parts of my own editing in nlab's entry where more references can be found). In my understanding, whatever the definition is, the category of formal schemes is a realization of certain subcategory of Ind-schemes. Typically one requires at least that the ind-object in the subcategory may be represented by a diagram whose connecting morphisms are closed immersions of schemes. A pretty modern treatment is in
- A. Beilinson, V. Drinfel'd, Quantization of Hitchin's integrable system and Hecke eigensheaves on Hitchin system, preliminary version (pdf)
Some subcategories of Ind-objects in many algebraic categories can be described by putting the topology on algebraic objects. Thus the complete local rings, or more general the pseudocompact case, in the Grothendieck's approach to local schemes. One can use a topological version of Yoneda on rings to get a nice theory of formal schemes, over an arbitrary ring:
- B. Pareigis, R. A. Morris, Formal groups and Hopf algebras over discrete rings, Trans. Amer. Math. Soc. 197 (1974), 113--129 (doi:10.2307/1996930).
Nikolai Durov suggests to use directly the Gabriel-Demazure approach but not over Aff but over the opposite to the category of pairs (commutative ring, nilpotent ideal). Formal schemes should be an appropriate subcategory of that category of presheaves. That larger category (but without singling out there the smaller subcategory which would correspond more precisely to Grothendieck's formal schemes) is sketched in ch. 7-9 of
- N. Durov, S. Meljanac, A. Samsarov, Z. Škoda, A universal formula for representing Lie algebra generators as formal power series with coefficients in the Weyl algebra, Journal of Algebra 309, n. 1, 318--359 (2007)