Here are some perhaps silly ideas.
(i) Think of simplicial sets as presheaves on $\Delta$, now look up when a subpresheaf can be subtracted. (E.g. is there a universal property involved?) What os the subobject classified in the presheaf topos here? I am sure that these things are known, but possibly do not have a simple answer.
(ii) One related point from Peter May's answer is that $X\setminus A$ should be the maximal subsimplicial set of $X$ with trivial intersection with $A$. The condition on the realisations is then a possible red herring.
(iii) A final point is that if you are handling a simplicial complex (say a PL manifold, withawith a given triangulation) there would be questions of subdivision, and if I remember rightly some sort of regular neighbourhoods, and I think that this is geometrically more significant that merely looking at the complement of $|A|$ in $|X|$.