It depends on context. In the physics literature, there is a term "exactly solvable" meaning that a closed form for the solution can be written; it is never used to indicate that the solution exists in an abstract sense. E. g., see Baxter's classical book "Exactly solvable models in Statistical mechanics". So, in this context "exact solution" does mean "closed form solution".
In other context, you may talk about approximate or perturbative solution, and then I feel it is fine to contrast it to the "exact solution" even when the closed form for the latter is not known.