The "manifold picture" can be applied to physics in the context of the Brillouin zone, see for example On Brillouin Zones. The reason that discreteness and smoothness appear inverted, is that the Brillouin zone describes reciprocal space. The distinction is not fundamental, one can equivalently describe a crystal in real space, where discrete features appear on short distance, or in reciprocal space, where discrete features appear at large distance.

The "manifold picture" can be applied to physics in the context of the Brillouin zone, see for example On Brillouin Zones. The reason that discreteness and smoothness appear inverted, is that the Brillouin zone describes reciprocal space. The distinction is not fundamental, one can equivalently describe a crystal in real space, where discrete features appear on short distance, or in reciprocal space, where discrete features appear at large distance.

So to answer the specific question in the OP: I don't think there is a need to abandon the manifold picture to describe physical matter, you just want to apply it to reciprocal space rather than to real space.