It is easy to give infinite sets of smooth manifolds that are homotopy equivalent but not diffeomorphic. They are distinguished by their Pontrjagin classes and the argument requires little more than the theory of characteristic classes. It is much easier than Milnor’s construction of exotic spheres and I think it is easier than the analysis of lens spaces, but it is little known, I think because of the popularity historical recapitulation. Proving theorems about homeomorphism is much more difficult than diffeomorphism. It is a hard theorem of Novikov that the (rational) Pontrjagin classes are an invariant of topological manifolds, from which it follows that these actually yield infinite sets of manifolds which are homeomorphic but not diffeomorphic. In fact, there are only finitely many smooth manifolds homeomorphic to a given one.
Surgery theory approximately says manifolds homotopy equivalent to a given simply connected smooth manifold are determined by their tangent bundle, which can be anything, in particular arbitrary Pontrjagin classes. In particular, there are infinitely many smooth manifolds homotopy equivalent to $S^2\times S^4$ with differing first Pontrjagin class and thus clearly not diffeomorphic (nor even homeomorphic).
That’s a big hammer, but there is also an elementary construction, not for $S^2\times S^4$, but for $S^3\times S^4$. Consider $S^3$ bundles over $S^4$ with structure group $SO(4)\sim SO(3)\times SO(3)$. Such a bundle is trivial on the two hemispheres and classified by the gluing map on the equator: $\pi_4(BSO(4))=\pi_3(SO(4))=\pi_3(SO(3))\oplus\pi_3(SO(3))$. Two independent characteristic classes are the Euler class and the first Pontrjagin class. The Euler class controls the homology of the total space of the bundle. If it vanishes, the homology is that of the trivial bundle: $H^*(M)=H^*(S^3\times S^4)$; in particular $H^4(M)=\mathbb Z$ and there is room for infinitely many values of the Pontrjagin class. The Pontrjagin class (of the tangent space) of total space of the bundle is the Pontrjagin class of the (vector or sphere) bundle over $S^4$, so that gives infinitely many manifolds which are not pairwise diffeomorphic (nor homeomorphic).
However, these infinitely many manifolds aren’t all homotopy equivalent to $S^3\times S^4$. We must show that there are finitely many homotopy types. One way to see this is that we glued trivial bundles over the hemispheres by a map from the equator to the structure group $S^3\to SO(4)$. We could generalize to gluing by a family of homotopy equivalences: $S^3\to hAut(S^3)\subset Map(S^3,S^3)= S^3\times \Omega^3S^3$. A homotopy between maps $S^3\to Map(S^3,S^3)$ yields a homotopy equivalence between the total spaces of their bundles. Thus the set of homotopy classes of the total spaces of bundles is (at most) $\pi_3(S^3)\oplus \pi_6(S^3)$. The first factor is the Euler class and determines the homology of total space, while the second is a more subtle invariant, but is a finite group. Thus there are only finitely many homotopy types of these bundles, so there are infinitely many manifolds in one homotopy type with different Pontrjagin classes. The map of spaces $SO(4)\to Map(S^3,S^3)$ yields a homomorphism on homotopy groups $\pi_3$, so the element of $\pi_6(S^3)$ is additive in the Pontrjagin class, so every homotopy type realized is realized infinitely many times; in particular, there are infinitely many choices of Pontrjagin class with total space homotopy equivalent the product $S^3\times S^4$.
(Alternately, instead of restricting to linear bundles, one can show that there are only finitely many homotopy types of simply connected spaces with the homology of $S^3\times S^4$. Such a space receives a map from $S^3\vee S^4$ with homotopy cofiber a sphere. Thus its homotopy type is determined by how the top cell is attached. The attaching map lives in $\pi_6(S^3\vee S^4)$ which Hilton-Milnor computes as $\pi_6(S^3)\oplus \pi_6(S^4)\oplus \pi_6(S^6)$. This is the sum of an infinite cyclic group and a finite group. The infinite cycle group controls the cup product. Thus spaces with the desired homology are parameterized by a coset of a finite group, and thus finite.)