Aside from the conceptual challenge of functions themselves, students find limits difficult because of their quantifier complexity. I have never understood why standard algebra pedagogy suppresses quantifiers, thus, for example, leaving many students unable to distinguish between unknowns (literals bound by existential quantifiers), variables (literals bound by universal quantifiers) and constants (literals that belong to the language itself). Students who miscalculate the derivative of $\pi^2$, mentioned elsewhere, don't get this distinction. People who become mathematicians usually "got it" without anyone spelling all this out, and then they learned about quantifiers studying logic in college, so they regard quantifiers as sophisticated and advanced. But most students don't "get it," and I think this accounts for the huge attitude downturn when they get to algebra.