Why do people think that abelian varieties are the hardest case for the Hodge conjecture? Today, I heard that people think that if you can prove the Hodge conjecture for abelian varieties, then it should be true in general.  Apparently this case is important enough (and hard enough) that Weil wrote up some families of abelian 4-folds that were potential counterexamples to the Hodge conjecture, but I've never heard of another potential counterexample.
Anyway, in short:
1) Does the Hodge Conjecture for abelian varieties imply the full Hodge conjecture?
2) If not, is there an intuitive reason why abelian varieties should be the hardest case?
 A: I would say the answer to both questions is no. In fact, abelian varieties should be an "easy" case. For example, it is known that for abelian varieties (but not other varieties), the variational Hodge conjecture implies the Hodge conjecture. It is disconcerting that we can't prove the Hodge conjecture even for abelian varieties, even for abelian varieties of CM-type, and we can't even prove that the Hodge classes Weil described are algebraic. So if the Hodge conjecture was proved in one interesting case, e.g., abelian varieties, that would be a big boost.
Added: As follow up to Matt Emerton's answer, a proof that the Hodge conjecture for abelian varieties implies the Hodge conjecture for all varieties would (surely) also show that Deligne's theorem (that Hodge classes on abelian varieties are absolutely Hodge) implies the same statement for all varieties. But no such result is known (and would be extremely interesting).
A: The class of Abelian varieties is the simplest class of varieties  where the Hodge conjecture is not known to be true. So naturally a certain amount of effort is directed toward them.
However, it's not clear me that one can make an obvious reduction from more general smooth projective varieties  to Abelian varieties. The reason I'm skeptical is because the Hodge structures for such varieties  need not lie in the tensor category generated by Hodge structures of Abelian varieties. 
One last thing. The Hodge conjecture is false for compact Kaehler manifolds, and in fact
for complex tori! Cf. Voisin, IMRN vol 20 (2002). There is also an older example due to Zucker. 
A: My next door neighbor (in the math department) is a Hodge theorist, and I have never heard her say that abelian varieties are the hardest case of the Hodge conjecture.
However, they are certainly a demonstrably "rich" case of the Hodge conjecture.  My neighbor did tell me once that the Hodge conjecture is presumably true "most of the time" even for compact Kahler manifolds, because a generic Kahler manifold doesn't have enough nontrivial cohomology groups to make the Hodge conjecture an interesting statement.  On the other hand, an abelian variety of large dimension has lots of large dimensional cohomology groups, and there are many known families of abelian varieties with sufficiently small Mumford-Tate group so that the Hodge conjecture cannot be verified just by intersecting divisors together.
For a brief exposition around the Hodge conjecture and abelian varieties, you may consult
http://alpha.math.uga.edu/~pete/mtnotes.pdf
Caveat lector: I am not an expert on this subject.
A: Related to Jim Milne's answer, one might mention that Deligne proved that for abelian varieties, all Hodge cycles are "absolutely Hodge" (i.e. when you think of them embedded diagonally inside the product of the algebraic de Rham cohomology and  $\ell$-adic cohomology
(for every $\ell$) and apply an automorphism of $\mathbb C$, the resulting cycles are again diagonally embedded rational cycles, and are in fact again Hodge).  Note that 
if the Hodge conjecture holds, then this is certainly true (since the conjugate under any automorphism of $\mathbb C$ of an algebraic cycle is again an algebraic cycle).
On the one hand, this is much more than is known about the Hodge conjecture for more general classes of varieties.  
On the other hand, one can't immediately extend this to other classes of varieties
 because the motives of abelian varieties don't generate all motives over a field of char. 0
(in fact, far from it, as far as I know), a fact already brought up in Donu Arapura's answer.
