When your paper makes a borderline case for a top journal Say you write a paper that you feel makes a borderline case for publication in a prestigious journal (Annals, Acta, Inventiones, JAMS, etc).  What are the advantages and disadvantages of submitting the paper to the prestigious journal over a less prestigious journal where your paper is very likely to be accepted? My novice take is that there is little downside since if the paper is rejected you can always then submit it to a less prestigious journal. Are there factors I'm not considering? Of course, in most cases, the paper is Arxiv'ed early on so one's claim to the result is never hurt by a lengthy publication process. 
I'd be interested to also hear about the implications on job applications. How much does it hurt a job applicant to have a very good result Arxiv'ed and submitted versus accepted? (Presumably if the validity of the result was a make or break factor for a job the hiring group could take a look at the preprint or talk to an expert in the field. Maybe that is an unrealistic expectation).
 A: I somewhat disagree with some of the earlier answers on the "jobs applications impact".  First, I think it makes absolutely no difference whether you have "submitted to Annals" or "preprint" on your CV.  Everyone knows the acceptance chances and will ignore this line.  Second, it is really important that all your papers are available on the arXiv or your personal web page.  Often, the hiring committee can't judge the applicant's work, and will ask a local expert at the department to take a look at the papers and give an impartial opinion.  If the papers are not available, the committee is forced to trust the applicant on their existence, a bad situation all around.  Finally, except for the really top journals, having a paper published vs. having it still in a preprint form is of minor difference - if the local experts and/or reference letters are all saying that these recent papers are really good, that's sufficient.  From that point of view, you should basically ignore the job application considerations, and always do what's best for the paper. 
P.S.  If I may make a suggestion, I think it's much more important to choose the right people to write reference letters than worrying about minor points in your CV.  Especially now, in the mathjobs era of mass applications, it is the letters that really help people stand out from the crowd.  So my advice would be to start thinking well in advance who can you ask for the letters, and learn how to better communicate your results (to them and everyone else).  
A: This requires a conditional response.  First, who is submitting?

*

*Tenured professor:  Not much repercussion either way.
Worth the wait if you think it deserves publication.


*Graduate student:  Some downside to waiting a long time
and not having the thing published come application time, but this can happen
at a lesser journal, too (for grads, the paper is often
submitted only shortly before they are graduating).
And postdoc hiring is not
as publications-based as tenure-track hiring.  Letters
of recommendation mean more.


*Postdoc.  Here, submitting with a high probability of
a rejection after a long wait can be a major gambit.
Mitigating factor:  does the postdoc have other worthy
publications?  If this will be the flagship result,
it's hard not to think that a slightly lesser journal
would have a higher expected yield (in terms of jobs).
Note that it is true that some journals may reject
your paper quickly -- then you can turn around and
submit somewhere else -- but those papers are not really
the marginally-great ones being asked about.
In the end, you are left with a hard decision.
I don't think there's a formula which can help.
In this case, you either go for broke or you don't.

*

*Collaboration:  decide based on the member with the
most to lose.

A: I think this is a really tough question, and I've never been on a hiring committee, but I'll try answering it anyway.
If your paper deserves to be published in a top journal (the Annals, let's say), then there should be an expert in your field who holds that opinion.  If fact, if your paper actually gets accepted by the Annals, the referee will be such a person.  If some such person has suggested the Annals as a place to submit your paper, then you could ask that person for a letter of recommendation expressing their opinion of your paper, and then submit it.  I think it's much more likely that a hiring committee will carefully read a letter from an expert than that they will look carefully at the details of your paper.  Seeing "submitted to Annals" on a CV at least shows you think you've got a great result, but that probably needs to be backed up by either an acceptance or a second opinion.
A: We shouldn't forget the most obvious and glib answer:  If your paper makes a borderline case for a top journal, send it to a journal that itself makes a borderline case for being a top journal.  There is no dichotomy between "top journal" and "other journal".
A: If you've written an excellent paper, that's still no guarantee that it'll get into the prestigious general journals (such as those listed in your question).  For example, the editors or referees might declare your topic to not be of general interest.  If this turns out to be the case, you should consider publishing in a prestigious specific journal (not necessarily a less prestigious journal).
The worst possibility is the long rejection -- that is, having a paper refereed for a year or so, only to be rejected.  This seems to happen a lot with the general journals as the referees are trying to maintain a high standard (and the editors can't always tell if a paper is worthy or not).  The refereeing process is confidential, so the only downside is time wasted, and you might get some excellent feedback too.
For job applications, it's much easier for a potential employee to gauge the merit of a general journal (LMS, AMS, etc.) then a specific journal.  They can be 100% a result published in these general journals is decent, regardless of which field they are in.  Whereas, it can be difficult to explain the importance of a specific journal.
I heard from one university that they had 600+ applications for one position.  If the hiring committee looked at one application per minute, they would still take 10 hours.  They won't have time to look at arXiv, chat to colleagues, etc. for the vast majority of applicants.  Moreover, even if the paper is on the arXiv, unless the committee are experts in your field also, they won't know whether or not your paper is decent.  You can explain the merits of your work in-person at a job interview.
Having a paper on the arXiv (vs. not having it on the arXiv) counts for nothing in job applications.  Generally, you want not-yet-accepted papers listed as either "in preparation" or "submitted".
A: As suggested in passing in Douglas Stones' answer, one advantage of submitting to a better journal is you're likely to learn more mathematics from the referee reports, accepting or no. 
A: There is one disadvantage I know of, but it's a biggy: waiting.  It can take as long as a year for a journal to turn around a paper, and if you do this several times, it can be very aggravating.  I recently had a paper accepted on its third go-round, and while it still got in a pretty good journal, the whole process took about 2 years.  In particular, I went on the job market with this paper as a pre-print rather than an accepted paper as a consequence.  I doubt this fact was consequential, but obviously it would have been better to have had it accepted.
That said, I always aim high (within reason), and generally think it's the right policy.  There's so much uncertainty about what belongs in what journal that one might as well give it a shot.
