I do not know of any active set theorists who think large cardinals are inconsistent. At least, within the realm of cardinals we have seriously studied. [Reinhardt suggested an ultimate axiom of the form "there is a non-trivial elementary embedding $j:V\to V$." Kunen quickly afterward showed that this is inconsistent, using choice. It is not known whether choice is needed, but current research suggest that, even without choice, natural strengthenings of this axiom may be inconsistent in ZF alone.]
There are several active set theorists who do not commit themselves one way or the other to the consistency of large cardinals, but use them if necessary, and do not object on mathematical grounds to arguments that involve them. For some of them, set theory is about all possible (natural) extensions of ZFC, and there are certainly many interesting such extensions (such as V=L) that rule out large cardinals. Thus, it is not that they consider the cardinals inconsistent.
(I confess, this may be ignorance on my part.)
And I know of only two mathematicians who years ago were serious set theorists and who have expressed doubts about the consistency of (certain) large cardinals. Neither is currently active within the field, and so their position should be taken with a grain of salt, since it missed the significant results from the late 80s that could very well have forced them to reconsider.
Why do we expect ZFC to be consistent, to begin with? We expect more than mere consistency, of course, but doubting large cardinals usually means distrust in set theory as a whole. I am not a philosopher, so I will not discuss philosophical positions or justifications. A good reference for the heuristics behind the basic ZFC axioms is Penelope Maddy's wonderful paper "Believing the axioms. I", J. Symbolic Logic 53 (1988), no. 2, 481--511. Large cardinals are discussed in its follow-up, "Believing the axioms. II", J. Symbolic Logic 53 (1988), no. 3, 736--764. Maddy's two books on Platonism and Naturalism discuss extensively why the view of a set theoretic universe with large cardinals rather than not is the reasonable choice, given our current understanding. The books present several subtle technical points that can only be completely understood once one is aware of the deep connections between large cardinals and (generic) absoluteness.
How do set theorists measure the internal plausibility of large cardinal assumptions, beyond their usefulness in proving results? The point of the inner model program is to develop fine structural ("L like") models for large cardinals. These models are canonical in several precise ways, and have a rich internal structure that many set theorist take as evidence of the consistency of the large cardinals under consideration. Thanks to its advances, we have a much clearer view of the set theoretic universe nowadays (for example, we now have the different covering lemmas, and several generic invariance results) than when the program began, motivated by what we now call Gödel's program.
The program has currently reached well past Woodin cardinals, but is not yet at the level of supercompact cardinals. This can be interpreted as saying that, using the strongest tools currently at our disposal, we are fairly certain of the consistency of, say "there is a Woodin limit of Woodin cardinals". Time will tell whether the program will reach supercompactness. If it doesn't, this will provide us with strong evidence of their inconsistency, though I am not sure anybody actually expects this to be the outcome.
John Steel and Tony Martin have over the years refined something they call "the speech", were they explain their position towards large cardinals. It is well worth reading, and trying to summarize it in a few lines would be an injustice. It can be found in these two postings to the FOM list:
1, 2 (the notation here is $P_T =$ set of $\Pi^0_1$ consequences of $T$), and in the papers from the "Does mathematics need new axioms?" panel, see here. (Steel's own views are also presented in some detail in Maddy's books.)
At the risk of not being balanced, let me point out some highlights: We have a coherent picture of the universe of sets, with large cardinals. We can, within this picture, interpret theories where there are no such cardinals. However, we do not have such a coherent picture in the opposite direction. The consequences of large cardinals, at the arithmetic level (and more, as we climb up through the hierarchy) are compatible. The arithmetic consequences of any natural extension of ZFC fall somewhere within this hierarchy (as far as the theories we can currently analyze), even if the theory does not mention large cardinals. In fact, determinacy statements, incompatible with choice, also fall within this hierarchy and are bi-interpretable with large cardinals (again, as far as those theories we can currently analyze).
Large cardinals provide us with generic absoluteness, and generic absoluteness, a natural requirement if we are interested in understanding the projective theory of the reals, requires the consistency of large cardinals. See this answer for a bit more on this issue; let me emphasize that this is not some technical or artificial requirement, but rather a natural extension of basic results in classical descriptive set theory.
Large cardinals seem inherently necessary to mathematical practice, not just set theory. Harvey Friedman has written extensively on this issue.
In short: We have a very clear measure of progress understanding large cardinals and their consequences. By this measure, we can now understand many set theoretical issues that do not involve large cardinals but for which they are necessary in deeper ways (not just consistency-wise). This measure actually requires the large cardinals, we do not have anything like that without them. This measure is meaningful even in settings that are not set theoretical, and seems unavoidable even within mathematical practice (though it is perhaps too soon to tell how significant it will be at the end for "practicing mathematicians"). We do not have any serious mathematical model where large cardinals would be inconsistent, however, we have a serious program of research that would ultimately teach us that, were this the case. The program has provided us, instead, with many positive results (we have nice inner models for measurability, for strong cardinals, for Woodin cardinals, we have nice inner models of models of determinacy, that capture the large cardinals that provide us with the consistency of the determinacy statements).
To conclude, we understand (motivate/explain) large cardinals within the larger context of reflection principles, the simplest of which follow already from ZFC. (So, we have a natural generating principle for them.) On the other hand, I know of no objections to large cardinals beyond "they are too large" or "they do not feel right", neither of which seems mathematical to me. The first also seems particularly artificial.
The only 'program' towards their inconsistency that I am aware of has instead produced many interesting consequences for the partition calculus at the level of $0^\sharp$ (and is perhaps responsible for the early theory of $0^\sharp$ itself).