First of all, why is it so unreasonable to think that ZFC itself is contradictory? Because we have a good intuition about sets and we have a lot of experience with ZFC. The same basically applies to large cardinals. What I am trying to point out here is that assuming large cardinals is not much more unreasonable than passing from Peano Arithmetic to ZFC. (Edit: David Roberts points this out is his answer: If you doubt the consistency of large cardinals, why not start earlier and question the existence of the set of natural numbers?)
A lot of work has been done on the subject of large cardinals and besides
Reinhardt cardinals, nothing has ever turned out to be inconsistent.
There is the so-called inner model program where one assumes the existence of a certain large
cardinal and tries to build an (easily controllable) smallest model of set theory in which there is such a large cardinal and which contains all the ordinals.
The idea is that because we have a good understanding of the final inner model,
we would notice during the construction of the model if there were any problems with the consistency of the large cardinal in question.
This program has worked out so far to rather high levels of the hierarchy of large cardinals.
Another important point to believe in the consistency of large cardinals is the fact that
the consistency strengths of large cardinals are apparently linearly ordered.
In other words, it has not yet happened that there is a natural notion of a large cardinal which cannot be compared to the other known large cardinals in terms of consistency strength (a large cardinal is stronger that the other if the consistency of the first implies the consistency of the second). That indicates that there is a natural direction in which
set theory can be strengthened, which is a remarkable (heuristic) observation.
(For certain types of large cardinal axioms it can actually be proved that they form a
linearly ordered hierarchy.)
I find it unlikely that we should see this linearly ordered structure above a large cardinal whose existence is actually inconsistent.
So, if there is a natural direction to strengthen our basic theory, why not go all the way and work in the strongest theory in that direction, by assuming the existence of all
(consistent) large cardinals. Among other things, it has turned out that the existence of large cardinals implies a rather nice structure theory for the subsets of the real line
that are definable is a certain sense (projective sets).
And as pointed out above, there are good reasons to believe in the consistency of large cardinals.
Concerning arguments against large cardinals, I would think that the main objection is that large cardinals have no effect on ordinary mathematics. But as I have pointed out in the previous paragraph, this is not entirely true. Moreover, even though this is cumbersome, most of "ordinary mathematics" can actually be carried out in weak systems of number theory.