Suppose $C$ is a non-small model category. As in Hovey's book, a category is taken to mean a class of objects with a set of maps between any two. Note that $End(C)$ will not in general be a category because it can have a class worth of maps between any two objects (natural transformations take a value for each object in $C$). You can get around this in a few ways:
- Pass to a larger Grothendieck Universe. Basically the idea is that you can start set theory by fixing an inaccessible cardinal $\kappa$ and declaring a set to be a set if its cardinality is smaller and a class if its cardinality is bigger. If a construction takes you out of the realm of sets you simply go up to a larger inaccessible cardinal. This is what Lurie does in Higher Topos Theory (page 51).
- Place a smallness hypothesis before passing to $End(C)$. If you require $C$ to be small then $End(C)$ is a category but there is no interesting homotopy theory. You could also restrict attention to small functors and then the machinery of Chorny mentioned in the comments applies.
You've remarked that you don't want to do (2), so I think you're stuck with doing (1). Unfortunately, this can interfere with your ability to do homotopy theory in $End(C)$, because if you pass to a larger Grothendieck Universe then you lose (co)completeness. Again you have several options I can see:
Forget the model structure on $C$ and treat $C$ as an infinity category instead. One of the punchlines of Lurie's machinery is that functors between two infinity categories again form an infinity category (after passage to a larger Grothendieck Universe). The reason is that there is no (co)completeness requirement in the definition of an infinity category. Depending on your choice of model for $(\infty,1)$-theory you can define the weak equivalences to be the Dwyer-Kan equivalences of relative categories or simplicial categories, the weak equivalences of simplicial sets in the Joyal model structure if using Quasi-categories, etc. You no longer have a notion of cofibration or fibration, so choosing the weak equivalences tells you what the infinity category structure on $End(C)$ is.
Work with Hovey's 2-category of model categories, then restrict attention to the 1-morphisms between $C$ and $C$. Before there was a class worth of these, as well as a class of 2-morphisms between the 1-morphisms. But now that we're working in this larger Grothendieck Universe $End(C)$ will be a small category in that universe. The weak equivalences will be whatever the 2-weak equivalences in the 2-category of model categories are. According to the Vistas section of Hovey's book this should be natural transformations which are natural weak equivalence when restricted to cofibrant objects. Unfortunately, Hovey comments that he doesn't know what the cofibrations or fibrations should be, and even worse this category doesn't appear to be (co)complete.
Invent a new notion of model category, which has only (co)limits up to some cardinal. For example, let's say $M$ is a $\kappa$-model category if it's $\kappa$ (co)complete. If $\kappa$ is the inaccessible cardinal indexing your universe then $M$ is (co)complete. Once you pass to a larger Grothendieck Universe (indexed by $\kappa'$ say) you may not have all (co)limits. But because (co)limits of functors are computed objectwise you should still have all $\kappa$-colimits. So you can safely call $End(M$) a $\kappa$-model category even though it's not a model category. Presumably $\kappa$-colimits would be enough for whatever constructions you'd like to make because they cover all the colimits coming from $M$.
If you choose option 1 you're done. Since you asked for a model category I'll assume you've chosen option 2 or 3. Choosing option 2 will force you into option 3 anyway if you want to be (co)complete in some sense. Since I just made up the notion of a $\kappa$-model category no one has worked out the theory of these things (I'm interested in thinking about this for a future research project, but it might be non-trivial). In any event, if the theory turned out nicely then option 3 would give you several choices of model structure. Forgetting that the domain category is a model category, and treating it just as an index category you could take either the projective or injective model structures on $End(M)$. In the former weak equivalences and fibrations are functors $F$ such that $F(f)$ is a weak equivalence or fibration for all $f$. The latter is similar but with weak equivalences and cofibrations. Once Chorny got around the set theoretic issues by choosing small functors these were the model structures he found.
If you remember that the domain category is a model category then you could say $F$ is a weak equivalence if it takes weak equivalences to weak equivalences and $F$ is a fibration if it takes fibrations to fibrations. Of course there's also the dual thing with cofibrations. I'm not sure if these give model structures; I suspect not. It seems we're having the same trouble as in option 2, i.e. we don't have a good sense of what the fibrations and cofibrations should be. For weak equivalences a natural choice is left Quillen equivalences or perhaps DK equivalences. Figuring out an answer to this seems like hard work but work worth doing. Again, I'd be interested in this as a research project, and indeed it's related to a few things in the research statement hosted on my website.
