Let $k$ be an algebraically closed field of characteristic zero and let $X$ be a projective variety over $k$. To try and answer your first two questions, let us try formalize the property that Hom-schemes $\mathrm{Hom}(Y,X)$ are finite unions of quasi-projective schemes.
Definition. We say that $X$ is bounded over $k$ if, for every normal projective variety $Y$ over $k$, the Hom-scheme $\mathrm{Hom}(Y,X)$ is of finite type over $k$.
We can then show the following result; see [1] and [2].
(I think that your question is only concerned with 1 and 6 in the following result, but the rest might also be useful to you.)
Theorem. The following are equivalent.
- $X$ is bounded over $k$.
- For every algebraically closed field $L$ containing $k$, the projective variety $X_L$ is bounded over $L$.
- For every smooth projective curve $C$ over $k$, the Hom-scheme $Hom(C,X)$ is of finite type.
- For every normal projective variety $Y$ over $k$, the Hom-scheme $Hom(Y,X)$ has only finitely many connected components.
- For every normal projective variety $Y$ over $k$, the Hom-scheme $Hom(Y,X)$ is quasi-projective.
- For every normal projective variety $Y$ over $k$, the Hom-scheme $Hom(Y,X)$ is a projective scheme over $k$.
- For every normal projective variety $Y$ over $k$, the Hom-scheme $Hom(Y,X)$ is a projective scheme over $k$. The subscheme of non-connected morphisms $Hom^{nc}(Y,X)$ is a bounded projective scheme of dimension $<\dim X$ which maps finitely to $X$.
Boundedness is most likely equivalent to "hyperbolicity". In fact, it is implied by hyperbolicity (Kobayashi, Brody, ...). More precisely:
Theorem. Assume $k=\mathbb{C}$. If $X$ is hyperbolic (i.e., every holomorphic map $\mathbb{C}\to X^{an}$ is constant), then $X$ is bounded.
Conjecture. Assume $k=\mathbb{C}$. If $X$ is bounded over $\mathbb{C}$, then $X$ is hyperbolic.
This should answer your first question. For your second question: if $X$ has no rational curves, then every Hom-scheme $Hom(Y,X)$ has projective components (but it can have infinitely many components). Conversely, if all Hom-schemes $Hom(Y,X)$ have (only) projective components, then $X$ has no rational curves. (Use that the components of $Hom(\mathbb{P}^1,\mathbb{P}^1)$ are affine varieties of increasing dimension.)
Abelian varieties have no rational curves and give examples of non-finite type Hom-schemes with each component projective. Note that non-trivial abelian varieties are far from being hyperbolic, so there's no contradiction to the above conjecture.
References.
[1] R. van Bommel, A. Javanpeykar, L. Kamenova.
Boundedness in families with applications to arithmetic hyperbolicity
https://arxiv.org/abs/1907.11225
[2] A. Javanpeykar and L. Kamenova
Demailly's notion of algebraic hyperbolicity: geometricity, boundedness, moduli of maps
https://arxiv.org/abs/1807.03665