There must not be a general result, as even a scalar 2nd-order equation with variable coefficients does not have explicit solutions. The operator $u\mapsto Lu:=u''-Au'-Bu$ from $H_1^0(0,1)$ to $H^{-1}(0,1)$ is Fredholm of index $0$, as a compact perturbation of $u\mapsto u''$. Therefore the existence of a non-trivial solution is the exception, and the non-existence is the rule. Notice that non-existence means that the non-homogeneous BVP
$$u''(t)=A(t)u'(t)+B(t)u(t)+f(t),\qquad u(0)=u(1)=0$$
is uniquely solvable for every $f\in H^{-1}(0,1)$, by Fredholm alternative.
When the equation is scalar (not a system), the maximum principle can be used to get information. For instance, the eigenvalues of $L$ are real. Also, if there exists a $w>0$ such that
$$w''\le A(t)w'+B(t)w,$$
then the only solution is the trivial one. But this is not a necessary condition, because it is equivalent to the fact that the spectrum of $L$ is negative.
It happens, in the stability theory of travelling waves, that we are interested in the sign of the real parts of the eigenvalues of $L$~; the reason is that we have linearized systems of the form
$$\frac{dv}{dt}=\frac{d^2v}{dx^2}-A(x)\frac{dv}{dx}-B(x)v.$$
All of the eigenvalues but finitely many have negative real parts. When $A$ and $B$ are real-valued, and have limits $A_\pm$ and $B_\pm$ as $x\rightarrow\pm\infty$, it is possible to define, and sometimes compute explicitly, an index which gives the parity of the number of eigenvalues in the right half of the complex plane. This is Evans function theory. It gives a necessary condition for stability (the index must be even), which is not sufficient. An important open problem is to extend this theory in the direction of indices $\mod 2^m$, like the one we have for finite dimensional operators ($=$ matrices, Routh-Hurwitz theory).
Late edit. In the case of the stability of travelling waves, suppose that the original problem, which was nonlinear, had constant coefficients. In other words, it was invariant under space-time translation. Then if $u(x-ct)$ is a travelling wave (say $c=0$, up to the change of a reference frame), then $u(x+h)$ is an other travelling wave, for every constant $h$. This implies that $Lu'=0$. Because $u$ has finite limirs at infinity and is the solution of some autonomous ODE, $u'$ vanishes at infinity, generically with an exponential decay rate. Therefore $\lambda=0$ is an eigenvalue of $L$. Here, we do have a non-trivial solution of the linear BVP. But it does not fit the MO question, because the domain is $\mathbb R$, not an interval.
