There must not be a general result, as even a scalar 2nd-order equation with variable coefficients does not have explicit solutions. The operator $u\mapsto Lu:=u''-Au'-Bu$ from $H_1^0(0,1)$ to $H^{-1}(0,1)$ is Fredholm of index $0$, as a compact perturbation of $u\mapsto u''$. Therefore the existence of a non-trivial solution is the exception, and the non-existence is the rule. Notice that non-existence means that the non-homogeneous BVP
$$u''(t)=A(t)u'(t)+B(t)u(t)+f(t),\qquad u(0)=u(1)=0$$
is uniquely solvable for every $f\in H^{-1}(0,1)$, by Fredholm alternative.

When the equation is scalar (not a system), the maximum principle can be used to get information. For instance, the eigenvalues of $L$ are real. Also, if there exists a $w>0$ such that
$$w''\le A(t)w'+B(t)w,$$
then the only solution is the trivial one. But this is not a necessary condition, because it is equivalent to the fact that the spectrum of $L$ is negative.

It happens, in the stability theory of travelling waves, that we are interested in the sign of the real parts of the eigenvalues of $L$~; the reason is that we have linearized systems of the form
$$\frac{dv}{dt}=\frac{d^2v}{dx^2}-A(x)\frac{dv}{dx}-B(x)v.$$
All of the eigenvalues but finitely many have negative real parts. When $A$ and $B$ are real-valued, and have limits $A_\pm$ and $B_\pm$ as $x\rightarrow\pm\infty$, it is possible to define, and sometimes compute explicitly, an index which gives the parity of the number of eigenvalues in the right half of the complex plane. This is *Evans function theory*. It gives a necessary condition for stability (the index must be even), which is not sufficient. An important open problem is to extend this theory in the direction of indices $\mod 2^m$, like the one we have for finite dimensional operators ($=$ matrices, Routh-Hurwitz theory).

**Late edit**. In the case of the stability of travelling waves, suppose that the original problem, which was nonlinear, had constant coefficients. In other words, it was invariant under space-time translation. Then if $u(x-ct)$ is a travelling wave (say $c=0$, up to the change of a reference frame), then $u(x+h)$ is an other travelling wave, for every constant $h$. This implies that $Lu'=0$. Because $u$ has finite limirs at infinity and is the solution of some autonomous ODE, $u'$ vanishes at infinity, generically with an exponential decay rate. Therefore $\lambda=0$ is an eigenvalue of $L$. Here, we do have a non-trivial solution of the linear BVP. But it does not fit the MO question, because the domain is $\mathbb R$, not an interval.