It partly depends on how much you want to be able to integrate and how much machinery you can take for granted (factorization of polynomials, rewriting trig functions in terms of tan for example.) Is this mainly curiosity?
This is probably not news to you, but if differentiation is automatic and you can put in a suggested form for the answer, then undetermined coefficients is "easy." So if you want to evaluate $$\int x^3\cos^5xe^{3x}dx$$ and can tell the program that the answer should be of the form $$\sum_{n=0}^2 [(a_{n}\sin(x)\cos^{2n}(x)+b_{n}\cos^{2n+1}(x))e^{3x}+(c_{n}\sin(x)\cos^{2n}(x)+d_{n}\cos^{2n+1}(x))]$$ where the $a,b,c,d$ are all polynomials of degree $3$ in $x$, then the program has, after differentiation, a system of linear equations in 48 variables which either has a solution or does not. If we knew less about the solution we might have thrown in more terms. In fact we might know that the $c$ and $d$ polynomials are $0$ so this cuts it down to a more manageable $24$ unknowns and linear algebra. It might be smoother to use multiple angles like $\cos(5x)$ in place of powers.
My impression is that a large part of machine symbolic integration consists of having the machinery to efficiently give a form for the possible solution and check for one, along with the theorems to say that when this fails there is no closed form solution in terms of the menu of functions allowed.