Let $X$ be a smooth projective complex variety and $t\in H^{2p}(X,\mathbf{Q})(p)$ a rational cohomology class.
Assume that there exist $$t_1,\ldots,t_N\in H^{2*}(X,\mathbf{Q})(*)$$ algebraic classes (i.e. in the image of the cycle map $cl : A^*(X)\to H^{2*}(X,\mathbf{Q})(*)$) and an algebraic subgroup $$G\subset\text{Aut}(H^*(X,\mathbf{Q}))\times\text{GL}(\mathbf{Q}(1))$$ (automorphisms as a graded $\mathbf{Q}$-algebra) fixing the $t_1,\ldots, t_N$ and largest with respect to this property, such that $G$ also fixes $t$.
Is $t$ algebraic?
From papers of Milne, one can deduce this is true under the variational Hodge conjecture. However, I wonder if this is already known unconditionally.