Are any examples known of an integer matrix $A$, such that the largest prime divisor of some specified entry of $A^n$ grows exponentially in $n$? How about where it just grows strictly faster than any polynomial in $n$?

For example, it is a conjecture that there are infinitely many primes among the Fibonacci numbers, and if that was proven, the matrix $$\begin{pmatrix} 1 & 1 \\ 1 & 0\end{pmatrix}$$ would work. But as far as I can tell, it is not even known if the largest prime divisor of the Fibonacci numbers grows exponentially (at least nothing turned up in my attempt to search for such results).

My motivation for asking is the work of Geordie Williamson on counterexamples to Lusztig's conjecture (http://people.mpim-bonn.mpg.de/geordie/Torsion.pdf), where he obtains counterexamples in $SL_m$ for primes dividing certain entries of the $n$'th power of some specific matrices (and where $m$ is linear in $n$). To be able to rule out a polynomial bound above which the conjecture is true, one would like to find a matrix as described above which can also be obtained in the way described in the linked paper.

Finding which matrices can be obtained as described in that paper seems very hard, but it might help if one had a certain goal in mind, hence why I am looking for examples like above.