Francesco Polizzi's idea is enough to solve the problem:
Lemma. Let $f = a_3x^3 + a_2x^2 + a_1x + a_0 \in \mathbf Z[x]$ nonconstant with $a_i \geq 0$ and $a_0 > 0$, such that $f(1)$ is a prime $p > 2$. Then $f$ is irreducible.
Proof. Suppose $f = gh$ for $g,h \in \mathbf Z[x]$; we must show that $g \in \{\pm 1\}$ or $h \in \{\pm 1\}$. Write $g = b_nx^n + \ldots + b_0$ and $h = c_mx^m + \ldots + c_0$ with $b_n \neq 0$ and $c_m \neq 0$.
First assume $\deg g > 0$ and $\deg h > 0$. Since $\deg f \leq 3$, without loss of generality we may assume $\deg g = 1$ and $m = \deg h \in \{1,2\}$, as well as $b_1, c_m > 0$. If $b_0 < 0$, then $f$ has a positive real root, which contradicts Descartes's rule of signs (this can be easily done by hand in this case as well). Moreover, $a_0 > 0$ rules out $b_0 = 0$, so we must have $b_0 > 0$. But then $g(1) \geq 2$, so in fact $g(1) = p$ and $h(1) = 1$. By symmetry, this rules out the case $\deg h = 1$. Finally, if $\deg h = 2$, it is convenient now to adopt easier notation: \begin{align*} g = ux + v, & & h = ax^2+bx+c, \end{align*} with $u,v,a > 0$. The formulas \begin{align*} a_3 = au, & & a_2 = bu + av, & & a_1 = cu + bv, & & a_0 = cv \end{align*} give \begin{align*} a > 0, & & av \geq -bu, & & cu \geq -bv, & & c > 0.\label{1}\tag{1} \end{align*} We will show that $h(1) \geq 2$, contradicting the earlier assertion that $h(1) = 1$. This is clear if $b \geq 0$, since we already have $a > 0$ and $c > 0$. If $b < 0$, then \eqref{1} shows $$\frac{c}{-b} \geq \frac{v}{u} \geq \frac{-b}{a} > 0,\label{2}\tag{2}$$ hence $$h(1) = a + b + c \geq \left(1 - \left(\frac{-b}{a}\right) + \left(\frac{-b}{a}\right)^2 \right)a.$$ The function $1-x+x^2$ attains a minimum of $\tfrac{3}{4}$ at $x = \tfrac{1}{2}$, and is bigger than $1$ on $(1,\infty)$. Thus, we only have to consider $x = \tfrac{-b}{a} \in (0,1]$. For $x < 1$, we must have $a \geq 2$, so $(1-x+x^2)a \geq \tfrac{3}{2} > 1$. Finally, for $x = 1$, i.e. $b = -a$, \eqref{2} shows that $v \geq u$. Since $p = v+u$ is odd, we must have $v > u$, so \eqref{2} gives $c > -b$, so $h(1) = c \geq 2$.
Thus, we conclude that $\deg g = 0$ or $\deg h = 0$; say $g = c$ for $c \in \mathbf Z_{>0}$ without loss of generality. If $c > 1$, then $c = p$, which is impossible since $f(0) = d \in \{1, \ldots, p-1\}$ is not divisible by $p$. So we conclude that $g = 1$. $\square$