I came up with the following proposition, but don't know how to prove it. I used Maple to see that it holds when $ a + b + c + d <300 $.
Let $a,b,c$ and $d$ be non-negative integers such that $d\geq1$ and $a+b+c\geq1$.
If $a+b+c+d$ is a prime number other than $2$, the polynomial $ax^3+bx^2+cx+d$ is irreducible over $Z[x]$.
Francesco Polizzi's idea is enough to solve the problem:
Lemma. Let $f = ax^3 + bx^2 + cx + d \in \mathbf Z_{\geq 0}[x]$ nonconstant with $d > 0$, such that $f(1)$ is a prime $p > 2$. Then $f$ is irreducible in $\mathbf Z[x]$.
Proof. Suppose $f = gh$ for $g,h \in \mathbf Z[x]$; we must show that $g \in \{\pm 1\}$ or $h \in \{\pm 1\}$.
First assume $\deg g > 0$ and $\deg h > 0$. Since $\deg f \leq 3$, without loss of generality we may assume $\deg g = 1$ and $m = \deg h \in \{1,2\}$; say \begin{align*} g = ux + v, & & h = \alpha x^2 + \beta x + \gamma , \end{align*} with $u > 0$ and $\alpha > 0$ or $\alpha = 0$ and $\beta > 0$. If $v \leq 0$, then $f$ has a nonnegative real root, which is impossible because $f$ has non-negative coefficients and $f(0) > 0$. Thus, $v > 0$, and therefore $g(1) \geq 2$, forcing $g(1) = p$ and $h(1) = 1$. By symmetry, this rules out the case $\deg h = 1$. Thus, we may assume $\alpha > 0$. The formulas \begin{align*} a = \alpha u, & & b = \beta u + \alpha v, & & c = \gamma u + \beta v, & & d = \gamma v \end{align*} give \begin{align*} \alpha > 0, & & \alpha v \geq -\beta u, & & \gamma u \geq -\beta v, & & \gamma > 0.\label{1}\tag{1} \end{align*} We will show that $h(1) \geq 2$, contradicting the earlier assertion that $h(1) = 1$. This is clear if $\beta \geq 0$, since we already have $\alpha > 0$ and $\gamma > 0$. If $\beta < 0$, then \eqref{1} shows $$\frac{\gamma}{-\beta} \geq \frac{v}{u} \geq \frac{-\beta}{\alpha} > 0,\label{2}\tag{2}$$ hence \begin{align*} h(1) = \alpha + \beta + \gamma &= \left(1 - \left(\frac{-\beta}{\alpha}\right) + \left(\frac{\gamma}{-\beta}\right)\left(\frac{-\beta}{\alpha}\right)\right)\alpha\\ &\geq \left(1 - \left(\frac{-\beta}{\alpha}\right) + \left(\frac{-\beta}{\alpha}\right)^2 \right)\alpha. \end{align*} The function $1-x+x^2$ attains a minimum of $\tfrac{3}{4}$ at $x = \tfrac{1}{2}$, and is bigger than $1$ on $(1,\infty)$. Thus, we only have to consider $x = \tfrac{-\beta}{\alpha} \in (0,1]$. For $x < 1$, we must have $\alpha \geq 2$, so $(1-x+x^2)\alpha \geq \tfrac{3}{2} > 1$. Finally, for $x = 1$, i.e. $\beta = -\alpha$, \eqref{2} shows that $v \geq u$. Since $p = v+u$ is odd, we must have $v > u$, so \eqref{2} gives $\gamma > -\beta$, so $h(1) = \gamma \geq 2$.
Thus, we conclude that $\deg g = 0$ or $\deg h = 0$; say $g = n$ for $n \in \mathbf Z_{>0}$ without loss of generality. If $n > 1$, then $n = p$, which is impossible since $f(0) = d \in \{1, \ldots, p-1\}$ is not divisible by $p$. So we conclude that $g = 1$. $\square$
Remark. The argument above relies crucially on the assumption $\deg f \leq 3$. More importantly, for higher degree polynomials the same result is false without a lower bound on $p$ that grows at least linearly in $\deg f$:
Example. Let $p$ be an odd prime. Then the polynomial $f = x^{2p-2} + x^{2p-4} + \ldots + x^2 + 1$ has $f(1) = p$, but it factors as $$\Big(x^{p-1}+x^{p-2}+\ldots+x+1\Big)\Big(x^{p-1}-x^{p-2}+\ldots-x+1\Big).$$ Indeed, the above reads $$\zeta_p(x^2) = \zeta_p(x)\zeta_{2p}(x) = \zeta_p(x)\zeta_p(-x),$$ which is true because both sides are monic and have the same roots in $\mathbf C$ (namely the $2p^\text{th}$ roots of unity except $\pm 1$, all with multiplicity $1$).
