The answer is yes. (I suppose that $V=E$ in your statement.) This might be too complicated, and it's messy, but it's the first that came to mind. (The idea is really quite simple.)
I will write $j_*$ for the derived functor and ${}^\circ j_*$ for its ordinary $H^0$.
1/ First suppose that your local system $E$ is trivial. Then the statement you want is well-known, and a reference is lemma 4.3.2 of Astérisque 100.
2/ General case. Write $K=E[d]$, where $d=\dim X$ and $K'={}^\circ j_* E[d]$. So we're trying to prove that $j_{!*}K=K'$.
The problem is local, so you can assume that $D$ is defined by a global equation $t_1\dots t_r=0$. Let $X'=X[t_i^{1/N}]$ (ie the subscheme of $X\times\mathbb{A}^r$ defined by the equations $T_i^N=t_i$, where $T_1,\dots,T_r$ are the coordinates on $\mathbb{A}^r$). Let $\pi:X'\rightarrow X$ be the projection, $U'$ be the inverse image of $U:=X-D$ in $X'$ and $j':U'\rightarrow X'$ the inclusion. Then $\pi$ is finite, and its restriction to $U'$ is étale. As the monodromy of $E$ is finite, if we take $N$ big enough, then $\pi^* E$ is trivial; write $E'$ for its obvious extension to $X'$, i.e., ${}^\circ j'_*\pi^*E$. Then $j'_{!*}\pi^*K=E'[d]$ by 1/. Note that $j_{!*}\pi_*\pi^*K$ is a direct summand of $\pi_*j'_{!*}\pi^*K$ (this is a very particular case of the decomposition theorem).
Using the trace map, we see that $E$ (resp. $K$) is a direct summand of $\pi_*\pi^* E$ (resp. $\pi_*\pi^*K$). So $j_{!*}K$ is a direct summand of $\pi_* E'[d]$ and ${}^\circ j_* E$ is a direct summand of $\pi_* E'$ (note that $\pi_*$ is exact in the ordinary sense and in the perverse sense). In particular, $j_{!*}K[-d]$ is an ordinary sheaf and ${}^\circ j_*E[d]$ is a perverse sheaf.
Using that $j_{!*}K[-d]$ is an ordinary sheaf, we get that the canonical morphism $j_{!*}K\rightarrow j_*K$ factors through a morphism $j_{!*}K\rightarrow {}^\circ j_* E[d]$, and this morphism is the identity on $U$ so it has to be injective. By the decomposition theorem again, ${}^\circ j_*E[d]=j_{!*}K\oplus L$, where $L$ is a perverse sheaf supported on $D$ and is such that $L[-d]$ is an ordinary sheaf. Let $i:D\rightarrow X$ be the inclusion. Then $i^!L[-d]$ is still an ordinary sheaf. But it is a direct factor of $i^! {}^\circ j_* E$, and, applying $i^!$ to the exact triangle ${}^\circ j_* E\rightarrow j_* E\rightarrow \tau_{\geq 1}j_* E$ (and using that $i^! j_* E=0$), we get $i^! {}^\circ j_* E=i^!\tau_{\geq 1}j_* E[-1]$. This is concentrated in (ordinary) degree $\geq 2$, hence $L=0$.
