I assume you want to fix the prime $p$, and I assume you want $f$ to be the minimal polynomial of $\alpha$ over $\mathbb{Q}$. If you don't fix $p$, then $x-6$ is a counter-example (there are two valuations for which $v(6)>0$, but $x-6 \neq (x-6)^2$, even up to units.) If $f$ is not the minimimal polynomial then $(x-2)(x-4)$ is a counterexample.

Subject to these caveats, you are right.

I find it really confusing to multiply over all places as you are doing. What I would rather do is the following: Fix a normal extension $L$ of $\mathbb{Q}$ in which $f$ splits. Let $\alpha_1$, ..., $\alpha_N$ be the roots of $f$ in $L$. Factor $f$ over $\mathbb{Q}_p$ as $g_1 g_2 \cdots g_k$ and group the $\alpha_i$ together if they are roots of the same $g_j$. Fix an extension $v$ of the $p$-adic valuation to $L$. Then there are $k$ different $p$-adic valuations of $\mathbb{Q}(\alpha)$. They each arise as follows: For each $g_i$, choose a root $\alpha_i$ and let $\phi_i : \mathbb{Q}(\alpha) \to L$ be the map induced by $\alpha \mapsto \alpha_i$. Then your valuation is $v \circ \phi_i$.

Suppose that $v(\alpha_i)>0$ for the roots of $g_1$, $g_2$, ..., $g_s$ and $v(\alpha_i) \leq 0$ for $g_{s+1}$, \cdots, $g_r$. So you are interested in the first $s$ valuations.

Here are the lemmas you need; I'll leave them as exercises. I'm pretty sure they work.

**Lemma 1:** For $i>s$, the polynomial $g_i$ is a unit in $\mathbb{Z}_p[[x]]$

**Lemma 2:** For $i \leq s$, the polynomial $g_i$ can be written as $h_i u_i$ where $u_i$ is a unit in $\mathbb{Z}_p[[x]]$ and $h_i \in \mathbb{Z}[[x]]$.

**Lemma 3:** The kernel of $\mathbb{Z}[[x]] \mapsto L_v$, where $x$ goes to a root of $g_i$, is generated by $h_i$.

Proof of Lemma 3: If the map were from $\mathbb{Z}_p[[x]]$, then the kernel would be generated by $g_i$. Since $h_i = g_i u_i^{-1}$, we see that $h_i$ is in the kernel. If $q$ is in the kernel, then $q=g_i v = h_i u_i v$ in $\mathbb{Z}_p[[x]]$. Since $h_i$ and $q$ are both in $\mathbb{Z}[[x]]$, we see that the coefficients of $u_i v$ are in $\mathbb{Q}$, so they are in $\mathbb{Q} \cap \mathbb{Z}_p = \mathbb{Z}$. Thus, $q$ is a multiple of $h_i$ in $\mathbb{Z}[[x]]$.

We take $h_i$ as your representative of the kernel. Then $f$ is $\prod h_i$ times a unit $u$ of $\mathbb{Z}_p[[x]]$. But $f$ and $\prod h_i$ are both in $\mathbb{Z}[[x]]$, so $u$ and $u^{-1}$ are in $\mathbb{Q}[[x]]$ and we deduce that $u$ is a unit of $\mathbb{Z}[[x]]$, as desired.

Remark: Lemma 2 should be thought of as an analogue of the Weierstrass preparation theorem, although it is not the standard $p$-adic analogue. To see this, note that the (formal) Weierstrass preparation theorem is

If $g$ is a power series in $x$ and $y$ then $g=uh$ where $u$ is a unit of $k[[x,y]]$ and $h$ is in $k[[y]][x]$.

The standard $p$-adic analogue is: If $g \in \mathbb{Z}_p[[x]]$ then $g=uh$ with $u$ is a unit of $\mathbb{Z}_p[[x]]$ and $h$ is in $\mathbb{Z}_p[x]$.

Lemma 2 goes in the "other" direction: If $g \in \mathbb{Z}_p[[x]]$ then $g=uh$ with $u$ is a unit of $\mathbb{Z}_p[[x]]$ and $h$ is in $\mathbb{Z}[[x]]$.