The following counterexample is very similar to exercise 2 to Bourbaki's Algèbre commutative ch. VI §8 and works as well. Actually it is example 4.9 in Scholze's Perfectoid Spaces; there, the main point is that $B$ is an "almost finitely generated" $A$-module in the sense of "almost ring theory", so there is a way around the fact that it is not finitely generated. This fact, however, makes up our counterexample.
Let $p$ be an odd prime and for all $n \ge 1$, adjoin a $p^n$-th root of $p$, call it $\pi_n$, to the field of $p$-adic numbers $\mathbb{Q}_p$. The $p$-adic valuation uniquely extends to this field; take its completion with respect to this value. This is our field $K = Frac(A)$ with its local valuation ring $(A, \mathfrak{m})$. Note that the valuation $v: K^\times \rightarrow \mathbb{R}$ is of rank 1, but non-discrete; in fact, its value group is $\bigcup_{n \ge 1} \frac{1}{p^n} \mathbb{Z} = \mathbb{Z} [\frac{1}{p}]$. Because the rank is 1 and the field is complete, the valuation is Henselian (exercise 6b in Bourbaki, loc. cit.; first sentence of paragraph 4.1 in Engler-Prestel).
Now adjoin a square root of $p$ to get the quadratic field extension $L|K$, and extend $v$ to $L$. Convince yourself that for each $n \ge 0$, there is an element in $L$, call it $\rho_n$, which is a $2p^n$-th root of $p$ and thus is a primitive element for $L|K$. The value group of $L$ is $\frac{1}{2} \mathbb{Z}[\frac{1}{p}]$ and it is not hard to see that the valuation ring $B = \lbrace x \in L: v(x) \ge 0 \rbrace$ can be written as
$$B = A \oplus \bigcup_{n \ge 0} \rho_n A$$
as $A$-module. But because $v(\rho_n) = \frac{1}{2p^n} > \frac{1}{p^{n+1}} = v(\pi_{n+1})$ and $\pi_{n+1} \in \mathfrak{m}$, we have $\rho_n \in \mathfrak{m}B$ for all $n$, hence whole union on the right is contained in $\mathfrak{m}B$. So as $A$-modules,
$$B = A + \mathfrak{m}B .$$
Now if $B$ were finitely generated over $A$, Nakayama's lemma would imply $A = B$, which is absurd since $\rho_n \in B \setminus A$.
Positive results: We have theorem 2 in Bourbaki's Algèbre commutative ch. VI §8 no. 5 which gives (in a more general setup, without the Henselian assumption) equivalent criteria for $B$ being a finitely generated $A$-module, especially the famous equality $[L:K] = \sum e_i f_i$. In Neukirch's Algebraic Number Theory ch. II §6, where the setup has a Henselian rank 1 valuation $v$, this equality is proven for $L|K$ separable and $v$ discrete; it is remarked that "both conditions are really necessary" but that for a complete field, one may drop the separability. Corresponding statements can be found in Serre's Local Fields ch. I §4 and ch. II §2, and also in Engler-Prestel, theorem 3.3.5. I guess that exercise 3 in Bourbaki loc. cit. is a counterexample with $v$ discrete and $L|K$ inseparable, I just do not see right away whether $v$ is Henselian there.