Not much of a criterion, but here is a family of examples: this holds for $F=\mathbb Q_p$ and $E$ any finite extension. To see that $F$ is rigid and any automorphism of $E$ fixes it, it is enough to note that any automorphism of $E$ is continuous, since $\mathbb Q$ will then be pointwise fixed by an automorphism, and its closure is $\mathbb Q_p$.

To see any automorphism of $E$ is continuous, first we show that its ring of integers $O_E$ is first-order definable in $E$. Indeed, let $e$ be the ramification degree of $E/F$. I claim that there exist $a,b,c\in\mathbb N$ such that for $x\in E$ we have $x\in O_E$ iff $1+p^ax^b=y^c$ has a solution $y\in E$.

First note that if $x\in O_E$, the polynomial $f(T)=T^a-(1+p^bx^c)$ satisfies $|f(1)|=|p^bx^c|\leq |p^b|=p^{-b}$, while $|f'(1)|=|a|$ (throughout, the absolute value is the $p$-adic one). If $p^{-b}<|a|^2$, then from a suitable version of Hensel's lemma we get that $f$ has a root.

On the other hand, if $x\not\in O_E$, then $|x|=p^{d/e}$ for some $d\geq 1$. If $c>be$, then $|p^bx^c|=p^{-b}|x|^c\geq p^{-b}p^{c/e}>1$, which shows $|1+p^bx^c|=|p^bx^c|=p^{-b+cd/e}=p^{(cd-be)/e}$. If we arrange that $cd-be$ is never divisible by $a$, then this will restrict $|1+p^bx^c|$ from being an $a$-th power. All of these conditions can be fulfilled by taking $a$ to be a prime not dividing $e$, $b$ sufficiently large and not divisible by $a$, and $c$ a multiple of $a$ larger than $be$, and this gives us the Diophantine description of $O_E$ in $E$.

By above, any automorphism of $E$ must preserve $O_E$, and hence also every set $p^nO_E,n\in\mathbb N$. Since the latter form a basis of neighbourhoods of $0$ in $E$, this gives the desired continuity.

This can be extended to some finite extensions $F/\mathbb Q_p$ and extensions $E/F$ such that no automorphism of $E/\mathbb Q_p$ is identity on $F$. Those can be without much difficulty constructed using Galois theory.