Counterexample. Let $E_1=\{e_0,e_1\}$ and $E_2=\{e_0,e_2\}$ where $$e_0=\{0,1\}\cup\{2,4,6,8,\dots\},$$ $$e_1=\{0\}\cup\{3,5,7,9,\dots\},$$ $$e_2=\{0,1\}\cup\{3,5,7,9,\dots\}.$$ Then $(\omega,E_1)\not\cong(\omega,E_2)$, but $(E_1,V_1^*)\cong(E_2,V_2^*)$ since $V_1^*=\{\{e_0\},\{e_1\},\{e_0,e_1\}\}$ and $V_2^*=\{\{e_0\},\{e_2\},\{e_0,e_2\}\}.$
On the other hand, if $H=(V,E)$ is a hypergraph with the property that $E_v\ne E_w$ whenever $v\ne w$, then the dual hypergraph $H^*$ has the same property, and $(H^*)^*\cong H$. Hence, if $H_1$ and $H_2$ are two such hypergraphs, then $H_1^*\cong H_2^*\implies H_1\cong H_2$.