If you allow $M$ to be any class, you get a contradiction by taking $M=V$.

Meanwhile, dropping transitivity doesn't actually add anything: given $M\supseteq\kappa$, just consider $\hat{M}=$ the Mostowski collapse of $M$. Given $\alpha<\kappa$ and $j:\hat{M}\rightarrow \hat{M}$ nontrivial elementary with $\alpha<crit(j)<\kappa$, $j$ lifts to a nontrivial elementary embedding $\hat{j}$ of $M$ into itself with $\alpha<crit(\hat{j})<\kappa$. So if $\kappa$ is Berkeley in the usual sense, it's Berkeley in the nontransitive sense too.

Finally, if you drop transitivity and replace "$\kappa\subseteq M$" by the original "$\kappa\in M$," then the argument above breaks down but the notion becomes inconsistent: taking $M=\{\kappa\}$ doesn't leave room for any nontrivial elementary embeddings, for example, let alone ones with critical points (and we can cook up less-silly examples too).

Dropping transitivity doesn't actually add anything: given $M\supseteq\kappa$, just consider $\hat{M}=$ the Mostowski collapse of $M$. Given $\alpha<\kappa$ and $j:\hat{M}\rightarrow \hat{M}$ nontrivial elementary with $\alpha<crit(j)<\kappa$, $j$ lifts to a nontrivial elementary embedding $\hat{j}$ of $M$ into itself with $\alpha<crit(\hat{j})<\kappa$. So if $\kappa$ is Berkeley in the usual sense, it's Berkeley in the nontransitive sense too.

Finally, if you drop transitivity and replace "$\kappa\subseteq M$" by the original "$\kappa\in M$," then the argument above breaks down but the notion becomes inconsistent: taking $M=\{\kappa\}$ doesn't leave room for any nontrivial elementary embeddings, for example, let alone ones with critical points (and we can cook up less-silly examples too).

I had a stupid moment earlier, where I mixed up ZF and ZFC - it's not immediately obvious to me now that replacing "set" with "class" results in inconsistency. I suspect it does however.

If you allow $M$ to be any class, you get a contradiction by taking $M=V$.

Meanwhile, dropping transitivity doesn't actually add anything: given $M\supseteq\kappa$, just consider $\hat{M}=$ the Mostowski collapse of $M$. Given $\alpha<\kappa$ and $j:\hat{M}\rightarrow \hat{M}$ nontrivial elementary with $\alpha<crit(j)<\kappa$, $j$ lifts to a nontrivial elementary embedding $\hat{j}$ of $M$ into itself with $\alpha<crit(\hat{j})<\kappa$. So if $\kappa$ is Berkeley in the usual sense, it's Berkeley in the nontransitive sense too. (If you drop $\kappa\subseteq M$ this breaks down, but you also get a much less interesting notion as far as I can tell.)

If you allow $M$ to be any class, you get a contradiction by taking $M=V$.

Meanwhile, dropping transitivity doesn't actually add anything: given $M\supseteq\kappa$, just consider $\hat{M}=$ the Mostowski collapse of $M$. Given $\alpha<\kappa$ and $j:\hat{M}\rightarrow \hat{M}$ nontrivial elementary with $\alpha<crit(j)<\kappa$, $j$ lifts to a nontrivial elementary embedding $\hat{j}$ of $M$ into itself with $\alpha<crit(\hat{j})<\kappa$. So if $\kappa$ is Berkeley in the usual sense, it's Berkeley in the nontransitive sense too.

Finally, if you drop transitivity and replace "$\kappa\subseteq M$" by the original "$\kappa\in M$," then the argument above breaks down but the notion becomes inconsistent: taking $M=\{\kappa\}$ doesn't leave room for any nontrivial elementary embeddings, for example, let alone ones with critical points (and we can cook up less-silly examples too).