$\newcommand{\R}{\mathbb R}\newcommand{\T}{\mathcal T}$There would be no reason for you to ever stop searching unless you would have to incur a cost/penalty with each new observation/trial.
Suppose, for simplicity, that this cost is a real constant $c>0$, for each observation. Let $[k]:=\{1,\dots,k\}$ be the (say finite) set of the types of shells. Let $U\colon[k]\to\R$ be a utility function such that, for each $j\in[k]$, $U(j)$ is the utility of having exactly $j$ types of shells.
Let $T_1,T_2,\dots$ be the types of shells obtained on trials $1,2,\dots$. Assume that the $T_i$'s are iid random variables with values in $[k]$. For $J\subseteq[k]$, let
\begin{equation}
p_J:=P(T_1\in J).
\end{equation}
For natural $n$, consider the random subset
\begin{equation}
\T_n:=\{T_i\colon i\in[n]\}
\end{equation}
of $[k]$, so that $|\T_n|$ is the number of (distinct) types of shells obtained on trials $1,\dots,n$.
Then our expected gain after $n$ trials is
\begin{equation}
G_n:=EU(|\T_n|)-cn,
\end{equation}
which is the expected utility obtained on trials $1,\dots,n$ minus the cost of the $n$ trials. We want to stop at a time moment $n$ maximizing the expected gain $G_n$.
Let us explicitly express the expected gain $G_n$ in terms of the distribution of $T_1$, that is, in terms of the probabilities $p_J$. To do so, consider the events
\begin{equation}
A_J:=\{\T_n=J\},\quad B_J:=\{\T_n\subseteq J\}
\end{equation}
for $J\subseteq[k]$. Then $A_J=B_J\setminus\bigcup_{j\in J}B_{J\setminus\{j\}}$ and hence
\begin{equation}
P(A_J)=P(B_J)-P\Big(\bigcup_{j\in J}B_{J\setminus\{j\}}\Big).
\end{equation}
Noting that $P(B_J)=p_J^n$ and expressing $P\Big(\bigcup_{j\in J}B_{J\setminus\{j\}}\Big)$ by the inclusion-exclusion, we get
\begin{equation}
P(A_J)=\sum_{r=0}^{|J|} (-1)^{|J|-r}\sum_{R\in\binom Jr} p_R^n,
\end{equation}
where $\binom Jr$ denotes the set of all subsets of $J$ of cardinality $r$.
Next, for $j\in[k]$,
\begin{equation}
P(|\T_n|=j)=\sum_{J\in\binom{[k]}j}P(A_J).
\end{equation}
Thus,
\begin{equation}
\begin{aligned}
G_n&=-cn+\sum_{j=0}^k U(j)P(|\T_n|=j) \\
&=-cn+\sum_{j=0}^k U(j)\sum_{J\in\binom{[k]}j}\sum_{r=0}^j (-1)^{j-r}\sum_{R\in\binom Jr} p_R^n.
\end{aligned}
\end{equation}
Assuming (say) some kind of parametric model for the distribution of $T_1$, one can estimate the probabilities $p_J$ based on a comparatively small sample of shells and then estimate $G_n$ for all $n$. Alternatively, one may be successively updating the estimates of the $p_J$'s with each new trial.
Below is the image of a Mathematica notebook with calculations of $G_n$ for $k=10$, $P(T_1=j)\propto0.6^j$ and $U(j)=\ln(2+j)$ for $j\in[k]$, and $c\in\{0.05,0.1,0.2\}$. It appears that the sequence $(G_1,G_2,\dots)$ has an increasing-decreasing pattern, with maximum attained at $n=2$ for $c=0.2$, at $n=3$ for $c=0.1$, and at $n=6$ for $c=0.05$. (Click on the image to magnify it.)
