You don't say what your motivating application is. It would be easier to sample without replacement but maybe that is not an option. You also suggest that you might be willing to limit the maximum number of times the most frequent item occurs. Any assumptions on the model change what can be said. I am just going to assume that a puzzlemaster, unknown to us, has put $n$ colored balls in a bag according to some private scheme she made up. We sample $s$ times with replacement and must draw conclusions as best we can. I will assume that $n$ is known but very large and that $0.1 n \lt s \lt 0.9 n$ but that is just a vague thought in the back of my mind. You do seem to want $s \lt n$

I think that you don't find a good match with species richness because you are not asking the right questions. Do a thought experiment about various outcomes and what you can conclude.

  Suppose all $s$ balls turn out to be green. What can you conclude? Most of the balls are green. You can say with $90\%$ confidence that the number of green balls is at least $\alpha n$ for some $\alpha(n,s)$ which I won't attempt to suggest that I personally know how to compute. Suppose that you do know and you announce "Of the $n=1000000$ balls, we sampled $s$, they were all green, so I am $95\%$ confident that at least $870,000$ are green. You get a surprise hint "good guess!, exactly $900000$ are green. Now you know for sure that $2 \le N \le 100001.$ You get a further hint that the answer is either $n=2$ or $N=10001$. Can you say anything about the chance that $N \gt 10?$ I say no. OK now suppose you got just the first hint or even no hints at all. Does that make it easier to decide if $N \gt 10?$ How could it? And with no hints how confident can you be that there are any balls which are not green? I don't see why you would be.

NOW suppose that of your $s$ sampled they were green and red in the ratio 17 to 9. Then you can be $90\% $ confident that at least $\alpha n$ are red or green for the same $\alpha $ as before. But as far as saying if there are any other colors and if so how many, things are no better and no worse than before. What you can say with good confidence is that there are about twice as many green as red..

SO I think you can answer these questions (which you don't seem that interested in): Estimate how many balls might be a different color than any we have seen. For each color seen, estimate how many balls are this color.

You don't say what your motivating application is. It would be more effective to sample without replacement, but maybe that is not an option. You also suggest that you might be willing to limit the maximum number of times the most frequent item occurs. I think a lower bound on the least frequent color might be more helpful (equivalently, posit that anything below a certain relative frequency is an "anomaly" and does not count). I am just going to assume that a puzzle-master, unknown to us, has put $n$ colored balls in a bag according to some private scheme she made up. We sample $s$ times with replacement and must draw conclusions as best we can. I will assume that $n$ is known.

I think that you don't find a good match with species richness because you are not asking the right questions. What you can say something about is the possible frequency of the colors you have already seen. Indirectly that tells you something about the number of colors, but very little.

Do a thought experiment about various outcomes and what you can conclude. Suppose first that as you keep drawing (then replacing) balls they are all green. You can say that there seem to be lots of green balls. You can say "With $95\%$ confidence the number of green balls is at least $m=m(n,s)$" , equivalently the number of non-green balls is estimated to be no more than $n-m.$ However I do not think that you can make any conclusion about their colors. Even if you get a hint: " There are at least $q$ balls which are not green and they are either all the same color or else each has a color unique to itself" I do not think you have any way of saying which is more likely. And with no hint the only change is that perhaps $q=0$.

I don't think that a limit such as "no color occurs more than one third of the time" will help. Suppose instead that (for a not too small $s$) the balls come out "red, white, blue, yellow, pink" with frequencies roughly $2:2:2:1:1$. Then you can say "With $95\%$ confidence the number of red,white,blue,yellow and pink balls" balls is at least $m$" for the same $m=m(n,s)$ as in the all green scenario. AND you can say that whatever the number of balls of those five colors the frequencies will be roughly 1/4,1/4,1/4,1/8,1/8. But you are no better of on the question of how many unseen colors might there be.

I am less confident what one might say in the $2:2:2:1:1$ scenario above if along the way you say a single black ball, but I do not think that you could say much.

Another extreme is if you have made a reasonable number of samples and never seen the same color twice. Would that occur in your application? Then you might be able to say with some confidence that few, if any, colors repeat. Then you could (I would think) answer the "more than x colors" question as likely having seen many less than $x$ balls.