Pick an infinite-degree field extension $k \to L$. Then $\mathrm{Vec}_L$ is $k$-linear abelian monoidal, and the unique simple object is the monoidal unit, which enjoys $\dim_k \mathrm{End}(L) = \dim_k L = \infty$.
Maybe you insist that the unit object $1_{\mathcal{C}}$ in your category have $\mathrm{End}_{\mathcal{C}}(1_{\mathcal{C}}) = k$. Then things can still go wrong. Here are two examples of a similar infinitary flavour.
Example 1: Recall that Deligne's category $\mathrm{Rep}(\mathrm{GL}_t)$ is the universal symmetric monoidal category with a dualizable object $V_t$ of dimension $t$. It is defined over $\mathbb{Z}[t]$. To make things easy, let's work over the field of fractions $k := \mathbb{Q}(t)$. Then, after Karoubi completing, Deligne's category is semisimple and in particular abelian. I'll call the unit object $1_{\mathcal{C}}$.
By universality, there is a map $\mathrm{Rep}(\mathrm{GL}_t) \to \mathrm{Rep}(\mathrm{GL}_{t-1})$ sending $V_t \mapsto V_{t-1} \oplus 1_{\mathcal{C}}$. Consider the colimit
$\mathrm{Rep}(\mathrm{GL}_{-\infty}) = \operatorname{colim} (\mathrm{Rep}(\mathrm{GL}_t) \to \mathrm{Rep}(\mathrm{GL}_{t-1}) \to \mathrm{Rep}(\mathrm{GL}_{t-2}) \to \dots )$
You can take this colimit in, say, Karoubian monoidal categories. In this colimit, the image of $V := V_t$ remains dualizable but has infinitely many direct summands, and so has infinite-dimensional endomorphism algebra. The unit object, on the other hand, is simple with endomorphisms $k = \mathbb{Q}(t)$.
You can base-change this example to any field containing $k = \mathbb{Q}(t)$, for example $\mathbb{C}$.
Example 2: Pick a nonprincipal ultrafilter $\mathcal{U} \subset \mathcal{P}(\mathbb{N})$. Recall that, for any algberaic object $X$, the ultrapower $^* X$ is the quotient of $X^{\mathbb{N}}$ by the relation that $(x_0,x_1,\dots) \sim (y_0,y_1,\dots)$ if the set $\{n \text{ s.t. } x_n = y_n\}$ is in $\mathcal{U}$. Ultrapowers of fields are fields, ultrapowers of categories are categories, etc. More generally, any first-order axiom satisfied by $X$ will be satisfied by $^* X$. The notation "$^* X$" is from Robinson's nonstandard analysis.
Now consider the ultrapower $^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$. Because $\mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ is symmetric monoidal fully dualizable abelian, so is $^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$. Moreover, because every exact sequence in $\mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ splits, also every exact sequence in $^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ splits. Note: This does not mean that $^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ is semisimple: the formula that says "every object is a direct sum of simples" is not first-order.
The (isomorphism classes of) objects of $\mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ are indexed by $\mathbb{N}$, the (isomorphism classes of) objects of $^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}$ are indexed by $^*\mathbb{N}$. Let $V \in {^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}}$ be an object of dimension $\omega \in {^*\mathbb{N}} \setminus \mathbb{N}$. Then $V$ has infinitely many direct summands, and so $\mathrm{End}(V)$ is infinite-dimensional.
The field at the identity is $\mathrm{End}_{^* \mathrm{Vec}^{\mathrm{fd}}_{\mathbb{C}}}(\text{unit object}) = {^*\mathbb{C}}$. You wanted the field at the identity to be specifically $\mathbb{C}$, you say? Well, $^*\mathbb{C}$ is an algebraically closed field of characteristic zero, and hence determined up to noncanonical isomorphism by its transcendence degree over $\mathbb{Q}$. Moreover, this field has infinite transcendence degree, and hence its transcendence degree equals its cardinality. But it is not too hard to show that $^*\mathbb{C}$ is of the same cardinality (continuum) as $\mathbb{C}$. So there exists a noncanonical isomorphism $^*\mathbb{C} \cong \mathbb{C}$.
Remark: These two examples are basically the same example: that isomorphism $^*\mathbb{C} \cong \mathbb{C}$ will take $\omega \in {^*\mathbb{N}} \subset {^* \mathbb{C}}$ to some transcendental number in $\mathbb{C}$, and hence select an inclusion $\mathbb{Q}(t) \subset \mathbb{C}$. In both cases, what I did was to engineer a category with a dualizable object of dimension this transcendental number, and which can split off arbitrarily many copies of the unit object.
I do not know an example of a rigid abelian symmetric monoidal category with infinite-dimensional hom-spaces but with $\mathrm{End}(\text{unit object}) = \mathbb{Q}$ specifically. Engineering infinite-dimensional hom spaces tends to require some "analytic" constructions (colimits, ultrapowers, etc.) and so one expects it to move you away from purely-algebraic numbers. It's possible — I haven't thought about it — that the first example can be adjusted to work over $\overline{\mathbb{Q}}$, by taking $t$ to be some algebraic number which is not an algebraic integer, say, rather than taking it to be transcendental. I haven't thought about it though.
