For me, a fusion category (over $\mathbb{C}$) is just a tensor $F$ (the associator, with $6$ simple-object labels and $4$ fusion space indices) and a tensor $d$ (the quantum dimensions, with one simple-object label), such that the $F$-tensor fulfils the pentagon equation. (This seems to be the most useful language for physics.)

If furthermore the fusion category is unitary, this means that (one can find a basis for the fusion spaces such that) the $F$-tensor is unitary when interpreted as a linear map between two groups of its indices.

What extra structure/conditions do I have to add in this tensor language to make the (unitary) fusion category pivotal/spherical?

For example, it seems that a unitary fusion category is spherical if the $F$-tensor is symmetric under the full tetrahedral symmetry group (after normalizing with the square roots of the quantum dimensions of four of its simple-object labels). This seems not to be the most general case, however.