Return to Answer

[EDITED] At least when the group is simple (or semisimple), finite dimensional irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, whose irreducible representations are 1-dimensional; in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze (see Dan's comment below).

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse (meaning non-existent, though I haven't checked this rigorously). So I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural tool here. For instance, in the rank 2 case $\mathrm{SU}(3)$, highest weights are given by ordered pairs of non-negative integers $(r,s)$ and the corresponding dimension is $(r+1)(s+1)(r+s+2)/2$, which takes value 3 only for the dual weights $(1,0), (0,1)$. Duality here comes from the Dynkin diagram automorphism reversing nodes.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.

[ADDED] Maybe I'm having trouble translating your compact group language into the representation framework for semisimple Lie algebras over $\mathbb{C}$, but in any case I don't see why your example $\mathrm{SU}(2) ]times \mathrm{SU}(2)$ should yield representations which are dual to each other. Instead they look self-dual to me. At least in the Lie algebra setting, the approach I outlined involving Weyl group orbits of weights should rule out further simple examples at least. Your third compact group example doesn't involve a semisimple group, by the way, which further complicates matters.

[EDITED] At least when the group is simple (or semisimple), finite dimensional irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, whose irreducible representations are 1-dimensional; in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze (see Dan's comment below).

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse (meaning non-existent, though I haven't checked this rigorously). So I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural tool here. For instance, in the rank 2 case $\mathrm{SU}(3)$, highest weights are given by ordered pairs of non-negative integers $(r,s)$ and the corresponding dimension is $(r+1)(s+1)(r+s+2)/2$, which takes value 3 only for the dual weights $(1,0), (0,1)$. Duality here comes from the Dynkin diagram automorphism reversing nodes.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.

[ADDED] Maybe I'm having trouble translating your compact group language into the representation framework for semisimple Lie algebras over $\mathbb{C}$, but in any case I don't see why your example $\mathrm{SU}(2) \times \mathrm{SU}(2)$ should yield representations which are dual to each other. Instead they look self-dual to me. At least in the Lie algebra setting, the approach I outlined involving Weyl group orbits of weights should rule out further simple examples: for this you'd look case-by-case at the "minuscule" highest weights and the resulting dimensions. Your third compact group example doesn't involve a semisimple group, by the way, which further complicates matters.

[EDITED] At least when the group is simple (or semisimple), finite dimensional irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, whose irreducible representations are 1-dimensional; in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze (see Dan's comment below).

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse (meaning non-existent, though I haven't checked this rigorously). So I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural tool here. For instance, in the rank 2 case $\mathrm{SU}(3)$, highest weights are given by ordered pairs of non-negative integers $(r,s)$ and the corresponding dimension is $(r+1)(s+1)(r+s+2)/2$, which takes value 3 only for the dual weights $(1,0), (0,1)$. Duality here comes from the Dynkin diagram automorphism reversing nodes.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.

[EDITED] At least when the group is simple (or semisimple), finite dimensional irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, whose irreducible representations are 1-dimensional; in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze (see Dan's comment below).

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse (meaning non-existent, though I haven't checked this rigorously). So I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural tool here. For instance, in the rank 2 case $\mathrm{SU}(3)$, highest weights are given by ordered pairs of non-negative integers $(r,s)$ and the corresponding dimension is $(r+1)(s+1)(r+s+2)/2$, which takes value 3 only for the dual weights $(1,0), (0,1)$. Duality here comes from the Dynkin diagram automorphism reversing nodes.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.

[ADDED] Maybe I'm having trouble translating your compact group language into the representation framework for semisimple Lie algebras over $\mathbb{C}$, but in any case I don't see why your example $\mathrm{SU}(2) ]times \mathrm{SU}(2)$ should yield representations which are dual to each other. Instead they look self-dual to me. At least in the Lie algebra setting, the approach I outlined involving Weyl group orbits of weights should rule out further simple examples at least. Your third compact group example doesn't involve a semisimple group, by the way, which further complicates matters.

At least when the group is simple (or semisimple), irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, so in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze.

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse, so I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural starting point here.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.

[EDITED] At least when the group is simple (or semisimple), finite dimensional irreducible representations are classified precisely by their highest weights and classical formulas for dimensions and tensor product multiplicities apply. (The group can't be a torus, whose irreducible representations are 1-dimensional; in the remaining reductive cases one has to factor in a power of det or other such character to get all irreducible representations.)

Concerning property 1, you have to have a group whose 3-dimensional irreducible representations all fail to be self-dual (ruling out for instance $\mathrm{SU}(2)$): the multiplicitiy of the trivial module can be computed by the dimension of the Hom space from it to the tensor product in question, which is easy to analyze (see Dan's comment below).

It helps to give one or more starting examples where both properties hold. One of these would be $\mathrm{SU}(3)$, which has just a single irreducible 3-dimensional representation (the natural one) and its non-isomorphic dual. (Allowing a reductive group with a nontrivial torus as center will complicate things a bit.) Beyond rank 2 the 3-dimensional irreducibles seem to be very sparse (meaning non-existent, though I haven't checked this rigorously). So I don't immediately see any further examples among the simple or semisimple groups. Weyl's dimension formula is a natural tool here. For instance, in the rank 2 case $\mathrm{SU}(3)$, highest weights are given by ordered pairs of non-negative integers $(r,s)$ and the corresponding dimension is $(r+1)(s+1)(r+s+2)/2$, which takes value 3 only for the dual weights $(1,0), (0,1)$. Duality here comes from the Dynkin diagram automorphism reversing nodes.

P.S. I'm not sure about the exact setting of the question (or its motivation). For instance I'm taking for granted that irreducible representations are over $\mathbb{C}$.