Optimal prefix-free code design with a complex objective function We have a long message $m$ to encode. The message is composed of a set of symbols $\{s_i\}$. Let $p_i$ denote the number of appearance of $s_i$ in $m$. We seek to find a prefix-free code for each $s_i$ so as to minimize $\frac{N_0N_1}{N}$, where $N_0$ and $N_1$ denote the number of bits $0$ and $1$ in the coded message, respectively, $N=N_0+N_1$ denotes the length of the coded message. The prefix-free code system is a set of codes where any code is not a prefix of another. Our problem is to find such optimal coding system. Our problem resembles Huffman coding but with a more complex objective function.
 A: Proof for 3 symbols: We can assume that the codes are 0, 10, 11, any other choice is suboptimal.
Let the occurrences be $p_1\le p_2\le p_3$. We have 3 sensible options for the encodings:
Option 1. $p_1$: 10, $p_2$: 0, $p_3$: 11
After normalization, $N_0=p_1+p_2$, $N_1=p_1+2p_3$, $N=2p_1+p_2+2p_3=2-p_2$.
The objective function is $\frac{(p_1+p_2)(p_1+2p_3)}{2p_1+p_2+2p_3}$.
Option 2. $p_1$: 10, $p_2$: 11, $p_3$: 0
After normalization, $N_0=p_1+p_3$, $N_1=p_1+2p_2$, $N=2p_1+2p_2+p_3=2-p_3$.
The objective function is $\frac{(p_1+p_3)(p_1+2p_2)}{2p_1+2p_2+p_3}$.
Option 3. $p_1$: 11, $p_2$: 10, $p_3$: 0
After normalization, $N_0=p_2+p_3$, $N_1=2p_1+p_2$, $N=2p_1+2p_2+p_3=2-p_3$.
The objective function is $\frac{(p_2+p_3)(2p_1+p_2)}{2p_1+2p_2+p_3}$.
These are not compared so easily.
To compare the last two, the denominators are the same, so using $p_3=1-p_1-p_2$, so we get
$$(1-p_2)(p_1+2p_2)~VS~(1-p_1)(2p_1+p_2)$$
$$p_2-p_2(p_1+2p_2)~VS~p_1-p_1(2p_1+p_2)$$
$$p_2(p_3-p_2)~VS~p_1(p_3-p_1)$$
So we need to choose between options 2 and 3 depending on whether $p_1$ or $p_2$ is closer to $p_3/2$.
Comparing the other options leads to more complicated calculations, so your problem won't have a nice and simple answer, like the Huffman coding, but rather requires the solution of several inequalities.
