I don't have anything like a complete answer. BUT, consider the case where $B$ is very large in absolute value (and assume the $\gamma$s are all positive or all negative). Then (assuming that $\gamma$s are positive, and $B \gg 0$) your matrix elements are approximately $$\frac{\exp(B\gamma_j)}{\mu_i - \gamma_j}.$$ The exponentials factor out of the determinant to give you a multiplicative factor of $\exp(B \sum \gamma_j),$ and what remains is a Cauchy matrix of which the determinant can be evaluated explicitly, and it seems like your conjecture is correct. Similarly, when $B\ll 0$ you get $\exp(-B\sum\gamma_j)$ times a (different) Cauchy determinant. You should check that your conjecture is correct then too. If, for some reason, you knew that your matrix was always nonsingular as $B$ varies you would be golden, but this is not clear. It is clear that the determinant is zero whenever some $\gamma_i$ is equal to some $\gamma_j,$ and also clear that the determinant has a pole whenever some $\mu_i$ equals some $\mu_j,$ so if the determinant were a rational function, I think you would know that these were all the zeros and poles, but it is not. Still, this might be close.
NOT true for negative $B$ For $2\times 2$ matrix with $\mu_1 = 1, \mu_2=3, \gamma_1 = 2, \gamma_2=4$ the determinant is positive for positive $B$ and very negative $B$ but is negative in a range of negative $B$ (but not VERY negative, which makes me think that the conjecture is actually false, and the countexemples the OP found might be real counterexamles. In fact, this is also borne out by a (sort-of) closed form for the determinant:
closed form Do the dumbest thing possible and use the multilinearity of the determinant, to get $$ D = \sum_{\mbox{subsets of $1, \dots, n$}} \exp(B\sum \mathbf{\gamma}_S) C_{\mathbf{\mu}, \mathbf{\gamma}_S},$$ where $\mathbf{\gamma}_S$ is the set of $\gamma$s with the ones in $S$ negated, while $C_{\mathbf{\mu}, \mathbf{\gamma}_S}$ is the Cauchy determinant, with $\mu$s and the some-signed-flipped $\gamma$s.