I assume your are talking about a symmetric strict monoidial category $\mathcal{M}$. I think, you have to choose, beside $\bar{A}$, an isomorphism $\iota_A: A \otimes \bar{A} \to I$, say. I think $\iota_A = \mathrm{id}$ for all $A$ is not possible.
For any $f: X \to Y$ and $g:Y \to X$ follows, with your definition of $\bar{\square}$ and the interchange law:
$\bar{f} \circ \bar{g} := (\bar{X} \otimes f \otimes \bar{Y}) \circ (\bar{X} \otimes g \otimes \bar{Y}) = \bar{X} \otimes f \circ g \otimes \bar{Y}: \bar{X} \otimes X \otimes \bar{Y} \to \bar{X} \otimes Y \otimes \bar{Y}$
(By the symmetry, both of your definitons of $\bar{\square}$ 'agree', that is they fit in suitable commutative diagram. However for composing bars you need to insert the symmetry isomorphism $\gamma: A\otimes B \otimes C \mapsto C \otimes B \otimes A$ in suitable places, like this $ \bar{f}:= \gamma \circ (\bar{X} \otimes f \otimes \bar{Y}): \bar{X} \otimes X \otimes \bar{Y} \to \bar{Y} \otimes X\otimes \bar{X}$.)
That is, $f$ and $g$ are inverse then $\bar{f}$ and $\bar{g}$ are inverse to each other. Well, of course, this is nothing the compatability laws in your monoidial category.
As you suggested, define $\hat{f}:= \rho_{\bar{X}} \circ (\bar{X} \otimes \iota_Y)\circ (\bar{X} \otimes f \otimes \bar{Y}) \circ (\iota_{\bar{X}} \otimes \bar{Y})^{-1} \circ \lambda_{\bar{Y}}^{-1}: \bar{Y} \to \bar{X}$
Where $\lambda_Y:I \otimes Y \to Y$ and $\rho_X: X \otimes I \to X$ are the designed natural isomorphims.
Exploiting the compatablity laws, this gives you $\hat{g} \circ \hat{f} = \mathrm{id}_X$ for $g:= f^{-1}$. Thus you obtain one implication.
I see, in the meantime Todd Trimble gave an answer which completly subsume mine. However as it took me some time to type it, i post it as a 'down to earth' version.
Edit: I wonder if you obtain an endofunctor $\mathcal{M}^{op} \to \mathcal{M}$ via $X \mapsto \bar{X}$ and $f \mapsto \hat{f}$.
Well, probably you need some conditions like $\bar{\bar{X}} = X$ and $\iota_{\bar{X}} = {\iota_X}^{-1}$. I will think about. This may be also related to other direction.