Essentially it's this. Let $\sigma : V \xrightarrow{\sim} V$ be the initial labeling of your vertices. Let $E$ represent your algorithm which is to return a canonical form $E(G) \in X$ some poset $(X,\leq)$ (usually a total ordering).
Then we say that $E(G)$ is a canonical form of $G$ if whenever $G \simeq H$ as graphs, we have that $E(G) = E(H)$ and conversely if $E(G) = E(H)$ then $G \simeq H$. So it is an injection from $\left(\textbf{YourGraphSpace}/\simeq\right)\rightarrowtail X$.
$X$ can be any set because we didn't refer to $\leq$ in the definition. However usually you have many $x \in X$ which correspond to the various labelings of $G$, and there could even be infinitely many! Usually in the optimization problem the least such $x \in X$ is designated as the standard form. I.e. $\text{argmin}^{\leq}_{x \in X} (E(G) = x)$.
And doing so, finding the canonical form of any general class of graphs in a reasonable amount of running time is like any other NP-complete problem you'll find. It's extremely hard to solve.