How (if at all) does category theory deal with situations where the usual notion of isomorphism isn't right?

Question

This question will potentially rub some people the wrong way; I can't do much about this, except state right here at the outset that this question is motivated by a genuine desire to understand, and to conceptualize things better and more clearly. I realize that this isn't the "usual" way of conceptualizing things under category theory; all I can say is that maybe there are people out there who have thought about this point of view already and that, if so, I want to understand what they already understand.

There's a category-like object that can be described (in vague terms) as follows.

Objects. Well-ordered sets.

Arrows. Functions.

"Special" arrows. Monotone functions.

Monoidal product. Given well-ordered sets $X_0$ and $X_1$, we define that their monoidal product $X_0 \otimes X_1$ is the result of taking the coproduct of these objects viewed as posets, and then declaring that each element of $X_1$ is greater than each element of $X_0$. Given functions $f_0 : X_0 \rightarrow Y_0$ and $f_1 : X_1 \rightarrow Y_1$, there is a corresponding function $f_0 \otimes f_1 : X_0 \otimes X_1 \rightarrow Y_0 \otimes Y_1$, defined in the obvious way. If $f_0$ and $f_1$ happen to be "special" (i.e. they're monotone), then so too is $f_0 \otimes f_1$.

Call this objects $\mathbf{Wos}$ (for "well-ordered sets"). Then we have appear to have a "bifunctor" $$\otimes : \mathbf{Wos}, \mathbf{Wos} \rightarrow \mathbf{Wos}.$$

The weird thing about this situation is that under the usual definition of "isomorphic," $\mathbf{Wos}$ has isomorphic objects that aren't the same. For example, the ordinal $\omega$ and $\omega 2$ are "isomorphic" as objects of $\mathbf{Wos}$, in the sense that there exists an arrow $\omega \rightarrow \omega 2$ that has a two-sided inverse, despite that $\omega$ and $\omega 2$ aren't isomorphic as ordered sets.

The standard solution, of course, is to just "throw out" all the morphisms that aren't monotone. This seems a bit drastic though; after all, the arrow $f_0 \otimes f_1$ makes sense even if $f_0$ and $f_1$ aren't monotone. In other words, the bifunctor $\otimes$ is defined on the whole of $\mathbf{Wos}$, not just the wide subcategory of special morphisms. Therefore, let us instead change our notion of isomorphism; let us simply define that two objects of $\mathbf{Wos}$ are isomorphic iff there exists a special arrow $X \rightarrow Y$ that has a special two-sided inverse.

That's fine, but I want a more systematic viewpoint here. As I see it, two objects $X$ and $Y$ of a category-like structure ought to be "the same" iff $\mathrm{Hom}(X,-)$ and $\mathrm{Hom}(Y,-)$ are naturally isomorphic. However, and here's the rub, I wish to view $\mathrm{Hom}(X,-)$ as a functor defined on the whole of $\mathbf{Wos}$, not just the subcategory of special arrows. And I want $\mathrm{Hom}(X,-)$ and $\mathrm{Hom}(Y,-)$ to be naturally isomorphic iff there is a special arrow $X \rightarrow Y$ with a special inverse.

My question is as follows.

Question. How (if at all) does category theory deal with (or think about, or conceptualize) these kinds of situations, where the usual notion of isomorphism isn't right?

Having motivated the question, let me finish up by offering a "minimal example." There is a category-like object given as follows.

Objects. Sets equipped with a distinguished subset.

Arrows. Functions.

"Special" arrows. Functions that preserve membership in the distinguished subset.

Denote this $\mathbf{Sds}.$ The neat thing about this is that its "self-enriched," in the sense that given objects $X$ and $Y$ of $\mathbf{Sds}$, the collection of functions $X \rightarrow Y$ forms an object of $\mathbf{Sds}$ in a natural way, by taking the distinguished subset of $\mathrm{Hom}(X,Y)$ to consist of precisely those functions that preserve membership in the distinguished subset. In fact, $\mathbf{Sds}$ seems to have "hom-tensor adjunction," in the sense that the object of functions $X \times Y \rightarrow Z$ is the "same as" the object of functions $X \rightarrow \mathrm{Hom}(Y,Z).$

Answer 1

One standard way of looking at your first example would be to see it as two categories: the usual category $\newcommand{WOS}{\mathbf{WOS}}\WOS$ of well-ordered sets and monotone maps, and the category $\newcommand{\fn}{\mathrm{fn}}\WOS^\fn$ with arbitrary functions as morphisms. Then $\omega$ and $\omega^2$ are not isomorphic as objects of $\WOS$, but they are isomorphic as objects of $\WOS^\fn$ (equivalently, their underlying sets are isomorphic).

But if you really want to see the pair $\WOS \subseteq \WOS^\fn$ as a single category (and you reasonably might) then as Finn Lawler suggests in comments, you could consider it as a category enriched over the cartesian closed category $\newcommand{\SDS}{\mathbf{SDS}}\SDS$ you describe. Giving such an enrichment is equivalent to giving a subcategory containing all objects (sometimes called a lluf subcategory).

One well-studied case like this is that of dagger categories and relatives. These generalise examples like the category of finite-dimensional Hilbert spaces, where one wants to consider all linear maps, but wants to consider two such spaces the same only if they are unitarily isomorphic, not just linearly isomorphic. This has been viewed in various ways in the literature, including both the ways above: a distinguished lluf subcategory of unitary (or self-adjoint) maps, or equivalently as a distinguished subset of each hom-set.

---

Edit: I originally missed the part where you say you want “special isomorphisms” to correspond to “natural isomorphisms $\newcommand{\Hom}{\mathrm{Hom}} \Hom(X,-) \to \Hom(Y,-)$, where $\Hom(X,-)$ is viewed as a functor on the whole category, not just the special morphisms”. This is a bit ambiguous, so let’s take some notation: call the larger category $\newcommand{\C}{\mathcal{C}}\newcommand{\D}{\mathcal{D}}\newcommand{\sp}{\mathrm{sp}}\C$, and the special subcategory $\C^\sp$.

If by $\Hom$ you mean $\Hom_\C$, then the Yoneda lemma says that natural isomorphisms $\Hom_\C(X,-) \to \hom_\C(Y,-)$, considered as functors on $\C$, will always correspond to isomorphisms in $\C$ (not generally special ones as you want). Similarly, natural isos $\Hom_\sp(X,-) \to \hom_\sp(Y,-)$, as functors on $\C^\sp$, always correspond to isomorphisms in $\C^\sp$.

We can restrict $\Hom_\C(X,-)$ to a functor on just $\C^\sp$. Considered as such, there may be more natural isomorphisms $\Hom_\C(X,-) \to \Hom_\C(Y,-)$ (natural just w.r.t. special maps) that don’t correspond to any isos in $\C$, special or otherwise. (Consider the case where $\C$ is a group, viewed as a one-object category, and only the identity is designated as special; then any automorphism of the underlying set will give a natural isomorphism of this form.)

It sounds most like what you want is to consider $\Hom_\sp(X,-)$ as a functor on the whole of $\C$. But this is not possible: the functorial action doesn’t extent, because when you compose a special arrow with an arbitrary map, it doesn’t generally stay special (except in the trivial case where all maps are special).