Veblen function with uncountable ordinals & beyond Disclaimer: I am not a professional mathematician.
Background: I have been researching large countable ordinals for awhile & I think the Veblen function is particularly eloquent. My understanding is that $\Gamma_0$, the small Veblen ordinal & the large Veblen ordinal are all significantly smaller than the first uncountable ordinal $\omega_1$. Having some extra time during quarantine, I had an idea to extend the Veblen function to the domain of uncountable ordinals & created the following notation. I would like to know how far this notation reaches & if anything similar already exists. 
Note: For the sake of brevity I have omitted numerous steps from the hand written derivation of this notation.

Consider $\phi_0'(\alpha)=\omega_\alpha$ such that:
$$\phi_0'(0)=\omega_0=\omega$$
$$\phi_0'(1)=\omega_1$$
Nesting these functions results in:
$$\phi_0'(\phi_0'(0))=\omega_\omega$$
$$\phi_0'(\phi_0'(\phi_0'(0)))=\omega_{\omega_\omega}$$
Next, consider the supremum of the previous nestings:
$$\phi_1'(0)=\sup\{\omega, \omega_\omega, \omega_{\omega_\omega},...\}$$
$\phi_1'(0)$ is then the first fixed point of $\phi_0'(\alpha)$ which correlates to $\phi_1(0)=\varepsilon_0$ being the first fixed point of $\phi_0(\alpha)=\omega^\alpha$ in the original Veblen function.
Continuing as in the original case, we eventually hit the limit of our single variable function. At this point ($\Gamma_0$ in the original), we turn to the multivariable function:
$$\phi_{1,0}'(0)=\phi'(1,0,0)=\sup\{\phi_1'(0),\phi_{\phi_1'(0)}'(0),\phi_{\phi_{\phi_1'(0)}'(0)}'(0),...\}$$
Again, like in the original case with the small Veblen ordinal, we eventually get stuck. At this point we move to the version of the Veblen function with a transfinite number of variables.
$$\phi'(1@\omega)=\sup\{\phi'(1,0),\phi'(1,0,0),\phi'(1,0,0,0)\}$$
Eventually this notation reaches as cap as well. In the orginal case, this is called the large Veblen ordinal & is the cap of the original Veblen function. In the expansion, we simply iterate our 'jump' operator:
$$\phi_0''(0)=\sup\{\phi'(1@0),\phi'(1@\omega),\phi'(1@\varepsilon_0),...\}$$
We can keep going by iterating the base function such that:
$$\Phi_0(0)=\sup\{\phi_{0}'(0), \phi_0''(0), \phi_0'''(0),...\}$$
Given the general form $\alpha_\gamma^\beta(\delta)$ we are essentially: 


*

*maxing out $\delta \leadsto$ incrementing $\gamma$

*maxing out single variable $\gamma \leadsto$ multivariable $\gamma$

*maxing out multivariable $\gamma \leadsto$ incrementing $\beta$

*maxing out $\beta \leadsto$ incrementing $\alpha$
Repeating the process a couple more times results in:
$$\sup\{\Phi_0(0),\Phi_0'(0),\Phi_0''(0),...\}=\psi_0(0)$$
$$\sup\{\psi_0(0),\psi_0'(0),\psi_0''(0),...\}=\Psi_0(0)$$
Looping repeatedly reminded me of the original Veblen function process & so I created the following function:
$$\Xi(\alpha, \beta, \gamma, \delta)=\alpha_\gamma^\beta(\delta)$$
Such that:
$$\Xi(0,0,0,0)=\phi_0(0)=1$$
$$\Xi(0,0,0,1)=\phi_0(1)=\omega$$
$$\Xi(0,0,1,0)=\phi_1(0)=\varepsilon_0$$
$$\Xi(0,1,0,0)=\phi_0'(0)=\omega$$
$$\Xi(0,1,0,1)=\phi_0'(1)=\omega_1$$
$$\Xi(1,0,0,0)=\Phi_0(0)$$
$$\Xi(2,0,0,0)=\psi_0(0)$$
$$\Xi(3,0,0,0)=\Psi_0(0)$$

If you made it this far, thank you for taking the time. To reiterate, how far does this notation reach & does anything like this already exist?
 A: It's a bit long for a comment, but I'll make several points.

These are not uncommon ordinals.
I've seen them used in Rathjen's ordinal collapsing function involving Mahlo cardinals, which he denotes $\Phi$. As the comments point out, they appear in various places.

This is not at all how the multivariable Veblen function behaves (before the edit).
Your $\phi_{1,0}'(0)$ is simply $\phi_{\phi_1'(0)}'(0)$. It would be akin to saying that $\Gamma_0=\phi(\phi(1,0),0)$, which is not at all true.
To explain how the multivariable Veblen function works, I recommend seeing it as recursively closing over itself on lexicographically smaller arguments. In short, left-most arguments are more significant than right-most arguments. That is, we have things like $(1,0,0)>_L(\omega,0)>_L(3,0)>_L(2,\omega)>_L(1,0)$. From this, one can see that $\Gamma_0=\phi(1,0,0)$ is greater than $\phi(\alpha,\beta)$ for any $\alpha,\beta<\Gamma_0$. This can be shown to be equivalent to
$$\phi(1,0,0)=\sup\{\phi(1,0),\phi(\phi(1,0),0),\phi(\phi(\phi(1,0),0),0),\dots\}$$
but makes more sense when considering transfinitely many arguments.

As far as I can tell, it's significantly smaller than the usual Veblen function modified with $\phi(\alpha)=\omega_\alpha$.
The Veblen function is already optimal, as far as this kind of recursion goes. Thus, the fact that your functions have significantly less arguments than the general Veblen function will make it much smaller. A quick look and I'd say only 5 or 6 arguments of the Veblen function would be needed to outperform your functions.
A: I am hoping an expert would answer this question so as to shed light on deeper or more profound points. As such, this is a basic answer covering some easy to understand points. This is based upon number of things I thought about years ago (it seems that some of those observations can be used in this question).
So let's start with your question "how far does this notation reach". I don't know what would be the answer to the question. It seems that to be able to answer though one would have to frame the question much more precisely (and I am not certain what that framing would be). Meanwhile the specific constructions you are posting (and far beyond that) are easily understood thinking in terms of generalized notion of being able to do complex calculations on ordinals. 
For example, let's talk about something specific. In the beginning of your post you mention a way of starting with the function $x \mapsto \omega_x$ and how to arrive at an ordinal that is analogous to $\Gamma_0$. This analogy can be made precise using infinite programs that are sufficiently powerful. How so? Assume that a function $f:\mathrm{Ord} \rightarrow \mathrm{Ord}$ is "given" to the program. Exactly the same program that takes one to $\Gamma_0$ (given $f(x)=\omega^x$) will take one to "analogue of $\Gamma_0$" that you mention in your question. The only difference is that the function $f$ "given" to the program now is $f(x)=\omega_x$.

Now the same observations apply to bigger ordinals. I haven't studied the original Veblen paper so I am not 100% sure if the correspondences that I mention below are exact or not (so please correct if they aren't). 
One way to think about SVO is in terms of a function $F:(\omega_1)^\omega \rightarrow \omega_1$. For example, writing $\omega_1=w$, we will have $\mathrm{SVO}=\mathrm{sup}\{\,F(w^i) \,\, | \,\, 1 \leq i<\omega\}$. This is analogous to thinking $\Gamma_0$ in terms of $F:(\omega_1)^2 \rightarrow \omega_1$. So, we will have $\Gamma_0$ as the first fixed point of the ordinal function $x \mapsto F(\omega_1+\omega_1 \cdot x)$.  Quite informally, I use the term "storage-functions" for these functions $F$. The $\omega_1$ isn't quite relevant in the sense that we just need an ordinal "big enough" ($\omega_{CK}$ would be sufficient in the above two cases). But anyway, that's besides the point. The point here is that when a function $x \rightarrow \omega^x$ alongside with a command of form $u:=\omega_1$ is given to us, then there is a specific infinite program which can compute the storage function (in input-output sense).
Is this relevant to your question? Yes. The same program that gives us SVO when given the function $x \mapsto \omega^x$ will take us to the "analogue of SVO" in the question (using the function $x \mapsto \omega_x$). But the issue of "storage function" seems to become trickier in this "analogue case".
EDIT: I am not suggesting to gloss over several important aspects such as equivalence of different definitions. If we are being fully detailed, I will admit the paragraphs above are quite insufficient. END

Finally, very briefly, towards the end you mention "extension" of transfinite variable. In the case of original hierarchy these kind of basic extensions would be handled by extending the domain of the "storage function" by a very modest amount. For example, from $F:(\omega_1)^{\omega_1}  \rightarrow \omega_1$ to $F:(\omega_1)^{\omega_1} \cdot \omega \rightarrow \omega_1$ etc. Similarly observations made earlier in this post about the "same" program taking us to the "analogue" of corresponding ordinal would apply (when given $x \mapsto \omega_x$ instead of $x \mapsto \omega^x$).
EDIT2: To OP (as a precaution): Please note that just writing $F:(\omega_1)^{\omega_1}  \rightarrow \omega_1$ (or anything of that sort) doesn't mean that the underlying function has been fully well-defined and neither I meant to imply that. In the given specific cases, precise definition can either be descriptive or based upon a (infinite) program which computes the function (given an extra command of form $u:=\omega_1$). Showing that the given def. satisfy certain desirable/required properties is bound to be more work. END
How time consuming it would be to write the detail of storage functions? For $(\omega_1)^2 \rightarrow \omega_1$ (starting with $x \mapsto \omega^x$) taking us to $\Gamma_0$ it should be fairly simple (though still a bit long to post all of it here). And then it gets lengthier, as it gets more complicated.
