It may be easier to first apply a clustering algorithm to the data to split into clusters and then apply PCA to reduce the dimensions of your clusters. Also, if we are using something similar to PCA, we need to reduce vectors to objects in an inner product space and we need to forget about any other structure that the data has, but if you are looking at something like atmospheric pressure, it may be helpful to take into consideration the extra structure of the data.
I have developed a technique similar to PCA that one can use to find not the highest dimensions for data but instead find the highest clusters of dimensions of the data.
Suppose that $A_1,\dots,A_r$ are real $n\times n$-matrices and $d\leq n$. Then we say that $(X_1,\dots,X_r)\in M_d(\mathbb{R})^r$ is a real $L_{2,d}$-spectral radius dimensionality reduction (LSRDR, and there is also a notion of a complex and quaternionic LSRDR) of $A_1,\dots,A_r$ if the following quantity is maximized:
$$\frac{\rho(A_1\otimes X_1+\dots+A_r\otimes X_r)}{\rho(X_1\otimes X_1+\dots+X_r\otimes X_r)^{1/2}}.$$
Here, $\rho$ denotes the spectral radius while $\otimes$ denotes the tensor product. One can maximize this quantity using a variation of gradient ascent.
LSRDRs tend to be unique in the sense that if $(X_1,\dots,X_r),(Y_1,\dots,Y_r)$ are $L_{2,d}$-SRDRs of $A_1,\dots,A_r$, then one can usually find an $\alpha$ and an invertible $C$ where $Y_j=\alpha CX_jC^{-1}$. Furthermore, if $(X_1,\dots,X_r)$ is an LSRDR of $(A_1,\dots,A_r)$, then one can typically find matrices $R,S$ where
$X_j=RA_jS$ for $1\leq j\leq r$. In this case, there will typically be a constant $\alpha$ where $RS=\alpha\cdot I$. One can choose the constant $\alpha$ so that $RS=I$, and in this case, we say that $(X_1,\dots,X_r)$ is a normalized LSRDR of $(A_1,\dots,A_r)$. If $(X_1,\dots,X_r)$ is a normalized LSRDR of $(A_1,\dots,A_r)$, then set $P=SR$. Then we usually have $P=P^2$, so $P$ is non-orthogonal projection. If we also assume that each $A_j$ is symmetric, then $P$ will most likely also be symmetric, so $P$ will most likely be an orthogonal projection.
Suppose that $\mathbf{v}_1,\dots,\mathbf{v}_r\in\mathbb{R}^n$ are vectors. If you want, you may assume that these vectors are normalized so that $\mathbf{v}_1+\dots+\mathbf{v}_r=\mathbf{0}.$ Let $1\leq d\leq n$. Let $A_j=\mathbf{v}_j\mathbf{v}_j^*$ for $1\leq j\leq r$. Then if $P$ is the orthogonal projection matrix as we have above, then $P\mathbf{v}_1,\dots,P\mathbf{v}_r$ will be your vectors of reduced dimension.
We can now characterize the projection $P$ without needing to use the spectral radius. $P$ is simply the orthogonal projection where the following quantity is maximized:
$$\frac{\|(v_1\otimes Pv_1)(v_1\otimes Pv_1)^*+\dots+(v_r\otimes Pv_r)(v_r\otimes Pv_r)^*\|}{\|(Pv_1\otimes Pv_1)(Pv_1\otimes Pv_1)^*+\dots+(Pv_r\otimes Pv_r)(Pv_r\otimes Pv_r)^*\|^{1/2}}.$$
You can read more about LSRDRs at my site here and here.
An example:
In the following example, $r=200,n=30,d=15$, $u_1,\dots,u_r\in\mathbb{R}^n$ are selected according to the standard multivariate Gaussian distribution, and $A,B$ are the projection matrices where $A(x_1,\dots,x_{30})^T=(0,\dots,0,x_{15},\dots,x_{30})^T$ and
$B(x_1,\dots,x_{30})^T=(x_{1},\dots,x_{16},0,\dots,0)^T$. We set $v_j=Au_j$ for $1\leq j\leq 100$ and $v_j=Bu_j$ for $101\leq j\leq 200$. When we computed the LSRDR, the projection matrix $P$ was very close to the matrix $A$. Here is the heatmap of the orthogonal projection matrix $P$.
We observe that the projection matrix $P$ projects onto what it finds to be the largest cluster of dimensions instead of the largest dimensions themselves.
Notes:
The notion of the LSRDR is almost but not quite transitive. The LSRDR of an LSRDR of $(X_1,\dots,X_r)$ is almost but not quite an LSRDR of $(X_1,\dots,X_r)$.
The LSRDR of $v_1v_1^*,\dots,v_rv_r^*$ seems to take a lot of steps to converge, but maybe one can improve the time to convergence by using a slightly different algorithm than I have used. I have personally had better luck when the matrices $A_1,\dots,A_r$ are general matrices rather than positive semidefinite matrices.
Unlike PCA and the SVD, LSRDRs are not always unique, and they only sometimes satisfy the properties that we want them to satisfy. For example, if $(X_1,\dots,X_r)$ is an LSRDR of $(A_1,\dots,A_r)$, then sometimes there are no $R,S$ with $X_j=RX_jS$.
In my experience, modifications of the notion of an LSRDR may behave better than LSRDRs for particular tasks (such as making graph embeddings).
