0
$\begingroup$

I am an engineer working on speech signal processing and I have a problem that I have encountered while trying to model speech signals. The mathematical formulation is not entirely pure and I try to explain this verbatim wherever I find difficult to put in the form of equations or proper mathematical abstraction.

I have a random variable $R$ which takes values from an infinite dimensional Hilbert space. The exact probability distribution of $R$ is unknown, but known to be supported on an open ball. There is a large collection $S$ of size $N$, of the outcomes of the random variable $R$, but don't have access to them. All i have access to, is the inner products between each one of them. I mean I only know $\langle \phi_i,\phi_j \rangle$ $\forall i,j \in \{1,2,3...N\}$ where $\phi_i \in S$. Now I need to find a vector representation, that is a map $\alpha : S\to\mathbb{R}^d$, such that the correlation coefficient $\rho$ of the data $(\langle \phi_i,\phi_j \rangle,\langle \alpha(\phi_i),\alpha(\phi_j) \rangle)$ is maximum. I 'd like to know any algorithm to do this. Right now, the choice of $d$ can be taken as desired.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Strictly speaking it doesn't make much sense to talk about defining a function on $S$ when you've explicitly said you don't know any of the elements of $S$. I'll assume rather that you have a sequence $\phi_1,\ldots, \phi_N$ of vectors which you don't know, but whose inner products you do know, such that you would like to find a sequence $\alpha_1,\ldots,\alpha_N$ of vectors of length $d$ which have the same inner products. That is, I'll talk about $\alpha_i$ instead of $\alpha(\phi_i)$.

Let $M\in\mathbb{R}^{N\times N}$ (here I'm assuming you're working over the reals, but the situation works out the same for complex coefficients) be the "Gram matrix" defined by $M_{ij} = \langle\phi_i,\phi_j\rangle$. For any column vector $x$, $x^TMx = \lVert\sum_k x_k\phi_k\rVert^2\geq 0$, so $M$ is positive semidefinite. Therefore there exists a "square root" $F\in\mathbb{R}^{N\times N}$, a matrix which satisfies $M = F^T F$.

Define $\alpha_i$ to be column number $i$ of $F$. Then $\langle\phi_i,\phi_j\rangle = \langle\alpha_i,\alpha_j\rangle$ holds exactly, so the correlation is $1$. But we have used the relatively large $d = N$. If you only care about the inner products being approximately equal, you can apply the Johnson-Lindenstrauss lemma and randomly project the $\alpha(\phi_i)$ to get as good an approximate version as you'd like with $d = O(\log n)$.

Do you really care about maximizing the given correlation of those two inner products, or is making them nearly equal good enough?

Improve this answer
$\endgroup$
11
  • $\begingroup$ First of all, thanks for the reply and effort for the answer. I got your point and the technical pit fall in my question. As you might be aware of the usual problem setup in a statistical machine learning, my usual intention is to use a large collection of examples and try to learn something which we can use over examples coming in the future. So my intention is to learn from a large collection of size $N$, and then use what ever learnt from it over the future examples. So my mistake is stating $\alpha : S \to \mathbb{R}^d$. If this is the case, then I am not learning anything!... $\endgroup$
    – Rajesh D
    Commented May 12, 2016 at 18:10
  • $\begingroup$ (continued).... My main intention is to learn and estimate the hypothetical map $\alpha_h : H \to \mathbb{R}^d$, where the $H$ is the Hilbert space associated with the random variable $R$. I intended to use a large collection $S$ of size $N$, for the purpose of estimating/learning the map $\alpha_h: H \to \mathbb{R}^d$. I am interested to maximize correlation coefficient over the training set $S$. $\endgroup$
    – Rajesh D
    Commented May 12, 2016 at 18:15
  • $\begingroup$ With your due permission I would like to mend my question as per this, otherwise the point you have raised would make my question dumb. $\endgroup$
    – Rajesh D
    Commented May 12, 2016 at 18:16
  • $\begingroup$ @RajeshDachiraju: At first I thought $\alpha_h$ was what you were after, but how could you say anything about that when you don't know anything about the $\phi_i$ themselves, just their inner products? That is to say, how do you go about defining a function on $H$ with properties you want when you don't have any elements of $H$ or functions on $H$ about which you know something? In what form would you hope to express $\alpha_h$? $\endgroup$
    – Noah Stein
    Commented May 12, 2016 at 18:25
  • $\begingroup$ @RajeshDachiraju: If you clarify your question to address this, I'm happy to delete this answer and comment thread. $\endgroup$
    – Noah Stein
    Commented May 12, 2016 at 18:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .