# Analogue of the integral Fourier operator with angle in some cone

Let $\Phi(x,y,\theta)$ be a phase function defined on $X \times X \times (\mathbb R^n-0)$ where $X$ is some domain in $\mathbb R^n$, let $A(x,y,\theta)$ be an amplitude function. As usually an integral Fourier operator is defined by the formula $$(Fu)(y)=\frac{1}{(2\pi)^n} \int\limits_{\mathbb R^n}\int\limits_{X} e^{i\Phi(x,y,\theta)} A(x,y,\theta)u(x) \; dxd\theta$$ I'm currenly looking for the generalization of the integral Fourier operator on the case when $\theta$ belongs to some open convex cone $\Gamma$ in $\mathbb R^n$, for example $\Gamma = \mathop{\mathsf{int}} \mathbb R^n_+$: $$(F_{\Gamma}u)(y) = \frac{1}{(2\pi)^n} \int\limits_{\Gamma}\int\limits_{X} e^{i\Phi(x,y,\theta)} A(x,y,\theta)u(x) \; dxd\theta$$ At first I thought that this "little" modification of definition will not sufficiently affect ideas of proofs of some results in theory of integral Fourier operators and I can easily modify them. But it was a mistake. So my question is if there are some results on this generalization yet?

An example of arising problem is further. I have an operator $$(Au)(y)= \frac{1}{(2\pi)^n} \int\limits_{\mathbb R^n_+} \int\limits_{\mathbb R^n_+} e^{i \theta(x-y)+iH(x,y,\theta)} u(x) \; dx d\theta$$ where $H(x,y,\theta)$ is a function with some specific properties. I have to show that $A = I +K$, where $I$ is the identity operator, $K$ is a compact linear operator from $L^2_{0}(\mathbb R^n_+)$ to $L^2_{loc}(\mathbb R^n_+)$. If $\Gamma$ is equal to $\mathbb R^n$ then it can be done using representation $$(Au)(y) = I + \frac{i}{(2 \pi)^n}\int\limits_{0}^{1} \int\limits_{\mathbb R^n} \int\limits_{\mathbb R^n_+} e^{i \theta(x-y) + itH(x,y,\theta)} H(x,y,\theta) u(x) \; dxd\theta$$ where $$I = \frac{1}{(2\pi)^n} \int\limits_{\mathbb R^n} \int\limits_{X} e^{i \theta (x-y)} u(x) \; dx d\theta.$$ If we try to do tha same in the case of our "generalized" Fourier integral operator we will obtain instead $I$ in this formula an operator $$\frac{1}{(2 \pi)^n} \int\limits_{\Gamma} \int\limits_{X} e^{i \theta(x-y)}u(x) \, dxd\theta.$$

-

Why don't you assume that the amplitude $A(x,y,\theta)$ is actually supported for $\theta$ in your cone $\Gamma$ along with the standard symbolic properties? You will always run into trouble with a definition like yours, which amounts to deal with singular amplitude. In particular any integration by parts will produce boundary terms that you will not be able to control.
You should take a look at Hörmander's definition of an oscillatory integral where the amplitude is indeed a symbol and the phase $\phi$ is a first-order symbol, real-valued for the classical FIO, with a nonnegative imaginary part for FIO with complex phase.