Questions tagged [bayesian-probability]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
9 votes
3 answers
450 views

What does the KL being symmetric tell us about the distributions?

Suppose two probability density functions, $p$ and $q$, such that $\text{KL}(q||p) = \text{KL}(p||q) \neq 0$. Intuitively, does that tell us anything interesting about the nature of these densities?
HesterJ's user avatar
  • 123
9 votes
1 answer
293 views

Who introduced the term hyperparameter?

I am trying to find the earliest use of the term hyperparameter. Currently, it is used in machine learning but it must have had earlier uses in statistics or optimization theory. Even the multivolume ...
AChem's user avatar
  • 803
9 votes
1 answer
457 views

In what sense is the Bayesian posterior mean a “convex combination”?

I asked this on math.stackexchange with no response, I'm hoping someone here might have something. Suppose I want to estimate $x \in \mathbb{R}^n$ from two signals with zero mean, normally ...
Ronaldo Carpio's user avatar
8 votes
1 answer
318 views

Base schemes and Bayesian priors

One of Grothendieck's dicta about algebraic geometry is to consider "the relative situation", where one doesn't consider the category of schemes but of schemes over a fixed base scheme. In Bayesian ...
Allen Knutson's user avatar
7 votes
1 answer
1k views

Rate of convergence of Bayesian posterior

Suppose a data generating process (DGP) is parameterized by some unknown parameter $\theta_0$, say $P_{\theta_0}$, and we want to estimate the value of $\theta_0$ using Bayesian method. Let $\pi(\...
Herr K.'s user avatar
  • 173
6 votes
1 answer
2k views

What can be said about an infinite linear chain of conjugate prior distributions?

We can sample a discrete value from the multinomial distribution. We can also sample the parameters of the multinomial distribution from its conjugate prior the dirichlet distribution. Since the ...
DoubleJay's user avatar
  • 2,353
6 votes
0 answers
193 views

Existence of stick breaking representations for random measures

The Dirichlet process has a roughly size ordered representation in terms of beta random variables, called a stick-breaking representation (Sethuraman, 1994). Similar results hold for the beta process, ...
Shannon S.'s user avatar
5 votes
6 answers
2k views

Are all probabilities conditional probabilities? [closed]

We know that $P(A\mid B) = \frac{P(A \cap B)}{P(B)}$. So $P(B) = P(A\mid B)P(A \cap B)$. Thus are all probabilities conditional probabilities? Can one make a probability more accurate by introducing a ...
Tony 's user avatar
  • 59
5 votes
3 answers
1k views

Probability estimates for pairwise majority votes

This is related to the rank aggregation question I asked previously. I have items $I_1, \ldots, I_N$ and the observations of a number of pairwise trials which pit pairs $I_i$ and $I_j$ against ...
David R. MacIver's user avatar
5 votes
1 answer
325 views

Bounding the sensitivity of a posterior mean to changes in a single data point

There is a real-valued random variable $R$. Define a finite set of random variables ("data points") $$X_i = R + Z_i \; \text{for } i\in\{1,\ldots,n\},$$ where $Z_i$ are identically and independently ...
Ben Golub's user avatar
  • 1,058
4 votes
2 answers
215 views

Do these distributions have a name already?

In playing with some math finance stuff I ran into the following distribution and I was curious if someone had a name for it or has studied it or worked with it already. To start, let $\Delta^n$ be ...
Jess Boling's user avatar
4 votes
1 answer
450 views

Gaussian process kernel parameter tuning

I am reading on gaussian processes and there are multiple resources that say how the parameters of the prior (kernel, mean) can be fitted based on data,specifically by choosing those that maximize the ...
john's user avatar
  • 141
4 votes
0 answers
227 views

Convergence of the expectation of a random variable when conditioned on its sum with another, independent but not identically distributed

Suppose that for all $n \in \mathbf{N}$, $X_n$ and $Y_n$ are independent random variables with $$X_n \sim \mathtt{Binomial}(n,1-q),$$ and $$Y_n \sim \mathtt{Poisson}(n(q+\epsilon_n)),$$ where $q \in (...
as1's user avatar
  • 91
4 votes
0 answers
636 views

Bayesian Networks and Polytree

I am a bit puzzled by the use of polytree to infer a posterior in a Bayesian Network (BN). BN are defined as directed acyclic graphs. A polytree is DAG whose underlying undirected graph is a tree. ...
Bremen's user avatar
  • 41
3 votes
2 answers
607 views

Parametrising a sparse orthogonal matrix

I need to find a way to parametrise a matrix that is both sparse (to some degree) and orthogonal, i.e., I am looking for a parametrisation that describes $A \in \mathbb{R}^{n\times m}$ such that $AA^𝑇...
HesterJ's user avatar
  • 123
3 votes
2 answers
408 views

Multivariate normal concentration

If $X\sim N(0,\Sigma)$ for some $d$-dimensional normal distribution, then $X = \Sigma^{1/2} Z$ where $Z\sim (0,I)$. How to compute the following quantity? $$ \operatorname{var} (X^T X) = \...
Nikolayevich's user avatar
3 votes
1 answer
267 views

A quantity associated to a probability measure space

Let $(S,P)$ be a (finite) probability space. We associate to $(S,P)$ a quantity $n(S,P)$ as follows: The probability of two randomly chosen events $A,B\subset S$ being independent is denoted by $n(S,P)...
Ali Taghavi's user avatar
3 votes
1 answer
408 views

Updating Geman and Geman (1984) on image restoration

I am reading the seminal paper Stuart Geman and Donald Geman, Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images, IEEE Transactions on Pattern Analysis and Machine ...
rjm's user avatar
  • 75
3 votes
1 answer
169 views

Parameter estimation using bayesian update on moduli space?

Scientists take a set of data points, say in ${\mathbb R}^2$, and, assuming that this data should fit a polynomial of degree $d$ (or an exponential, etc.), they estimate parameters. I would think ...
David Spivak's user avatar
  • 8,549
3 votes
1 answer
4k views

Derivatives of conditional expectations

Let $X$, $Y$ and $Z$ be independent, real-valued random variables, probably with continuous density functions. Define $A = X + Y$ and $B = X + Z$. Consider the regular conditional expectation $E_Y(a,...
Tom LaGatta's user avatar
  • 8,372
3 votes
1 answer
2k views

Bayesian Inference with Student-t likelihood

Suppose I've observed $x$ from a Student-t distribution with unknown $\mu$, and I'd now like to infer $\mu$. Since the t-distribution isn't exponential family, there's no conjugate prior available, ...
Luke's user avatar
  • 41
3 votes
0 answers
69 views

Confusion with implementation of PDE constraint Bayesiain inverse problem

Consider a PDE, $$\partial_t u -a \nabla u - ru (1-u) = 0$$ at a given snapshot in time. The inverse problem is to find the diffusion coefficient $a \in L^{\infty}$ from a noisy measurement $$Y = \Phi(...
Jarwin's user avatar
  • 81
3 votes
0 answers
152 views

Minimizing an f-divergence and Jeffrey's Rule

My question is about f-divergences and Richard Jeffrey's (1965) rule for updating probabilities in the light of partial information. The set-up: Let $p: \mathcal{F} \rightarrow [0,1]$ be a ...
jw7642's user avatar
  • 91
3 votes
0 answers
82 views

Have stick-breaking priors with non-iid atoms been considered, and if not, why not?

Roughly speaking, a stick-breaking prior is a random discrete probability measure $P$ on a measurable space $\mathcal X$ of the form $$P=\sum_{j\ge1}w_j\delta_{\theta_j}$$ where $(w_j)_{j\ge1}$ is a ...
mathducky's user avatar
2 votes
3 answers
34k views

A "simple" explanation of the concept of D-separation in a Bayesian Network?

Hello everyone. I'm looking for a "simple" explanation of the concept of D-separation in a Bayesian Network. As far as I know the definition is "two variables (nodes) in the network are D-Separated ...
Manuel's user avatar
  • 139
2 votes
2 answers
461 views

Bayes statistics precisely formulated

I am trying to learn something about Bayesian statistics, however, I am struggling already with the simplest equations and, moreover, with the very basic questions: What are we given? What is our goal?...
Fabian Werner's user avatar
2 votes
1 answer
175 views

Derive equation for regularized logistic regression with batch updates

I am trying to understand this paper by Chapelle and Li "An Empirical Evaluation of Thompson Sampling" (2011). In particular, I am failing to derive the equations in algorithm 3 (page 6). ...
denvercoder9's user avatar
2 votes
1 answer
505 views

Adaptive priors

A lot of recent literature in Bayesian approach to inverse problems involves Adaptive priors, i.e - priors that depend on noise level. A lot of articles deal with optimization of contraction rates ...
Madhuresh's user avatar
  • 157
2 votes
1 answer
87 views

What is the problem with this model parameter estimation algorithm?

In a statistical model with parameters $\theta$ and unobserved laten variables $Z$, the model likelihood is $$L(\theta;X)=Pr(X|\theta)=\sum_ZPr(X,Z|\theta)$$ The standard way to estimate $\theta$ ...
Yicong Liang's user avatar
2 votes
0 answers
134 views

Sum of arrival times of Chinese Restaurant Process (CRP)

Suppose that a random sample $X_1, X_2, \ldots$ is drawn from a continuous spectrum of colors, or species, following a Chinese Restaurant Process distribution with parameter $|\alpha|$ (or ...
Grandes Jorasses's user avatar
2 votes
0 answers
59 views

Concentration of posterior probability around a tiny fraction of the prior volume

In the context of approximating the evidence $Z$ in a Bayesian inference setting $$ Z = \int d\theta \mathcal L (\theta)\pi (\theta) $$ with $\mathcal L$ the likelihood, $\pi$ the prior, John Skilling'...
long_john's user avatar
2 votes
0 answers
95 views

Is there any good reference on the Bayesian view that can be helpful for reading papers on the number theory using heuristic arguments?

Nowadays there are many papers on the number theory using heuristics. I have read some of them. But I have no clear understanding of the Bayesian Probability(subjective probability). The concept of ...
gualterio's user avatar
  • 1,043
2 votes
0 answers
102 views

Convergence of Bayesian posterior

Let $\Delta [0,1]$ denote the set of all probability distributions on the unit interval. Let $\mu \in \Delta [0,1]$ denote an arbitrary prior. Importantly, $\mu$ does not necessarily admit a density ...
user_newbie10's user avatar
2 votes
1 answer
717 views

Bayesian methods in online setting

Imagine the following (very concrete) model: We have a series of random variables $x_k$ with values in $\lbrace 0, 1\rbrace$. We assume $x_k \mid p_k \sim \operatorname{Alt}(p_k),$ where $p_0 \sim R(0,...
Joe's user avatar
  • 151
2 votes
2 answers
500 views

Quantifying the effect of noise on the posterior variance in Gaussian processes / multivariate Gaussian vectors

Consider a real-valued Gaussian process $f$ on some compact domain $\mathcal{X}$ with mean zero and covariance function $k(x,x') \in [0,1]$ (also known as the kernel function). This question concerns ...
jmscarlett's user avatar
2 votes
0 answers
58 views

A canonical example of the non-existence of predictive probability distribution

Section 3 of Fortini et al. (2000) states that Given $(X^\infty, \mathcal X^\infty,P)$, a predictive probability distribution of $x_n$ given $(x_1, \dots, x_{n-1})$ with respect to $P$ need not ...
Mohamad's user avatar
  • 131
2 votes
0 answers
256 views

Can truncated/non-smooth distributions be used as priors/posteriors in Variational Bayesian methods?

Variational Bayesian methods can sometimes be a good alternative to Markov Chain Monte Carlo numerical evaluation of probability distributions. They do this, as I understand it, by approximating the ...
David O's user avatar
  • 21
2 votes
0 answers
1k views

Estimating Wiener process parameters

Consider a Wiener process with zero drift, infintesimal variance $\sigma^2$, and an unknown starting value $\nu$. That is, \begin{align} Y_t \sim \mathcal{N}(\nu, t\sigma^2). \end{align} Now, ...
Neil's user avatar
  • 598
1 vote
1 answer
109 views

Bayesian inverse problems on non-separable Banach spaces

I am now studying Bayesian inverse problems. In the note of Dashti and Stuart https://arxiv.org/abs/1302.6989, they mentioned that "... when considering a non-separable Banach space $B$, it is ...
T. Huynh's user avatar
1 vote
1 answer
305 views

Posterior expected value for squared Fourier coefficients of random Boolean function

Let $f : \{0, 1\}^{n} \rightarrow \{-1, 1\}$ be a Boolean function. Let the Fourier coefficients of this function be given by $$ \hat f(z) = \frac{1}{2^{n}} \sum_{x \in \{0, 1\}^{n}} f(x)(-1)^{x \cdot ...
RandomMatrices's user avatar
1 vote
1 answer
1k views

Convolution of two Gaussian mixture model

Suppose I have two independent random variables $X$, $Y$, each modeled by the Gaussian mixture model (GMM). That is, $$ f(x)=\sum _{k=1}^K \pi _k \mathcal{N}\left(x|\mu _k,\sigma _k\right) $$ $$ g(y)=\...
wuhanichina's user avatar
1 vote
1 answer
74 views

A problem with elementary inequality involving probabilities and Brier scoring rule

I am trying to prove certain relations between certain values of the so called Brier inaccuracy measure (Brier scoring rule). Given a vector $p = (p_1, \ldots p_n)$, where $p_1 + \ldots p_n = 1$ and $...
mtg's user avatar
  • 135
1 vote
1 answer
117 views

Accounting for unobserved events in baysian learning

I wanted to use Bayes theorem to help me automate the task of deciding if I should ignore events, but I am not sure how to update the posterior if I do The simple story goes like this: An event $y_i$...
Tobias's user avatar
  • 213
1 vote
1 answer
392 views

Conditional probability and independence

Suppose that we have vectors of events $\{H_1,...,H_n\}$ and $\{D_1,...,D_m\}$. Consider the following two sets of conditions: Condition set 1 (1) $P(H_i H_j)=0$ for any $i\neq j$ and $\sum_iP(H_i)=...
Eric's user avatar
  • 2,601
1 vote
1 answer
125 views

Conditional Gaussians in infinite dimensions

I asked this over on cross validated, but thought it might also get an answer here: The law of the conditional Gaussian distribution (the mean and covariance) are frequently mentioned to extend to the ...
user2379888's user avatar
1 vote
1 answer
123 views

Conditional density for random effects prediction in GLMM

I am currently working on generalized linear mixed models (GLMM) and need some help concerning the prediction of the random effects. More specifically, I don't understand the given representation of ...
e4e5ke2's user avatar
  • 13
1 vote
1 answer
329 views

convergence of Bayesian posterior with non iid data

Let $(\epsilon_t)_t$ be a sequence of iid random variables, distributed according to the density $f:\mathbb{R}\to (0,\infty)$ and $$ x_t = q( \theta^\star, x_1,x_2, \ldots, x_{t-1}) + \epsilon_t \,. ...
Peter's user avatar
  • 355
1 vote
1 answer
730 views

Exploiting conditional independence for inference in Bayesian networks

How is conditional independence used for making probabilistic inference in Bayes networks easier or more efficient? For example, given the following Bayes network: Let's say I want to compute ...
Johan Hirsch's user avatar
1 vote
1 answer
45 views

Bayesian estimation with lower dimensional prior

Let a statistical model of a random variable $X$ with parameter $\theta \in R^m$ be represented by a density function $p(X=x|\theta)$. Assume that the prior, $q(\cdot)$, is on a lower dimensional ...
hchen's user avatar
  • 27
1 vote
2 answers
1k views

Probability spaces involved in using Bayesian Inference

I am currently reading "Statistical and Inductive Inference by Minimum Message Length" by C.S. Wallace. In this, Wallace gives a fairly informal account of Bayesian Inference which, in the case ...
12qu's user avatar
  • 19