Suppose, for simplicity, that $N=\infty$, so you are just taking values from a distribution. In order to answer your question, you should have some prior knowledge about this distribution (i.e. a probability distribution in the space of all distributions). I will give you two examples of this.

Example 1: Suppose, you are doing (many times) some experiment, which has 2 results: 0 or 1. You don't know the probability p of "1", any value $p$ from 0 to 1 is possible. Then you can formalize your knowledge as the following: $p$ is uniformly distributed on $[0,1]$. Suppose, that after a few experiments you have got the sequence w="1010110111", you can write the formula, for a posterior density $f(p)$ of $p$. In general, if you have $n$ zeros and $m$ ones then $$f(p)=\frac{(1-p)^np^m}{\int_0^1(1-p)^np^mdp}=\frac{1}{n+m+1}\begin{pmatrix}n\\n+m\end{pmatrix}(1-p)^np^m.$$ This formula is just a continuous version of Bayes' theorem

Because mean of the result of experiment is exactly $p$, the formula, written above, is exactly formula for the distribution, you are searching for. If w="1010110111", then n=3, m=7 and this distribution looks like this:

Dotted line is at p=0.7.

Example 2: Suppose, result of your experiment is a real number. Then based on your case, you, for example, can consider it to be distributed normally with expectation $a$ and standard deviation $\sigma$. You don't know $a$ and $\sigma$, but you can consider some distribution law for them. For example, you can consider them to be independent, $a$ distributed with density $f_a(a)=\frac{10/\pi}{100+a^2}$ and $\sigma$ --- with density $f_\sigma(\sigma)=exp(-\sigma)$. After some experiments you can calculate posterior density of $f_\sigma$ and $f_a$. Since $a$ is exactly mean of the result of an experiment, $f_a$ is the density you are searching for.

1

Suppose, for simplicity, that $N=\infty$, so you are just taking values from a distribution. In order to answer your question, you should have some prior knowledge about this distribution (i.e. a probability distribution in the space of all distributions). I will give you two examples of this.

Example 1: Suppose, you are doing (many times) some experiment, which has 2 results: 0 or 1. You don't know the probability p of "1", any value $p$ from 0 to 1 is possible. Then you can formalize your knowledge as the following: $p$ is uniformly distributed on $[0,1]$. Suppose, that after a few experiments you have got the sequence w="1010110111", you can write the formula, for a posterior density $f(p)$ of $p$. In general, if you have $n$ zeros and $m$ ones then $$f(p)=\frac{(1-p)^np^m}{\int_0^1(1-p)^np^mdp}=\frac{1}{n+m+1}\begin{pmatrix}n\\n+m\end{pmatrix}(1-p)^np^m.$$ This formula is just a continuous version of Bayes' theorem

Because mean of the result of experiment is exactly $p$, the formula, written above, is exactly formula for the distribution, you are searching for. If w="1010110111", then n=3, m=7 and this distribution looks like this: Dotted line is at p=0.7.

Example 2: Suppose, result of your experiment is a real number. Then based on your case, you, for example, can consider it to be distributed normally with expectation $a$ and standard deviation $\sigma$. You don't know $a$ and $\sigma$, but you can consider some distribution law for them. For example, you can consider them to be independent, $a$ distributed with density $f_a(a)=\frac{10/\pi}{100+a^2}$ and $\sigma$ --- with density $f_\sigma(\sigma)=exp(-\sigma)$. After some experiments you can calculate posterior density of $f_\sigma$ and $f_a$. Since $a$ is exactly mean of the result of an experiment, $f_a$ is the density you are searching for.