Probability distribution on Python-dictionary-like objects?

Question

I would like to examine information-theoretical properties of random variables that take as values objects which are akin to dictionaries in the Python programing language. That is, each sample of the random variable has a various number of numerical attributes which are not exchangeable and uniquely identifiable across samples. In other words, the random variables are neither random sets, as the attributes can be uniquely identified, nor random vectors, as the number of attribtutes may differ across samples.

Formally, and using a bit of Python nomenclature and syntax, I define these objects in my work as follows: Given a random vector $X\in\mathbb{R}^n$ (providing the possible values) and a map $K:\mathbb{R}^n\rightarrow\mathcal{P}(\{1,\dots,n\})$ (determining the set of keys contianed in the dictionary) a random dictionary $D(X,K)$ is defined as $$D(X,K)=\{k:X_{k}\text{ for k in }K(X)\}$$ where $X_k$ is the $k^{th}$ component of $X$.

To study the information-theoretical properties of my "random dictionaries" I need to define a probability density on them, and it is fairly straightforward to verify that for a sample $x\in\mathbb{R}^n$ the density of $D(x,K)$ is given by $$p(D(x,K))=p((x_k)_{k\in K(x)})\cdot p(K(x)|(x_k)_{k\in K(x)})=p((x_k)_{k\in K(x)},K(x))$$ where $p((x_k)_{k\in K(x)})$ is the marginal density of the components of $x$ included in the dictionary, $p(K(x)|(x_k)_{k\in K(x)})$ is the conditional probability of $K(x)$ being the key set under the observed components $(x_k)_{k\in K(x)}$ of x, and the rightmost term is just the joint density obtained by the definition of the conditional density. So far, so good.

Now, as for my question: Say, one has to work not with the joint density but only the density of the included values $p((x_k)_{k\in K(x)})$. Then, one faces the problem that the set of included keys $K(x)$ may be different for different values $x$ of $X$, each of which inducing a different marginal density, and, hence, integrating this density over all of $\mathbb{R}^n$ would yield a value of $|K(X)|$, i.e. the number of different key sets selected by $K$ over the values of $X$ $$\int_{\mathbb{R^n}}p((x_k)_{k\in K(x)})dx=|K(X)|.$$ This is a problem, however, as a probabillity density should integrate to 1 insted of $|K(X)|$. The obvious way to fix this is to just normalize the density by dividing it by $|K(X)|$, but I am not certain whether this is the right thing to do and I could not find any relevant literature to such a problem. Thus I would like to ask if somebody knows any existing theory or literature that could be relevant to such objects, or maybe has a more reasonable approach to fixing this issue?

Many thanks!

Answer 1

Expanding on @user76284's comment. Assume that $K$ is the set of possible keys (it seems you assume $K = \{1, \dots, n\}$ for some $n$) and assume $V$ is the set of possible values for each key (you seem to assume $V = \mathbb{R}$).

A likely better way to construct the space is to first define the space of all dictionaries with a fixed set of keys $S \subset K$. This is exactly the set of functions from $S$ to $V$, denote this set $\mathcal{F}_{S \to V}$. If $S$ is finite (most likely, but not necessarily), then $\mathcal{F}_{S \to V}$ is isomorphic to $V^{|S|}$.

Thinking of the object as a function is however IMHO notationally useful, as you no longer need to worry about the specific ordering of elements in $S$.

Now, our set of dictionaries $D$ is just the union of dictionaries with all possible key sets, i.e.:

$$ D = \bigcup_{S \in \mathcal{P}(K)} \mathcal{F}_{S \to V} $$

where $\mathcal{P}(K)$ is the power set of $K$. Assuming $K$ is finite and we fix an ordering over $S$, we can also rephrase this as:

$$ D = \bigcup_{S \in \mathcal{P}(K)} \left\{(S, v) | v \in V^{|S|} \right\} $$

However, I'd argue that the function notation is somewhat better as for any given $d \in D$ you can use the usual notation/concept of domain ($\operatorname{dom} d$) to denote the keys of the dictionary and image ($\operatorname{im} d$) to denote the set of values and $d(s)$ to denote the value stored for key $s \in \operatorname{dom} d$.

Now, if $K$ is discrete but $V$ is continous, you have a mixed object with some discrete elements and some continuous elements. If you are willing to dig a bit into some measure theory, then probability distributions over $D$ are defined the usual way.

If not, we can build this up ourselves. We can say that a mixed discrete-continuous "density" $f_D : D \to \mathbb{R}^+$ is defined as:

$$ f_D(d) = f_{\mathcal{P}(K)}(\operatorname{dom}d) f_{\operatorname{dom}d \to V}{}(d) $$

where $f_{\mathcal{P}(K)}$ is a probability mass function over the powerset $\mathcal{P}(K)$ and for any $S \in \mathcal{P}(K)$ the $f_{S \to V}$ is a probability density function over the corresponding space of functions.

For any measurable $A \subset D$ lets define $\operatorname{dom} A = \{\operatorname{dom}f | f \in A\}$.

Now for any random variable $X$ over $D$ with "density" $f_D$ we have:

$$ P(X \in A) = \sum_{S \in \operatorname{dom} A} f_{\mathcal{P}(K)}(S) \int_{v \in A, \operatorname{dom} v = S} f_{S \to V}{}(v) \text{d}v $$

Unless I messed up my notation somewhere, this should satisfy all the usual properties of probability distributions.

See also https://math.stackexchange.com/questions/4463349/joint-distribution-of-continuous-and-discrete-random-variable/4463435#4463435

EDIT: The definition of "density" above assumes only continuous distributions over values given a key set. If you wish to handle both continuous AND discrete distributions over values in the same notation, you'll no longer strictly speaking have a density and you'll need to rely only on the multivariate CDF (or just do the measure theory thing)