Semantics of neural network-like structures

Question

Background

Language (of mathematicians and most other people) has a sequential surface structure and a tree-like deep structure. So semantics usually is the semantics of such syntactical structures: sequences and/or trees of symbols.

Model theory - as the standard way of doing mathematical semantics - is an unequaled success story which produced many deep results of great generality, handling structures of almost arbitrary complexity, provided they are represented language-like.

Other kinds of representations of objects (than closed terms), placeholders (than open terms), properties and relations (than open formulas), or facts (than closed formulas) can be thought of, e.g. special diagrams drawn on a sheet of paper or on a discrete grid, obeying certain rules. As long as there is a systematic way of translating such representations into sequential, language-like ones, they don't bring anything new into play.

But:

For some people it's a matter of fact, that for a single person the most immediate representation of objects, relations, facts, etc. - be it mathematical or not - is in his brain, as a spatio-temporal pattern of interacting units.

The simplest model of such a brain in action is a sequence of directed labelled graphs (vertex labels = neuron activity, edge labels = synaptic weights, both subject to change, as is the number of vertices and edges).

Doing semantics, i.e. giving meaning to states, or sequences of states, or whatsoever of such a system can in principle follow two tracks:

1. translating the neural representation in a systematic way into a language-like representation and doing "classical" semantics on the latter

2. doing semantics directly on the neural representation

Question

Be it given that there is probably no systematic way of translating neural representations into language-like ones:

Is there nevertheless something like a mathematical semantics of neural or graph-like representations as sketched above?

This is in fact a reference request: I am looking for advanced, elaborate, and general treatments of the matter. (Neural Binding does not count as a general treatment.)

The treatment doesn't have to mention neural networks explicitely: everything graph-like, but more general than a sequence or a tree would be appreciated.

^{More and better tags are welcome!}