A semantic tokenization scheme where token geometry reflects semantic relationships [R]

I have been thinking about an alternative tokenization and representation scheme for language models and would be interested in hearing whether similar ideas have been explored before, as well as potential advantages or flaws.

The core observation is that modern tokenizers (BPE, SentencePiece, etc.) primarily capture statistical structure in text. While this is highly effective, the resulting token assignments are not explicitly organized according to semantic relationships. Concepts that are semantically related may end up with completely unrelated token identifiers, and semantic structure is learned later through embeddings and training.

The idea is to construct a tokenization scheme in which the symbolic representation itself carries semantic information.

For example, instead of assigning arbitrary identifiers to concepts, we could learn a mapping from concepts to short character strings such that semantically similar concepts receive similar codes. A concept like “dog” might receive a code close to those assigned to “wolf” and “fox”, while more distant concepts such as “car” would receive codes that are farther away in the code space.

One possible implementation would be:

1) Build a semantic graph using resources such as WordNet, embedding similarity, or a combination of both.
2) Learn a compact symbolic encoding for concepts.
3) Optimize the encoding so that distances between codes correlate with semantic distances in the graph.
4) Train language models directly on these codes.

An extension of the idea is to treat a standard keyboard layout as a fixed geometric space. The keyboard itself is not semantically meaningful, but it provides a globally agreed-upon metric structure. The learned encoding could exploit distances between characters and positions when constructing semantic codes.

For example, if two concepts are semantically close, their symbolic representations would differ only slightly. Ambiguous concepts could potentially occupy positions that reflect their relationships to multiple semantic regions. Context would still determine the intended meaning, but the representation itself would encode semantic structure rather than relying entirely on downstream embedding learning.

My intuition is that such a representation could act as an inductive bias, potentially improving:

- Sample efficiency
- Training efficiency
- Interpretability
- Cross-lingual concept sharing
- Compression of semantic information

However, it is also possible that sufficiently large models already learn these structures efficiently, making such an encoding unnecessary.

I would be interested in feedback on several questions:

1) Has similar work been explored in tokenization, representation learning, or NLP?
2) Are there theoretical reasons why such a representation should or should not help?
3) Would a semantically structured symbolic space provide a useful inductive bias for transformer-based models?
4) Are there related approaches involving semantic hashing, vector quantization, discrete latent spaces, graph embeddings, or other forms of structured tokenization that I should look into?

I am particularly interested in understanding whether explicitly embedding semantic structure into the symbolic representation could provide measurable benefits over learning that structure entirely through embeddings and model training.