Experiment

Do Language Models Pick Up Synesthetic Associations?

A small study on grapheme-color synesthesia and whether embedding spaces reflect our most common letter–color associations.


Background

Synesthesia & this project

Grapheme-color synesthesia is a perceptual phenomenon in which written letters and numbers automatically evoke specific color experiences. For a synesthete, for instance, A might always appear (or feel like) red. The associations are stable across time and some can be surprisingly consistent across individuals.

The color mosaic above shows each letter in its modal synesthetic color; the color most commonly chosen by synesthetes in a large empirical study. Hover any tile to see the percentage.

This project asks whether embedding models, the mathematical representations of language inside modern AI, have picked up these associations from text. We computed cosine similarity between letter and color embeddings across five model families, then compared the resulting 26 × 11 similarity matrices to empirical data from 6,588 synesthetes collected by Witthoft, Winawer & Eagleman (2015). Each RGB color pick was mapped by Witthoft et al. to one of Berlin & Kay's 11 basic color categories, producing a frequency matrix of how often each letter gets each color.

286
letter–color pairs tested
13
embedding conditions
ρ = .29
best baseline-corrected Spearman correlation
9 / 26
best hit rate (35%, chance ≈ 2.4)

Interactive

Explore the similarity matrices

Toggle between the synesthesia frequency data and each embedding condition. The matrices show raw cosine similarities, before the baseline correction described below. Color intensity reflects the value. Red is for frequency, green for Word2Vec, purple for RoBERTa, teal for Voyage, blue for GPT. Hover any cell for details.

Results

How well does each model predict synesthetic associations?

Both matrices carry baselines that say nothing about which letter goes with which color. Synesthetes pick some colors (blue, orange, yellow) more often for every letter, and some color words sit closer to every letter in embedding space. GPT with "letter A" / "red", for instance, puts orange closest to 16 of the 26 letters. When these two baselines happen to line up, a raw correlation goes up even if the model knows nothing letter-specific.

To correct for this, both matrices are double-centered (the mean of every letter row and every color column is removed) before computing Spearman ρ across the 286 letter–color pairs. Significance comes from a permutation test: the model's letters are shuffled 10,000 times, which keeps every baseline intact and breaks only the letter–color pairing. p-values are Holm-corrected across the 13 conditions. The 95% confidence intervals come from resampling the 26 letters with replacement; with only 26 letters they are wide, so small differences between models should not be read as a ranking. Hit rate counts how often the color that is unusually close to a letter (similarity minus that color's average over all letters) matches the most common synesthetic color.

Raw ρ = uncorrected correlation, shown for comparison. Chance for hit rate: shuffling letters gives about 2–3 / 26 on average; always guessing the single most common modal color (yellow) gets 5 / 26. Only the two GPT phrased conditions have hit rates above chance after correction (p < .01).

Key finding

The phrasing effect (GPT)

The three GPT conditions use the same model and the same 26 letters, changing only the wording. Asking for "the letter A" / "the color red" instead of bare "A" / "red" doubles the baseline-corrected correlation, from a weak ρ = 0.15 to ρ = 0.29. Voyage-3 shows the same increase (0.13 → 0.22). RoBERTa-large trends the same way (0.07 → 0.17), but its increase is within noise. A paired bootstrap over letters confirms the gain from bare to full phrasing for GPT (Δρ = +0.15, 95% CI +0.03 to +0.27) and Voyage-3 (+0.09, CI +0.01 to +0.15), but not for RoBERTa-large (+0.10, CI −0.08 to +0.27). Word2Vec and GloVe show no letter-specific signal in any phrasing. The bars show the corrected ρ and the faint marker shows the raw value. Uncorrected, the jump looked much larger (0.10 → 0.43), mostly because of a color baseline in the "letter A" condition.

Per-letter breakdown

Which letters does GPT get right?

For each letter, did the color that is unusually close to it in the model (after removing each color's average similarity) match the most common synesthetic color? Cell color = that letter's modal synesthetic color. No model gets A → Red, the most consistent association in the data (54% of synesthetes).

Left = GPT "letter A" / "red"  |  Right = GPT "the letter A" / "the color red"


Methods

Embedding models used

Five model families were tested, each in one or more phrasing conditions. For each condition, cosine similarity was computed between every letter embedding and every color embedding, producing a 26 × 11 matrix. All ρ values below are baseline-corrected.

Static
Word2Vec

Distributional embeddings trained by predicting surrounding words. Tested on Google News (3B words) and a custom literary corpus (~50 Project Gutenberg novels). Vectors are fixed — context has no effect. Phrasing a query as "letter A" is done by averaging the vectors for "letter" and "A". No letter-specific signal in any phrasing (ρ ≈ −0.08).

Static
GloVe Twitter

Global Vectors for Word Representation, trained on 2B tweets (200-dimensional). Tested on lowercase letters ("a"), since the vocabulary is lowercased. Like Word2Vec, it shows no letter-specific signal (ρ = −0.02).

Contextual · Large
RoBERTa-large

all-roberta-large-v1 via Sentence Transformers. A 24-layer BERT-style encoder fine-tuned for semantic similarity. Shows the same phrasing effect as GPT: near-zero with bare letters, ρ = 0.17 with "the letter A". Its raw correlation in that condition (0.41) was mostly baseline.

Contextual · Retrieval
Voyage-3

Voyage AI's general-purpose retrieval model. The second-strongest model after correction: ρ = 0.22 with full phrasing (p < .01) and 0.18 with "letter A". Bare letters are weaker (0.13, not significant). Its raw numbers barely changed, so its signal was never a baseline artifact.

Contextual · LLM
GPT (text-embedding-3-small)

OpenAI's text-embedding-3-small. The strongest overall performer: ρ = 0.29 with the full "the letter A" / "the color" phrasing, and the only model whose hit rate beats chance (9/26 and 8/26 for the two phrased conditions, against about 2.4 expected). Misses A → Red in all three conditions.