Semantic IDs—and How to Train Them

Over the past few weeks, I’ve been shifting some of my attention from relational foundation models to generative recommenders (GenRecs). Working at Spotify, the idea of generating recommendations naturally interests me: how can we connect listeners with the right audio content?

What semantic IDs are

One concept that keeps coming up is semantic IDs: short sequences of discrete tokens obtained by quantising dense item embeddings.

Think of quantisation as approximating a continuous vector using a finite collection of representative vectors. Their indices become tokens. An item can then be represented as something like `(12, 45, 8, 19)`, which a generative model learns to predict one token at a time.

Learning IDs with residual quantisation

Google’s TIGER trains a transformer to generate the next item’s semantic ID from a user’s interaction history. Its RQ-VAE starts with pretrained content embeddings, transforms them through an encoder, and quantises the resulting latent vectors. Each quantisation level approximates the residual left by previous levels. A decoder reconstructs the original embedding, with reconstruction, codebook and commitment losses training the components together. (Rajput et al., 2023)

The residual idea reminds me of gradient boosting with squared-error loss: each stage addresses what previous stages failed to capture. The analogy is useful for understanding successive correction, although the training algorithms differ.

When simpler quantisation wins

What surprised me in Spotify’s GLIDE paper was that residual K-Means outperformed RQ-VAE: 9.52% higher relative HitRate@30, with fewer training complications. R-KMeans still learns centroids, but quantises the item embeddings directly, without an additional neural encoder–decoder. GLIDE’s embeddings come from a podcast-specific text encoder following BGE-M3’s architecture. (D’Amico et al., 2026, §§4.1 and 5.2.2)

My hypothesis is that sufficiently good initial embeddings may leave less room for an additional learned transformation to help. That transformation could introduce distortion or optimisation difficulties without improving retrieval.

What if a generated ID is invalid?

Another question bothered me: what if the model generates a token sequence that doesn’t identify any real item?

TIGER’s nominal space contains 256⁴ = 4,294,967,296 combinations—approximately 4.29 billion—versus roughly 12,000–18,000 catalogue items. Yet only 0.1%–1.6% of top-10 predictions were invalid across its datasets. The authors suggest widening beam search and filtering invalid IDs. (Rajput et al., 2023, §§4 and 4.5)

GLIDE also uses beam search, with 30 beams in production, and reports poorer retrieval with simpler decoding alternatives. Its evidence supports beam search for retrieval quality, rather than guaranteeing that every output is valid. (D’Amico et al., 2026, §§4.4.4 and 5.2.5)

Why the representation is appealing

The appeal for me is the combination of a compact output vocabulary and shared semantic structure. Instead of learning a separate output token for every item, a model learns reusable code tokens and their combinations.

For intuition, imagine two running shoes sharing an early prefix, with later tokens distinguishing finer details. This isn’t a manually assigned taxonomy: individual tokens need not translate neatly into “running,” “red” or a particular brand.

There is still a learning problem after quantisation. Adding tokens to an LLM’s vocabulary doesn’t automatically give them meaning: their embeddings and predictive relationships need training. That can involve adapting a pretrained LLM or training a generative model from scratch.

Beyond recommendation

I’m also curious about applications beyond recommendation—for example, representing customer behaviour as discrete codes for an LLM to model. That’s a hypothesis to test against alternatives such as dense-vector inputs, especially where quantisation might discard information the task needs.

This is an independent exploration of publicly available research, written in a personal capacity. This post is not affiliated with or endorsed by Spotify and does not represent Spotify’s views.

References

D’Amico, E., De Nadai, M., Chandar, P., Vohra, D., Lin, S., Lefarov, M., Gigioli, P., Penha, G., Kopysitsky, I., Senese, I. J., Mei, D., Fabbri, F., Semerci, O., Zhao, Y., Tang, V., St. Thomas, B., Ranieri, A., Smith, M. N. K., Bernkopf, A., . . . Bennett, P. N. (2026). Deploying semantic ID-based generative retrieval for large-scale podcast discovery at Spotify [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2603.17540

Rajput, S., Mehta, N., Singh, A., Keshavan, R. H., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V. Q., Samost, J., Kula, M., Chi, E. H., & Sathiamoorthy, M. (2023). Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36. https://doi.org/10.52202/075280-0452