AI Has a Default Setting for Measuring Similarity And It Might Not Be the Right One
Every time Claude finds a relevant answer to your question, every time Spotify decides two artists sound alike, every time a search engine ranks one page above another, a single mathematical operation is almost certainly making that call. It’s called cosine similarity, and it has become so deeply embedded in how AI systems work that most of the people building these tools apply it without question. A 2024 paper from researchers at Netflix and Cornell is now questioning whether that default was ever as reliable as everyone assumed.
Cosine similarity measures the relationship between two things by representing them as arrows in mathematical space and calculating the angle between them. Smaller angle means more similar. The appeal is intuitive — it ignores magnitude and focuses purely on direction, which seems like a principled way to compare meaning rather than scale. When language models convert words into vectors and measure how related concepts are, cosine similarity is almost always the tool doing that measurement. When recommendation systems decide what you might want to watch next, cosine similarity is comparing your preferences to available options. It became standard because it worked often enough, was computationally efficient, and nobody looked too carefully at when it didn’t work.
The Netflix paper looked carefully. What the researchers found is that cosine similarity scores are not stable properties of the relationships in your data — they are partly artifacts of invisible choices made during the model training process. Specifically, when a model learns its internal representations, it has a degree of freedom in how it scales different dimensions of those representations. Those scaling choices leave the model’s actual predictive accuracy completely unchanged, but they can dramatically alter the cosine similarity scores that get computed afterward. The same training data, the same learned relationships, the same model — but three different scaling choices produced three completely different similarity matrices in their experiments. In one case, the cosine similarity between any two distinct items came out at exactly zero, meaning the metric concluded that nothing was similar to anything else. That result was mathematically valid given the training configuration. It was also completely meaningless.
This isn’t a theoretical edge case confined to academic simulations. Consider what cosine similarity failure looks like in production. A content recommendation system that concludes two highly related articles share zero similarity will systematically fail to surface relevant content — and the engineers maintaining it may never identify the source of the problem because the model’s predictive accuracy metrics remain unchanged. A semantic search tool that returns irrelevant results despite correct underlying embeddings may be misattributed to poor data quality or insufficient training rather than a miscalibrated similarity metric. A fraud detection system comparing behavioral vectors may incorrectly assess risk profiles as dissimilar when the underlying patterns are closely related. In each case the model is working correctly by one measure and failing silently by another — and cosine similarity is the layer where the failure lives.
The alternatives the paper points toward are not exotic. The unnormalized dot product — cosine similarity’s simpler cousin that skips the normalization step — was found to produce more stable and sometimes more accurate results in the same scenarios where cosine failed. The key distinction is that the dot product preserves magnitude information that cosine similarity discards. When the scale of a vector carries meaningful signal — and in many real-world embedding spaces it does — throwing that information away in the name of direction-only comparison actively degrades the measurement. Training the model to directly optimize for similarity, rather than computing it as an afterthought from embeddings trained for a different objective, produced unique and consistent results in the paper’s experiments. Projecting embeddings back into the original data space before measuring similarity was identified as a third remedy that sidesteps the problem entirely by removing the embedding space where the instability originates.
For anyone building with AI tools or evaluating vendor systems, the practical implication is straightforward: similarity scores are not objective measurements. They are outputs of a pipeline that includes modeling choices, regularization decisions, and normalization steps that most end users never see and most vendors never disclose. When a retrieval system tells you two things are 94% similar, that number is not a fact about the data — it is a fact about how the model was configured. Asking which similarity metric a system uses, and whether it was validated against alternatives, is a reasonable question with a meaningful answer. Most systems won’t have a good one.
The deeper issue the paper surfaces is one of inherited defaults. Cosine similarity didn’t become universal because research demonstrated it was optimal. It became universal because early systems used it, it worked well enough to avoid scrutiny, and the field moved fast enough that questioning foundational tools felt like a distraction. The result is that AI systems making consequential decisions — about what information you see, what content gets recommended, how language gets understood — are built on a measurement approach that researchers are still working to fully validate. The metric got there first. The scrutiny is catching up.
Works Cited
Steck, H., Ekanadham, C., & Kallus, N. (2024, March 11). Is cosine-similarity of embeddings really about similarity? https://arxiv.org/abs/2403.05440
