Computers have traditionally compared texts by matching character strings. A search for "request vacation" then finds no document that only mentions "report absence". Embeddings solve this problem by translating the meaning of content into numbers. This makes it possible to compare texts by similarity of meaning, even if they use different words or are written in different languages.
Embeddings are a building block of many AI applications, especially semantic search and retrieval-augmented generation.
How does an embedding work?
A specialized embedding model, usually a neural network, converts a text into a vector, that is, a list of often several hundred to a few thousand numbers. Each number describes a dimension in an abstract space of meaning. During training, the model has learned to map texts with similar meaning to similar vectors.
To compare two pieces of content, the distance or angle between their vectors is calculated, often using what is called cosine similarity. The closer the vectors, the more similar the content. For large data volumes, the vectors are stored in a vector database that performs this search efficiently.
Besides text, images, audio, or product data can also be represented as embeddings. Some models map text and images into the same space, so that a text description can find matching photos.
Typical examples
- Semantic search in wikis, manuals, and file shares
- Detecting duplicate or similar tickets in the service desk
- Grouping customer feedback by topic
- Assigning incoming requests to known categories
- Recommending similar products or documents
What companies should keep in mind
- One model per index: Embeddings from different models are not comparable with each other. If you switch models, all content must be recalculated.
- Check the language: Not every model captures German technical language equally well. Tests with real examples from your own content are advisable.
- Data protection: Embeddings are not automatically anonymous. Under certain circumstances, conclusions about the source text can be drawn from them, so they should be protected like the original data.
- Consider chunk size: The size of the text sections for which an embedding is calculated significantly affects search quality.
How it differs
An embedding is not the same as a token. Tokens are the building blocks into which a language model splits text. Internally, each token is also translated into a vector, but in a business context, embedding usually means the vector that summarizes the meaning of an entire sentence or section.
View AI software development