Text becomes tokens. The choice of vocabulary quietly sets the context limit and how well unusual strings survive.
Technology path
From raw text to retrieval
The retrieval half on its own, which is where the quality of a RAG system is actually decided.
4 steps · curated, not generated
-
-
Tokens become vectors positioned by meaning, so similar things end up near each other.
-
Those vectors are indexed for fast nearest-neighbour search.
-
The first pass is fast and approximate; a reranker reads the top candidates properly and fixes the order.