Semantic Search with Embeddings: Making a Company Knowledge Base Findable with AI

Finding one piece of information among thousands of documents, contracts, manuals and tickets is a problem keyword search only half solves: if a user types «early withdrawal» and the contract says «termination before expiry», nothing is returned. Embedding-based semantic search turns texts and questions into numeric vectors that represent meaning, so relevant content is found even when the words differ, and it is the foundation of enterprise assistants, chatbots and LLM-based RAG systems.

What Embeddings Are and How Semantic Search Works

An embedding is a numeric representation, a vector with hundreds or thousands of dimensions, produced by a machine learning model: texts with similar meaning end up close together in vector space. Search follows three steps: each document is split into chunks and converted to vectors, vectors are stored in an index or vector database, and for each user question an embedding is computed and the chunks with the highest similarity, usually measured with cosine similarity, are retrieved. The result is not an answer but a ranked list of relevant passages that an LLM can then use to write an answer with source citations.

Chunking: the Decision That Matters Most for Quality

Search quality depends more on how documents are split than on the model chosen. Chunks that are too long dilute meaning and consume context; chunks that are too short lose track of what they refer to. Good practice is to split by logical structure (headings, sections, paragraphs), keep a small overlap between chunks and attach metadata such as document title, section, date, author and confidentiality level. Metadata lets you filter by department or validity and respect existing access permissions, an essential requirement when the knowledge base contains documents not meant for everyone.

Hybrid Search and Reranking: Combining Keywords and Meaning

Embeddings do not replace keyword search: product codes, acronyms, proper names and contract numbers are found better by lexical search (BM25), while natural-language questions benefit from semantic search. The most robust approach is hybrid: run both searches, merge the results and reorder them with a reranking model (cross-encoder) that scores each question-passage pair more precisely but on a small set of candidates. This two-stage architecture, fast retrieval then accurate reordering, is today's reference setup for production systems.

Measuring Results: Metrics and Evaluation

Without measurements every change is an opinion. Build a small evaluation set of 50-100 real questions with the documents that should be found, then measure metrics such as recall@k (the share of correct answers present in the top k results) and MRR (the average rank of the first correct result). The set lets you compare embedding models, chunk sizes and reranking strategies with data rather than gut feeling, and spot regressions immediately when documents or models change. For sensitive data, also consider open-source models that run locally, to avoid sending confidential text to external services.

• Split documents by logical structure, with a small overlap and metadata on every chunk.
• Use hybrid search (lexical plus semantic) with reranking on the top results.
• Respect access permissions by filtering results per user or department at query time.
• Build an evaluation set of real questions and measure recall@k and MRR on every change.
• Plan reindexing when documents change and when the embedding model is replaced.

Key takeaway: semantic search works when treated as a system to design and measure, not a feature to switch on: sensible chunking, hybrid search, respected permissions and continuous evaluation make the difference between a convincing demo and a tool employees actually use.