The Problem With Naive RAG
When I architected the Medical Knowledge RAG platform, the first prototype had a critical flaw: chunk size was fixed at 512 tokens. Medical literature has highly variable information density — a 512-token drug interaction paragraph has completely different retrieval needs than a 512-token table of lab reference ranges.
What Actually Works
Semantic chunking over fixed-size: Split on headings and paragraph boundaries first. Only fall back to token limits when a section is too large.
Hierarchical retrieval: Retrieve at the document level first, then re-rank passages within those documents. A two-stage approach reduced hallucinations by ~40% in my testing.
Metadata filtering before vector search: If the user asks about dosage for Drug X, filter to documents tagged "pharmacology" before FAISS similarity search. Vector search is expensive — give it a smaller, relevant haystack.
Elasticsearch + FAISS Hybrid
Using Elasticsearch for keyword recall and FAISS for semantic similarity, then fusing scores with Reciprocal Rank Fusion (RRF) gave us the best accuracy. Neither alone was sufficient for medical terminology where exact terms matter as much as semantic similarity.
The 60% Latency Win
Redis caching of embeddings for identical or near-identical queries. Medical professionals ask very similar questions. Cache hit rate was 38% on day one.