PolicyLens

PolicyLens lets a user upload an insurance policy PDF and ask questions about it in plain English. The system splits the PDF into clause-sized chunks and stores them as vectors in Postgres with pgvector. For each question it finds the best chunks, re-ranks them, and gives them to an LLM that answers only from that text. The answer streams to the screen. It also makes a structured summary on upload.

A policy can be 50 pages. Sending all of it on every question costs many tokens, adds delay, and the model can miss details in the middle. Fine-tuning changes how a model writes, not what it knows, and we cannot train a model for every upload.

With RAG we send only the 3 most relevant chunks. It is cheaper and faster. We can show which section the answer came from. Each user’s data stays in the database and is filtered by policy ID.

The AI work needs Python libraries: PyMuPDF for PDFs, sentence-transformers for the re-ranker, tiktoken for token counts. The product work (users, file storage, history) was quick to build in Express. Node is the single public API for the browser. Python is an internal engine that Node calls.

The trade-off is one more network hop, two deployments, and two places where auth can go wrong. A smaller team could put everything in FastAPI. I would consider that if I started again.

• The browser posts the file to Node. Node checks the JWT and makes a policy ID.
• Node saves the file in Supabase Storage and adds a row to policies.
• Node forwards the file to Python without waiting, and answers the browser at once.
• Python runs a background task: parse with pymupdf4llm, clean, chunk by clause, embed with Jina in batches of 100, insert into pgvector in batches of 50.
• The browser polls the status every 2 seconds and shows a progress bar. When chunks exist, the status is ready.
• A separate thread then makes the summary with the LLM. The dashboard polls for it every 6 seconds.

• The browser posts { question, policy_id } to Node. Node adds k = 8 and forwards it to Python.
• Python cleans the question, picks metadata filters, and embeds it with Jina.
• pgvector returns the 8 nearest chunks for that policy (cosine distance).
• A cross-encoder scores each chunk against the question. We keep the top 3.
• We build a context block, add the system prompt, and call the LLM with streaming.
• Tokens flow back through FastAPI and Node to the browser. Node saves the full answer in chat_history when the stream ends.

A fixed-size splitter can cut a clause in the middle, and a half clause can reverse its meaning (“…does not cover” without the rest). Insurance text has structure: sections, numbered clauses, tables. My chunker splits at SECTION, PART, ARTICLE, CHAPTER and “##” headings first. A section that fits the size limit stays whole. A larger one is split at clause markers such as 3.1 or (a), with a 3-line overlap. Only if a piece is still too large do we split by tokens.

Each chunk also gets metadata: section name, clause type, coverage category, flags for deductible and limit, page and token count.

Different clauses have different shapes. A definition is short and exact, so a 384-token maximum keeps it focused. Coverage and endorsement clauses are longer and need more room (768). A schedule is a table, so it gets 1,024 tokens and no overlap. Smaller chunks give sharper matches. Larger chunks keep more context. The clause type lets us choose the right balance for each part.

If 30% or more of the lines in a section hold a “|”, the section goes to the table chunker. It finds the header row and the separator row, groups the data rows into batches up to 1,024 tokens, and repeats the header in every batch. This way a row like “Fire | $500” never loses its column names. These chunks are marked as type schedule with table_chunk = true.

An embedding is a list of numbers that represents the meaning of a text. Texts with similar meaning have vectors that point in similar directions, so we can search by meaning instead of by exact words. “Is fire covered?” can match a chunk that says “loss caused by fire, smoke or explosion”.

Jina v3 gives 1024 numbers by default. It supports shortened vectors, so we ask for 768. This matches the database column and uses less storage. Smaller vectors lose a small amount of accuracy but search faster. If I changed the size, I would need to change the column and re-embed all chunks.

pgvector adds a vector column type and distance operators to Postgres. Cosine distance (<=>) measures the angle between two vectors. A small angle means similar meaning. The SQL function returns 1 − distance as the similarity.

An exact search compares the query with every row. HNSW is an index that links each vector to near neighbours in layers. A search walks the graph and finds close matches much faster, with a small chance of missing the true best one. I used m = 16 and ef_construction = 64, which are common defaults.

Vector search uses a bi-encoder: the question and each chunk are turned into vectors separately. This is fast but rough. A cross-encoder reads the question and a chunk together and gives one relevance score. It is more accurate but slower, so we run it only on the 8 candidates.

We keep 3 because the context must be short. Less text means lower cost, lower delay and less noise for the LLM. The numbers 8 and 3 were reduced from 8 and 5 to cut response time. I would tune them with a test set.

• The system prompt says: answer only from the context, never guess, and use a fixed sentence when the answer is missing.
• Temperature is low (0.3).
• The context holds only the 3 best chunks, so there is less unrelated text.
• The non-stream API returns the source chunks so a person can check them.

The limit: nothing checks the answer after generation. A next step is a faithfulness check, where a second model confirms that each claim appears in the sources.

Every chunk stores policy_id in its metadata. Every search passes a filter { policy_id }, and the SQL uses metadata @> filter. The policy ID starts with the user ID and a random UUID, so it is hard to guess.

I should be honest about the gap: the server does not yet check that the logged-in user owns that policy ID. The fix is a lookup in the policies table before every query. I would also move to row-level security in Supabase.

Today the tests check the pipeline: chunk sizes, filters, formats. They do not measure quality. To measure it I would build a golden set of 30 to 50 questions with the correct answer and the correct source chunk for several real policies. Then I would track recall@k (is the right chunk in the top 8?), MRR, and answer faithfulness (using a judge model or manual review). I would run it when I change the chunker, the embedding model or the prompts.

pymupdf4llm reads the text layer. A scan has none, so the text is empty and nothing useful is stored. The project keeps the old LlamaParse path behind USE_LLAMAPARSE=true, which does OCR in the cloud but takes 60 to 120 seconds. A better design detects an empty text layer and runs OCR only for those files.

Right now each question is searched alone, so “and what is the limit?” has no subject. I would send the last few messages with the request. Then I would do query rewriting: ask a small model to turn the follow-up into a stand-alone question (“What is the fire coverage limit?”) before embedding. The rewritten question goes to search. The recent messages also go in the final prompt.

The filters are keyword rules. A question with “flood” adds coverage_category = flood. But a chunk has only one category, the first match, so a chunk about fire and flood may be tagged “fire” and be excluded. The fallback runs only if the search returns zero rows. A safer design uses the filter as a boost, or runs a search with and without it and merges the results.

The policy text is untrusted input. The model has no tools, so it cannot take actions, which limits the damage. The worst case is a misleading answer. I would wrap the context in clear delimiters, tell the model that text inside them is data and not instructions, and scan the output for odd content. For a bigger system I would also log sources for audit.

After ingest, a thread runs a fixed search query (“policy overview benefits coverage exclusions surrender death benefit premiums”), takes the 10 best chunks, and asks the LLM for a JSON object with fields such as key benefits, exclusions, death benefit and free-look period, in plain English. The code strips code fences, parses the JSON and saves it in policy_summaries. If parsing fails, it saves nothing. The prompt suits life insurance best.

Ingest takes seconds to minutes: parse, many embedding calls, many inserts. A normal HTTP request would risk a timeout, and the user would see nothing. With a background task the upload returns fast and the UI shows real progress. Polling is simple and works on any host.

The alternatives are server-sent events or WebSockets (push instead of poll) and a job queue such as Celery, RQ or Cloud Tasks. A queue also survives restarts and allows retries, which the current in-process tasks do not.

Abstract base class plus factory for the LLM, embedder and vector store, so I can switch provider with an environment variable. Lazy imports for heavy libraries. A start-up hook that loads the query service once. A retry decorator with exponential backoff. A singleton status tracker. A backend-for-frontend proxy in Node. Pydantic for settings and chunk metadata.

The @with_retry decorator calls the function, and if it raises, waits and tries again. The wait grows each time: 1 s, 2 s, 4 s for Jina and Supabase (3 tries); 5 s, 10 s, 20 s for LLM calls (4 tries). This helps with short rate limits and network errors. One known problem: on the Supabase insert, a retry repeats batches that already succeeded. I would retry per batch instead.

It was the simplest thing that worked on one server. It breaks on restart (progress is lost), with several workers (each has its own dictionary), and it has no failed state. The fix is a status column or table in the database, or Redis. The chunk count already serves as a durable “ready” signal, which is why the endpoint checks it first.

One database holds users, policies, chat history and vectors. I can filter vectors with normal JSON and SQL conditions, join them with other data, and use one free-tier service. There are fewer moving parts. A dedicated vector database can scale to many millions of vectors with more tuning options. For this size, Postgres is enough.

The Python service runs on a free Hugging Face Space with a small CPU. A local model needs torch and a large download, and it embeds slowly on CPU. The Jina API is fast, has a free tier (about 1 million tokens per month) and gives strong quality. The cost is a network call, a third party that sees the text, and a limit on usage. The code keeps a local BGE option for offline use.

All LLM classes follow the BaseLLM interface: complete, stream and the same with message lists. Groq and Kimi use the OpenAI-compatible API, so they share most of the code. The factory picks the class from LLM_PROVIDER. The query service never knows which provider it uses. Gemini needs a small adapter to convert message formats.

I use fetch and read response.body.getReader() in a loop. Each chunk is decoded with TextDecoder and passed to a callback. The callback updates state: text = text + token for the AI message with a known ID. React re-renders and the bubble grows. I used a plain text stream, not SSE, to keep it simple.

The app has only five screens and no nested routes. A state variable plus history.pushState with a hash works on any static host without server rewrites. It also let us handle the OAuth redirect, which returns tokens in the URL hash. The cost is manual work for deep links and no route-level code splitting. react-router is installed but unused. If the app grows, I would switch to it.

In development StrictMode runs effects twice. The upload effect then sent two uploads. I removed StrictMode and added guards (a ref flag, a ref for the callback, a narrow dependency list). It works, but the better fix is an effect that is safe to run twice: for example, start the upload from a button click handler instead of an effect, and cancel with an AbortController on cleanup.

After the upload call returns a policy ID, a timer calls the status endpoint every 2 seconds and sets the bar to the progress number from the server. When status is ready, the app moves to the dashboard. Then another timer asks for the summary every 6 seconds, up to 20 times, while a skeleton shows. The chat button is already usable at that point.

Email and password: bcrypt hashes the password, and Node returns a JWT signed with a secret, valid for 7 days. Google: the browser signs in through Supabase, then asks Node for its own JWT. Every protected route runs a middleware that verifies the JWT, and falls back to checking a Supabase token. The browser stores the token in localStorage and sends it as a Bearer header.

1. The Google login route issues a JWT for any e-mail without checking the Supabase token. Fix: verify the token with Supabase on the server.
2. The summary route has no login check. Fix: add the middleware and an owner check.
3. No ownership check on query and status, and the Python API is public. Fix: check owner in Node, add a shared secret, restrict CORS.
4. An example env file holds what looks like a Google client secret. Fix: rotate it and remove it from the repo history.

I would also add rate limits, server-side validation, and delete all data for a policy when the user deletes it.

localStorage is easy, but any script on the page (an XSS bug) can read the token. An httpOnly cookie cannot be read by scripts, but then you need CSRF protection (SameSite and a CSRF token). The most robust choice is a short-lived access token in memory plus a refresh token in an httpOnly cookie. We used localStorage for speed of development, and I know the risk.

Today: files are in a private Supabase bucket and shared by signed URL (valid for 1 year). Chunk text goes to Jina, and question plus chunks go to the LLM provider. So data leaves our system. I would shorten URL life, tell users clearly, offer delete-everything, remove personal data before sending where possible, and choose providers with no-training and no-retention terms. For strict customers, I would use the local embedding model and a self-hosted LLM.

PDF parsing is fast (0.2 s). The slow parts are external: the embedding calls during ingest, and the LLM calls (the summary takes 30 to 40 s). For many users I would: move ingest to a job queue with separate workers; run several API instances behind a load balancer; keep status in a shared store; cache embeddings of common questions; add connection pooling for Postgres; and add rate limits per user. The re-ranker runs on CPU, so I would watch its latency and could move it to a GPU or a hosted API.

The prototype uses free tiers: Hugging Face Space, Vercel, Supabase, Jina (about 1M tokens per month) and Groq. One upload costs a few embedding calls and one LLM call for the summary. One question costs one embedding call and one LLM call. The main cost risk is the LLM at scale. Short context (3 chunks) keeps tokens low.

The Python API is a Docker image on a Hugging Face Space. A GitHub Action uploads the repo to the Space on every push to main. The Node backend has its own Dockerfile. The React app deploys on Vercel with an environment variable for the API URL. Secrets are environment variables, never committed. Known gaps: the Dockerfile caches a different re-ranker than the code loads, the health check points to a missing route, and there is no automatic test step in the pipeline.

PDF parsing. The first version used LlamaParse, a cloud service that took 60 to 120 seconds per file. I replaced it with pymupdf4llm, which runs locally in about 0.2 seconds, and made the output format identical (same page markers) so the chunker did not change. I kept the old path behind a flag as a rollback. Other changes: summary moved to its own thread so the dashboard opens sooner; embeddings sent in parallel batches; retrieval reduced from 8-and-5 to 8-and-3 to cut latency.

Duplicate uploads. In development, React ran the upload effect twice and the same file was sent twice. I found it by watching the network tab and the double log lines. I fixed it with a ref guard and by removing a changing callback from the dependency list, and I left comments in the code explaining each fix. I now prefer to start uploads from a click handler, not an effect.

1. Fix the security items: verified Google login, owner checks, a Node-to-Python secret, secret rotation.
2. Add a failed state, a durable status store and a job queue.
3. Build an evaluation set and track recall@k and faithfulness.
4. Add conversation memory with query rewriting.
5. Show sources in the chat and jump to the page in the PDF (the API already returns page and highlight text).
6. Support more policy types with a detected type and matching prompts.

I would write the evaluation set first, so every choice (chunk size, k, re-ranker) has a number behind it. I would use one backend language, or define the contract between Node and Python early. I would design auth and ownership checks at the start instead of adding them later. And I would keep the README in step with the code, because it already differs in several places.