
Running RAG with a (small) language model in the browser
Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element A few weeks ago MicroLLM lab was featured in Hacker Newsletter. It shows off how you can run small language models (25 to 360 million parameters) in a browser on the users’ own GPU (or CPU)1. No server, no API key, no account. Pretty nifty, so it got me thinking about the steps after semantic search I added to this blog: that already provides the “R” in RAG2 for a question, and it does that entirely in the browser, from a 4 MB lookup table. What if a tiny model, also in the browser, read those passages and answered the question? ...
