Machine Learning

Pencil sketch of a Mac mini and a laptop showing a loss chart and an answers chart both going down, a mug reading one sentence, and sticky notes: v1, v2 and v4 crossed out, v3 ticked

Fine-tuning a small language model for RAG in a browser on a Mac

Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element Last week we tried to summarize semantic search results with small language models running in your browser. They weren’t great at it, hallucinating often. But a 0.6B-parameter model doesn’t need to know everything, it just needs to read four passages, and either summarize them or say they don’t answer the question. “That sounds like a fine-tuning problem, and maybe a follow-up post.” ...

Pencil sketch of a laptop running ask my blog, a mug reading two sentences max, and sticky notes: retrieve, summarize, check the sources

Running RAG with a (small) language model in the browser

Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element A few weeks ago MicroLLM lab was featured in Hacker Newsletter. It shows off how you can run small language models (25 to 360 million parameters) in a browser on the users’ own GPU (or CPU)1. No server, no API key, no account. Pretty nifty, so it got me thinking about the steps after semantic search I added to this blog: that already provides the “R” in RAG2 for a question, and it does that entirely in the browser, from a 4 MB lookup table. What if a tiny model, also in the browser, read those passages and answered the question? ...

Narrating your blog with Kokoro, a local and opensource text-to-speech model

Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element Seven years ago I used Google’s text-to-speech API to create audio versions of my posts. There was a Python script that parsed the Markdown file, cleaned it up, chunked it to stay under Google’s 5000 byte limit, convert each chunk to an MP3, stitch them together, and finally embed the resulting file with a Hugo shortcode that is essentially a fancy <audio> tag. It worked, it was free1, and I was pretty pleased with it. The little audio player has been sitting at the top of my posts ever since. ...

Sketch of a laptop running a search for 'how does pgvector indexing work?' next to a scatter plot of embeddings, with sticky notes 'embed locally', 'search instantly', 'stay private'

Client-side semantic search for your static site

Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element Eight years ago I added client-side search to this blog with Lunr.js. It creates an inverted index at build time, ships it as JSON, and matches strings in your browser. No server-side engine required. It has worked fine ever since, in the sense that it finds a post if you type a word that is actually in it. ...

Cosine similarity on a dark background: vectors A (an orange) and B (an apple) separated by angle theta, with the formula cos(theta) = A·B / (||A|| ||B||)

Building a semantic search engine in ±250 lines of Python

Listen to this article instead ▶ 0:00 / 0:00 0.75× 1× 1.25× 1.5× 2× Your browser does not support the audio element Once upon a time I wrote a post about building a toy TF-IDF keyword search engine. It has been one of the more popular posts I’ve written, and in this age of AI I felt a sequel has been long overdue. It’s pretty fast (even though it’s written in pure Python), it ranks results with TF-IDF, and it can rank 6.4 million Wikipedia articles for a given query in milliseconds. But it has absolutely no context of what words mean. ...