Small models that answer with a number
Since September, the models in the tools I build may answer only with a number, or not at all. I do not want a model to speak in them, and it has nothing to say. Synapse is the method I wrote down for building that way, and Koher published it on 20 October with its code open for anyone to copy.
I start by naming the one thing a piece of work needs to know, then split it into operations that each do one thing. Some need no model at all. Where one does, it is small enough to run on a computer I already have, and its answer is the number of a passage, the number of an item on a list fixed before it runs, or nothing. Code checks that number, and a table people wrote decides where the work goes next, including when the model does not know.
Take a farmer asking why the new leaves of a groundnut crop are turning yellow between the veins. A search by meaning puts passages on nitrogen, iron and waterlogging side by side, because all three describe yellow leaves. Someone who knows crops reads “new leaves” and “between the veins”, knows that groundnut is often grown on chalky soil, and picks the passage on iron as the likely one. In Synapse, people who know crops would label pairs of questions and passages, and a small model trained on those labels would answer with the number of the passage it picks, or with nothing. Agriculture is only my example here, and I have not trained that model.
By small I mean the size of the model, and that running it should need no rented server and no charge per answer. I also keep models small because of copyright. The models that read text in my tools start from a base model trained partly on thousands of self-published books, used without asking the people who wrote them, and some of that also sits in the large model that helped write their extra training rows. A smaller model is trained on less text, so less is taken, and I still count what is taken as theft.
When a job looks too hard for a small model, I take it to mean I have not yet split it finely enough, and I do not reach for a bigger one.
The method:
https://github.com/koherarchitecture/synapse/blob/main/METHOD.md
The code: