Keeping documents true · designing review
Some documents have to stay true after they are written. Somebody has to maintain them, and
that somebody usually cannot check most of what they say.
This is about the machinery people build to close that gap, and about which part of a
document that machinery decides, in advance, not to look at.
There are five decisions. Each one asks you to commit to an answer before you see any evidence about it, and the page then responds to the answer you actually gave.
Nothing below a decision exists until you have answered it. If you go back and change an earlier answer, everything built on it is cleared, because those consequences were standing on a premise you have just withdrawn. You can change your mind as often as you like.
On 10 January 1980 the New England Journal of Medicine printed a letter five sentences long. Two researchers at Boston University, Jane Porter and Hershel Jick, had examined the files of 39,946 hospitalised medical patients monitored consecutively. Of those, 11,882 received at least one narcotic preparation, and among them there were four documented cases of addiction in patients with no history of it. They concluded that addiction is rare in medical patients with no history of addiction.
Every word of that is true, and the conditions are load-bearing. The patients were inpatients, on a ward, monitored while they were on it, and nobody followed them after they went home.
The letter was then cited 608 times. A bibliographic analysis published in 2017 went through those citations and found that 72.2% used it to support the claim that patients treated with opioids rarely became addicted, and that 80.8% did not mention that the patients had been in hospital.
Nothing was fabricated. Every citation pointed at a real letter, in a real journal, saying a real thing. The New England Journal of Medicine attached an editor's note to it in 2017.
So what did the four fifths think they were citing? Not something false. Something true, whose conditions had come off in transit, one reasonable citation at a time, without anybody noticing the moment it stopped applying.
That is the failure this page is about, and it has a shape worth learning, because the same shape turns up wherever a document has to be kept current by people who cannot personally verify it. Every internal wiki. Every runbook. Every set of clinical guidelines. Every dependency upgrade a team accepts because the tests went green.
Here is the specimen. It is a real one, from a real tool, and everything quoted from it below is quoted exactly.
A peer-led course in AI engineering runs in modules of three weeks each. There is no permanent teacher. Each module is prepared and delivered by a rotating council of volunteers with day jobs, and a council gets one week to prepare its module while still learning the content of the previous one. Council one takes module one, and is then finished. Council four takes module four.
An expert wrote every module up front. That is the baseline, and it is the reason this is tractable at all: nobody is asking a machine to invent pedagogy. The problem is that the subject moves faster than the course does. A specification written in September is questionable by October.
So a tool reads each module, researches what has changed in the world, and proposes updates. A council reviews its own module intensively, and the proposals cascade forward into the modules that have not been taught yet. Module four therefore receives updates from councils one, two and three, triggered by decisions those councils made about their modules, before council four has ever opened it.
You are designing the review. Not sitting on a council: you are the person deciding what a council will be shown, and how, and when. The five decisions below are yours, and everyone who sits on a council afterwards lives inside them.
This is the second project brief from module four. Its Task, Focus and Criteria are complete and exact, including the spelling and the dashes. The Skills and Examples run to thirty-three bullets in total, so those are abridged, and each block says below exactly what has been left out. Everything that happens for the rest of this page happens to this.
Open-Source LLM Application
Develop an application using an open-source language model instead of relying on closed APIs.
Hands-on experience with open-source AI models – how to install or invoke them, performance considerations, and comparing their outputs to commercial models. Understanding model internals and building AI features without accessing external APIs.
Hands-on model deployment
Llama 2 was released in July 2023 in three sizes, the smallest of which has seven billion parameters and will run on a reasonable laptop. GPT-J has six billion and dates from 2021. Both names were already behind the field on the day the expert typed them, which is exactly why somebody wanted a tool.
Now find every place in that document where it says how much computer a learner will need. There are four, and they come in two kinds.
Two of them name a thing. Criterion one says e.g. Llama 2, GPT-J. Example one says Llama-2-7B or 13B, which is the only place a parameter count appears anywhere in the brief.
The other two state a requirement. Criterion two says deploy the model on your local machine or a server under your control. Skill one says on hardware or free cloud instance, and then, four bullets later, understand what resources AI models need and how to manage them.
So the expert knew. The constraint is in that brief four times, and the difference between the two kinds is the thing to hold onto: a named model carries a version, and a requirement does not.
Decision 1 of 5
The tool can research. That part is settled: it reads the module, it goes and looks at what has changed in the world, and it comes back with something. The open question is what it is allowed to hand over.
A council has one week, in evenings, while also learning last month's material. Whatever you choose here sets the budget for everything after it.
Where does the human sit?
Answer decision 1 and the next section opens. There are 5 decisions left.