One number or four

Talata gives learners feedback on their written English. The scorer behind it is a fine-tuned DeBERTa-v3-large with attention pooling, four regression heads and an ordinal head. The four heads are task achievement, coherence and cohesion, vocabulary and grammar. The ordinal head produces one holistic score.

There was a period where the API returned only the holistic score and the app showed no scorecard. There's now a scorecard with four traits on it. Nobody retrained anything to make that happen.

A trait scorecard is the thing learners actually want. Being told you're around B2 doesn't tell anyone what to do next. Being told your organisation is the weak part does. Four bars also fill a screen, and I'd already built the component. It was switched off deliberately, and the rest of this explains why.

Sitting underneath that is a problem I haven't solved. The numbers all sit on a scale nobody has validated. Fixing that means paying experienced markers to band a few hundred essays, which is real money up front, and I have no clear route to earning it back. So for now the thing is free, and being free is what makes the trait-labelled data usable at all. The money problem and the licensing problem happen to point the same way.

The checkpoint I built to keep a commercial option open was trained on permissively licensed data: ASAP 2.0 and KLiCKE, both CC BY, both carrying human holistic ratings. One human score per essay. Nothing in that lineage tells a model what coherence is, so the four heads were being optimised against an objective nobody had ever labelled.

The acceptance record for the run puts it flatly. Human trait supervision: false. Result: invalid for serving. Across the frozen benchmark the raw trait outputs average -0.2096. The reported score scale starts at 2.0. The heads were producing a number that wasn't a score.

So that checkpoint declares its dimensions as unserved. The API omits them, and the app falls back to a plain estimate with no scorecard. If it had shipped, learners would have received four confident-looking numbers, and the first person to ask how they'd been validated would have had a very bad answer.

The holistic head of that checkpoint was worth serving, and it was measured against a frozen benchmark of 514 essays held out of every training and calibration split, checksum-protected in CI so it can't drift quietly. The pass marks were fixed before the run: QWK at least 0.70, adjacent agreement at least 0.80, MAE no worse than 0.85. A threshold picked after you've seen the result isn't a threshold.

Two checks that aren't accuracy ran alongside it. Eight essays were degraded through three increasing error densities, giving 24 transitions where the score should never rise; it held on 22. The cheaper CPU comparator already in production held on 23. That one-step gap went into the record as a difference, not as a clean pass. Separately, within a single prompt the model ordered essays the way the human raters did 88.65% of the time, worst prompt 83.73%. That one is a diagnostic and not a gate, because every prompt in it appeared in training, and the record says so.

0.8495 is narrower than it looks. The frozen benchmark holds out essays, not questions. Every prompt in it appeared somewhere in training. Every prompt a real learner writes to has not.

So I retrained seven times, holding out one prompt each time, and scored each fold on the prompt it had never seen. Macro QWK 0.6785, worst fold 0.5665, macro MAE 0.5598, adjacent 0.8770.

Same model, same task, same metric. The only difference is whether the question was in training. For that checkpoint, when somebody asks how good it is, 0.6785 is the honest answer.

This isn't academic. Serving a large encoder on a GPU is the expensive part of a free tool, so I tested a cheap replacement: frozen bge-base-en-v1.5 embeddings concatenated with surface features I already compute, gradient boosting on top, CPU only, no fine-tuning. On the frozen benchmark it reached QWK 0.8429 against the GPU model's 0.8495, with a lower MAE. That's the point where you write the migration ticket.

On leave-one-prompt-out it got macro QWK 0.6028 and a worst fold of 0.3116. The plain feature model, with no embeddings at all, got 0.6606 and 0.4478. Adding the embedding improved the in-distribution benchmark and made cross-prompt generalisation worse than the baseline it was meant to beat. The embedding carries topic, and topic is a shortcut when the test questions are also the training questions.

With only the frozen benchmark I'd have replaced a working scorer with a worse one and had a metric telling me it was fine. Split your evaluation by whatever varies in production. For essay scoring that's the prompt, not the row.

Talata is free. Nothing is sold, there's no paywall, and there's no donations link. That last one was a decision, not an oversight. It also isn't a principled stand. I can't see a route to revenue I'd bet on yet, so free is where it sits.

Which matters, because the best trait-labelled data in this field is non-commercial. ELLIPSE is about 6,500 essays by English language learners, each carrying an overall rating plus six human trait ratings, under CC BY-NC-SA 4.0. BAWE is 2,700 proficient academic essays under the same family of licence, which is where the advanced end of the scale comes from. Both are usable while the app is genuinely free and unpaywalled, and neither is usable in anything that takes money.

The checkpoint trained on that data has real trait supervision. Coherence and cohesion comes from ELLIPSE's cohesion rating. Vocabulary is the mean of its vocabulary and phraseology ratings. Grammar is the mean of syntax, grammar and conventions. Task achievement has no ELLIPSE equivalent and currently uses overall as a proxy, which is the weakest of the four and is written down as such.

It scores held-out QWK 0.76, MAE 0.63. That's lower than 0.8495, and I don't put the two side by side as though they were comparable. Different training data, different held-out set, different definition of a band.

It also has the weakness I've just spent two sections on. 0.76 is a held-out split, not leave-one-prompt-out, so by my own argument it's the optimistic figure and the cross-prompt number for this checkpoint doesn't exist yet. That's on the list.

So production now serves four traits plus Overall, and the app shows the scorecard. The commercially clean Overall-only bundle stays deployed, tested and switchable, because if money ever changes hands the model profile has to flip first. Some rules that keep that option open: consent for any essays collected has to permit commercial use even though nothing is sold today, the clean benchmark has to stay reported so the commercial path always has current evidence, and there's no distilling a cheap student from the non-commercial teacher. A student model inherits its teacher's lineage. Distillation doesn't launder anything.

Grammar checking has the same shape of problem, more starkly.

I put LanguageTool and a GECToR token-tagging model through a small hand-built probe: 30 sentences each containing one known error, plus 15 clean sentences to catch false positives. GECToR found 28 of the 30. LanguageTool found 22. Neither flagged anything in the clean set except one non-error note from LanguageTool. It's a 45-sentence probe and I describe it as one, not as a benchmark.

GECToR is clearly better and I can't sell it. Its checkpoint was trained on a restricted mix of GEC corpora, so it runs in the free app and is filtered out of the commercial-safe assessor profile by configuration. LanguageTool is the clean one and the weaker one.

That's a normal state of affairs in applied ML, and it's why provenance is tracked per artifact and not per project. "Is this model any good" and "am I allowed to use this model for this purpose" are separate questions with separate evidence, and you can't train your way out of the second one.

Having real human trait labels makes the four heads defensible. It doesn't make them correct. The grammar head is the example.

Measured against production on 27 July 2026: a clean 286-word essay, carrying 0.35 error spans per 100 words, returned Grammar 7.0. The same essay with nine planted agreement and plural errors, at 4.89 spans per 100 words, also returned Grammar 7.0. The head reads register and sentence complexity. It doesn't read accuracy.

That's worse than an inaccurate number. The revision loop is the product. A learner who fixes every flagged error, resubmits and watches the grammar score sit still has good reason to stop trusting the tool.

The fix uses marks the assessors have already produced, so it costs no extra inference. Distinct error-span density drives a reduction on the Grammar trait: nothing below one span per 100 words, ramping to a full deduction at eight, capped at 1.5 bands, downward only, rounded to the same half-band steps as everything else. Overall is untouched and still comes from the holistic head.

Three details in there are the actual engineering. It counts merged overlapping spans instead of raw marks, because LanguageTool and the neural model routinely flag the same word; one production essay produced 19 marks over 13 distinct regions, and counting marks would penalise a learner for being caught twice. It only ever moves the score down, because finding no errors is not evidence that there are none. And it's bounded, so a noisy assessor can dent a score but can't destroy one.

There's an obvious tension here, and it's worth being explicit about. Earlier I evaluated a LanguageTool residual correction on the holistic score and rejected it. On a stratified sample of 600 essays it looked useful. Fitted on all 2,444 development essays its coefficient collapsed to -0.00346, frozen-benchmark QWK moved from 0.759 to 0.758 and MAE moved from 0.692 to 0.693. It was fitting noise in the subsample, and it claimed to be a measurement. So it went, and LanguageTool stayed a feedback layer that doesn't touch the score. The other half of that same decision record did ship: a tail-preserving isotonic calibration on the holistic output, with the adjustment capped at one point and tapered to nothing by 8.5 because the fitting data has no trustworthy evidence that high. It's live on the model production serves today.

The grammar adjustment survives because it claims much less. It's labelled in the code as a product heuristic and not a calibrated measurement, it's bounded and monotonic and one-directional, and it stays away from the number the holistic head has actual evidence for. A heuristic is fine as long as you don't dress it up as a measurement.

The scorer ranks writing well. Whether its numbers mean anything is unproven, and that's the largest open problem in the product.

Look at what the frozen benchmark is. 514 rows, all ASAP 2.0, and the band label is the ASAP holistic grade plus one. 100 rows each at bands 2.0 through 6.0, 14 rows at 7.0, and nothing above that at all. No examiner has rated anything in the pipeline. Every headline accuracy figure on this page is measured against that mapping.

So the top of the served scale is unmeasured, which is why the model under-scores strong writing by roughly 0.7 of a band no matter how the calibration is fitted. That's a data problem and calibration cannot reach it. Fitting a better curve to a scale nobody has checked produces a more confident wrong answer.

What fixes it is 150 to 300 essays carrying real human band scores, with the label provenance recorded per row so a set that mixes teacher ratings with self-reported exam results can be audited later. Spread matters more than volume: fifty essays above band 7 are worth more than another two hundred in the middle, because the middle is the part the model already handles. Once those exist, the calibration refit, conformal band ranges and a defensible accuracy claim are all CPU work.

The hard part isn't collecting them. It's the consent wording, which has to permit commercial use at the moment of collection. Contributors are anonymous, so a set gathered under research-only phrasing can never be upgraded and would have to be collected again from nothing. That single paragraph decides whether Talata can ever be sold. It's had more attention than most of the model code.

Until the anchor set exists the app leads with a cautious level description and a range, the numeric output is optional detail carrying an explicit note about the benchmark's limits, and the history view shows no numbers at all. The model ranks better than it measures, so that's how it's presented.

The anchor set is a hiring problem and not a modelling one. Someone with real marking experience has to sit down and band a few hundred essays, and they should be paid properly for it. IELTS markers and tutors exist and can be hired. That's money out before anything comes back.

And I don't have a clear plan for making it back. There's no paywall, no subscribers, no institutional contract, and no evidence yet that anyone wants this enough to pay for it. Buying labels first and looking for demand afterwards is a bet, and it's not one I can justify at the moment.

So for now it stays free and I see whether anybody uses it. That's the whole decision. I'm not going to dress it up as a strategy. If usage turns up, the case for the anchor set makes itself, and the consent wording is already drafted so the option stays open. If it doesn't, I've spent nothing finding out.

So the number that matters operationally is the cost floor, not the throughput. Keeping the bill near zero while nobody is using it is one of the main things driving the architecture.

The cold start is written down as a decision, so nobody optimises it away later. A warm container removes it, and the standing cost isn't worth paying at this level of use. Anyone meeting that wait fresh will read it as a defect, which is exactly why the reasoning has to be recorded somewhere other than my head.

There's a useful accident in all this. More GPU time cannot improve a scale that has never been measured, so the cheap option and the correct option are currently the same option: stop training and go and find labels. That won't stay true, and I didn't plan it.

Scoring is one shape of this problem. Generation is the other, and the question changes from "how close is this number to a human's number" to "is this safe to put in front of a learner".

Comprehendo generates reading exercises: a passage, one multiple-choice question, four options, an explanation for each, and the exact span of text that proves the answer. A freshly generated item passes two automated checks before it can enter the shared cache that other learners are served from.

Two details do most of the work. The correctness verifier never sees the marked answer, because a model shown the key will find a reason the key is right. And the publish flag decides nothing on its own: the code re-applies every threshold itself and rejects the item if the issues list is non-empty. The judge's verdict is an opinion. The gate is code.

The two checks also fail in opposite directions, deliberately. If the correctness verifier is unavailable or returns something unparseable, the item is allowed through and structural validation still applies. If the quality verifier does the same, the item is rejected and regenerated. A wrong answer key is visible to a learner and gets reported. Fluent, plausible, subtly machine-written prose sitting in a shared cache is visible to nobody, so an absent grader must never be the thing that promotes it.

None of that is difficult. The difficult part is the last mile, where the evaluation says no and there are four working heads in the checkpoint, a scorecard component already built, and competitors whose marketing pages have four bars on them.

For a while I could account for one number, so I shipped one. I can now account for four, so the app shows four. If the licensing position changes, it goes back to one, which is why the other profile stays tested.

None of which is settled. The licence on the training data, what I can afford to measure and what I'm ever allowed to charge for turn out to be the same problem, and I've picked the cheapest corner of it. It's free, it costs close to nothing while it sits idle, and I'd rather find out whether anyone wants it before spending money on making it accurate. Ask me again if anyone does.