Article
How we measured text difficulty and fooled ourselves three times
A teacher needs to know whether a text is within a pupil's reach. The question sounds simple, but the answer is a number — and numbers can lie convincingly.
We set out to teach a machine to place an English text on the CEFR scale, from A1 for beginners to C2 for fluent readers. The practical point is telling B1 from B2, which in an Israeli school is the difference between a 4-point and a 5-point bagrut.
The work took one day. The valuable part is not the result but three errors, each of which produced rising, plausible numbers while there was nothing to measure.
The first error: the instrument was measuring length
Early versions scored 44 percent correct, then 51, then 56. The numbers were climbing, the features looked sensible. We were ready to believe them.
Something else gave us pause. We began looking at where the instrument was wrong, and the picture was too tidy: three misses in a row fell on short texts — 134, 151 and 218 words — and three hits in a row on long ones: 441, 919 and 1045.
We checked the dataset. In it, the average length of a text rises with the level: about a hundred and twenty words at A2, more than eight hundred at C2. This is not a property of language but the design of the exams: the easy paper gives a short passage, the hard one a long passage.
We selected texts of equal length and ran the instrument again. It scored 49 percent — exactly what you get by always answering "B2" without looking at the text at all.
The instrument was measuring length. Everything else was decoration.
Something unpleasant also emerged about the very B1/B2 boundary the whole exercise was for: in this dataset B1 texts run from 134 to 461 words and B2 from 451 to 759. The overlap is one text out of seventy-one. On such data the question "B1 or B2" is undecidable in principle, and any published accuracy across that boundary is accuracy on length.
The second error: an instrument that could not disagree
We asked a language model to describe texts and to name their level along the way. Two independent runs agreed with each other 100 percent of the time. Agreement with the exam was also 100.
A perfect result should alarm you more than a poor one.
We checked across the whole corpus: of 8464 descriptions, 8454 named exactly the level already stored in the database — including a source that has only two levels. The explanation was humbling: the level was being fed to the model in its input. It was not assessing anything. It was repeating.
Hence a rule we now apply to everything: an instrument showing perfect agreement is suspect until it has been proved capable of disagreeing at all.
The third error: direction taken from a textbook
We wanted to know whether grammar helps. We counted the density of constructions rated B2 and above — according to the reference book. We got 51, 36 and 53 percent, which is pure noise, and concluded that grammar says nothing about level.
The conclusion was wrong, and the error was hiding in the grouping itself.
We began measuring each rule separately, and saw that complexity and the rule's level in the reference book diverge systematically. The Past Simple, an A1 construction, turned out to be the strongest marker of a difficult text: in 92 percent of cases there was more of it as the text grew harder. Subject pronouns, also A1, were the strongest marker of simplicity — they vanish as text grows harder.
The bucket "B2 and above" was mixing rising features with falling ones, and they cancelled each other out completely.
Once the direction of every rule had been measured rather than taken from a book, grammar alone began ordering nine out of ten triples of texts correctly.
How we checked ourselves honestly
Comparing different texts is meaningless: length, genre and topic get in the way. We were lucky with a dataset in which the same Guardian article has been rewritten by editors at three levels of difficulty. The content is fixed and only the execution changes — so everything that separates the versions belongs to difficulty, not to subject matter.
Features were selected on one half of those triples and measured on the other half, which took no part in the selection. Otherwise the instrument memorises the answers instead of learning.
What came out of it
81 percent correct on the school scale, against 41 for the strategy "always answer with the most common class". A language model asked directly gives 44.
The strongest feature was an unexpected one: the distance between a word and the word it depends on for meaning. The harder the text, the further related words drift apart, and the more the reader has to hold in memory. At the upper boundary this feature separates texts flawlessly — a hundred percent.
A pattern emerged that we had not expected. At the bottom of the scale difficulty grows by form: longer sentences, longer words, wider vocabulary. At the top it grows by substance: words become rarer and the links between them longer. Features that work excellently at school levels lose half their power higher up.
What the instrument cannot do
It does not tell C1 from C2. For school this hardly matters — the programme ends at B2.
It stumbles on polysemy: the word minute is A1 meaning "sixty seconds" and C1 meaning "tiny", while dictionaries are keyed by spelling.
And it runs into the ceiling of agreement among the dictionaries themselves: ours and the Oxford list agree exactly on only 39 percent of words. There is nothing on which to demand more accuracy from an instrument than the authorities have among themselves.
On novelty — honestly
The client asked directly whether this can be called our own method. We checked the literature, and the answer is no.
In the first edition of our report we presented the paired verification method and the weighted sum as our own working solutions. They are not: back in 2016 Xia and co-authors contrasted pairwise comparison with classification, gave the same argument — text difficulty is continuous rather than divided into steps — and obtained the same difference in numbers.
The features we arrived at are listed in the work of Vajjala and Meurers. The danger of confusing length with level is known. The paired corpus was built precisely for comparisons of this kind.
Arriving at the same conclusions independently has the value of replication, but not of priority. This cannot be called a method of our own.
What may be of interest are three particulars. A demonstration, in numbers, that a well-known dataset is unfit for the question "B1 or B2". Measuring the direction of each grammatical construction instead of taking it from a reference book. And a merged dictionary as the median of three independent sources with disputed words excluded — the exclusion raised accuracy from 58 to 66 percent.
Why we are telling this
All three errors were found the same way: we read the cases in which the instrument was wrong. We did not audit the code or reason about the method — we read particular texts and looked at what was wrong with them.
A rising number proves nothing. What proves something is a control experiment designed so that the instrument could fail — and the particular cases, read one by one, where it did.