Reports

Read-only analysis passes over the dictionary database

Reports answer “what does the data look like right now?” — coverage gaps, vocabulary distribution, term age. They live in src/reports/ and are run from the command line, since several take minutes over the full lemma table and print tables meant to be read in a terminal or redirected to a file. Pass --help to any of them for the full argument list, and --persona or --backend to choose a database.

Broadly Common Words

Top 150 in several corpora, and the next most common

Lists the tokens in the top 150 of at least three corpora, by combined rank, then the 50 best combined ranks not in that list: words common everywhere without leading enough corpora to qualify.

PYTHONPATH=src python src/reports/broadly_common_words.py

Options:

--top-n N
--min-corpora N
--runner-ups N
--output report.json

View report

src/reports/broadly_common_words.py

Database Integrity

Orphans, duplicates, missing fields, bad levels

Runs the structural checks in storage.integrity: orphaned derivative forms and form-token links, lemmas missing required fields or inflections, duplicate GUIDs and words, out-of-range difficulty levels, unpunctuated sentence translations, and sentence levels that disagree with their linked words. Read-only; the checks' fix paths are deliberately not exposed here.

PYTHONPATH=src python src/reports/integrity.py

Options:

--check NAME (repeatable)
--verbose
--output report.json

src/reports/integrity.py

Stale Audio

Audio whose text no longer matches the translation

Finds audio whose expected_text has drifted from the current translation, so the file still says the old word. The release round-trip does not catch this -- the file is present and valid, just wrong. Flagging rows for regeneration is done from the Audio Hub.

PYTHONPATH=src python src/reports/stale_audio.py

Options:

--language CODE
--verbose
--output report.json

src/reports/stale_audio.py

Translation Coverage by Level

Which languages owe words, per difficulty level

Cross-tabs curated lemmas by difficulty level against every language, showing how many words each language still owes. Used to choose the language set for a batched Voras run, and to confirm a level range is complete before a release.

PYTHONPATH=src python src/reports/translation_coverage_by_level.py

Options:

--max-level N / --min-level N
--languages es-419 zh-tw de ...
--per-level
--missing-for LANG
--output report.json

src/reports/translation_coverage_by_level.py

Vocabulary Distribution

Word counts across levels and parts of speech

Reports how curated vocabulary is spread over difficulty levels and POS types, which shows whether a level is over- or under-filled.

PYTHONPATH=src python src/reports/vocabulary_distribution.py

Options:

--min-subtype-count N
--output report.json

src/reports/vocabulary_distribution.py

Missing Words

Expected words absent from the dictionary

Lists words that should exist in the dictionary but do not.

PYTHONPATH=src python src/reports/missing_words.py

Options:

--top-n N
--min-rank N
--output report.json

src/reports/missing_words.py

Wordlist Coverage

Dictionary coverage of an external word list

Parses a wikitext-format English word list and reports which of its content words the dictionary already covers, ignoring grammatical words.

PYTHONPATH=src python src/reports/wordlist_coverage.py WORDLIST

Options:

WORDLIST (positional)
--output report.json

src/reports/wordlist_coverage.py

Term Age

How long terms have been in the database

Reports the age distribution of terms, from their added-at timestamps.

PYTHONPATH=src python src/reports/term_age.py

Options:

--level N / --level 3-8
--pos-type noun
--output report.json

src/reports/term_age.py

Level Warnings

Problems in the curriculum's level assignments

Warns about core levels outside 20-50 words (ideal 25-40), headwords with two senses in the core, scattered subtypes, words far rarer than their level-mates, and senses taught out of order. Warnings only; it moves nothing.

PYTHONPATH=src python src/reports/level_warnings.py

Options:

--language en
--max-level 30
--output report.json

src/reports/level_warnings.py

Words at Each Level

Diffable listing of the curriculum

Writes one text file listing every level's words, grouped by subtype, for review and diffing.

PYTHONPATH=src python src/reports/level_words.py --output FILE

Options:

--output FILE (required)
--bands core,named,topic

src/reports/level_words.py

Looking for the LLM-backed verification and integrity tools? Those live on the Quality Hub and the Data Quality page.