Skip to content

Rust Readability Crate: Data Inside the Binary

4 min read Notes

Rust Readability Crate: Data Inside the Binary

A scoring job that runs on a build agent cannot depend on a data volume being mounted. A rust readability crate that compiles its language data into the binary removes that dependency, and readsight does exactly that with 86 languages.

Code: ReadSightRS, MIT licensed.

Rust 1.74, edition 2021, #![forbid(unsafe_code)].

What is embedded

Eighty-six language JSON files and 86 hyph-*.tex pattern files ship inside the crate through include_dir. Compiled patterns are built from the same data at startup, so a first score needs no cache warm-up and no writable directory.

Config::from_dirs() and ReadSight::with_config() exist for the case where the data must change without a rebuild: a custom language file, a correction to a hyphenation pattern, a test fixture.

The API asks for a formula name

let rs = ReadSight::new("en-us")?;

for formula in rs.supported_formulas() {
    let result = rs.score(&formula, text)?;
    // result.score, result.grade_level, result.interpretation
}

supported_formulas() returns the slice that belongs to the language: nine for en-us, eleven for es, eight for de-1996, five universal ones for a language like th.

score() dispatches by name, and the typed methods (gunning_fog(), lix(), wiener_sachtextformel(text, variant)) return Result<FormulaResult, Error>. Empty input gives Error::EmptyText instead of a zero, which is the failure mode a batch job actually needs to see.

FormulaResult carries its inputs

Every result exposes score, grade_level, interpretation, formula_name, language_code and a BTreeMap of intermediate values: words, sentences, syllables, letters, polysyllables.

Intermediate values are what makes a number reviewable. A grade of 18 on Gunning Fog is a verdict; the map says whether long sentences or long words produced it, and that decides the edit.

Three syllable modes and a named trade

tex is the default for 80 languages: the Liang algorithm against hyph-utf8 patterns. composite covers en-us and en-gb. heuristic covers ru, uk, be and bg, where one vowel is one syllable and a TeX pattern over-counts.

split_word() always returns exact hyphenation points regardless of mode; split_syllables() may return the heuristic split. The readme states that syllable counts in tex mode are approximate, which is the sentence most libraries leave out.

Tests that prove parity

The suite runs 90 integration tests: golden vectors against the PHP reference for supported_formulas, analyze, every applicable formula across 86 languages and per-word syllable vectors for twenty languages, plus ported unit tests and a smoke test that builds every language and runs every supported formula.

Doc tests are part of it, because the examples in the readme are compiled and executed by cargo test. A readme that compiles is a readme that cannot drift.

Building and running a batch

let rs = readsight::ReadSight::new("de-1996")?;
let grade = rs.flesch_reading_ease(text)?.score;

One engine per language, reused across the batch. supported_languages(None) returns the 86 codes sorted, which is what a loop needs to cover a multilingual export without a hardcoded list.

cargo test runs the parity suite, clippy runs with -D warnings, and the crate forbids unsafe, so a batch tool can be reviewed by someone who is not a Rust specialist.

Where the crate fits

A CLI that rates a content export, a service endpoint that returns grades for a document, a warehouse job that scores a catalogue monthly. Anything that has to run offline, in a container, or on a machine where nobody can install a data package.

It does not do layout, grammar or translation quality. It scores text and names its own approximations, and the batch owner decides what a grade is worth.

Related: nodejs readability for the same engine in a JavaScript build, and islands architecture for another design where the payload carries everything it needs.

Custom work

Need an API that does not exist yet?

Describe the task and the response format. We will estimate the endpoint, the integration and the maintenance.

Start a conversation