GitHub · by Akash Priyadarshi

jev-curate

A Rust dataset sifter that keeps clean rows verbatim and drops the rest.

Akash Priyadarshi’s pipeline streams Parquet and JSONL through Jev to filter and score synthetic, instruction-tuning and web-scraped data. It has a Rust streaming core with a Python API, evaluates rows in multi-threaded batches over speculative fan-out rather than one HTTP round trip per row, and emits the records it keeps without altering them.

The published figure is 24.0 rows per second on a single node, measured on a mock bench in the repository, with a 1,500-plus rows per second cluster target rather than a result. Read it as the honest version: a measured number for the harness and a stated target for the thing that has not been run yet. The project also quotes TypeSafe’s own benchmark figures for the cost argument, which are TypeSafe’s and not the author’s.

The case it makes is about the arithmetic of cleaning ten million to a billion rows. Judging synthetic rows with a large generative model is slow and expensive; regex and keyword filters are fast but blind to circular reasoning, hallucinated derivation steps and robotic sycophancy, because those are properties of meaning rather than of syntax. Typed decisions sit in between: a calibrated probability, an ordinal rubric score and a categorical choice per row, with no generated prose to parse.

Nothing is rewritten. Clean records are emitted exactly as they came in, which matters for a dataset containing mathematical notation that a rewriting pass would quietly damage. It is published on crates.io and PyPI, with a live demo site.