✎ Edit this page

Tutorial 4: Add a benchmark

← Learn by building

You will: turn real material your team already uses (exam papers, corrected OCR, a reviewed translation) into a small benchmark, so that AI models and skills can be measured on it. You need: some real examples with correct answers, and the right to share them. No code is needed for the first step. About one hour for a first draft.

What makes a good benchmark

Benchmarks move up four maturity levels: 1 Example (format check only), 2 Draft (real items, one author), 3 Reviewed (checked by a second reader), 4 Stable (held-out set, used in practice).

1. Propose it (no code)

Open a Propose a benchmark issue. Say which task, which language, what material, who owns it, who can check the answers, and how results should be scored. It goes on the Benchmarks board, and a maintainer replies.

Not sure which task it fits? Read the task pages, or describe it in your own words.

2. Make 5 items by hand

Write 5 items in the standard format, one per line in a file test.jsonl:

{"id": "my-set-001", "input": "the text or question", "target": "the correct answer", "metadata": {"source": "where it comes from"}}

Attach the file to your issue. A maintainer checks it with the scorer (as in Tutorial 1) and tells you if anything is missing.

3. Grow it and write the card

Add the rest of the items. Then write a short README.md (the dataset card): what the set is for, how many items, where the material comes from and its license, who wrote and who checked the answers, and one worked example. The template is toc-extraction/bo/v2.

4. Publish

A maintainer (or you, with a pull request) adds it to buddhist-benchmarks following its CONTRIBUTING.md. It gets a version (1.0.0), a git tag and a copy on Hugging Face. Every skill for the task is then scored on it, and we announce the new benchmark, crediting you and your team.

5. Keep it alive

Groups may publish competing benchmarks for the same task. We keep both, and new skills are scored on all of them.

✎ Edit this page · History · Discuss