How-to guides
Each guide lists the repo, the steps and the check to run. In every repo: run python3 scripts/build.py (where it exists), then python3 scripts/check.py, before you push. CI runs the same check.
- Create a new repo
- Run an evaluation
- Add a result and update the leaderboard
- Promote a winner to buddhist-skills
- Handle a submission
- Announce a new leader
- Turn a Share issue into a prompt library entry
- Add a benchmark
- Add a task, a language or a model
- Add an external leaderboard
- Release a BuddhistBench version
- Knowledge base: pages, Directory, experiments
- Change the design
- Add a contributor or a backer
- Add a partner organization
- Write provenance records
- Handle an expert review
- Handle a compute or API credit donation
- Post news
Create a new repo
Always create a new BuddhistAI repo from a template, so it starts with the agent notes (CLAUDE.md, AGENTS.md), the license, CI and the right layout.
| New repo for | Template | Name it |
|---|---|---|
| One skill (a workshop repo: variants, runs, provenance) | BuddhistAI/template-skill-repo | <task>-<language>, for example segmentation-tibetan |
| One benchmark (test data, scorer, maturity level) | BuddhistAI/template-benchmark | <task>-<language>-benchmark |
| Anything else | Start empty, then copy CLAUDE.md from knowledge-base scripts/agent-notes/ |
— |
Steps: on GitHub, New repository → Repository template → pick the template, owner BuddhistAI, public. Then replace every <task>, <language> and <code>, write the repo’s own line in CLAUDE.md and AGENTS.md (“This repo: …”), and add the repo to the Architecture table.
A safety net: the weekly agent-notes workflow in knowledge-base opens a pull request in any org repo that has no CLAUDE.md (Architecture).
Run an evaluation
Today a maintainer runs every evaluation on their own machine with their own API key. Automatic scoring is planned (Status).
Generic steps (any task):
- Read the task’s
TASK.mdin buddhist-benchmarks for the input, the output format and the headline metric. - Run the skill (or the plain model, for the “No skill” row) on every item’s
inputintasks/<task>/data/<code>/v<major>/test.jsonl. Savepredictions.jsonl, one line per item:{"id": ..., "output": ...}. - Log real token counts from the API (input and output, every call, retries and tool calls too). Cost on the leaderboard comes from these.
-
Score it:
python3 tasks/<task>/scorer/score.py --language <code> \ --data tasks/<task>/data/<code>/v<major>/test.jsonl \ --predictions predictions.jsonl > scores.json - Repeat at least 3 times when the model is not deterministic (Claude 5.5 models only accept temperature 1.0).
- Never tune a skill on the test set. Use separate training data (for example toc-extraction-tibetan
benchmark/data/train.jsonl). - Run the same model with no skill too, so the leaderboard can show the gain.
- Save the exact command, library versions and run logs, and write a provenance record per leaderboard row in the skill’s own repo.
Worked example: Tibetan TOC (toc-extraction-tibetan):
python -m venv .venv && .venv/bin/pip install -r benchmark/requirements.txt
export ANTHROPIC_API_KEY=...
python benchmark/bench_skills.py --model anthropic/claude-haiku-5-5 --temperature 1.0 \
--conds ours,colleague,colleague+tool,seed,skill:variants/sabcad-outliner-toc \
--out results/<date>-haiku-5-5-run1.json # run 3 times: run1, run2, run3
python scripts/summarize.py results/<date>-*-run*.json
python scripts/to_result_json.py --date <date> --commit <short-commit> --benchmarks ../buddhist-benchmarks
ours is the GEPA variant, colleague the sa-bcad outliner, seed the no-skill prompt; skill:<folder> runs any variant folder. to_result_json.py writes the result.json files straight into a buddhist-benchmarks checkout. The benchmark version is pinned in benchmark/benchmark.json and downloaded on first use. GEPA optimization (benchmark/gepa_toc.py) burns many tokens: always set an iteration limit.
Add a result and update the leaderboard
Repo: buddhist-benchmarks. Format: docs/results.md.
- Make
tasks/<task>/results/<code>/runs/<run-id>/withresult.json(required), plusscores.jsonand the outputs so anyone can re-score.<run-id>=<date>-<skill>-<variant>-<version>-<short-commit>-<model-slug>. - Check that the skill’s repo has a provenance record for the run (from 2026-11-17 on).
result.jsonmust citeskill_ref(owner/repo@commit:path),model(must be inmodels.json),data_version,scorer_version,repeatsandusage(real tokens,estimated: false).- Add the variant to that language’s
skills.jsonif it is new:status(promoted,challenger,baselineorreference),name,approach(two sentences),made_by,source. - Run
python3 scripts/build_leaderboards.pyandpython3 scripts/check.py. Commit and push tomain(or open a PR and merge it). - CI rebuilds
LEADERBOARD.md, the README table,STATE.mdand the site, and publishes the results table to Hugging Face. - Update the homepage by hand: the leaderboard preview on https://buddhistai.github.io/ is a
rowsarray in buddhistai.github.ioindex.html(search forconst rows=). Copy the new numbers there. (Planned: read them from the leaderboard data instead.)
Top performance = highest headline score among shippable skills on the latest data version. Best value = highest score among options that cost at most 20% of the top performance. The script picks both.
Promote a winner to buddhist-skills
Repo: buddhist-skills. Full steps in its CONTRIBUTING.md.
- Check the leaderboard: the new skill beats the current one on the same data version, by more than the run-to-run noise (compare the ± values).
- Copy the skill into
skills/<task>-<language>/with metadata:task,language,language-code,benchmark(<task>/<code>@<version>),score,version,source(owner/repo@commit:path),promoted(date),authors. - Run
python3 scripts/build.py(regeneratesSKILLS.mdand.claude-plugin/marketplace.json) andpython3 scripts/check.py. - Add an entry at the top of
CHANGELOG.md. - If this is the first skill for this task in any language, add the task to
skills/ask-buddhistai/SKILL.md. - In buddhist-benchmarks
skills.json, set the new variant topromotedand the old one tochallenger, and setpromoted. Rebuild the leaderboards. - Update the task page here (
skills:in the front matter), runpython3 scripts/build.py. - Announce the new leader.
Never edit a released SKILL.md directly. Improvements start as a challenger (a variant in the task’s workshop repo) and come back through promotion.
Handle a submission
Today (manual):
- A submission arrives as a pull request to buddhist-benchmarks (following SUBMIT.md), a Share issue, a homepage “paste your repo” link (opens a prefilled buddhist-skills issue), or a post in Discussions.
- Check: the result cites an exact commit, the right data version, real token counts, and includes outputs. The author’s repo has a provenance record (for a closed tool, the same facts in the PR description). The skill follows the one-language rule. The author is named.
- If only a skill was submitted (no run), run the evaluation yourself (above) on the standard models (today Claude Haiku 5.5 and Claude Sonnet 5.5).
- Merge. Move the item on the Skills board to “Being measured”, then “Released” if it is promoted.
- Thank the author on the issue or PR.
Planned (decided at the 2026-11-17 call): submit by linking a GitHub repo; a GitHub Action reads the repo, guesses task and language, runs the benchmark on a sponsor key, and posts the score; a maintainer approves (like BDRC’s OCR leaderboard). See Building the leaderboard app.
Announce a new leader
When a submission becomes the Top performance or Best value pick for a task and language, or a new benchmark goes live:
- Congratulate the author on the issue or PR.
- Add a post at the top of News (Post news), and post the same news in Discussions → Announcements.
- Post on LinkedIn and X, crediting the person and their organization, with a link to the leaderboard row. Who owns the social accounts is not decided yet (Status); until then, ask NT.
- Keep posts plain and neutral: what the task is, the score and cost, who made it, how to install it.
Example: “New Best value for Tibetan TOC extraction: [skill] by [author, org] scores 0.82 on Claude Haiku 5.5 at about $0.05 per 100 pages. Install it from buddhist-skills. Leaderboard: [link]”
Turn a Share issue into a prompt library entry
Repo: buddhist-prompts (until 2026-11-18 this was community/ in buddhist-skills).
The Share your files form takes a link or uploaded files: a skill, or any file someone gives to AI (personal AI file such as CLAUDE.md, project instructions, style guide, glossary, prompt, workflow). Name and picture are optional. It says that the form is public and needs a GitHub account. The homepage links to it from the hero, the Contribute section and the “Art, practice and teaching” cards.
- Create
<topic>-<language>/<author>-<name>/withCARD.md(fromtemplates/CARD.md) and the file unchanged. Setkind(skill, workflow, prompt, personal, project-instructions, style-guide, glossary, other) andfilefor kinds other than skill, workflow and prompt. For a repo link, copy the files at one commit and record the repo and commit inSOURCE.md. - Credit the author in
authors(the name they gave, or their GitHub name). Setgithubto the issue author’s GitHub name,avatarto the picture they attached (if any) andlicenseto their choice. WriteSOURCE.md(where the files came from, author, date, license). One language per entry. - Run
python3 scripts/build.pyandpython3 scripts/check.py. Open the pull request withCloses #<issue>; after the merge, post the link to the entry in the issue. The issue is on the Prompt library board already: move it to Reviewing while you work, then Listed. - A skill for a task that has a benchmark: also add it as a variant in the task’s workshop repo so it can be measured. If the task has no workshop repo yet, note it on the task’s Skills board card.
- A paper or a guide written for people to read goes to the knowledge base Papers instead.
Authors ask to change or remove an entry in their issue; do it within a few days.
Add a benchmark
Repo: buddhist-benchmarks (see its CONTRIBUTING.md).
- Agree on it in a Propose a benchmark issue. Prefer real data already in use. Favor what active translators need.
- Add
tasks/<task>/data/<code>/v1/test.jsonlandREADME.md(dataset card; template:toc-extraction/data/bo/v2). Addtasks/<task>/results/<code>/skills.json. - New task: also
TASK.md,scorer/score.py, the entry intasks/index.json(setactive),CHANGELOG.md. - Have the answer key checked by a reader of that language and tradition.
- Run
python3 scripts/check.py, merge. CI tags<task>/<code>@1.0.0and publishesBuddhistAI/<task>-<language>on Hugging Face. - Run the current skills and plain models on it. Update the task page here (
benchmarks:,leaderboard:,status: measured).
When two groups have different benchmarks for the same task, keep both and score new skills on both.
Add a task, a language or a model
- Task: a page
tasks/<id>.mdhere (copytemplates/task.md) and aplannedentry in buddhist-benchmarkstasks/index.json. Runpython3 scripts/build.pyhere. - Language: add it to buddhist-skills
languages.jsonand buddhist-benchtaxonomy/languages.json. - Model: add it to buddhist-benchmarks
models.jsonwith its display name, provider, price tier and list prices (input_per_mtok,output_per_mtok,date,source) before adding its first result. When a provider changes prices, update them there: every cost follows.
Add an external leaderboard
Add an entry to buddhist-benchmarks external.json. The overview shows it as a card that links out. To show its live top rows, add a parser in build_leaderboards.py (--fetch-external). Nothing external is scored by us.
Release a BuddhistBench version
Repo: buddhist-bench. Bump VERSION and each changed track.json, add a CHANGELOG.md entry, run python3 scripts/check.py, push to main. hf-sync tags vX.Y.Z and publishes. Real items need human authors and two human reviewers (same tradition and another tradition). Never commit test.jsonl, permission letters or private papers. Never publish an intervention that is not accurate, sourced and reviewed.
Knowledge base: pages, Directory, Languages, experiments
Repo: knowledge-base.
- Task or workflow page: copy from
templates/. Runpython3 scripts/build.py(index tables, “Found in workflows”, Directory tables), thenpython3 scripts/check.py. - Directory entry: add to
directory/projects.ymlordirectory/datasets.yml, never todirectory/README.md. Verify the link first. - Language page:
languages/<id>.md, copied fromtemplates/language.md. Keep every frontmatter key and section.directory_namesmust match the language names used in the Directory YAML: the “In the Directory” block on each page and the counts inlanguages/README.mdare generated from them byscripts/build.py. Countries live inlanguages/countries.yml(each with a source link). The speakers chart onlanguages/README.mdis drawn byscripts/charts.pyfromspeakers_n,speakers_year,speakers_source(and optionalspeakers_basis,short_title) in each page; the map usesisoandbuddhists_nincountries.ymland is drawn in the browser byassets/buddhist-map.js(d3 and world-atlas from jsdelivr). Leave a number out rather than guess it. Every fact needs a numbered source; write “not verified” when you cannot check a fact. Re-check the pages about every six months, because AI support changes fast. - Experiment: folder
experiments/<yyyy-mm>-<slug>/fromtemplates/experiment/. WritePLAN.mdbefore running and never edit it after the start; record changes inLOG.md. Numbers inREPORT.mdlink to results at an exact commit or version. Update the table inexperiments/README.md. - This handbook: update it in the same PR as any change it describes.
Change the design
Repo: design.
- Edit
tokens.jsonorcss/*.css. python3 scripts/build.py, thenpython3 scripts/check.py(WCAG AA in light and dark).- Bump
versionintokens.json, add aCHANGELOG.mdline. Never rename a class without a major bump. - Copy
buddhistai.cssinto buddhist-benchmarkssite/and knowledge-baseassets/(and the leaderboard app, when it exists).
Add a contributor or a backer
- The homepage contributor row (
#people) is generated every day byscripts/update_contributors.pyin buddhistai.github.io. It adds commit authors of public BuddhistAI repos, public org members, authors of prompt library entries (github,authorsandavatarin each buddhist-prompts<topic>-<language>/<entry>/CARD.md; people with no GitHub name show their picture or initials), and the GitHub names incontributors-extra.json. To credit someone by hand, add them tocontributors-extra.json. Never edit the row between thecontributors:start/contributors:endmarkers. - Backers (“Supported by”) are edited by hand in both the org profile (BuddhistAI/.github
profile/README.md) and the homepage (#people), as text or logo cards. - Only add a backer after NT confirms it. No logos without permission. WeBuddhist stays one card among others.
- Compute and API credit donors can be listed with the backers if they wish (Conventions).
Add a partner organization
A group that contributes often (for example a translation team) gets its own GitHub team with write access to its own task repos only. The full steps (invite as Member, create <org>-team, give Write on their repos, optional Triage, CODEOWNERS lines, credit) are on the public Governance page. Never give a partner team write access to a core repo (buddhist-skills, buddhist-benchmarks, buddhistai.github.io, design, knowledge-base, .github). Before the first partner team gets access, set up the Maintainers team and the branch rules (Architecture).
Write provenance records
Repo: the skill’s own repo. Standard: Provenance records.
- Run with a script that saves the exact command and library versions (toc-extraction-tibetan
bench_skills.pywrites<out>.meta.json). - After
to_result_json.py, runpython scripts/provenance.py --benchmarks ../buddhist-benchmarks, thenpython scripts/provenance.py --check. - Commit
provenance/together with the run files and logs. Never change a run file after its record is made. - For a new skill repo, copy the workshop template and adapt
provenance.pyfrom toc-extraction-tibetan.
Handle an expert review
An expert review arrives through the Review a benchmark form (label expert-review).
- Thank the reviewer. Agree on the scope (all items, a sample, or one subset) and how they will mark problems (a table in the issue, a spreadsheet, or comments on a pull request).
- Share the items in a readable form. For a private benchmark, share only what its permission record allows.
- Turn each confirmed problem into a fix: wrong answers → patch version, missing hard cases → minor version (Versions). Record the review in the task
CHANGELOG.md. - Name the reviewer in the dataset card (
reviewers: name, version reviewed, date). A full review by a second reader from the tradition moves the benchmark to maturity level 3, Reviewed. Update the maturity shown on the homepage. - Re-run the best skill and the challengers on the new version. Move the card on the Benchmarks board.
Handle a compute or API credit donation
An offer arrives through the Donate compute or API credits form (label compute). Only NT accepts a donation.
- NT replies on the issue and agrees the details in private (never ask for a key in an issue).
- Store keys only as repo or org secrets (
OPENROUTER_API_KEYor a provider key), never in a file or chat. Note in Status which runs the donation pays for and any limits (amount, end date, models). - Use donated credits for evaluations of everyone’s submissions in the order they come, not only the donor’s.
- Thank the donor in News. If they wish and NT confirms, add them to “Supported by” (Add a contributor or a backer).
Post news
Page: knowledge-base news/README.md.
- Add a section at the top:
## <day> <Month> <year> · <title>, then two to five plain sentences with links (leaderboard row, skill, benchmark, person). - Post for: a new Top performance or Best value, a new benchmark or data version, a new best skill release, a new partner organization or backer, a big change to the sites.
- Credit people and organizations by name. Neutral wording: what changed, who made it, where to find it.
- Copy the post to Discussions (Announcements) and to X (@buddhist_ai, created 2026-11-17). LinkedIn: not set up yet.

BuddhistAI Knowledge base