Eval complementary manifest (JevBench + open text sets)¶
Kind: reference.
This page documents the durable inventory filed under
evals/complementary-manifest.yaml.
It implements the accepted spec on
#63 from the
#54 research ranks.
Parent epic: #51.
Primary bench — JevBench¶
JevBench stays the primary typed-decision evaluation path for typevet (law: #24, TypeLLM, Jev and judgevet). The published protocol uses 231 tasks (139 Choice, 74 Noul, 18 Score) without permutation averaging. Upstream lives in TypeLLM/TypeLLM.
Complementary sets below are stress and regression only. They do not replace JevBench task mix or Score coverage for v1.
Ranked complementary sets (machine source of truth)¶
The YAML file lists ranks 1–7 with license, Decision primitive (Noul / Choice),
split notes, sample_size: TBD, loader status, and blockers. Defaults from
accepted #54:
- Prefer permissive licenses; sample large corpora.
- Prioritize primitive gaps (especially Noul) over topic-matching JevBench.
- Require a pair
stateschema before BoolQ, SNLI, or PubMedQA loaders. - CLINC uses domain shards only (#66 accepted):
CLINC150 domain shard map. Separate loaders from
Banking77 (#58); do not
reuse Banking77 intent maps or finvet
FRAUD_INTENTS(#53).
| Rank | Dataset | License | Decision |
|---|---|---|---|
| 1 | google/boolq | CC BY-SA 3.0 | Noul |
| 2 | google/civil_comments | CC0 1.0 | Noul (toxicity threshold) |
| 3 | qiaojin/PubMedQA (pqa_labeled) | MIT | Choice |
| 4 | SemEval hyperpartisan | CC BY 4.0 | Noul |
| 5 | clinc/clinc_oos | CC BY 3.0 | Choice (10×15 domain shards; map) |
| 6 | go_emotions | Apache 2.0 | Choice (conversion + prune policy) |
| 7 | stanfordnlp/snli | CC BY-SA 4.0 | Choice 3-way |
Edit ranks in the YAML when research accepts a new revision; bump
manifest_version.
Out of this manifest (cross-linked)¶
| Scope | Issue | Where |
|---|---|---|
| finvet public sets (Banking77, DIFrauD, CFPB, ABCD, UCI SMS, …) | #53 | Eval partner data policy |
| PSAI computer-use (multimodal deferred) | #52 | YAML out_of_manifest.psai_computer_use |
| collections NBA partner data | #61 | Eval partner data policy |
| OSWorld-class GUI | — | Deferred with PSAI; not #54-ranked |
Non-goals¶
- Downloading or vendoring corpora from this repo.
- Implementing loaders (tracked on #58, #59, and research children from #54).
Open questions¶
Sample sizes, share-alike derivatives, and state JSON conventions remain
TBD in YAML open_questions until child issues close them.