CFPB and synth collections — seed-only (no public gold intent)¶
Kind: reference. This page states how typevet may use CFPB narratives in public eval work. It also covers a private partner project’s synthetic collections intent data.
Parent: #51. Baseline: #53 research return. Public eval direction: eval partner data policy.
What the partner project uses them for¶
CFPB Consumer Complaint Database (US public record, debt-collection product filter) supplies unlabelled complaint narratives. The partner project reads a local CSV export. The loader does not attach collections intent labels.
Synthetic collections intent data is derived. The partner project builds it from CFPB seeds plus a rewriter and Jev filtering. It is not a fixed public corpus with stable gold labels unless a separate regen or publish spec accepts one.
typevet policy (v1)¶
typevet treats CFPB and synth collections as seed-only for public eval claims:
| Path | Allowed in typevet public artifacts | Allowed eval claim |
|---|---|---|
| CFPB CSV (local, operator-owned) | No checked-in CFPB bulk in this repo | No “CFPB eval” or gold-intent benchmark |
| Partner synth collections output | Only if a later issue accepts fixtures | No public gold intent without regen/publish spec |
| Optional contract fixtures from synth | Yes, when an eval issue adds them | Must follow fixture labeling below |
Adopt Banking77 and DIFrauD for shippable public eval direction per
53. Do not rank CFPB or synth beside them as labeled benchmarks.¶
Fixture labeling¶
When typevet adds optional contract fixtures built from synth (or from CFPB-shaped text), label them as synthetic fixtures per verified evidence and inferred claims. State in the fixture metadata or test docstring that the text is synthetic and that no gold collections intent is claimed from CFPB alone.
A green contract test on such a fixture verifies adapter wiring for that case. It does not verify model judgment quality or a public leaderboard score.
Claims to avoid¶
Do not write or imply:
- “CFPB eval”, “CFPB benchmark”, or “gold intent from CFPB”.
- A frozen synth corpus as a public dataset without an accepted publish spec.
- Agreement, ECE, or calibration claims on synth-derived labels before #50 and the Noul/Choice logprob path (#11, #22).
When this changes¶
A child issue under #51 may accept a regen spec (how to rebuild synth) or a publish spec (how to freeze and name a public fixture set). Until then, seed-only stands. This page does not document the partner synthesis pipeline; the partner project documents that workflow.