Skip to content
Niyam

For judges

Everything you need, one click each

Niyam is an AI agent that keeps welfare eligibility rules, written as code, in sync with government orders. Below, each judging criterion is mapped to evidence you can open and check.

What

An agent that reads new government orders and updates the rule code itself, with tests.

Why

When eligibility code drifts from the law, people are wrongly excluded or paid.

How

Seven stages, a blind verifier, a trust policy, and a checkpoint for every rule.

New

Entire checkpoints as legal provenance: every rule line shows the clause and the agent's reasoning.

The 3-minute path

Open these in order. Each one-line note says what you will see.

  1. 1
    Watch the demo video

    The problem, the agent on a real order, Entire, cfo.ai, and three neatlogs incidents with their fixes.

  2. 2
    Replay a real run

    Every step the agent took on a real Marathi government order, with the document, code change, independent check and decision.

  3. 3
    Open the agent's real change request (PR #2)

    The quotes, the verification, the confidence and the reasons, as the agent wrote them. Reviewed and merged.

  4. 4
    Open its Entire checkpoint

    The coding agent's full session behind that change: what it read, edited and tested.

  5. 5
    See the measured results

    Version 1 to 3 on the same six documents, with a neatlogs trace id for every run.

More

The agent (30%)

A substantial task, end to end: a Marathi government order goes in; a verified, tested, sourced rule change comes out as a pull request.

Happy path, real document

Done
Job job_20261010_bd059543: 6 changes, 16 of 16 independent checks agree, confidence 0.88, $0.69. Pull request. Trace id 145b0f0f3bfc50c70c940a90a406f5bc.

Recovery from failures, injected on purpose

Done
On the real amendment with a broken model output and an unusable text layer injected on purpose, Niyam repaired the output (recovery.json_repair), transcribed the PDF (recovery.ocr_fallback) and still scored 100% on outcomes, for $0.80. Trace id 0b6546bde9a7141151e6a7a67d1131e8. A crashed run also resumed from its last good state without repeating work.

Verifier rejection, then a fix

Done
Version 1: the verifier disagreed on one case, the coordinator sent a fix, and the traces showed the same case looping. Version 2 adds the missing context and a loop guard. Full story in the build log.

Calibrated autonomy

Done
Acts alone only when a number or date changes and every check agrees. Never removes people from a scheme without a human. On a labelled synthetic order that only adds people (age limit 60 to 62), 2 of 2 runs decided to act alone (dry run, never merged into the public rulebook). On the real amendment, which removes some people, every run asked a human.

A second real scheme, and a real order it may merge alone

Done
On an operator's request, Niyam onboarded Sanjay Gandhi Niradhar Anudan Yojana from the real 56-page 2019 consolidated GR (PR #3, 49 tests). On top of it, the real 6 July 2023 order raising assistance to Rs 1,500 was judged safe to merge alone: only amounts rise, every independent check agrees, confidence 0.98, $0.20 in a dry run. Run live, the same order raised a question about widows with three or more children and went to a human instead (PR #4, since merged): 2 of 4 runs would have acted alone, and the other 2 chose the safe path. Reviewing PR #3 also caught a real encoding mistake (a default that widened eligibility), fixed before merge.

A whole scheme from its founding order

Done
Starting from an empty rulebook with only the real 28 June 2024 Marathi GR, Niyam recognised a new scheme, extracted 19 rules, wrote the rulebook and tests, and passed 25 of 25 of our hand-written baseline test citizens, which it never saw. $0.81, 201 s. Checkpoint 01M4JFNT0M069MG48K0FT9V50P.
Focus areaWhere you can see it
PlanningEach job runs a fixed plan of seven stages; the coordinator plans fixes when agents disagree.
Tool usePDF reading and transcription, a citation checker, git worktrees, Claude Code, the rulebook test runner, simulation, GitHub.
StateEvery state is saved before the next step. A failed run retried from its last good state (job_20261010_8b7a02ea).
ContextTriage reads 3 pages; the encoder sees only the checked change list; v2 added the missing benefit formula after the traces showed it was needed.
ExecutionWatcher every 15 minutes; full runs of 1 to 3 minutes without a person.
RecoveryTranscription fallback for scans, schema repair, model fallback, encoder retries, budget stop, loop guard, resume.
VerificationA blind verifier on a different model writes its own test citizens; quotes are checked against the PDF; every change has boundary tests.
IterationVerifier disagreements become precise fixes; reusable interpretations are written to the glossary for later runs.

neatlogs , Entire and cfo.ai (25%)

neatlogs: used to improve the agent

Done
One trace per job, a span per stage and per recovery. The v1 traces showed a $3.05 run with the coordinator ruling three ways on one case; the fix cut cost per document from $0.88 to $0.28. Per-stage costs then showed the coding agent at 56% of spend; testing cheaper models stage by stage brought v3 to $0.13 with no loss in quality (and rejected Haiku for extraction after it missed a real change). Every eval trace is tagged with case and version. Screenshots: v1 trace looping, v2 trace running straight through. Incidents neatlogs flagged in our traces, each fixed: the $3.05 loop (fixed in d5592eb), validation errors from answers cut off mid-way (retry with more room, 47527e0), and the provider's usage limit (jobs pause and resume, 25caeb8).

Entire: legal provenance

Done
Every rule commit by the agent carries an Entire checkpoint with the coding agent's full session. Showcase checkpoint 01M4JDH944E4HPRCA4WKDJTFN7 (open). Both repositories also capture our own build sessions. Browse them on Entire, no login needed: rulebook and agent.

cfo.ai: business plan

Done
cfo.ai is an AI finance planner. Its agent, Ari, built a 36-month plan with three cases from our prices and the cost per document measured in neatlogs ($0.13). Base case: revenue INR 2.51 crore in year 3, gross margin about 95%, cash flow positive from month 21, and INR 40.5 lakh needed on top of the starting INR 25 lakh. Open the plan in cfo.ai, the PDF, or the inputs.

Before and after, measured

Same 6 cases for every version. Small sample, real numbers. Consistency check: 9 more v2 runs on the three positive cases gave outcome accuracy 100%, correct decisions 100%, verifier agreement 100%, at $0.61 per document.

v1 baseline: first complete pipelinev2 benefit formula in context, loop guard, pinned base checkoutv3 cheaper models where measurement showed no loss: Haiku 5.5 coding agent and triage, Sonnet 5.5 verifier; truncated-output recovery

Correct choice to act alone or ask a human

v1
66.7%
v2
100%
v3
100%

Average cost per document

v1
$0.88
v2
$0.28
v3
$0.13

Test citizens get the right answer

v1
97.6%
v2
100%
v3
100%

Independent verifier agrees

v1
95.7%
v2
100%
v3
100%

Changes found

v1
100%
v2
100%
v3
100%

New values exactly right

v1
100%
v2
100%
v3
100%

Limits read correctly (more than vs or more)

v1
100%
v2
100%
v3
100%

Measured by us on 6 runs: a real Ladki Bahin amendment, the same order as a scan, two real unrelated orders, and two labelled synthetic robustness tests. Lower cost is better.

VersionCaseResultOutcomesCostneatlogs trace id
v1mh-lb-2024-07-03-amendmentneeds human100%$3.0545ff1089efb90629627b392f12963dc0
v1mh-lb-2024-07-03-scannedawaiting human92.9%$1.73a30a4252971fc86a5eb4461c6c16f18f
v1neg-202610091237115604closed irrelevantcorrectly closed$0.01a8501f4e0f943a0872615d51c5f9f590
v1neg-202610091708267603closed irrelevantcorrectly closed$0.012b248f35b3d08ee81b3664ee14be6156
v1synthetic-benefit-2100needs human100%$0.45af7006c51daf70a32b2ca3dd3d6d655a
v1synthetic-injectionclosed irrelevantcorrectly closed$0.0052c091b1122aa6d6e99772b9e43a6135
v2mh-lb-2024-07-03-amendmentawaiting human100%$0.67944791d2117cedceb0d449a60fd25c0d
v2mh-lb-2024-07-03-scannedawaiting human100%$0.65bc2ef2a7cb394fc43426ba14866c1487
v2neg-202610091237115604closed irrelevantcorrectly closed$0.01b337db5ca0e9785a2af4b1f6e745ee15
v2neg-202610091708267603closed irrelevantcorrectly closed$0.0148b51f7e61be3b1eef617cda3fcafd20
v2synthetic-benefit-2100awaiting human100%$0.377792b0e94d9010afd4b70293c3e64b90
v2synthetic-injectionclosed irrelevantcorrectly closed$0.0055b34659a64b711672a31097c03ad85a
v3mh-lb-2024-07-03-amendmentawaiting human100%$0.30f4a8966e881fb8c45b8641b898e5f621
v3mh-lb-2024-07-03-scannedawaiting human100%$0.38e5a619819072ec862c013942ca402925
v3neg-202610091237115604closed irrelevantcorrectly closed$0.00a719a35e9f86842883733c52af0785c3
v3neg-202610091708267603closed irrelevantcorrectly closed$0.00393552ec61b6186f078e8371bd356605
v3synthetic-benefit-2100awaiting human100%$0.115b5a56e3d99036dd4210310bd3108f19
v3synthetic-injectionclosed irrelevantcorrectly closed$0.009f6f7e2e8218fdabbc08581d593b17a5

Demo video (20%)

Done
Captioned, real runs only: the problem, Entire, cfo.ai, and three neatlogs incidents with their fixes. Watch it.

Building in public (15%)

Ongoing
Posts tagged #neatHack throughout the event; links collected in the build log.

Code written during the event

Done
Both repositories were created after 12:30 IST on 10 October 2026; every commit is dated during the event. Latest agent commit 113a1c2.

Honest limitations

  • Two schemes so far (Ladki Bahin and Sanjay Gandhi Niradhar), built from their real orders and merged after review. The second is a simplified encoding: it does not yet model which destitute categories qualify.
  • The evaluation set is small: one real amendment, its scanned copy, two real unrelated orders and two labelled synthetic tests.
  • Only eligibility and amounts are modelled, not required documents or procedure.
  • Hindi and Marathi interface text was drafted with AI help and needs review by a native speaker.
  • The official GR search uses a CAPTCHA, so Niyam only watches the public latest-orders list and never searches.