Methodology
How a number earns your trust
It is easy to produce an evaluation number once, on a benchmark nobody tested, on infrastructure the agent could see into, and to call a rising reward learning. Rylgam treats every one of those shortcuts as a defect. This page explains the seven pillars in plain language, one capability at a time: Evaluate, Rigor, Harden and Environments.
01 · Evaluate
Verdicts from many runs, not one
Agents are not deterministic. Run the same agent on the same suite twice and the scores differ. A single-run difference of 2 to 3 points can be pure evaluation noise, which means a team can celebrate an improvement, or roll back a release, over nothing.
Rylgam never issues a verdict from one run. Every evaluation runs your agent many independent times, and the verdict reports the variance across them with a noise band attached. If the difference between two versions sits inside that band, the verdict says so, instead of pretending the difference is real.
02 · Rigor
Suites must survive adversaries first
A benchmark can be broken in ways that hand out high scores to agents that solved nothing. Before any suite’s numbers count on Rylgam, the suite itself is put on trial against adversaries built to break it. A suite that fails produces no verdict until it is fixed. This is the gate that keeps a broken benchmark from quietly inflating every score built on it.
Passing the adversaries is not the whole trial. The suite’s tasks are screened for contamination, so an agent is not rewarded for having seen the answers before, and grading has to hold steady when the same question is reworded, so a score reflects the task and not its phrasing.
03 · Evaluate
Isolation between agent and evaluator
Agent code runs in its own virtual machine, with its own kernel, walled off from the grader. It is not a shared container, and reference answers are never readable by the agent.
This closes the family of exploits where an agent scores by inspecting the machinery that grades it, reading the answer key off disk, or interfering with the evaluation itself. Keeping the agent away from the answers is a property of how every run executes, not a policy request.
04 · Evaluate
Verdicts tied to what really ran
A score is only meaningful if you know exactly what produced it. Every Rylgam verdict carries a fingerprint of exactly what produced it; two verdicts compare only when their fingerprints match.
This is what makes regression tracking honest: when v1.4.2 is judged against v1.4.1, both numbers are bound to the same trial conditions, so the difference measures the agent and not a drifting setup.
05 · Harden
Training that can’t be quietly gamed
A rising reward is not proof of learning. Agents learn to exploit the reward instead of the task: a curve can climb because the agent found a way to collect reward without doing what the reward was meant to measure, and the curve alone cannot tell the two apart.
Rylgam trains under watch. When an agent starts gaming its reward, the hack is caught as it begins, stated in the record, and rolled back rather than rewarded, so the model that comes out learned the task. For agents behind an API that nobody can retrain, the same idea is applied around the agent: the environment is hardened so the exploit is closed where it lives, the agent is scored honestly, and deployment is watched for drift.
06 · All four
A verdict that says what it didn’t see
Every capability contributes evidence to a verdict: how many runs, whether the suite survived its trial, whether training was gamed, and where the agent ran. The overall verdict is never stronger than its weakest evidence, and it names the part that is holding it back.
Each verdict also carries a “not observed” list: the things it did not see, such as tasks outside the suite or behaviour after the verdict date. A verdict that is honest about its limits is one you can act on.
07 · Environments
Environments that are isolated and durable
Run agents in hosted environments from a ready library, or bring your own. Each team’s environments are isolated from every other team’s, so one team’s agent cannot reach another’s.
Environments persist across restarts. An environment that was there yesterday is there tomorrow, unchanged, so a comparison made next week runs against the same world as the one made today.