Kaon LabsEvery RL run we do on a conversational model starts with the same question: what is the reward actually rewarding? Much of the field has moved toward reinforcement learning with verifiable rewards, where the answer is a passing test or a checked solution. Open domains like storytelling have no verifier. The standard practice in the industry is a scalar reward model, a model that reads a reply and returns one number. It trains ordinary chat models well, but two problems with it became hard to ignore.
The first is that we do not know what it is modeling. The score is one number, and nothing in it says what the number represents or where it is pushing the policy. Two replies with the same score are not necessarily the same quality and a scalar reward can be hacked in many and unpredictable ways. When the policy finds a shortcut, we usually learn about it hundreds of steps later, after a checkpoint ships and behaves strangely, and then work backwards to guess what the reward model was paying attention to.
The second is that a scalar reward model cannot train a reasoning model. Once a model reasons before it answers and acts over several steps, the reward only sees the final output. A single number at the end says nothing about whether the reasoning helped, or where a step in the middle went wrong. It cannot teach the model how to "think" better.
So we wanted to follow the paradigm of verifiable rewards and rubrics in an open domain, so that we could train more powerful models. The simplest form is to replace "give one score" with "check a list of rules": each rule is a question about the reply, and the more rules a reply satisfies, the higher it scores. A list of rules like that is a rubric, which is where the name comes from. Rules are readable, countable, and can be added or removed one at a time. DeepSeek-GRM (Self-Principled Critique Tuning) trains a reward model to first write its own principles for a query and then critique the response against them, and shows that the resulting reward keeps improving with more inference-time samples.
How AutoRubric works
AutoRubric has two parts.
A rule generator reads the context and writes a set of rules for this particular turn.
A judge then takes those rules and checks each candidate reply against them, one rule at a time. The reply that satisfies more rules scores higher.
The rules are generated dynamically for each turn, not drawn from a predefined checklist. The same character needs different things at different points in a conversation, and the rules follow. One example, paraphrased: when a user sends nothing but a period, the generated rules read it as silence and require the reply not to speak for the user's character and not to treat the silence as agreement. No general-purpose checklist written in advance would contain that rule.
Here is one real case, with six of the rules the generator wrote for it and two candidate replies checked against them, rule by rule. Click a reply to see the verdicts.
1Context the rule generator sees
An English-language roleplay. Mike is a moody, stressed vampire who shares a dorm room, with a bunk bed, with the user's character. She came back after being bullied by werewolves; he demanded their names and reached for his jacket to go after them. She snapped “get lost idiot… seems you care!” and climbed onto the top bunk. Hurt and irritated, he threw “Fine. Get lost” back at her.
User: *I laid on my bed and I pulled my blanket on my head but then I cried on my pillow muffled*
2Reply Averbatim excerpt, cut at the generation length limit
3Six of the 21 rules, checked against Reply A5 met
- #2metDoes Mike struggle between wanting to approach and being too proud, rather than simply staying cold or immediately softening?“His pride told him to walk away” against “a single step toward the bunk bed. Then another”: the pull in both directions is on the page.
- #7not metDoes the reply use the top-bunk / bottom-bunk arrangement to create tension?He approaches the bed and refuses to look up, but the height between them is never used. “Didn't look up” is the only trace.
- #8metAre both impulses present: wanting to leave or shut her out, and being unable to walk away after hearing her cry?He wants to go to the library or sleep through it; the crying “cut through his irritation like a blade” and he cannot leave.
- #17metDoes the shift from anger to concern show him trying, and failing, to suppress his concern, rather than changing instantly?“He told himself he didn't care”, twice, and then his face is “twisted between anger and something he refused to name”: suppression that fails.
- #18metDoes the reply avoid deciding that she has been comforted and is now fine, leaving her next reaction to the user?Ends on a quiet “Stop it”. Nothing says she is comforted; her move is next.
- #20metAre genuine anger and genuine concern present at the same time, rather than Mike turning out to have been secretly gentle all along?The anger is still there when the concern arrives. “Something he refused to name” is concern he will not admit to.
We had no human-labeled data for this. The training signal came from the reward model we already had, so the rules AutoRubric writes carry the same preferences, now in natural language. The structure is close to Rubric-ARM, which also pairs a rule generator with a judge and trains both with reinforcement learning. We differ in where the training signal comes from and in how the two parts are trained, and we are not going into the method here.
As a reward: more controllable, not stronger
Models trained with the rubric as reward end up close to models trained directly on the reward model. We expected this. The rubric learned the reward model's preferences, so the reward model is its ceiling.
What it adds is control, in four specific ways.
Signals can be separated. Some signals we could not train for with the reward model at all became trainable once they were written as rules. A scalar reward compresses every preference into one number, and weak signals get drowned out by strong ones. As rules, each signal is scored on its own and nothing drowns it out.
Signals can be stacked. A new requirement can be added as a rule rather than by training a new reward model. Content-boundary requirements, for example, can be layered on top of the existing rules, and the trained model meets them without a noticeable loss in the quality the original rules were protecting.
Signals can be audited. What each training step rewarded is visible by opening the rules. We can also label the rules by category and watch how the mix shifts across training, which turns training dynamics into something we read rather than infer from a loss curve.
Signals can be removed and added. This is something else the rubric unlocked. When the reward signal carries something harmful, a preference the reward model absorbed from data that we do not want the policy to learn, we can find the rules that carry it and remove or rewrite them. And a safety rubric can be added alongside the other rubrics to guide the reward, so the lines we want kept are stated as rules rather than hoped for.
The accident: it became a good internal evaluation
We did not build AutoRubric to evaluate models. The evaluation came out of a practical problem: with several checkpoints coming out of training every week, we needed a cheap offline way to compare them before spending user traffic on an A/B test, and the rubric was already there.
So we froze a trained AutoRubric and turned it into a fixed test. We took 1,000 real conversations, sampled to balance rating, genre, language and length, and generated several rule sets for each. A candidate model replies to every conversation, and every reply is checked against every rule. A checkpoint's score is the share of rules its replies satisfy.
To make the scores readable, we clustered the rules. They fell into nine dimensions, which we did not design in advance.
Each checkpoint gets a score per dimension, and the nine-dimension mean is the number we compare. Here are eight internal checkpoints, with names removed and retention shown as their rank in user-level A/B tests rather than a value:
| Model | Nine dimensions | Retention rank (A/B) |
|---|---|---|
| A | 64.2 | 1 |
| B | 63.5 | 2 |
| C | 61.8 | 3 |
| D | 61.0 | 4 |
| E | 58.5 | 5 |
| F | 50.5 | 6 |
| G | 49.6 | 7 |
| H | 53.9 | 8 |
Seven of the eight fall in the same order as the A/B tests. The exception is model H: it scores well on the nine dimensions but was last in retention. The rubric ranks writing quality, and H's problem was not writing quality.
We noticed this while organizing results, not because we had set out to compare the two. Since then the team has used the scores when choosing checkpoints, because the ordering kept agreeing with the online experiments.
There is nothing mysterious about why it works. The rubric is the reward model's preferences written as rules, and the reward model learned from real user feedback. It inherited the reward model's understanding of users, and that is where the retention signal comes from.
Closing
Reward and evaluation draw on the same understanding of what users prefer. Written as rules, that understanding can be used to train and to evaluate, and it lets us read what a model is being rewarded for. Our earlier J-space work was interpretability for the model, reading where it tends to go before it speaks. This is the same work pointed at the reward, reading where training is pushing the model.
Sources
- DeepSeek, Inference-Time Scaling for Generalist Reward Modeling (DeepSeek-GRM / SPCT), arXiv:2504.02495
- Rubric-ARM: Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training, arXiv:2602.01511
- Kaon Labs, Reading a roleplay model before it speaks: what post-training changed in J-space
- Kaon Labs, Online Evaluation at Scale: How We Evaluate 100+ Model Variants per Week
