What Is RLCD? The Secret Behind Jev

From pairwise reward modeling to calibrated, multiway decisions

Jev looks mysterious when viewed as an alternative to a language model. It becomes much simpler when viewed as the next step in reward modeling.

The core idea is:

\text{RLCD} = \text{multiway preference modeling} + \text{probability calibration}

More specifically, RLCD is a schema-conditioned Plackett–Luce objective. Jev turns that objective into a product by adding typed outputs and parallel inference.

That is the secret: the reward model is no longer hidden behind a generator. The reward model becomes the model.

Four-stage diagram showing scalar reward becoming pairwise preference, multiway choice, and finally a calibrated decision served through the Jev API. — Figure 1. The learned object changes at each step: a scalar reward becomes a preference, the preference becomes a multiway distribution, and calibration turns that distribution into a decision interface.

Reward Modeling Started with a Scalar

A conventional reward model receives a context $$x$$ and a candidate answer $$a$$ , then produces a scalar:

r_\theta(x,a)\in\mathbb{R}

Outcome reward models score the final answer. Process reward models score individual reasoning steps. In both cases, the learned object is an absolute-looking number.

The problem is that this number is not actually absolute.

A reward of $$0.8$$ does not have a stable meaning across problems, candidate pools, checkpoints, or model families. It is mainly useful for comparing candidates generated under similar conditions:

r_\theta(x,a_1) > r_\theta(x,a_2)

The operational signal was always relative preference. The scalar merely hid it.

PPRM Made the Preference Explicit

LLaMA-Berry’s Pairwise Preference Reward Model, or PPRM, exposes the comparison directly.

Given a problem $$x$$ and two solutions $$a_1$$ and $$a_2$$ , PPRM answers:

Is the first answer better than the second answer?

Its probability has the form:

P(a_1 \succ a_2\mid x) = \frac{\exp u_\theta(x,a_1)} {\exp u_\theta(x,a_1)+\exp u_\theta(x,a_2)}

Equivalently:

P(a_1 \succ a_2\mid x) = \sigma\left( u_\theta(x,a_1)-u_\theta(x,a_2) \right)

This is the Bradley–Terry model.

LLaMA-Berry implements the comparison as a constrained language-model decision over Yes and No tokens. It trains the evaluator on almost 7.8 million mathematical-solution pairs and uses DPO to improve the pairwise prediction task. The essential change is conceptual: reward modeling becomes preference-probability modeling. See the LLaMA-Berry paper.

PPRM still contains a latent scalar utility $u_\theta(x,a)$ , but that utility is no longer presented as an absolute reward. It becomes meaningful through a normalized comparison.

LLaMA-Berry subsequently uses Enhanced Borda Count to aggregate pairwise comparisons inside MCTS. That is downstream search machinery. EBC neither defines PPRM’s preference loss nor provides the bridge from PPRM to RLCD.

The relevant lineage is simply:

\text{scalar reward} \rightarrow \text{pairwise preference} \rightarrow \text{multiway preference} \rightarrow \text{calibrated decision}

Plackett–Luce Is the Multiway PPRM

PPRM compares two candidates. A real decision interface usually receives more than two.

Let the candidate set be:

A=\{a_1,a_2,\dots,a_K\}

Assign each candidate a context-dependent utility:

u_i=u_\theta(x,a_i)

Then normalize all candidates together:

P(a_i\mid x,A) = \frac{\exp u_i} {\sum_{j=1}^{K}\exp u_j}

This is the Luce choice model, also known as multinomial logit. It is the top-one form of the Plackett–Luce family.

When $$K=2$$ , it reduces exactly to Bradley–Terry:

P(a_1\mid x,\{a_1,a_2\}) = \frac{\exp u_1}{\exp u_1+\exp u_2}

PPRM is therefore the binary case of the same choice geometry.

If the supervision contains a complete ranking

a_{\pi_1}\succ a_{\pi_2}\succ\dots\succ a_{\pi_K},

the full Plackett–Luce likelihood repeatedly selects the next-best remaining candidate:

P(\pi\mid x) = \prod_{t=1}^{K} \frac{\exp u_{\pi_t}} {\sum_{j=t}^{K}\exp u_{\pi_j}}

The corresponding loss is:

\mathcal{L}_{\mathrm{PL}} = -\sum_{t=1}^{K} \log \frac{\exp u_{\pi_t}} {\sum_{j=t}^{K}\exp u_{\pi_j}}

When the label specifies only one correct choice $$y$$ , the loss becomes:

\mathcal{L}_{\mathrm{choice}} = -\log \frac{\exp u_y} {\sum_j\exp u_j}

That is the first stage of the Plackett–Luce likelihood: a multiway extension of PPRM.

This is the mathematical center of RLCD.

Side-by-side diagram of Bradley–Terry pairwise preference and Luce multiway choice sharing the same latent-utility normalization. — Figure 2. Bradley–Terry and PPRM are the two-candidate case of the same Luce choice geometry. Plackett–Luce extends that normalization from one choice to a complete or partial ranking.

RLCD Adds Calibration

Plackett–Luce gives us a probability distribution, but normalization is not calibration.

A softmax vector always sums to one. That does not mean a prediction reported as $$0.8$$ is correct 80% of the time.

Calibration adds that empirical meaning:

P(Y=\hat{Y}\mid \hat{P}=p)\approx p

Across predictions assigned probability $$0.8$$ , approximately 80% should be correct. This is also the contract TypeSafe gives for RLCD: Jev returns decisions and probabilities, and higher reported probabilities should correspond to higher observed accuracy. See TypeSafe’s RLCD primer.

A minimal implementation uses a proper scoring rule such as log loss:

\mathcal{L}_{\mathrm{NLL}}=-\log p_y

or the Brier score:

\mathcal{L}_{\mathrm{Brier}} = \sum_{i=1}^{K} \left(p_i-\mathbb{1}[i=y]\right)^2

A held-out calibration stage can then adjust the sharpness of the distribution:

p_i = \frac{\exp(u_i/T)} {\sum_j\exp(u_j/T)}

Here $$T$$ controls how concentrated the probabilities are without changing their ordering.

This separates two objectives that ordinary reward modeling often conflates:

Ranking asks whether the best candidate appears first.
Calibration asks whether the model knows how often that decision is right.

Automation needs both. Ranking selects an action; calibration determines whether software should execute it, defer it, or escalate it.

The useful abstraction is:

\text{RLCD} = \text{Plackett–Luce preference loss} + \text{calibration constraint}

Conceptual reliability diagram followed by a decision policy that gathers context, escalates, or executes according to calibrated confidence. — Figure 3. Calibration attaches empirical meaning to confidence, allowing application-specific policies to decide when to gather context, escalate, or execute. The reliability curve is conceptual, not a Jev benchmark.

Jev Turns the Reward Model into the Product

In the conventional RLHF stack, the reward model is an internal component:

\text{prompt} \rightarrow \text{generator} \rightarrow \text{candidate response} \rightarrow \text{reward model}

Users interact with the generator. The reward model only trains or evaluates it.

Jev reverses that architecture:

\text{state} + \text{candidate schema} \rightarrow \text{calibrated decision distribution}

There is no need to generate an explanation and parse it back into an action. The evaluator itself becomes the runtime interface.

Jev exposes three primitives:

Jev primitive	Preference-model interpretation
`Noul`	Binary Bradley–Terry decision between true and false
`Choice`	Luce distribution over $$K$$ unordered alternatives
`Score`	Distribution over an ordered set of levels

A Choice returns the selected option, the complete probability distribution, and a confidence value. A Score returns a position along user-defined levels together with the distribution across those levels. A Noul returns the probability that a proposition is true. See Jev’s primitive documentation.

These are not three unrelated capabilities. They are three schemas over the same underlying object:

P(\text{typed outcome}\mid \text{state},\text{question},\text{candidate set})

Jev is therefore a reward model generalized from “Which answer is better?” to “Which typed outcome should the program select?”

Architecture comparison showing a conventional RLHF reward model behind a text generator and Jev serving the evaluator directly as typed Noul, Choice, and Score outputs. — Figure 4. Conventional stacks use the reward model behind the generator. Jev serves the evaluator itself: state and schema in, typed probability distributions out.

Why Jev Can Run in Parallel

Autoregressive language models represent an answer as a token sequence:

P(y\mid x) = \prod_{t=1}^{T} P(y_t\mid x,y_{<t})

Every token depends on the previous tokens. Latency grows with output length.

A decision model already knows its output space. It only needs to estimate utilities and normalize them:

x,A \rightarrow (u_1,\dots,u_K) \rightarrow (p_1,\dots,p_K)

No sentence has to be decoded.

TypeSafe says Jev evaluates multiple questions sharing the same state independently and returns them in one request. Its launch announcement describes a new architecture, a parallel sampler, and RLCD as the three parts of the stack. For very high-cardinality choices, Jev uses a two-stage process: independent scoring followed by an explicit choice. See TypeSafe’s Jev announcement.

This gives us a second decomposition:

\text{Jev} = \text{RLCD} + \text{typed schemas} + \text{parallel serving}

RLCD explains what the model learns. The schema and sampler explain how that learned decision function becomes a fast software primitive.

RLCD Is Not a Third Kind of Reward Source

TypeSafe presents RLHF, RLVR, and RLCD as three post-training paths. They are not three mutually exclusive mathematical categories.

RLHF and RLVR primarily describe where the reward comes from:

RLHF: human preference.
RLVR: programmatically verifiable outcomes.

RLCD describes what the model is trained to return:

a constrained decision;
a probability distribution;
calibrated uncertainty.

Human comparisons can train RLCD. Verifiable outcomes can train RLCD. Synthetic judges can train RLCD. Logged production outcomes can train RLCD.

The word reinforcement learning describes the broader post-training pipeline. The statistical heart of the objective is preference estimation under a proper probabilistic loss. PPO is not required to obtain this structure.

The cleaner taxonomy is:

Method	Primary training signal	Product output
RLHF	Human preference	Generated response
RLVR	Verifiable reward	Generated reasoning or answer
RLCD	Decision outcome and calibration	Typed probability distribution

RLCD is defined by the output contract, not by a unique source of reward.

The Thesis Produces Testable Predictions

If Jev is a calibrated, schema-conditioned Plackett–Luce model, its behavior should expose several measurable properties.

1. Binary equivalence

A two-option Choice and an equivalent Noul question should produce closely aligned probabilities:

P(A\mid\{A,B\}) \approx P(A\succ B)

2. Pairwise–multiway consistency

For two candidates inside a larger set:

\frac{P(a_i\mid A)}{P(a_j\mid A)} \approx \exp(u_i-u_j)

Their relative odds should match a direct pairwise comparison when the context and wording are held constant.

3. Candidate-set sensitivity

Vanilla Plackett–Luce satisfies independence of irrelevant alternatives. Adding an unrelated candidate should preserve the odds between existing candidates:

\frac{P(a_i\mid A)}{P(a_j\mid A)} = \frac{P(a_i\mid A\cup\{a_k\})} {P(a_j\mid A\cup\{a_k\})}

Violations measure how strongly Jev’s utility encoder jointly represents the candidate set.

4. Empirical calibration

Predictions can be placed into probability bins. For the $$0.8$$ bin, observed accuracy should approach $$0.8$$ . This test distinguishes meaningful uncertainty from decorative softmax confidence.

5. Order symmetry

Permuting the order of candidate definitions should permute the returned probabilities without changing their values. Any systematic position effect reveals schema-order bias.

These tests turn the RLCD interpretation into a falsifiable model of Jev’s behavior.

Conclusion

Jev is not fundamentally a language model that learned to emit cleaner JSON. It is a preference model promoted into a software interface.

PPRM provides the first step:

\text{absolute reward} \rightarrow \text{pairwise preference probability}

Plackett–Luce provides the multiway extension:

\text{pairwise preference} \rightarrow \text{distribution over candidate actions}

Calibration makes that distribution operational:

\text{choice probability} \rightarrow \text{automation threshold}

Jev packages the result as typed, parallel inference. It is a calibrated multiway reward model served as an API.

The deepest shift is not from one reinforcement-learning algorithm to another. It is from generating an unconstrained answer to estimating a calibrated distribution over actions already defined by software.

Jev is what happens when the reward model stops grading the product and becomes the product.