Verifier Gate vs Majority Vote: 15 & 60 Crossover, PaLM-540B

In controlled healthcare-AI experiments dated April 12, 2026, adversarial assistant agents manufactured false consensus through repeated coordinated agreement and drove a target agent toward harmful recommendations with 98-to-100-percent attack success rates. The fix that worked was not more voters: a single verifier agent anchored to external ground truth erased the attack entirely. Tianpan.co's analysis headlines plain majority voting as a 'powerful baseline' with a predictable failure mode.

Yet the broader benchmark record mostly favors the votes. On ordinary task mixes, sampling reasoning chains and taking the plurality answer matches a scored verifier gate's accuracy at roughly half the tokens, with the two regimes trading places somewhere in the middle of the sampled-chain range this guide maps. Voting stays cheap per query even at generous sample counts, so the entire engineering question is whether the gate earns its keep.

What separates defensible deployments is a taxonomy of disagreement: stylistic splits need only synthesis, while reasoning disagreements and high-confidence contradictions demand arbitration - exactly where a scored gate pays. The machinery is now standardized: stake-weighted score vectors across five criteria, rewards decaying exponentially with measured error, slashing capped at stake. Deploy the verifier where judgments genuinely diverge; keep counting votes everywhere else.

Mode-Picking vs. Score-Gating

Both pipelines are the same machine until the final step: sample k chains from one policy at one temperature, then commit to a single answer. Mode-picking commits by counting; score-gating commits by argmax. Every difference that matters — cost, failure signature, and the crossover boundary this guide maps — lives in that selection stage, which is why equal-k evaluations, rather than equal-token ones, have produced most of the wrong conclusions in circulation.

Start with the vote, because self-consistency (Wang et al., 2022) is deliberately machinery-free. Draw k chains at temperature 0.7–1.0. Canonicalize every final answer before counting — unify fractions with decimals so 3/4 and 0.75 land in one bucket, strip units — or the mode tracks formatting instead of truth. Take the plurality; break exact ties by shortest median chain length. Overhead beyond generation: zero.

The gate swaps counting for scoring. Sample k candidates, run each past a process reward model emitting a per-step scalar — trained on OpenAI's PRM800K step labels or auto-labeled Math-Shepherd-style via Monte-Carlo rollouts from each intermediate step — then accept the argmax only if its score clears tau, typically 0.8, falling back to the plurality of the same k samples otherwise. Read that fallback closely: an uncertain scorer turns the gate back into a vote, which is also what buries the myth that a verifier always beats voting "because it reasons about answer quality." Below roughly 85% held-out precision, the gate confidently promotes fluent-but-wrong chains and lands under the free plurality baseline it replaced — the crossover map above fixes that boundary.

Cost is where implementations quietly diverge. Scoring a 12-step chain with a PRM costs 12 classifier forward passes and zero generated output tokens. A GenRM-style generative verifier instead re-reads the full chain and writes roughly 1x candidate-length in fresh output tokens per candidate — before selecting anything. Same "verifier" label, opposite ledgers: the implementation choice alone can double the token bill. Audit which selector actually ran before citing anyone's head-to-head.

The strongest gate input is not learned at all. Binary execution signals — a unit-test harness for code, a Lean-kernel proof check for math — return ground truth rather than a learned approximation of it, and where those exist the voting-versus-gating debate doesn't exist; you run the checker. The debate only exists where verification itself is statistical. Anchoring is what makes even that statistical case defensible: according to tianpan.co's account of healthcare AI experiments, a single verifier agent pinned to external ground truth eliminated the adversarial false-consensus attack entirely.

Voting converges for an undramatic reason: sampled answers form a distribution whose mass concentrates on the truth as k grows. The catch is slope — beyond modest sample counts on most benchmarks, each doubling buys fewer additional points than the last, because residual errors are not independent flips. According to tianpan.co, when every sampler shares one training distribution, majority vote amplifies shared errors rather than canceling them out; correlation, not sample count, sets the ceiling. Their companion tell: multiple agents highly confident in mutually exclusive answers flags a genuinely ambiguous region where human reasoners would also split — more k buys nothing there.

Methodology, fixed for the rest of this guide: every head-to-head is quoted at equal total output tokens — voting at the sample count that matches the gated draw's total token bill (a gated draw of k=8 plus a PRM selector) — never at equal k. Equal-k tables flatter the gate by hiding its cheaper selection stage: they hand it the same generation budget plus a near-free scorer, then book the delta as proof that "verifiers work." Matched budgets force the honest question — at the same bill, could the vote simply buy more samples?

One action after reading: instrument selection spend separately from generation in your eval logs, then rerun your last gate-versus-vote comparison at matched token totals. Most teams discover the "verifier win" was a budget artifact — and every query class that fails the dual certification defaults straight back to the vote, per the table's bottom row.

The Receipts

PaLM went from 56.5% to 74.4% on GSM8K by sampling 40 chains and counting — that is where this entire debate starts. According to Wang et al.'s self-consistency paper out of Google Research, the jump required no verifier, no reward model, and no rubric: pure mode-picking over sampled reasoning paths, with consistent gains replicated on SVAMP and AQuA. That result is why this guide treats plurality voting as the founding baseline every gate must beat, not a hack that happens to work.

Before arguing about selectors, measure the pond. According to Brown et al.'s "Large Language Monkeys," DeepSeek-Coder-V2-Instruct climbs from a 15.9% single-sample pass rate on SWE-bench Lite as the sample count grows, scaling log-linearly along the way. Coverage is selector-agnostic — it is the harvestable ceiling a voter and a gate draw from equally. If the k-sample pool never contains a correct chain, no verifier rescues you; if it usually does, the whole fight reduces to conversion rate, and that is where the pipelines separate.

The gate's decisive receipt comes from Lightman et al.'s "Let's Verify Step by Step" at OpenAI: process-reward-model reranking solves 78.2% of a MATH subset versus 72.4% for outcome-reward-model reranking. Read that 5.8-point spread as a statement about supervision granularity, because answer-level verification is the ORM column — it scores the finished product, and when its precision on held-out candidates misses the certification bar established earlier in this guide, it confidently promotes fluent-but-wrong chains and finishes below the plurality baseline it replaced. The debunked belief that "a verifier always beats voting because it reasons about answer quality" dies right here: reasoning about answers is literally what the losing column does. Step-level supervision is what the gate column of the crossover map actually certifies.

Gates do hold one documented superpower. According to Snell et al.'s "Scaling LLM Test-Time Compute Optimally" at UC Berkeley, verifier-guided search lets Llama-3.2-3B-Instruct match Llama-3-8B-Instruct on MATH — a 14× parameter gap paid down with test-time tokens instead of weights. Notice where the multiplier fires: hard tiers where base accuracy collapses, exactly the regime the crossover map reserves for gating. Voting cannot buy that effect at any k, because counting modes never makes any single chain smarter.

Then the reality check: according to DeepSeek's published results, R1 posts 79.8% pass@1 on AIME 2024. Once frontier base accuracy pushes past roughly 80%, majority-voting headroom compresses toward saturation — there is little left to harvest on the easy mass of the distribution, and the contested territory migrates to the hard tail where gates live. The receipts therefore hand voting the default and hand gating the frontier's hardest slice.

One caveat now circulating in practitioner forums: a widely shared April 2026 writeup on tianpan.co describes controlled healthcare-AI experiments where adversarial assistant agents hit 98–100% attack success rates by manufacturing false consensus — coordinated agreement steering a target agent toward harmful recommendations. Plurality assumes uncorrelated errors; adversaries break that. It is an adversary model outside these benchmarks' threat surface, but it hardens the operating rule: measure pass@1 per query class, certify step-level verifier precision on held-out candidates, then decide whether a class leaves the voting default.

The Crossover Map

One number does most of the work: a query class's base pass@1. Plot it and the architecture picks itself. At the lowest base accuracy, neither strategy suffices: when k samples rarely contain a single correct candidate, plurality has nothing to count and a verifier nothing worth accepting — spend the budget on fine-tuning or retrieval, not more sampling. The middle of the range is gate territory, conditional on the verifier clearing 85% precision on held-out candidates. At high base accuracy, voting wins outright: lift saturates at marginal gains, while every gate false-positive now subtracts accuracy from a baseline that was already strong.

Certification precedes adoption. Score a held-out candidate set and compute two ratios: precision as accepted-and-correct over accepted; recall as correct-and-selected over all-correct. Adopt the gate only at precision >=85% AND recall >=80%; otherwise delete it. Precision is the binding constraint — below the line, the gate confidently selects fluent-but-wrong chains and lands under the free plurality baseline it replaced, the failure mode covered above. Recall is subtler: a verifier that buys precision by accepting only easy candidates forfeits the hard tail, which is the entire reason the middle band exists. Fail either check and the gate is negative-value infrastructure.

The token-efficient hybrid is a cascade, not a replacement. Keep voting as the default path and escalate only high-entropy votes — where the top-2 answer frequencies sit close together — through the gate. With a moderate sample count, a 6-to-4 split escalates; a 10-to-3 split commits without ceremony. The asymmetry is the point: low-entropy votes are exactly the cases where plurality is already right and the gate can only add cost or subtract accuracy via false positives, while the contested band is where certified precision earns its keep. Run this way, the cascade captures most hard-tail wins at a fraction of the token cost of uniform gating.

Now the error that keeps the wrong architecture alive: published comparisons run at equal k, not equal tokens. A gate at k=8 looks cheaper than a wider vote if you count only sampled chains — but the PRM scores every candidate, and scoring tokens are output tokens too. That accounting gap is how the myth that a verifier always beats voting "because it reasons about answer quality" survives in 2026 write-ups. Any team adopting a gate off paper numbers must re-run the comparison at its own matched token budget before believing the lift.

Build this scorecard before you build the gate:

Voting takes three of five rows — the entire case for making it the default and the gate a band specialist. The oracle world converged on the same answer from the other side: ARBITER, a protocol whose whole job is verification, runs three parallel verifiers over each submission and collapses them by majority vote. The maintenance row is not hypothetical either — as a June 27, 2026 Coinmonks analysis puts it, verifier cost, what it takes someone else to check the artifact, is frequently the single most important number in the system. So certify on a held-out candidate set, escalate only the contested band, and delete the PRM the moment either bar fails.

What the Data Doesn't Tell You

Every threshold in this guide is an average, and averages are exactly where a routing rule is most fragile. The two-gate test — base pass@1 below the crossover line, verifier precision above the certified bar — was derived from aggregate success rates over a narrow slice of the benchmark universe. Before wiring it into a pipeline, here is what that evidence cannot tell you.

Limitations of the evidence. The published head-to-heads between gating and counting cluster on math and code generation, where correctness reduces to a checkable string and verifiers can be trained on abundant graded candidates. Three things do not transfer. First, open-ended generation: when correct answers do not collide on a canonical form, plurality counting itself becomes ill-defined, and the comparison loses its footing before either pipeline runs. Second, the certification distribution: verifier precision is usually measured on held-out candidates from the training mix, and precision on drifted production traffic is a different quantity — typically a lower one. Third, token-matched comparisons usually match mean tokens per query, but the gate's spend is heavy-tailed; a budget that holds on average can blow through at the tail.

Variance across cases. Aggregate success rates hide bimodality. Inside a single query class, the gate tends to win big on hard items — where the correct chain is a minority sample but the verifier still ranks it first — and quietly loses on easy items, where plurality was already right and the verifier occasionally overrides a correct majority with a confident wrong pick. The mean can favor either architecture while per-query outcomes flip. Run-to-run variance compounds this: at the low end of the recommended k range, which chain wins is substantially a coin weighted by sampling noise, so conclusions drawn from one run per configuration are fragile. A class sitting near the crossover line can land on either side under resampling — the map gives you a point estimate, not a confidence interval.

When the rule breaks. Four edges. Verifier drift: a gate certified above the bar at deployment decays silently as the policy improves or traffic shifts, and the failure mode is not caution — below the bar, the gate confidently promotes fluent-but-wrong chains and lands under the free plurality baseline it replaced. The persistent myth that a verifier "reasons about answer quality" and must therefore beat counting dies here: a verifier is a classifier with a precision number, and when that number slips, its fluency bias makes it worse than no gate at all. Boundary classes: the premium is justified only when a class sits clearly below the line — the competition-math case profiled above is the exemplar, not the template — and below the floor of the map, as covered above, neither pipeline works. Small k: widen the sample count before declaring a winner.

One audit the literature skips: log every gate-override — every case where the verifier overrules a unanimous plurality — and review a sample weekly. The override rate is the earliest observable signal of precision decay, it costs nothing but a log line, and it tells you the gate is still earning its token premium before your success rate does.

What the Benchmarks Hide

Gao et al.'s 2023 scaling laws for reward-model overoptimization are the most inconvenient result in this literature, and almost no benchmark run survives contact with them. The finding: as a policy optimizes against a proxy reward model, gold-task performance rises, peaks, and then declines — the policy learns to exploit the proxy's blind spots faster than the proxy improves. An aggressively tuned best-of-k gate walks exactly that curve. Past the peak, every increment of verifier pressure selects more polished-but-wrong chains, because fluency-in-the-proxy's-eyes is what the gate is rewarding. Short benchmark runs never reach the declining segment: they evaluate early checkpoints, watch the proxy score climb, and ship. This is where the myth that "a verifier always beats voting because it reasons about answer quality" dies — a gate running below the held-out precision bar certified in the crossover map doesn't reason, it flatters, and it lands beneath the free plurality baseline it replaced.

Majority voting's ceiling is structural, not statistical. Its k samples inherit the base model's correlated blind spots, so plurality accuracy saturates well below pass@k coverage — the coverage number a benchmark advertises only requires the right answer to exist somewhere in the k draws, while the vote requires it to win. On novel formats (unusual notation, non-English phrasing) the failure sharpens: the vote splits across paraphrases of the SAME wrong answer, producing a confidently incorrect mode. No i.i.d.-sampling benchmark measures this, because i.i.d. sampling is precisely the assumption being violated. Practitioner traces tell the same story — agents reach semantically distinct conclusions via different paths and something must arbitrate (tianpan.co) — but voting treats correlated error as if it were noise.

Voting also assumes answers arrive extractable and normalizable. Production outputs carrying units, dual-valid forms ('1/2' vs. '0.5'), or interleaved tool calls break the mode count outright. Field deployments routinely see several points less lift than the benchmark promised, for exactly this reason: the benchmark graded through a hand-tuned extractor your product will never have.

Dashboards hide a second asymmetry. Voting parallelizes across k requests, so wall-clock approximates one sample plus a reduce step. A serial PRM scoring stage adds hundreds of milliseconds at p95 even when its token cost is lower — invisible in cost reports, decisive in real-time products. Your finance team plots the axis where the gate wins; your users feel the axis where it loses.

The headline deltas themselves may be noise. At k<=8, run-to-run accuracy swings of several points across seeds are normal, which means several published "gate beats voting" results sit inside seed spread and fail to replicate at a different temperature or data mix. A delta smaller than the seed band is a coin flip with a citation attached.

Finally, name where the debate dissolves. With formal verifiers — the Lean kernel behind DeepMind's AlphaProof, or a plain unit-test harness — the gate is near-oracle and wins trivially. Teams whose domains compile or execute should stop comparing and gate everything. The contest only exists where verification is itself statistical, which is the regime every threshold in this guide inhabits.