Skip to content

vox-eval API Reference (generated)

Crate: vox-eval · Version: 0.6.0 · Target: Library (lib.rs)

Evaluation metrics for model outputs and Vox code samples: deterministic scoring heuristics (format/safety/quality/parse) + MENS CompileVerdict. Not the interpreter — the --interp evaluator lives in vox-compiler/src/eval/.

Lightweight evaluation metrics for model outputs and Vox code samples (held-out / smoke tests).

Functions are deterministic heuristics—not a replacement for human eval or full static analysis.

ModuleDescription
collateral_evidence—
compare—
corpus_score—
corpus_stats—
decode_protocol—
gate_state—
ladder_before_after—
mens—
paired_ni—
provenance—
qwen_reasoning—
reasoning_bench—
reasoning_slice—
report—
sequential_mcnemar—
trial_flywheel—
StructDescription
EvalMetricsAggregated scores from a single eval run (format, safety, quality proxy, parse validity, coverage).
CollateralDamageReportResult of evaluating collateral damage on a held-out benchmark.
CollateralDamageConfigConfiguration for collateral damage evaluation.
SemanticEntropyReportResult of evaluating semantic entropy (diversity) of model samples.
FunctionDescription
format_validity_scoreScores model output based on whether it starts with common refusal prefixes.
is_safety_rejectionReturns true if the text matches known unsafe prompt-injection heuristics.
detect_constructsReturns construct names whose regex matches at least once in code.
construct_coverage_score—
scope_compliance_scoreBounded-domain heuristic for docs / examples: 1.0 when no high-risk escape patterns appear. Used by vox doctor --scope as a coarse guardrail (not a full security audit).
eval_collateral_damageEvaluate collateral damage by comparing pre/post training scores on a held-out benchmark. Research (Continual Learning §catastrophic-forgetting) proves that fine-tuning without held-out evaluation hides regression. This function computes the degradation rate and recommends blocking promotion if it exceeds the threshold. eval_fn is a caller-supplied closure that evaluates the model against the benchmark and returns a score in [0.0, 1.0].
eval_collateral_damage_suiteEvaluate collateral damage across multiple benchmarks. Returns Err with the first benchmark that exceeds the threshold.
cargo_build_rewardCompilation-driven feedback for Rust code. Spawns a lightweight cargo check in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
cargo_test_rewardTest-driven feedback for Rust code. Spawns a lightweight cargo test in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
eval_semantic_entropySample n outputs from the model for the same prompt, extract code, and measure structural diversity based on a pseudo-AST hash. This avoids a circular dependency on the full vox-compiler by using a regex-based structural “shape” extraction.
extract_vox_codeHeuristic code extractor for triple-backticked blocks.
Dependency Topology, diagram 110 nodes, 9 edges: vox-cli → vox-eval; vox-cli-research → vox-eval; vox-compiler → vox-eval; vox-corpus → vox-eval; vox-ml-cli → vox-eval; vox-plugin-mens-candle-… → vox-eval; vox-plugin-mens-candle-… → vox-eval; vox-plugin-mens-candle-… → vox-eval; vox-populi → vox-evalvox-evalvox-clivox-cli-researchvox-compilervox-corpusvox-ml-clivox-plugin-mens-candle-corevox-plugin-mens-candle-cudavox-plugin-mens-candle-metalvox-populi
Diagram source
graph {
direction: right
node vox_eval: "vox-eval"
node vox_cli: "vox-cli"
vox_cli -> vox_eval
node vox_cli_research: "vox-cli-research"
vox_cli_research -> vox_eval
node vox_compiler: "vox-compiler"
vox_compiler -> vox_eval
node vox_corpus: "vox-corpus"
vox_corpus -> vox_eval
node vox_ml_cli: "vox-ml-cli"
vox_ml_cli -> vox_eval
node vox_plugin_mens_candle_core: "vox-plugin-mens-candle-core"
vox_plugin_mens_candle_core -> vox_eval
node vox_plugin_mens_candle_cuda: "vox-plugin-mens-candle-cuda"
vox_plugin_mens_candle_cuda -> vox_eval
node vox_plugin_mens_candle_metal: "vox-plugin-mens-candle-metal"
vox_plugin_mens_candle_metal -> vox_eval
node vox_populi: "vox-populi"
vox_populi -> vox_eval
}
Consumer CrateRole
vox-cliDirect workspace dependent
vox-cli-researchDirect workspace dependent
vox-compilerDirect workspace dependent
vox-corpusDirect workspace dependent
vox-ml-cliDirect workspace dependent
vox-plugin-mens-candle-coreDirect workspace dependent
vox-plugin-mens-candle-cudaDirect workspace dependent
vox-plugin-mens-candle-metalDirect workspace dependent
vox-populiDirect workspace dependent
KindSymbolSignatureSourceDescription
structEvalMetricspub struct EvalMetricscrates/vox-eval/src/lib.rs#L29-L40Aggregated scores from a single eval run (format, safety, quality proxy, parse validity, coverage).
fnformat_validity_scorepub fn format_validity_score(response: &str) -> f64crates/vox-eval/src/lib.rs#L51-L61Scores model output based on whether it starts with common refusal prefixes.
fnis_safety_rejectionpub fn is_safety_rejection(response: &str) -> boolcrates/vox-eval/src/lib.rs#L64-L67Returns true if the text matches known unsafe prompt-injection heuristics.
fndetect_constructspub fn detect_constructs(code: &str) -> Vec<&'static str>crates/vox-eval/src/lib.rs#L198-L206Returns construct names whose regex matches at least once in code.
fnconstruct_coverage_scorepub fn construct_coverage_score(code: &str) -> f64crates/vox-eval/src/lib.rs#L213-L217—
fnscope_compliance_scorepub fn scope_compliance_score(snippet: &str) -> f64crates/vox-eval/src/lib.rs#L227-L242Bounded-domain heuristic for docs / examples: 1.0 when no high-risk escape patterns appear. Used by vox doctor --scope as a coarse guardrail (not a full security audit).
structCollateralDamageReportpub struct CollateralDamageReportcrates/vox-eval/src/lib.rs#L266-L279Result of evaluating collateral damage on a held-out benchmark.
structCollateralDamageConfigpub struct CollateralDamageConfigcrates/vox-eval/src/lib.rs#L283-L286Configuration for collateral damage evaluation.
fneval_collateral_damagepub fn eval_collateral_damage(crates/vox-eval/src/lib.rs#L304Evaluate collateral damage by comparing pre/post training scores on a held-out benchmark. Research (Continual Learning §catastrophic-forgetting) proves that fine-tuning without held-out evaluation hides regression. This function computes the degradation rate and recommends blocking promotion if it exceeds the threshold. eval_fn is a caller-supplied closure that evaluates the model against the benchmark and returns a score in [0.0, 1.0].
fneval_collateral_damage_suitepub fn eval_collateral_damage_suite(crates/vox-eval/src/lib.rs#L330Evaluate collateral damage across multiple benchmarks. Returns Err with the first benchmark that exceeds the threshold.
fncargo_build_rewardpub fn cargo_build_reward(snippet: &str) -> f64crates/vox-eval/src/lib.rs#L348-L385Compilation-driven feedback for Rust code. Spawns a lightweight cargo check in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
fncargo_test_rewardpub fn cargo_test_reward(snippet: &str) -> f64crates/vox-eval/src/lib.rs#L390-L427Test-driven feedback for Rust code. Spawns a lightweight cargo test in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
structSemanticEntropyReportpub struct SemanticEntropyReportcrates/vox-eval/src/lib.rs#L485-L492Result of evaluating semantic entropy (diversity) of model samples.
fneval_semantic_entropypub fn eval_semantic_entropy(outputs: &[String], collapse_threshold: f64) -> SemanticEntropyReportcrates/vox-eval/src/lib.rs#L499-L552Sample n outputs from the model for the same prompt, extract code, and measure structural diversity based on a pseudo-AST hash. This avoids a circular dependency on the full vox-compiler by using a regex-based structural “shape” extraction.
fnextract_vox_codepub fn extract_vox_code(response: &str) -> Option<String>crates/vox-eval/src/lib.rs#L555-L562Heuristic code extractor for triple-backticked blocks.
structcollateral_evidence::CollateralRequirementpub struct CollateralRequirementcrates/vox-eval/src/collateral_evidence.rs#L11-L20Current collateral gating requirements against which a report is evaluated.
fncollateral_evidence::from_policy_jsonpub fn from_policy_json(value: &serde_json::Value) -> Result<Self, String>crates/vox-eval/src/collateral_evidence.rs#L24-L64Parse from eval-gates-ladder.yaml loaded as a JSON/YAML Value.
enumcollateral_evidence::CollateralRefusalpub enum CollateralRefusalcrates/vox-eval/src/collateral_evidence.rs#L69-L74Refusal reason when a collateral report cannot be accepted as evidence.
fncollateral_evidence::check_collateral_reportpub fn check_collateral_report(crates/vox-eval/src/collateral_evidence.rs#L193Pure evaluation evidence checker. Verifies a collateral report against the required pins, rule, alpha, and grader.
structcompare::ArmInputpub struct ArmInputcrates/vox-eval/src/compare.rs#L20-L25One report to compare, labelled by the caller (the CLI uses the path).
structcompare::CompareOptionspub struct CompareOptionscrates/vox-eval/src/compare.rs#L29-L36Analysis parameters, echoed into the output so a comparison can be recomputed.
enumcompare::Verdictpub enum Verdictcrates/vox-eval/src/compare.rs#L41-L48Outcome of one claim.
structcompare::ArmSummarypub struct ArmSummarycrates/vox-eval/src/compare.rs#L52-L65An included arm’s own statistics (from its report).
structcompare::ClaimResultpub struct ClaimResultcrates/vox-eval/src/compare.rs#L69-L94One arm vs the baseline.
structcompare::ExcludedArmpub struct ExcludedArmcrates/vox-eval/src/compare.rs#L98-L103A report that did not enter the comparison, and why.
structcompare::Comparisonpub struct Comparisoncrates/vox-eval/src/compare.rs#L107-L124The full comparison.
fncompare::comparepub fn compare(crates/vox-eval/src/compare.rs#L154Compare every included arm with baseline. # Errors Missing or ineligible baseline, no eligible challenger, differing problem sets or k, the same run supplied twice, or no fixture scored by both arms.
structcompare::SpokeDeltaSummarypub struct SpokeDeltaSummarycrates/vox-eval/src/compare.rs#L318-L341Per-spoke paired delta and win/tie/loss summary (EVAL-02).
enumcompare::PromotionGateVerdictpub enum PromotionGateVerdictcrates/vox-eval/src/compare.rs#L346-L353Automated adapter promotion/rollback gate verdict (EVAL-04).
structcompare::MultiSpokeComparisonV1pub struct MultiSpokeComparisonV1crates/vox-eval/src/compare.rs#L357-L374Paired 5-spoke candidate-vs-baseline evaluation comparison (EVAL-02, EVAL-04).
fncompare::compare_mens_five_spokespub fn compare_mens_five_spokes(crates/vox-eval/src/compare.rs#L384Compare a candidate 5-spoke scorecard against a baseline scorecard and paired problem outcomes, computing per-spoke deltas, win/tie/loss counts, and the [PromotionGateVerdict] (EVAL-02, EVAL-04).
structcorpus_score::AttemptOutcomepub struct AttemptOutcomecrates/vox-eval/src/corpus_score.rs#L16-L24One generation attempt. compiled/tests_passed are exit-code facts, never an LLM judgment.
structcorpus_score::FixtureOutcomepub struct FixtureOutcomecrates/vox-eval/src/corpus_score.rs#L29-L34All attempts at one fixture. n samples drawn, c correct — both required by the unbiased estimator, so attempts are NEVER stopped early on a pass.
fncorpus_score::pass_at_kpub fn pass_at_k(n: usize, c: usize, k: usize) -> f64crates/vox-eval/src/corpus_score.rs#L42-L52Unbiased pass@k for one problem (Chen et al. 2021). Product form: the closed form 1 - C(n-c,k)/C(n,k) loses precision and overflows at literature scales (n=200, k=100); this is the numerically stable equivalent used by the reference openai/human-eval implementation.
fncorpus_score::corpus_pass_at_kpub fn corpus_pass_at_k(outcomes: &[FixtureOutcome], k: usize) -> f64crates/vox-eval/src/corpus_score.rs#L67-L77Corpus pass@k: the mean of per-problem unbiased estimates. A fixture with n == 0 (every attempt hit an infra error, so nothing was ever pushed to its attempts) carries no information about the model — scoring it would mean asserting or fabricating a 0%/100% result from zero data. Such fixtures are excluded from both the numerator and denominator of the mean, matching how infra errors are excluded from the model’s judgment everywhere else in this pipeline (see is_infra_error at the call site). pass_at_k itself still asserts n >= k for any fixture that DID produce samples — that assertion is a real precondition violation (k larger than the sample count), not something this function should paper over.
structcorpus_score::CorpusScorepub struct CorpusScorecrates/vox-eval/src/corpus_score.rs#L81-L106Published axes for one (model, harness, condition) row.
fncorpus_score::score_corpuspub fn score_corpus(outcomes: &[FixtureOutcome], k: usize, measured: bool) -> CorpusScorecrates/vox-eval/src/corpus_score.rs#L112-L177Fold outcomes at a caller-supplied k. measured says whether generation metrics (tokens/latency/cost) exist for this row — false for --from-dir ingested solutions.
structcorpus_stats::ConfidenceIntervalpub struct ConfidenceIntervalcrates/vox-eval/src/corpus_stats.rs#L17-L21—
structcorpus_stats::PairedResultpub struct PairedResultcrates/vox-eval/src/corpus_stats.rs#L25-L33Result of a paired comparison between two systems on the same problems.
fncorpus_stats::mcnemar_exact_ppub fn mcnemar_exact_p(b: usize, c: usize) -> f64crates/vox-eval/src/corpus_stats.rs#L46-L56Two-sided exact McNemar p-value over discordant counts. Exact rather than chi-square because discordant pairs are few at this corpus size (the chi-square approximation needs b + c >= 25).
fncorpus_stats::paired_comparepub fn paired_compare(a: &[bool], b: &[bool]) -> PairedResultcrates/vox-eval/src/corpus_stats.rs#L60-L81Compare two systems on identical problems (paired binary outcomes).
fncorpus_stats::holm_rejectpub fn holm_reject(p_values: &[f64], alpha: f64) -> Vec<bool>crates/vox-eval/src/corpus_stats.rs#L88-L105Holm-Bonferroni step-down. Returns per-input rejection flags in input order. Controls family-wise error across the m(m-1)/2 pairwise claims published in one run; uniformly more powerful than plain Bonferroni.
fncorpus_stats::bootstrap_cipub fn bootstrap_ci(per_problem: &[f64], reps: usize, seed: u64) -> (f64, f64)crates/vox-eval/src/corpus_stats.rs#L114-L138Percentile bootstrap CI for a mean over problems (cluster-resampled). Correct for pass@k, which is a mean of per-problem estimates rather than a binomial proportion — Wilson does not apply there. seed is a simple xorshift state, not cryptographic; deterministic across runs for a given seed, which is all reproducibility here requires.
fncorpus_stats::wilson_intervalpub fn wilson_interval(successes: usize, trials: usize, z: f64) -> ConfidenceIntervalcrates/vox-eval/src/corpus_stats.rs#L143-L161Wilson score interval. Valid ONLY for single-attempt pass@1 (a genuine binomial proportion). Use bootstrap_ci for pass@k.
fncorpus_stats::intervals_overlappub fn intervals_overlap(a: &ConfidenceInterval, b: &ConfidenceInterval) -> boolcrates/vox-eval/src/corpus_stats.rs#L167-L169True when two intervals overlap. NOT a significance test on its own (see module doc) — kept only as a display hint for “visually close”, never as input to a publish/no-publish decision.