Uncertainty-quantification — Topic

📑 arXiv 3d ago

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

Split conformal prediction applied to LLM-as-judge frameworks reveals reliability issues masked by aggregate metrics: 33-67% of documents show transitivity violations despite low average rates, and prediction set width serves as a per-instance reliability indicator with strong correlation to actual uncertainty. The approach provides theoretically-guaranteed coverage bounds for judge outputs.

Evaluation Reliability Uncertainty-quantification

📑 arXiv 3d ago

IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation

IUQ quantifies uncertainty in long-form LLM generation by combining inter-sample consistency and intra-sample faithfulness. Addresses semantic coherence with factual inaccuracy in free-form text where answer sets can't be constrained.

Evaluation Safety Uncertainty-quantification

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations ↗

IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation ↗

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation