Financial QA Contrast-Set Generation and Evaluation Toolkit

Generating meaning-preserving contrast sets to measure QA answer consistency over financial reports

What it does#

A research pipeline for generating and evaluating contrast sets for a financial-domain QA system built on Volkswagen Group annual reports (2015–2024): perturbing validated QA evidence spans, judging each perturbation for meaning preservation, and measuring whether a QA model under test answers consistently when its input is reworded without changing its meaning.

Perturbation & judging#

Evidence spans are perturbed with a local, vLLM-served Qwen model (claim-aware preserve/alter pairs) and a rule-based perturbator (case, insertion, repetition), with an optional Gemini-based perturbator behind a flag. Each source/perturbed pair is then scored for meaning preservation by a smaller LLM judge run with self-consistency sampling and an NLI ensemble (RoBERTa-large-MNLI + ModernBERT-NLI), fused via a calibrated decision threshold into a final preserved/altered label.

Consistency evaluation#

A QA model under test (e.g. Llama-3.3-70B-Instruct, via vLLM) answers both the original and the meaning-preserved perturbed input for every retained item. A separate, decoupled scoring stage then compares original-vs-perturbed answers for consistency, and each against the gold answer for accuracy, using token-F1, the same NLI+judge fusion, and evidence-citation overlap.

Calibration#

Threshold and evaluator choices are calibrated against a human-annotated sample — collected with a Streamlit-based annotation tool over exported Excel files — using inter-annotator agreement and grid-search experiments to pick the NLI models, judge, and fusion threshold that best track human judgment.