Pratinav Seth

← Publications / Safety Post-Training & Alignment

Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems

Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth

EMNLP 2026 NLLP Workshop; NeurIPS 2026 Workshop2026My role: Senior authorWorkshop
PDFarXiv
Rule perturbations (delete, swap, negate) fed to five LLMs, measured behaviorally with OCS and representationally with ICS-Δ

Abstract

Large language model compliance systems are deployed on the assumption that a verdict depends on the regulatory rule it is given. We test this directly across five models and 20 regulatory and platform-policy domains: delete, swap, or negate the governing rule while holding the case fixed, and check whether the verdict changes (OCS) or the model's internal representation of compliance shifts at all (ICS-delta). Neither moves much: models' verdicts are often invariant to substantial perturbations of the supplied rule, and the guard model, evaluated here under a custom-rule adaptation of its native taxonomy, is the least rule-sensitive and least accurate of the five, barely above chance (51%, versus 90-92% for general-purpose models). This reflects easy cases more than blanket neglect: on cases where deleting the rule changes a previously correct model prediction, models do track it closely. Neither better prompting nor direct intervention on the model's internal representations closes this gap. Accuracy alone does not establish that a compliance verdict is grounded in the supplied rule.

Also on OpenReview.

Cite

@inproceedings{sadhu2026verdict,
  title         = {{Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems}},
  author        = {Saisab Sadhu and Aadit Sengupta and Vinay Kumar Sankarapu and Pratinav Seth},
  booktitle     = {EMNLP 2026 NLLP Workshop; NeurIPS 2026 Workshop},
  year          = {2026},
  eprint        = {2610.12313},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2610.12313}
}
All →