Beyond the Dom: Visual-First Web Accessibility Auditing Using Vision–Language Models and Semantically-Grounded Reports

Abstract

[Abstract] We introduce a semantically grounded reporting model for web accessibility audits, published as an RDF vocabulary with SHACL validation profile, JSON-LD context, and SPARQL competency queries, enabling interoperable, queryable, provenance-complete audit knowledge graphs. The vocabulary comprises eight core classes (AuditRun, Finding, Criterion, Target, Evidence, Recommendation, ModelInvocation, PageState) with 40+ properties, building on PROV-O, Web Annotation, Dublin Core, and SKOS standards. Twelve declarative SHACL shapes enforce structural integrity, semantic consistency and provenance completeness. The model supports dual anchoring of evidence (DOM selectors and visual bounding box coordinates) to bridge the code-based and perceptual assessment paradigms. We instantiated the framework with a vision–language model (LLaVA-1.5) to demonstrate visual-first detection for WCAG criteria 1.1.1, 1.4.3, 2.4.2 and 2.4.4 on 270 WebSight samples. For criteria 1.1.1 and 1.4.3, the pipeline achieves 84% human-verified precision (Cohen’s κ=0.74, N=50 gold-standard annotations). A real-world pilot on 42 of 50 public websites shows flag rates of 38.1% and 31.0% for the same criteria, constituting evidence of deployment feasibility. Comparative evaluation against axe-core 4.9 reveals that approximately 52% of VLM flags are candidate findings not detected by the specific axe-core 4.9 configuration, including violations unreachable by DOM attribute analysis. Semantic artifacts enable cross-run regression analysis, evidence filtering, and reproducibility in accessibility workflows, positioning accessibility auditing as infrastructure compatible with semantic web ecosystems.