Repository navigation
Replies: 2 comments
|
Status update, three weeks on. The vocabulary track (part A) has started moving through code rather than this thread. #4140 brings native AcroForm widgets into the layout stage as typed field items, #4144 stacks Working through that changed the framing of part A, and the change makes the ask smaller, so I want to state it plainly. This RFC asked for the trained label set to grow so detection could assert more PDF/UA structure types. The forms work showed a cheaper route for a large share of documents: when a PDF is tagged, read the structure tree and treat it as authoritative for heading levels, list and table structure, captions, alternate text, artifacts, and language, and run the models only for the untagged tail. That needs a structure-tree reader in docling-parse, which does not exist yet and which we plan to prototype, plus a handful of additive docling-core slots. The measurements in the table above still stand as the evidence that the untagged tail needs the model work: heading levels never populated on the PDF path, and captions, footnotes, formulas, and code downgrading to Two consequences for parts B and C. The remediation-corrections corpus becomes the training signal for exactly that fallback path, so the offer stands and gets sharper. And the tooling it targets lives in docling-eval, so I will move the dataset offer there as its own issue rather than leave it inside this thread. The tagged-structure proposal will follow as an issue beside #4143 once the reader prototype exists, so it has running code behind it the way the forms work does. |
|
As promised above: the tagged-structure proposal is now #4146, with the structure-tree reader it depends on open as a draft in docling-project/docling-parse#337, and the dataset offer moved to docling-project/docling-eval#221. |
Uh oh!
There was an error while loading. Please reload this page.
From: YRA Tech (document-accessibility tooling; Docling is the structural
backbone of our conversion pipeline). Prior upstream work: docling#3803
(supersample), docling-parse#299/#300/#301/#314, the MLX conversion kit for
docling#3943.
Summary. We convert documents into tagged, PDF/UA-conformant PDFs using
DoclingDocument as our canonical model. Docling's layout, table, and
reading-order components are the strongest open foundation we have measured
for this — and the accessibility use case needs a modestly larger semantic
vocabulary than the models currently detect. This RFC proposes (A) a
PDF/UA-aligned extension of the trained label set, (B) a consented,
openly-licensed dataset of human remediation corrections in
DoclingDocument-native form to help train it, and (C) an inter-rater
reliability protocol for that dataset that extends the redundant-annotation
method DocLayNet introduced.
A. The vocabulary gap, measured
PDF/UA (ISO 14289-1) requires a logical structure tree over a fixed
vocabulary of structure types.
DocItemLabeland DocTags express many ofthem, but several are either absent or not dependably populated by the
trained models — and a tagging pipeline can only assert what detection
supports. Our writer's honest fallback is instructive: captions,
footnotes, formulas, and code all currently downgrade to
Pin our tagtrees, because we cannot rely on their detection despite the labels existing
in the schema.
section_header+levelattr; DocTags already reservessection_header_level_0..5captionfootnotedocument_index(as a table-ish item)list_item+markerreferencecontent_layer: furnitureThe highest-value additions for accessibility, in order: heading levels,
Note linkage, THead/TBody/TFoot, Lbl/LBody boundaries,
TOC/TOCI, BlockQuote. Heading levels alone gate real conformance:
a flat H1 forest fails the intent of PDF/UA §7.4.
We benchmark ~40 engines over OmniDocBench/DocLayNet-derived structure
slices with element-level scoring (role accuracy per class, geometry,
reading order). Two findings worth sharing regardless of this RFC: docling
is the strongest structure scaffold we measure (element pass-rate 0.71/0.77
on our OmniDocBench/DocLayNet slices, n=50 each), and its reading order is
its weakest axis relative to specialized engines — we capture ~half of a
+6-point reading-order headroom by rank-consensus over IoU-aligned voters,
with granite-docling (fixed to run its VLM path on image inputs) as the best
open order specialist. Full methodology and per-class confusions available;
our ARD and structure-tree metrics are ports of docling-eval's, so results
are directly comparable.
Two observations suggesting the ground is prepared: the DocTags vocabulary
already reserves
section_header_level_0through_5— so for headinglevels the gap is training signal, not schema — and heading-hierarchy pain
is a recurring theme in the issue tracker (#3957's undocumented
Title→Section-header remap, #1023, #2121, #2335, #3555, #3612, discussion
#3676), while a search for "PDF/UA" across the org returns nothing: the
demand exists, and the accessibility framing that organizes it has not yet
been proposed.
B. The dataset: human remediation corrections, DoclingDocument-native
Our review tooling captures accessibility experts fixing model output:
per-element role corrections, reading-order repairs, table header scoping
(column/row), list wrap/unwrap, alt text — as an append-only edit-op log
over the DoclingDocument, with per-session consent flags. This is
complementary to DocLayNet-style annotation: it oversamples exactly the
hard tail where models fail (hard-example mining by construction), while
DocLayNet supplies the unbiased mass.
Proposed pilot, designed to be zero-friction on privacy:
target population) — no PII exposure, redistributable in full.
triple-remediated (see C), grown continuously by volunteer remediators.
CDLA-Permissive-1.0 that DocLayNet v1 ships under; same family,
simpler text.
docling-cvat-tools / docling-eval dataset formats so it drops into your
existing campaign tooling; we do the conversion work.
pilot; they follow only behind redaction tooling and an explicit
contributor license grant.
DocLayNet-v2 canonical taxonomy (per arXiv:2509.11720) as its baseline —
drop-in for your current training targets — with the PDF/UA types layered
as extensions. We note DocLayNet-v2 itself is described there as
proprietary; if source-document rights are part of why it could not be
released, the clear-federal sourcing and per-file license audit above are
designed to remove exactly that obstacle. Whether this corpus becomes a
DocLayNet v3, a "DocLayNet-A11y" companion track, or something else
entirely is your naming call — we are proposing the content and the
compatibility commitment, not claiming the lineage.
C. Inter-rater reliability protocol (extending the DocLayNet method)
DocLayNet established the key practice: of its 80,863 pages, 7,059 were
double- and 1,591 triple-annotated, with inter-annotator agreement computed
as pairwise mAP@0.5–0.95 on the triple-annotated subset (82–83 overall) and
model scores read against that human ceiling (baselines landed ~6–10%
behind it). We also note the DocLayNet label set was deliberately scoped to
visually recognizable, single-page-decidable classes — explicitly excluding
semantic and context-dependent types — and the agreement data vindicates
that caution: Title, the most semantic of the 11, scored the worst IAA
(60–72). The PDF/UA types this RFC cares about are context-dependent in
exactly the way that made them un-annotatable under free-standing
page-at-a-time labeling. Our claim is that the remediation setting changes
the conditions: remediators see the whole document, work in a task context
that forces a decision (a screen-reader traversal either works or it does
not), and their agreement is measured rather than assumed. We propose
evolving the DocLayNet method in three ways for this corpus:
Krippendorff's alpha for element roles and table-cell scoping (nominal,
≥3 raters, tolerant of partial overlap); rubric-based adjudication for
alt text (free text does not kappa). For reading order we propose
agreement over the successor relation rather than global rank
concordance: each remediator implicitly assigns every element a "next
element," and that successor assignment is itself a nominal label — so
the same Krippendorff's alpha applies, uniformly across dimensions.
This matches how assistive technology actually consumes order (a
linearized traversal: what matters is each element's neighbors, not its
absolute position), it penalizes a transposed-but-internally-coherent
block as the two broken transitions it really is rather than as N
displaced ranks, and — decisively for adjudication — it makes
agreement local: a 2-of-3 majority can assert most of a page's order
edge-by-edge even when no two remediators produced identical global
permutations. (Global statistics like Kendall's W remain computable as
a summary, but with 3 raters W is a noisy point estimate whose
significance leans entirely on the per-page element count; we would
report it aggregated across pages, not per page. The successor-edge
view is also the breakpoint distance of permutation analysis, so it
connects to established theory.)
ground truth. Majority (2/3) → assert, with the dissent retained as a
soft label. Three-way splits → expert adjudication or exclusion from
hard GT. The label-variation literature (LIDC-IDRI's retained
disagreements; Plank 2022) treats these dissents as training signal —
and DocTags-style models can consume them as calibrated targets.
triples cost. Our ensemble already emits per-element disagreement flags
that catch ~95% of role errors (measured on DocLayNet-class content at
62% precision), so we allocate triple remediation to the flagged tail
plus a small random calibration slice (for unbiased agreement
estimates, DocLayNet-style), and single-pass the machine-unanimous
remainder with spot checks. Per-annotator reliability (Dawid-Skene) is
estimated on the overlap. To our knowledge no document-understanding
dataset has used model-ensemble disagreement to target its redundancy
budget; we think it makes a volunteer-hours budget go ~3× further and
would welcome your scrutiny of the design.
What we're asking
DocItemLabel/DocTags along the table above — we will draft the concrete
schema PR if the direction is agreeable, and we bring the
DocItemLabel↔PDF/UA crosswalk from a production tagging writer.
for the layout, table, reading-order, and granite-docling models — and
your preferred formats/venue (docling-eval dataset? HF dataset under
the docling org? a cvat-tools campaign?).
DocLayNet's redundant-annotation effort.
We are happy to bring the pilot dataset sample to a TSC meeting or
maintainer call. Everything above (metrics, sims, per-class tables) is
reproducible and shareable.
All reactions