PDF internals research

Reading a PDF Object Graph Without Trusting the Cross-Reference Table

A PDF's cross-reference table maps object numbers to byte offsets, and a well-formed reader follows it. Our analyzer does not. It scans the raw bytes for the object pattern instead, and that choice is deliberate rather than lazy. The reason is that an analyzer built to explain file size must report what is actually in the bytes, including objects the cross-reference table does not admit to.

By BeanNest Studio Reviewed September 20, 2026

What this guide helps you decide

A PDF has a cross-reference table that is supposed to make object lookup fast. Why would an analyzer ignore it, and what does it find instead?

This research explains the byte-level object-location approach used by this repository's analyzer and the reasoning behind it. It is not a general PDF specification guide and does not claim to handle every malformed file.

Try it on the anchor product: BeanNest Tools /pdf/.

What the object scan does

For every corpus file the analyzer reports how many object markers it found and where each one starts. On the padded file it reports 45 objects, and on the three unpadded files it reports 5. Those counts match exactly how the corpus was constructed, which is the strongest available check that the scan is locating real structure rather than coincidence.

Why not follow the documented path

If an analyzer followed the cross-reference table, it would report exactly the objects the table lists. A file with a truncated, stale, or repaired table would then appear smaller and cleaner than it is. Since the guides discuss size, reporting a structure that the file actually contains matters more than reporting the structure the index claims.

What the scan reports

For each corpus file the analyzer emits an object count, a stream count, and a per-object breakdown including the object number. The padded file reports 45 objects and the others report 5, which matches how the corpus was built and gives a direct check that the scan found real structure.

Where this approach is weaker

Scanning bytes cannot distinguish an object marker inside a compressed stream from a real object, because the compressed bytes are not parsed at scan time. A file containing the literal characters of an object header inside a stream could overcount. Our corpus does not do that, so the padding test is a clean check, but the limitation is real and is stated here rather than hidden.

Why the limitation is acceptable here

The analyzer exists to support specific measured claims about small controlled files, not to be a general PDF parser. For those files the scan is exact, and the object counts match the generator's construction. Publishing the limitation alongside the result is what keeps the measurement usable.

What a reader should take from this

Two readers can disagree about a PDF's contents while both being correct, if one trusts the index and the other reads the bytes. When a file's size is the question, the bytes are the better witness. That is the entire justification for this design choice.

What the scan does not need

The scan needs no cross-reference table, no trailer, no startxref value, and no linearisation hint. It needs only the raw bytes and the object marker pattern. That independence is what makes it useful for size analysis: the object list it produces is a property of the file's contents rather than of the file's index. Two files with identical content but differently written indexes would produce the same object list, which is the behaviour a measurement tool should have.

Why the padded corpus is the correctness check

The generator writes exactly 5 objects for the three unpadded files and 45 for the padded one. The analyzer, reading only bytes, reports the same counts. That agreement is the check that the scan works on this corpus, because the expected answer was fixed before the analyzer ran. Without a constructed file like this, a scan could appear correct while silently missing or overcounting objects, and the guides would be built on a number nobody had verified.

The stream-marker overcount risk in detail

A compressed stream's bytes are arbitrary and could theoretically contain the characters that make up an object header. A naive byte scan would count that as a real object. The corpus avoids the case because its streams are small and their compressed bytes do not happen to contain the pattern, but a production analyzer must handle it. Compressed bytes are not parsed during the scan, so the analyzer cannot currently distinguish the two. Stating this bound is what keeps the object counts trustworthy for the files they describe.

How a stricter implementation would work

A stricter analyzer would read the cross-reference table, follow the offsets it names, and treat any object found outside the table as an anomaly worth reporting separately. That combines the index's precision with the scan's honesty about what is present, at the cost of failing on a damaged index. For the corpus used here the two approaches would agree, and the simpler scan is sufficient. The guides record the choice and its reason rather than presenting it as the only correct design.

What an object actually contains

Each object in the scan carries its number, its generation, its start offset, and the span up to the matching endobj marker. Those fields are enough to build a map of the file: how many objects exist, how large each is, and whether any are unexpectedly large. For a size investigation that map is the primary evidence, because a single oversized object can explain a surprising file size without any change in visible content.

Why generation numbers are reported but not used

The scan records the generation number because it appears in the object header, but the corpus uses generation zero throughout, as most modern files do. Generation numbers rise when an object is deleted and its number reused, which is a behaviour of incremental update. Reporting the field keeps the output faithful to the bytes while the guides avoid claiming anything about a mechanism the corpus does not exercise.

How the scan handles incremental updates

A file saved incrementally contains a new body for a changed object plus an updated cross-reference section pointing at the newer copy. The older body usually remains in the file. A byte scan finds both copies and would report a higher object count than the current document logically contains. The corpus has no incremental updates, so the guides do not make a claim about that case, but a production analyzer would need to resolve duplicates using the index it currently ignores.

The honesty cost of ignoring the index

Ignoring the cross-reference table buys independence from a damaged index at the cost of losing the index's authoritative view of which objects are live. The guides accept that trade explicitly for a four-file corpus where the two views coincide, and they name the case, incremental update, where they would diverge. Publishing the trade keeps the tool's output interpretable instead of presenting a raw object count as an unambiguous fact about every PDF.

Why object counts differ between files that look identical

Two files can render the same page while carrying very different object counts, because the object graph reflects how a writer chose to structure the document rather than what the document shows. A writer that embeds a font as a program plus a descriptor plus a mapping table uses several objects where another writer uses one. The count is a property of the file's construction, which is exactly why it is useful for a size investigation and misleading as a quality measure.

How the scan result is validated

The generator fixes the expected count before the analyzer runs: five objects for the three unpadded files and forty-five for the padded one. The analyzer, reading only bytes and using no index, produces the same counts on every file. That agreement is the validation. Without a constructed case whose answer is known in advance, a scan could be systematically wrong and nothing in its output would reveal it.

The specific way a byte scan can overcount

A compressed stream is arbitrary binary data. Its bytes could contain a sequence matching the object header pattern, and since the scan does not decompress streams during the walk, it would count that occurrence as an object. The corpus avoids the case by construction, which the guides state rather than conceal. A production analyzer would need to read the cross-reference table to distinguish real objects from coincidental byte patterns inside stream payloads.

How a hybrid analyzer would look

A more robust design would read the index first, follow the offsets it names, and treat any object found outside the index as an anomaly worth reporting separately. That combines the index's precision with the scan's ability to detect what the index omits. For the four corpus files the two approaches agree exactly, so the simpler scan is sufficient here, and the guides record the design choice and its reason rather than presenting it as the only correct approach.

What incremental updates would require

When a file is saved incrementally, a changed object gains a new body and an updated index entry while the old body often remains in the file. A byte scan finds both and would report a higher count than the document logically contains. The corpus contains no incremental updates, so the guides make no claim about that case and identify it as the first thing a production analyzer would need to handle.

First-party evidence and provenance

How we checked this

We generated four single-page PDFs with this repository's own writer, then parsed each file's raw byte structure with this repository's own analyzer. Both tools are dependency-free and deterministic, so the same commands reproduce the same numbers on any machine. For this guide we ran the byte-level object scan over all four corpus files and checked that the reported object counts match how the corpus was constructed, which is what demonstrates the scan finds real structure.

Method: We generated four single-page PDFs with this repository's own writer, then parsed each file's raw byte structure with this repository's own analyzer. Both tools are dependency-free and deterministic, so the same commands reproduce the same numbers on any machine.
Environment: Node.js on a desktop workstation. No PDF library, no network access, and no third-party file was used. The corpus is produced from text this repository owns.
Captured: September 20, 2026
Reviewed by: BeanNest Studio

Repository evidence artifacts:

tools/research-evidence/pdf/analysis-2026-09-20.json
tools/research-evidence/pdf/README.md