4  Documents, Images, and the Ingestion Boundary

Part I · The system in action

Ordered-input target: Open Responses 2026-04-24 · Xberg profile: v2 PDF/XLSX · Reference snapshot: 45a903c

A file is not text, evidence, or model input merely because the client attached it. The ingestion boundary must identify bytes, preserve order, detect format, enforce structural limits, produce one reviewed representation, and keep that representation invocation-local.

4.1 One path for every admitted input

The reference reuses one DocumentIngestionService across several entry points:

inline input_file ──────┐
public file_url ────────┤
read_source document ───┼─→ byte admission → format handler → representation
/inspect-document file ─┤
interpreter source ─────┘

input_image ───────────────→ byte admission → image decoder → image asset

Reusing the service prevents a local inspection command, URL reader, and model attachment path from quietly applying different PDF limits. Entry points still have separate authority: /inspect-document may read a local path selected by the user, while an HTTP request cannot turn a filename into a host path.

The implementation target follows the Open Responses ordered content shape for input_text, input_file, and input_image (“Open Responses Reference 2026-04-24,” n.d.). This is an implemented local profile with additional restrictions, not full protocol conformance.

4.2 Preserve content order or reject it

A mixed user message can interleave instructions, files, images, and more text. Flattening all text first and appending extracted documents later changes meaning. The parser therefore retains message and content indices as origin metadata.

A simplified internal order is:

input_text
input_file → admitted document marker/part
input_text
input_image → admitted image marker/part

Unknown items, unknown content parts, extra fields in supported message/content objects, malformed discriminators, and unsupported media fail explicitly. They are not silently discarded. System and developer arrays remain text-only in the reviewed profile; known function-call history has its own strict mapping.

The local file rule requires exactly one non-empty file_data or file_url. The pinned schema permits both or neither, so this exclusive-or rule is a documented local security restriction rather than a statement about what all conforming implementations must do.

4.3 Byte admission comes before parsing

Admission treats supplied metadata as a claim. It detects media from bytes, compares reviewed declared types where present, normalizes a bounded display filename, computes a digest, and refuses unsupported or ambiguous input before invoking a parser.

The checked-in profile includes ceilings for:

Resource Checked-in ceiling
files per request 4
encoded characters 7,000,000
decoded bytes per file 5 MiB
decoded bytes per request 10 MiB
extracted characters per file 5 MiB-equivalent character ceiling
extracted characters per request 10 MiB-equivalent character ceiling
retained invocation bytes 50 MiB
extraction deadline 60 seconds
concurrent extractions 1 process-wide
image bytes 5 MiB
image dimension 8,192 pixels
image pixels 40,000,000
image frames 1
filename length 240 characters

The HTTP application body limit is 8,000,000 bytes, below the larger schema-only base64 maximum. Near-schema-limit experiments showed material request-memory amplification, so the implementation publishes a narrower profile rather than pretending the schema maximum is operationally safe.

Limits are layered. A request can pass the listener ceiling and still fail decoded bytes, archive expansion, image pixels, extracted characters, or retained-invocation capacity.

4.4 Supported document profiles

The current profile admits:

  • HTML;
  • PDF;
  • DOCX;
  • XLSX;
  • UTF-8 text and Markdown;
  • JSON;
  • JSONL/NDJSON;
  • CSV; and
  • TSV.

JSON and line-delimited JSON are structurally parsed. Text-like formats retain complete admitted UTF-8 text. HTML, PDF, DOCX, and XLSX use pinned Xberg extraction in a killable worker. PDF and XLSX use explicit checked-in profiles because their security and fidelity risks differ from simple passthrough.

Every document result records:

  • invocation-local source ID;
  • content digest;
  • sanitized filename;
  • detected media type;
  • complete canonical text;
  • extraction mode and method;
  • warning count;
  • page and chunk counts;
  • ingestion/profile revision; and
  • original admitted bytes retained only for reviewed invocation-local uses.

The representation says completeInputExamined: true only after the complete admitted input passes its handler. A warning, timeout, malformed structure, unsupported encryption, partial extraction, or excess output fails instead of returning text labelled complete.

4.5 The Xberg worker boundary

Xberg 1.3.8 runs in a child worker rather than the main request process. The server passes one reviewed profile; callers cannot supply extraction options, model paths, plugins, or layout strategies.

Common profile controls include:

  • no extraction cache;
  • Markdown output;
  • one concurrent extraction;
  • 60-second parser timeout;
  • 20 MiB archive/content ceiling;
  • 20:1 compression ratio;
  • 100 archive entries;
  • 64 nesting levels;
  • bounded entities and parser iterations; and
  • 5 MiB embedded-file ceiling.

The PDF profile extracts pages, tables, metadata, annotations, footnotes, and document structure while removing repeating headers/footers and excluding form fields, watermarks, and embedded images. Markdown chunks use a 4,000-character target with 200-character overlap and repeated table headers.

PDF OCR uses local Tesseract fallback when native text is inadequate. Layout-model inference remains disabled because no model artifacts have been independently admitted. Enabling it would add packages, model bytes, resource use, and potentially new recipients; it is not a harmless configuration toggle.

The XLSX profile extracts structured tables/document shape with 6,000-character chunks and no overlap. It does not run OCR. A spreadsheet table is still untrusted data and may contain formulas, misleading labels, or instruction-like text.

4.6 Canonical-text fallback

Not every selected model route has reviewed native file support. For a user-message document, the server can replace the file part in place with its complete bounded canonical text and an explicit untrusted-document marker. This preserves order and allows a text-capable route to reason over the admitted representation.

Canonical-text fallback does not mean:

  • the provider received the original PDF or workbook;
  • native provider file semantics were tested;
  • layout was perfectly preserved;
  • page-level citation identity exists; or
  • extracted statements are verified facts.

PDF and XLSX are sent to provider adapters as bounded text/plain extraction in the reviewed Pi path. Original bytes remain invocation-local for reviewed mounts and inspection. Function-output files and images follow stricter capability declarations because their provider mappings differ.

4.7 Image admission and recipients

Reviewed image formats are PNG, JPEG, single-frame GIF, and WebP. Admission sniffs bytes, compares media claims, bounds dimensions/pixels/frames, and requires complete decoding through Sharp. SVG, animated images, and multipage images are outside the current profile.

An image asset records immutable bytes, digest, dimensions, frame count, detail level, origin, ingestion revision, and invocation-local source ID. The source ID—not a path, URL, or byte string—is the only selector accepted by the image_understanding tool.

For the mixed-provider bridge, an Anthropic coordinator sees a text marker containing the admitted source ID rather than image bytes. A separately authorized tool call can send exactly one admitted image and one focused question to a fixed OpenAI recipient selected by server configuration. The result identifies that recipient and labels its observations generated: true.

This is recipient minimization, not image truth. Generated observations are not provenance, authenticity, identity proof, coordinate-preserving OCR, or evidence that a public source was consulted. Image text remains untrusted data, never instructions.

4.8 Invocation-local identity

Source IDs combine request identity, content digest, and ingestion revision. Assets are immutable after admission and disappear on completion, failure, cancellation, or stream disposal. There is no trusted cross-request asset identity.

This design prevents a later request from guessing a source ID and retrieving another request’s bytes. It also means formal local-file citations remain limited: a transcript can retain extracted words, but it cannot claim a durable file/page/span identity that the application did not preserve.

A tool argument may select an admitted source ID. It cannot introduce a new path, URL, filename, provider, or byte payload. code_interpreter receives selected sources at fixed /inputs locations; it cannot request an arbitrary host mount.

4.9 /inspect-document reuses the same representation

The Pi command accepts one explicit local path, rejects control characters, symbolic links, directories, empty/oversized files, and unsupported formats, then invokes a fixed Bun child with execFile and no shell. The child imports the same ingestion service and checked-in Xberg configuration.

The Pi extension owns a private temporary directory. The child writes the complete extraction and an exact metadata record with owner-only permissions. The parent recomputes the digest before viewing or saving, and cleanup is idempotent across success, failure, cancellation, and session shutdown.

Current implemented behavior opens the complete already-bounded extraction at the first line in a read-only viewer. Navigation and explicit save do not add the content to model input, conversation messages, progress, ordinary notifications, or logs. Metadata-only notifications can report bounded counts and the digest.

This command proves representation inspectability. It does not add a second parser, a model-callable local-file tool, or a citation system.

4.10 Cancellation and worker termination

Cancellation has to cross every layer:

request/Pi cancellation
  → ingestion service abort
  → child worker termination
  → queue release
  → invocation registry cleanup
  → temporary directory cleanup

A queued extraction must stop waiting when cancelled. A running parser must be killable rather than trusted to poll cooperatively. A timeout or cancellation cannot publish partial text as complete. Cleanup may run more than once without widening its ownership scope.

A hard process kill cannot execute ordinary cleanup. The current implementation does not scan and delete unrelated temporary directories at startup; any future scavenger needs an ownership marker, age bound, and separate tests.

4.11 Malformed and hostile content

The boundary fails closed for:

  • malformed base64 and data URLs;
  • unsupported or mismatched media;
  • unsafe public URLs and redirects;
  • missing or invalid request framing;
  • malformed PDF/DOCX/XLSX archives;
  • encrypted or incomplete documents;
  • archive bombs and excessive nesting;
  • oversized entities or parser work;
  • decompression-ratio violations;
  • invalid UTF-8 where UTF-8 is required;
  • image decode failures and excess pixels/frames;
  • parser timeout or cancellation; and
  • extraction warnings when the caller requires a complete warning-free inspection.

The text that survives remains untrusted. A document can contain prompt injection, fabricated citations, misleading tables, or data exfiltration instructions. Admission permits bounded use; it does not grant authority or establish truth.

4.12 Demonstration: the BCAUSS paper

The approved model-independent recording uses the public paper Learning end-to-end patient representations through self-supervised covariate balancing for causal treatment effect estimation, supplied by an author. The book distributes the reviewed recording, poster, transcript, alt description, and metadata—not the PDF or extracted article text.

The recorded run reports:

16 pages
44 chunks
129,145 extracted characters
0 warnings
SHA-256 f5676a2e78eb050fcb13b8b1225d0f2476710bdd6ff981c81af7251fc96208b9
0 model/provider/public-network calls

This evidence is dated and format-specific. It demonstrates one local PDF path, navigation, explicit save, and metadata. It does not establish universal PDF fidelity, scientific validity, exact publisher layout preservation, page-span citations, or safe extraction of every malformed file.

4.13 Deterministic evidence

The current tests cover ordered mixed content, strict unknown-part rejection, file-source XOR, base64 admission, request and decoded limits, media detection, JSON/text handlers, generated PDF/DOCX/XLSX fixtures, archive limits, image decoding, provider translation, canonical-text fallback, asset cleanup, interpreter mounts, cancellation, worker timeout, inspection metadata, permissions, digest equality, and zero-provider inspection behavior.

Synthetic transport tests can show that ordered parts reach adapter mappings. They do not establish live model quality or universal provider-native file behavior. The official Open Responses suite currently has no comprehensive file matrix; file support alone cannot establish conformance.

4.14 Pattern summary

Admit bytes once, preserve content order, apply a reviewed format profile in a killable worker, produce one immutable invocation-local representation, and expose only source IDs to tools. Keep extraction, provider delivery, citation identity, and claim support as separate assertions.

4.15 Exercises

  1. Trace a PDF from Pi attachment to provider input. Mark every point where bytes become text or metadata.
  2. Explain why a 5 MiB file limit does not protect against every archive or image bomb.
  3. Design a retained-source identity that could support later local citations without exposing one tenant’s document to another.
  4. Compare native provider file delivery with canonical-text fallback. Name the lost semantics.
  5. Specify cancellation tests for a request waiting behind another extraction.

4.16 Chapter contract

  • Scope: ordered file/image admission, byte and structural limits, PDF/DOCX/XLSX/text profiles, canonical-text fallback, image decoding, extraction identity, local inspection, cancellation, and recipient disclosure.
  • Canonical claims: an admitted representation is bounded and identifiable but remains untrusted; extraction is not citation support or truth verification.
  • Migration inputs: ingestion material formerly mixed into grounding plus the current ordered-input, Xberg, image, and inspection implementations.
  • Evidence gates: deterministic malformed/oversized/cancellation fixtures, exact profile revisions, provider-mapping tests, and approved local inspection evidence.
  • Demonstration slots: the BCAUSS /inspect-document recording; the Northstar hero supplies mixed image/PDF/XLSX admission.
  • Source-claim ledger: profile claims are tied to Open Responses 2026-04-24, Xberg 1.3.8 profile v2, and reference snapshot 45a903c.
  • Exclusions: public URL acquisition belongs to Chapter 11, tool authorization to Chapter 8, claim support to Chapter 13, and durable cross-request identity remains unimplemented.
“Open Responses Reference 2026-04-24.” n.d. Accessed October 7, 2026. https://www.openresponses.org/reference/2026-04-24.