11  read_source: From URL to Admitted Representation

Part III · Skills and the compositional research toolchain

Implementation baseline: screened public-HTTPS source reader · reviewed reference snapshot 45a903c

Reading a URL is a network and parsing operation, not a synonym for trusting it. The application must decide which destination, bytes, representation, and lifetime it is willing to admit. In the Northstar workflow, this is the boundary between a search candidate and the official guidance actually examined.

11.1 One narrow public-read operation

read_source accepts exactly one bounded URL. It does not accept a method, body, caller headers, cookie, credential, proxy, local path, browser profile, provider, or extraction plugin. The reviewed configuration fixes public-HTTPS access and limits:

- name: read_source
  access: public-https
  maxCalls: 100
  timeoutMs: 15000
  maxResponseBytes: 5242880

Every admitted logical call spends invocation budget. Repeated successful reads may reuse one immutable representation within the same invocation, but authorization and current URL policy still run. Failed and cancelled calls retain their debit; they cannot become free remote probes.

Search and read remain separate. Chapter 10 discovers candidates. This chapter selects one candidate, authorizes a destination, retrieves complete bounded bytes, and derives a named representation. It does not decide whether the source is true or whether an answer follows from it.

11.2 URL admission before DNS

The reader admits only an unambiguous HTTPS URL using the default port and a DNS hostname. It rejects:

  • URL credentials;
  • explicit ports;
  • IP-address literals;
  • non-HTTPS and ambiguous schemes;
  • control characters and malformed forms;
  • local or internal hostnames; and
  • unsupported URL structure.

The fragment is not sent on the request and does not change fetch identity. Query order, repeated keys, encoding, and values remain significant because “cleaning” them can change the resource. Hostname case and other equivalences guaranteed by the URL parser may normalize; the reader does not invent semantic equivalence.

Syntax checks are only the first gate. A syntactically public hostname can resolve to a private destination.

11.3 DNS and address screening

The transport resolves the hostname and screens every returned address. If any answer is loopback, private, link-local, multicast, reserved, documentation-only, or otherwise outside the reviewed public set, the destination is rejected. Mixed public/private answers fail rather than selecting only the convenient public member.

The current profile retains a reviewed address-family policy; expanding it requires equivalent screening and tests rather than accepting a new family by default. Application-level checks are defense in depth, not an egress firewall or operating-system network sandbox.

Screening has to survive the time-of-check/time-of-use boundary. Resolving an acceptable address and then opening an ordinary hostname connection would permit a later lookup to select another destination. The implementation therefore pins the screened address for the actual connection.

11.4 Pinned transport with hostname TLS validation

Address pinning must not discard the original hostname’s security meaning. The connection targets the screened address while TLS Server Name Indication, certificate validation, and HTTP Host remain bound to the admitted hostname.

The reader does not honor ambient proxy variables, send cookies or authorization, inherit browser state, or add caller-controlled headers. This prevents ordinary process environment and user-agent state from silently widening the recipient boundary.

The transport requires complete, successfully framed responses. It does not guess the end of a close-delimited body. Deadlines and cancellation cover the bounded operation, but local cancellation does not prove that the origin did no work.

11.5 Redirects are new destinations

The current policy refuses redirects. This is stricter than following a redirect after validating only the first URL. A redirect can change scheme, hostname, address class, credentials, and disclosure destination.

A future redirect profile would need to bound hops and repeat full URL parsing, DNS/address screening, address pinning, and authorization for every hop. It would also need a retained safe lineage. Calling a client with automatic redirect support would bypass that review.

The practical consequence is visible failure on many otherwise ordinary web pages. The coordinator may search for a direct public representation; it may not relax policy or imply that a failed landing page was read.

11.6 Media type, framing, and encoding

A network success is not yet an admitted representation. The reader checks response status, framing, declared media type, content encoding, body size, and parser profile. Unsupported compression or encoding rejects rather than entering an unreviewed decompressor.

The current shared ingestion path supports bounded:

  • HTML;
  • PDF;
  • DOCX;
  • XLSX;
  • UTF-8 text and Markdown;
  • JSON and JSONL; and
  • CSV and TSV.

HTML is converted to inert content; scripts and subresources do not execute. Office and PDF formats use the same profiled document-ingestion boundary as request attachments. PDF OCR requires the configured local Tesseract path when applicable. Layout-model inference remains disabled. The reader is not a browser and does not claim visual page equivalence.

Media type and content have to agree sufficiently for the selected parser profile. Unsupported, malformed, encrypted, or corrupt representations fail with a bounded category rather than falling through to arbitrary parser behavior.

11.7 One byte ceiling, applied twice

maxResponseBytes is the public configuration ceiling for both response bytes and extracted canonical text. The implementation rejects oversized input or output instead of truncating a document and labelling it complete.

Shared ingestion adds format-specific controls over archive expansion, entry count, nesting, entities, parser iterations, pages, cells, images, dimensions, OCR, and extraction deadline. Those controls prevent a small wire body from becoming unbounded parsing work.

The result must also fit the shared tool-output envelope. These are distinct limits:

wire bytes
  → media/encoding admission
  → parser-specific expansion limits
  → canonical text bytes
  → model-visible serialized result bytes

Passing an early ceiling does not imply passing the later ones. A complete admitted representation is one that survives every applicable bound without silent truncation being misdescribed as completeness.

11.8 Shared ingestion, not a second parser stack

Request attachments and read_source use the same reviewed extraction profiles. This avoids a common inconsistency: a PDF attached locally produces one canonical representation while the same PDF fetched over HTTPS follows a more permissive parser path.

Network admission remains unique to this chapter. File-byte admission and extraction behavior belong to Chapter 4. The composition point is explicit: the public reader supplies screened bytes and media metadata to shared ingestion; shared ingestion returns the bounded canonical representation or a typed failure.

The source remains untrusted throughout. A document can contain imperative text, scripts represented as text, malicious instructions, false claims, or adversarial strings. Extraction changes representation, not authority.

11.9 Invocation-local source identity

Every successful read returns bounded content plus identity metadata, including:

  • canonical retrieval URL;
  • opaque sourceId;
  • body and/or representation digest as retained by the profile;
  • retrieval time;
  • media type;
  • representation/extractor revision; and
  • complete canonical text within the selected bounds.

The registry also binds the representation to the current invocation owner and request. The model can later pass the exact sourceId to code_interpreter; it cannot invent an ID, use one from another invocation, or turn it into a host path.

A source ID is not a URL, truth proof, publisher signature, or durable citation identity. A digest identifies bytes under the selected representation; it does not authenticate the origin, validate extraction, establish permission, or prove claim support.

The same URL can yield different bytes later. The same bytes can appear under different provenance. Retain address, retrieval event, representation identity, and source authority as separate fields.

11.10 Invocation-local caching and single flight

The live server has no authenticated cross-request Pi session identity suitable for shared cache ownership. The cache is therefore created once per HTTP invocation and destroyed at terminal cleanup.

Its key is the exact authorized retrieval identity: canonical URL including meaningful query, excluding the network-irrelevant fragment, under the current reader/extractor policy. It contains successful immutable representations and bounded in-flight entries only.

Concurrent identical misses share one underlying retrieval. Successful waiters receive detached immutable snapshots. Failures, malformed results, oversized representations, and late results after cancellation do not enter the completed cache. A cancelled waiter does not necessarily cancel work still needed by another active waiter; if every waiter is cancelled, late completion cannot publish authority.

A cache hit returns the original digest and retrieval timestamp. It is reuse, not a fresh network verification. There is no process-global, persistent, session, or cross-restart cache, no TTL freshness promise, and no durable snapshot archive.

11.11 Progress and privacy

Pi progress may show a bounded lifecycle, safe host/path preview without query or fragment, media category, representation length, duration, cache reuse, and safe failure class. It must not display document text, quotations, full sensitive URLs, resolved addresses, headers, digests, credentials, or raw errors.

Ordinary diagnostics similarly report category and counts rather than payload. “In memory only” does not mean absent from provider context, process memory, swap, crash dumps, or later answer text. Once the canonical representation is returned to the coordinator, it can enter model context under the configured provider policy.

11.12 Retrieved content remains data

The retrieved page cannot:

  • register a tool or skill;
  • choose a provider or endpoint;
  • widen a limit;
  • authorize another destination;
  • grant filesystem or network access;
  • approve a consequential action; or
  • override system and application policy.

Labels such as “untrusted source” help the model, but server enforcement must not depend on the model obeying them. Chapter 8’s policy runtime remains the authority boundary. Tool results can inform later proposals, never authorize their execution.

11.13 Retrieval is not truth or entailment

A completed read establishes a narrow fact: through the admitted transport and parser path, the application obtained this bounded representation at this time. It does not establish:

  • who authored the material;
  • whether it is current, complete in the broader publication, or accurate;
  • whether extraction preserved every layout-dependent meaning;
  • whether the selected passage supports a generated claim;
  • whether another source contradicts it; or
  • whether the application has permission to redistribute it.

A final URL citation can identify read provenance without pretending to be a provider-native annotation. The renderer must not fabricate page, section, quote, or source-span locators. Chapter 13 owns support assessment and completion criteria.

11.14 Northstar handoff

In the recorded workflow, web_search finds current official US guidance and read_source retrieves the selected official HTTPS representation. That sequence demonstrates why the tools are separate: search output chooses a candidate; source reading creates the invocation-local representation used by the answer.

The recording is dated evidence for the exact route and selected page. It does not prove that the page remains unchanged, that the retrieval policy reaches all official sites, that extracted guidance is universally applicable, or that the generated answer correctly entails every cited sentence.

11.15 Failure modes

SSRF by syntax. A URL parser accepts a hostname that resolves privately. Screen all addresses and pin the admitted one.

Redirect laundering. The first public URL redirects to an unreviewed destination. Refuse redirects or fully reauthorize each hop.

Partial-as-complete. The reader truncates a large response and presents it as the document. Reject or label an explicitly partial representation.

Cache-as-freshness. A reused snapshot is described as newly checked. Preserve the original retrieval time.

Text-as-instruction. Retrieved imperatives influence server authority. Keep policy and execution outside source content.

Digest-as-truth. A stable hash is treated as publisher authentication or semantic proof. Keep integrity, provenance, and support separate.

11.16 Pattern summary

Problem: A URL can conceal a prohibited destination, mutable bytes, dangerous parsing work, and hostile content.

Pattern: Admit public HTTPS narrowly, screen and pin addresses, refuse unreviewed redirects/encodings, apply shared bounded ingestion, and retain invocation-local representation identity.

Result: Discovery candidates can become concrete, reviewable source representations without becoming trusted instructions or factual proof.

Cost: Strict policy rejects some legitimate pages; durable identity, richer redirects, authenticated sources, and semantic verification require separate designs.

11.17 Exercises

  1. Design a redirect extension that reauthorizes every hop. List the fields required in safe redirect lineage.
  2. Explain why a mixed public/private DNS answer fails even if the client intends to use the public address.
  3. Compare a cache hit with a conditional fresh read. What new transport and freshness contracts would the latter require?
  4. Construct a successful retrieval whose final claim is still unsupported.

11.18 Chapter contract

  • Scope: public-HTTPS URL admission, DNS/address screening, pinned transport, redirects, media/encoding policy, response limits, ingestion, invocation caching, and source identity.
  • Canonical claims: successful retrieval identifies an admitted representation; it does not establish truth, authority, or entailment.
  • Migration inputs: current source-reader implementation, shared ingestion profile, invocation-cache probes, and deterministic transport tests.
  • Evidence gates: URL/DNS/TLS/redirect/media/size tests plus separately dated live retrieval evidence.
  • Demonstration slots: Northstar demonstrates candidate-to-representation handoff for one official URL only.
  • Source-claim ledger: retrieval claims record URL identity, time, media type, digests/revision where retained, and representation limitations.
  • Exclusions: discovery belongs to Chapter 10; support and research completion belong to Chapter 13; local attachment admission belongs to Chapter 4.