19 Production Case Study and Formal Conformance
Part V · Open ecosystem and validation
Functional baseline 45a903c · Open Responses suite 92c12d96 · 8 passed / 9 failed / 0 skipped · compatible subset; not conformant
The Northstar workflow is useful because it works across files, providers, public retrieval, local computation, skills, and artifacts. It is useful as evidence only when every boundary keeps its own claim. A polished answer and deck cannot erase a failed conformance test, an unreviewed recipient, ephemeral source identity, or missing production isolation.
19.1 The integrated system
The production-shaped case study begins in Pi with three synthetic inputs and one explicit research request:
- a generated image of the fictional Northstar Field Console;
- a one-page fictional specification PDF; and
- a synthetic measurements workbook.
The request asks the system to inspect visible labels, distinguish specification declarations from measurements, find and read current official public guidance, calculate mean runtime, preserve uncertainty, and avoid presenting the fictional product as real. A follow-up requests a short PowerPoint deck.
The execution topology is:
Pi 1.0.4 -- Open Responses HTTP/SSE --> loopback Bun server
→ ordered image/PDF/XLSX admission
→ Anthropic coordinator route
→ OpenAI image-understanding bridge
→ OpenAI hosted-search bridge
→ screened public-HTTPS read_source
→ local fresh Pyodide worker
→ evidence-labelled answer
→ integrity-bound business-tech-presentation skill
→ separately authorized local presentation_builder
→ short-lived artifact
→ Pi byte-count/SHA-256 verification and mode-0600 saveThis is one orchestrated task, not one provider capability. Configuration, server-owned tools, invocation-local assets, and recipient disclosures preserve the boundaries.
19.2 Exact implementation and evidence baselines
Several immutable or dated baselines coexist. They must not be merged into one “current version” claim.
| Baseline | Identity | Supports | Does not support |
|---|---|---|---|
| Current reviewed functional snapshot | 45a903c84c30ac1bcc9636c91e4166c5009f2f00 |
Published Northstar and document-inspection behavior | Immutable release identity or telemetry additions |
| Latest immutable release | v0.4.0 / 5ecc4f01e61858f16e42e5c2c34edac234855d6d |
Released package behavior | Every later presentation/inspection change |
| Historical Part II baseline | v0.3.0 / 1c7a18a8434e5143cbe9d050b8252a6ead1f9b51 |
Historical adapter/runtime evidence where explicitly cited | Current live server behavior |
| Telemetry implementation evidence | Reference revision ce47fe2cf5f109ef4939fd704e4956682d4af03d; OTel API 1.9.0, SDK/context 2.12.0; artifact SHA-256 2fbaf4d…0119 |
Deterministic span topology and redaction at that exact revision | Tagged release, OTLP, production collector |
| Open Responses suite | 92c12d96d7b61d6d15e2214daa5e9c6000ab6e1c |
Official pinned-suite result against reviewed target | Security, semantics, future suite revisions |
Telemetry is now tied to immutable commit ce47fe2cf5f109ef4939fd704e4956682d4af03d, but it is not part of the earlier v0.4.0 tag. The deterministic artifact and tests establish the bounded in-memory profile at that commit; they do not establish production OTLP or collector operation.
19.3 Input and document boundary
The server accepts ordered text, reviewed file types, and reviewed image types under one request-body and aggregate asset profile. It validates base64/data input, byte counts, media, pages/cells/images, expansion, parser work, and extraction deadlines. Invocation-local IDs bind admitted assets for tools.
For Northstar:
- the PDF supplies attached declarations;
- the XLSX supplies workbook measurements;
- the image supplies pixels for a generated observation;
- attachment IDs expire with the HTTP invocation; and
- the files are fictional fixtures with a public manifest.
This boundary does not authenticate real-world origin or create durable formal file citations. Attachments must be resent in a later invocation.
The separate BCAUSS recording demonstrates model-independent local inspection: 16 pages, 44 chunks, 129,145 characters, zero warnings, a retained digest, and zero provider/public-network calls for the reviewed run. It proves extraction/inspection behavior for that public author-supplied paper, not factual correctness or redistribution of the PDF.
19.4 Provider and recipient routes
The recorded Northstar coordinator is:
portable-agents/anthropic/claude-opus-5-5Selected sub-operations have separate recipients:
| Operation | Recipient | Released data | Result class |
|---|---|---|---|
| Coordinator | Anthropic route | Prompt, admitted text/markers, tool results | Generated orchestration/answer |
| Image understanding | OpenAI gpt-5.4 |
One admitted image + focused question | Generated, unverified observations |
| Hosted search | OpenAI gpt-5.4 |
Focused public query | Candidate URLs + generated notes |
| Source read | Selected public HTTPS origin | Bounded request | Admitted representation |
| Computation | Local Pyodide | Selected invocation snapshots + Python | Deterministic result under inputs |
| Presentation | Local PptxGenJS worker | Semantic deck + admitted images | Candidate .pptx artifact |
The coordinator receives an image marker, not original image bytes, until the image tool selects that asset. Search receives the focused query rather than the entire transcript. Recipient metadata is part of the result/evidence.
This demonstrates mixed-provider composition. It does not demonstrate semantic equivalence, automatic fallback, or a visible coordinator switch during the recording.
19.5 Search, retrieval, and research completion
web_search spends invocation-owned call and unique-URL budgets, requires completed native execution evidence, admits concurrent results in completion order, and returns bounded candidates. Those candidates are not source representations.
read_source then applies public-HTTPS URL, DNS/address, pinned TLS, redirect, response, media, encoding, shared-ingestion, and output limits. Its successful result records an invocation-local representation identity and retrieval time.
The final research answer keeps five classes separate:
- attached specification declarations;
- workbook measurements;
- generated image observations;
- deterministic calculations; and
- retrieved public guidance.
A citation URL identifies provenance; it does not establish truth or entailment. Research completion requires the task’s evidence matrix or an explicit gap.
19.6 Computation boundary
A fresh local Pyodide worker receives only fixed copies selected by invocation-local IDs. It has no declared network, package installation, arbitrary host paths, persistent state, or durable generated artifact.
For the workbook values 9 and 7.5 hours, the mean is:
(9 + 7.5) / 2 = 8.25 hoursThe calculation is deterministic under two finite hour measurements and the stated missingness policy. null is missing, never zero. The result does not prove the fixture values describe a real product.
The worker is not an OS sandbox. Production hostile-code isolation remains blocked on a separate process/container boundary and OS resource/egress controls.
19.7 Skill and presentation boundary
business-tech-presentation is an integrity-bound, non-executable procedure. The static catalog and manifest verify package identity, resources, byte counts, and SHA-256. Progressive load_skill changes context, not authority.
presentation_builder is selected separately. It accepts semantic slide data and rejects code, paths, coordinates, templates, package names, remote assets, macros, OLE, and external relationships. The server maps semantic slides through reviewed recipes/components and validates the OOXML package.
The Pi extension fetches the short-lived artifact, verifies exact byte count and SHA-256, saves with mode 0600, and requires confirmation before replacement. A verified structural package is not automatically visually approved. The published recording did not visually inspect the generated deck during that session; separate artifact review is exact-byte/version scoped.
19.8 Recovery and durable state
The integrated deterministic trace distinguishes safe replacement before output from unknown tool effects. A lost receipt after execution produces reconcile, not retry. Transcript reopen reconstructs records without dispatching pending calls. Cancellation cannot authorize repeating an uncertain effect.
The local SQLite experiment provides atomic append/read under trusted owner values and expected revision. It is not authentication, remote effect atomicity, or automatic resume authority. Provider continuation and Pi sessions remain separate state planes.
For presentation handoff, an unexpired artifact ID can be fetched and digest-verified after client interruption. Expiry/restart can make the exact artifact unavailable; rebuilding is a new operation and can produce different bytes.
19.9 Telemetry boundary
The OpenTelemetry implementation at ce47fe2cf5f109ef4939fd704e4956682d4af03d emits content-free spans for HTTP, ingestion, model invocation/round, tool proposal/policy/execution, queue/worker, and nested provider operations. Context propagates through async and host-worker scheduling boundaries.
The deterministic evidence artifact is byte-stable and excludes prompts, source bodies, image bytes/observations, tool values, Python code/output, credentials, raw errors, and opaque IDs. It uses an in-memory exporter. No production OTLP recipient, sampling, metrics, logs, retention, or collector-security claim is made.
Span success is operational completion only. It does not establish factual support, effect rollback, or task completion.
19.10 Four conformance statements
Conformance language must identify the level being claimed.
| Statement | Meaning in this project | Current status |
|---|---|---|
| API resemblance | Endpoint and payload/event vocabulary look like Open Responses | Present |
| Implemented compatible subset | Reviewed HTTP/JSON/SSE and input/tool paths work for documented clients/tests | Present, bounded |
| Official-suite result | Pinned public suite ran against the target and preserved failures | 8 passed, 9 failed, 0 skipped |
| Formal conformance | All required profile behavior and conformance criteria are satisfied | Not established |
API resemblance is the weakest statement. Compatible-subset operation is useful and testable but deliberately incomplete. An official-suite result is evidence, not a synonym for passing. Formal conformance cannot be claimed while required cases fail.
19.11 Official Open Responses suite result
The wrapper pins official suite revision:
92c12d96d7b61d6d15e2214daa5e9c6000ab6e1cIt accepts loopback targets, verifies the CLI digest, uses a frozen install with lifecycle scripts disabled, bounds report size, independently verifies summary counts, and preserves the official failing exit status.
Latest reviewed complete result:
8 passed
9 failed
0 skipped
compatible subset; not conformantSeven passes and all nine failures contacted the HTTP target. The result is credential/target-specific evidence under the named suite and configuration. It is not a live-provider quality result or security audit.
19.12 Missing required behavior
19.12.1 /v1/responses/compact
The server does not implement the compact endpoint/profile required by three pinned cases:
- compact response;
- compact missing-model behavior; and
- compact output starting a new WebSocket chain.
Producing schema-shaped summary items is not enough. A credible implementation needs quality/loss policy, state ownership, bounds, tenant isolation, cancellation, error handling, and compatibility with subsequent continuation.
19.12.2 WebSocket lifecycle and state
Six failures cover:
- basic WebSocket response;
- sequential responses;
- continuation on an active connection;
- reconnect recovery;
- missing previous-response state; and
- eviction after failed continuation.
The current HTTP/SSE lifecycle cannot be called WebSocket-compatible. Implementing a connection-local cache solely to satisfy fixtures would not establish durable, secure, multi-tenant state.
19.13 Decision on missing conformance features
Formal conformance is not the immediate implementation goal for this publication revision. The project will not add narrow compaction/WebSocket behavior merely to increase a pass count before state ownership, bounded concurrency, tenant isolation, and security semantics are designed and tested.
The decision is:
- retain and publish the failing result;
- keep classification compatible subset; not conformant;
- treat compaction and WebSocket lifecycle/state as blocked functionality;
- require an implementation plan covering state, isolation, cancellation, recovery, and conformance—not test recognition;
- rerun the exact pinned suite after any implementation; and
- evaluate newer suite/spec revisions separately rather than rewriting history.
If formal conformance later becomes a release objective, the work must include /v1/responses/compact, WebSocket lifecycle, active continuation, reconnect recovery, missing-state errors, failed-chain eviction, bounded concurrency, tenant isolation, and retained exact evidence.
19.14 Consolidated evidence matrix
| Boundary | Deterministic evidence | Dated/live evidence | Current limitation |
|---|---|---|---|
| Pi client | launcher/routing/extension tests | Northstar and inspection recordings | Pi-local session is not server tenant identity |
| Configuration | strict YAML/catalog tests | startup records | no production secret/entitlement service |
| Inputs | parser/admission/cancellation tests | exact BCAUSS inspection | ephemeral identity; no all-format formal citations |
| Coordinator adapters | synthetic OpenAI/Anthropic fixtures | Northstar Anthropic route | no provider equivalence |
| Search | receipt/budget/concurrency tests | Northstar OpenAI search | candidates/notes are not read evidence |
| Retrieval | URL/DNS/TLS/cache/ingestion tests | selected official page | no redirects/authenticated/private sources |
| Image | masking/recipient/termination tests | fixed OpenAI recipient | generated observation, not verified OCR/truth |
| Computation | real local Pyodide tests | Northstar calculation card | no OS-level hostile-code isolation |
| Presentation | schema/composition/OOXML/handoff tests | recorded save + separate reviewed artifact | exact-artifact visual scope only |
| Recovery/state | retry, unknown-effect, SQLite tests | none required | no durable remote effect ledger |
| Telemetry | ce47fe2 in-memory hierarchy/redaction tests + artifact |
no collector run | no production OTLP/metrics/logs |
| Security | deterministic controls/failure fixtures | privacy-reviewed recordings | not production approved |
| MCP/A2A | architecture only | none | unimplemented |
| Open Responses | parser/reducer/HTTP fixtures | pinned official suite | 9 failures; not conformant |
19.15 Blocked-work register
| Blocked claim/function | Missing evidence or implementation | Required next gate |
|---|---|---|
| Formal Open Responses conformance | Compact + WebSocket lifecycle/state | secure bounded implementation + pinned suite rerun |
| Production multi-tenancy | authentication, ownership, entitlements, isolation | adversarial cross-tenant and operational review |
| Hostile code execution | OS resource/network boundary | container/process implementation + escape/DoS testing |
| Production document parsing | stronger process isolation and corpus evidence | isolated parser service + fault/vulnerability tests |
| Durable local-file citations | retained identity and page/span mappings | authenticated source store + extractor mapping tests |
| Cross-request attachment/artifact identity | trusted ownership/lifecycle | durable store, retention/deletion, authorization |
| Exactly-once effects | remote idempotency/receipts/reconciliation | operation ledger + effect-system support |
| Production telemetry | OTLP, collector security, sampling/retention | exporter implementation + privacy/load tests |
| MCP/A2A support | transport/delegation implementations | pinned interoperability and lifecycle evidence |
| Provider semantic equivalence | task distribution and paired thresholds | repeated disaggregated evaluation |
| Universal presentation compatibility | suite/font/renderer matrix | exact-version render and manual review |
| Public gallery without lineage exposure | full review and neutralized metadata | 312-slide audit and independent-publication review |
19.16 What the case study proves
The integrated evidence supports this narrow statement:
Under the reviewed loopback configuration and exact dated evidence, a Pi client can route a fictional mixed-input research task through provider-neutral coordination, separately authorized bounded tools, screened public retrieval, local computation, an integrity-bound non-executable presentation procedure, and digest-verified local artifact saving while preserving recipient and evidence-class distinctions.
It does not prove deterministic model output, factual truth of external sources, real-product performance, provider equivalence, production security, exactly-once effects, every renderer’s compatibility, MCP/A2A support, or formal Open Responses conformance.
19.17 Release checklist for the integrated profile
Before publishing a new baseline:
- freeze reference commit and exact dependency lock;
- run credential-free checks and Pi-extension typecheck;
- regenerate deterministic telemetry evidence;
- build/verify presentation artifacts when required;
- rerun approved live workflows only with explicit paid-call authorization;
- review media frame by frame and retain transcript/alt/metadata/hashes;
- run the pinned official conformance suite and preserve failure exit;
- update the evidence matrix and blocked register;
- build the book from pinned Quarto and validate HTML/PDF/accessibility/links/public artifact;
- confirm license, secret, private-source, and generated-file exclusions.
Historical tags and evidence remain immutable. A new result receives a new identity; it does not edit the meaning of an old one.
19.18 Pattern summary
Problem: End-to-end success tempts a project to merge functional compatibility, semantic quality, security, and conformance into one claim.
Pattern: Reassemble the system through explicit component contracts, bind each observation to an exact baseline, publish official failures, and maintain a consolidated blocked-work register.
Result: The Northstar workflow can demonstrate real composition without overstating production readiness or formal protocol status.
Cost: Multiple evidence classes, immutable revisions, scoped demonstrations, and deliberately retained negative results require continuing maintenance.
19.19 Exercises
- Promote telemetry commit
ce47fe2into a tagged release. List every baseline and ledger entry that must change. - Design secure WebSocket continuation without trusting a caller-supplied response ID as ownership.
- Specify a compact representation that preserves unresolved effects and evidence qualifiers. Define rejection conditions.
- Add one provider to the Northstar coordinator evaluation without claiming semantic equivalence from one successful run.
- Convert one blocked-work row into a production gate with owner, expiry, tests, and rollback.
19.20 Chapter contract
- Scope: integrated Northstar architecture, exact baselines, recipient/tool/evidence boundaries, telemetry, official-suite reporting, and final supported-profile classification.
- Canonical claims: the reference implements a useful compatible subset and end-to-end research workflow; official failures and missing compaction/WebSocket behavior keep it not conformant.
- Migration inputs: Northstar/presentation evidence, document inspection, implementation-choice evidence, Open Responses records, and Chapters 1–18 component contracts.
- Evidence gates: immutable/datetime identities, deterministic checks, reviewed live media, telemetry artifact, official suite, and consolidated limitations.
- Demonstration slots: Northstar and model-independent inspection are admitted only under their published metadata and limitations; no provider-switch or MCP/A2A demonstration is claimed.
- Source-claim ledger: functional, live, telemetry, protocol, security, and semantic evidence remain separate and versioned.
- Exclusions: component mechanisms remain in their owning chapters; missing conformance and production controls remain explicit blocked work rather than implied features.