Durable product records

Infrahub Sync provides one storage contract for compact product records and immutable artifacts. It is independent of Prefect: a product run can retain any number of purpose-labelled execution links after Prefect no longer has the corresponding flow-run detail.

The Sync HTTP API and its worker store records in PostgreSQL and immutable artifacts in S3-compatible object storage. Each process creates its own clients from the same environment settings. CLI and public Python callers access these records only through the Sync API.

Service execution continues to use INFRAHUB_SYNC_CACHE_DIR as the absolute shared cache root for saved plans. This PH-2 seam is separate from product records and artifacts.

This is minimum product-projection configuration. It does not select a new provider type or add a release-management API.

For configured direct sync, the immutable review artifact is published after the saved plan commits and before the first destination write, in both serial and tiered execution. If publication fails, synchronization stops before contacting the destination write surface and retains typed failure evidence on a terminal failed ProductRun. A reserved artifact remains unavailable until an exact publication retry completes it.

Configuration registry

The product store holds an append-only registry of declared configuration packages. A package is a strict, JSON-native envelope containing declared content only: adapter names, settings, and references to credentials. It never contains a credential value.

Registration validates before it persists, so an invalid package is refused and no version is written. validate re-reads a version that is already registered and checks it against the adapter declarations installed now — which is why a package accepted at registration can report findings later, after an adapter's declared setting surface changes.

A package carrying several independent defects yields one finding per defect, not just the first. Each finding carries a stable machine-readable code, a severity of error or warning, a JSON Pointer location into the declared package, and a message. An error prevents execution; a warning records declared intent or a qualification gap and does not.

Finding codes

The code is the stable part of a finding. The message wording and the printed layout may change; the code will not. Three enumerations are emitted: the declared-content core's fifteen error codes below, the four destination-schema codes (emitted only when validate is given an explicit destination-schema opt-in), and the three warning-channel codes. One location can report more than one of them — a credential declaration naming an uninstalled provider and an invalid identifier reports both at /credentials/<name> — but only ever from one check: when two checks reach the same pointer, the first to judge it is the one that reports.

Code	What it means	Where it points
adapter-role-mismatch	The adapter named for this role cannot serve it — most often a source-only adapter declared as the destination.	/configuration/source or /configuration/destination
adapter-validator-finding	The adapter's own configuration check failed, or handed back something unusable. Nothing the adapter said is carried into the message, so read the adapter's own findings for detail.	/configuration/<role>
credential-path-not-declared	A $credential reference sits somewhere that does not accept one. Usually a misspelled setting name, or a reference placed in schema_mapping or order.	the referencing node
endpoint-not-absolute	A url or base_url setting is not an absolute http or https URL.	the setting
endpoint-not-relative	An api_endpoint or endpoint setting carries a scheme or a host. It names a path beneath the absolute URL, not a second address.	the setting
finding-limit-reached	The package carries more than 256 defects and the rest were not reported. It is reported first, not last, and it is counted as a finding: when it fires the reported set holds 257 items, not 256.	the whole package
inline-credential-value	A credential-bearing setting holds a literal value instead of a {"$credential": "<name>"} reference. A package never contains a credential value.	the setting
malformed-credential-reference	A credential declaration's environment identifier is not a valid variable name, or a $credential node carries keys beyond the reference itself.	the declaration or the node
missing-adapter	No adapter is installed under the declared name — check the spelling and the case. That role's settings are not judged at all, because there is no declared surface to judge them against, so expect exactly one finding for the role.	/configuration/<role>
missing-store-capabilities	The declared store type is unknown and carries settings. An unknown store type with no settings declares nothing unsafe and is accepted.	/configuration/store
setting-contains-credential-material	An endpoint setting carries user information, a query string, or a fragment. Credentials belong in a reference, not in a URL.	the setting
setting-not-a-string	An endpoint setting is declared as something other than a string.	the setting
undeclared-setting	The adapter or store does not declare that setting name. One finding per name, each at its own pointer.	the setting
unknown-credential-provider	A credential declaration names a provider that is not installed. env is the installed provider.	/credentials/<name>
unknown-credential-reference	A $credential names a reference the package's own credentials block does not declare.	the referencing setting

An adapter's own configuration check keeps its own codes, which are outside this set — unsafe-rest-request-endpoint is the one shipped today.

Destination schema validation codes

These five codes are emitted only on the explicit opt-in: validate given a destination-schema options object. The default validate path judges declared content only, performs no schema read and no network I/O, and never emits them. All five carry an error severity.

Code	What it means	Where it points
destination-schema-mismatch	A declared schema mapping disagrees with the destination's schema snapshot: an undeclared kind, a field that is neither an attribute nor a relationship, a relationship reference on an attribute, or a static value whose shape disagrees with the relationship's cardinality.	the mapping entry, field, reference, or static value
destination-schema-read-failed	The destination schema could not be read: a timeout, refused credentials, an unreachable server, a rejected or unusable response, an unresolvable declared token, or unusable declared client settings. The message names the failure class.	/configuration/destination
destination-schema-unsupported-semantics	The destination schema was read, but it declares semantics outside the supported schema domain — an unknown relationship cardinality, a member shape the domain does not define, or a default no JSON encoding can carry. A run of this configuration refuses the same schema.	/configuration/destination
destination-schema-validation-unsupported	Schema validation was explicitly requested against a destination adapter that does not declare it. A missing capability needed to determine safety is an error, not a warning.	/configuration/destination
unsupported-destination-write	The configuration requests destination write operations the destination adapter does not declare support for.	/configuration/destination

A successful opt-in read also returns destination_schema_fingerprint: the full SHA-256 digest of the schema semantics this configuration consumes — each mapped kind, its DiffSync identifiers, its ordered human-friendly ID and uniqueness-constraint component paths, every mapped field's type and required/default/unique properties, and every mandatory-without-default field on those kinds. Unmapped destination growth and differences in schema delivery order leave it unchanged. It is null whenever no snapshot was read — the default path, a non-declaring destination, or a failed read.

Warning-channel codes

The warning channel is closed: warnings are limited to intentional omissions and explicitly unqualified optional features, and nothing else.

Code	Severity	What it means	Where it points
intentional-omission	warning	An omissions entry declares that destination content is intentionally not synchronized. The declared reason, when present, is carried verbatim.	/omissions/<index>
omission-contradicts-mapping	error	An omission names content a schema mapping also maps. A contradictory declaration is a package defect, not a preference; the error replaces the warning at that location.	/omissions/<index>
optional-feature-unqualified	warning	The optional incremental feature is declared against a source adapter whose capability declaration does not qualify it, so extraction silently runs full.	/configuration/incremental

Storage profiles

Service deployment

The Sync API and worker construct the PostgreSQL/S3 profile from these settings:

Variable	Requirement
INFRAHUB_SYNC_DATABASE_URL	Non-empty PostgreSQL connection string accepted by Psycopg for product records.
INFRAHUB_SYNC_S3_BUCKET	Non-empty S3-compatible bucket for immutable artifacts.
INFRAHUB_SYNC_S3_PREFIX	Optional object-key prefix; defaults to infrahub-sync.
INFRAHUB_SYNC_S3_ENDPOINT_URL	Optional absolute http or https URL with no userinfo. The value reaches Boto3 unchanged; Boto3 owns any narrower SDK compatibility.
INFRAHUB_SYNC_S3_REGION	Optional region passed to Boto3.

S3 credentials use Boto3's standard credential-provider chain. Configure credentials in the API, worker, CLI, or smoke-process environment that constructs the client. Sync does not define access-key or secret-key settings.

Constructing a service process initializes the PostgreSQL schema. The PostgreSQL role must have the DDL privileges required to create the product-record tables in its configured schema, plus permission to read and write their rows.

Test injection

The local SQLite/filesystem projection is an injected test seam. It is not a service deployment option or a public CLI/Python execution mode. In-process service and store tests can inject it directly:

from pathlib import Path

from infrahub_sync.product_store import local_product_projection

records = local_product_projection(Path("/var/lib/infrahub-sync/product-cache"))

The path must be absolute after ~ expansion. Relative paths are rejected, and the seam never falls back to the process working directory.

The underlying PostgreSQL/S3 library constructor accepts a PostgreSQL DB-API connection factory and an S3-compatible client implementing the small S3Client protocol (put, get, copy, and delete):

from infrahub_sync.product_store import production_product_projection

records = production_product_projection(
    connect=postgres_connection_factory,
    s3_client=object_client,
    bucket="sync-product-artifacts",
    prefix="production",
)

Psycopg and Boto3 are installed only by the service extra. The record-store and artifact-store protocols accepted by ProductProjection are internal implementation seams, not a public custom-provider compatibility contract.

Record and identity contract

ProductRun owns the stable Sync run_id, requested operation, immutable configuration reference, actor and audit links, product phase and outcome, timings, summary and results, artifact references, and Prefect correlations. Relational child tables store artifact references and Prefect execution links separately from the compact run row.

Each PrefectExecutionLink records:

• a distinct flow_run_id;
• an optional deployment_id;
• its purpose or stage;
• its attempt number;
• optional last-observed state and timestamp.

Duplicate Sync run IDs and duplicate flow-run IDs within one record are rejected. A confirmed sync creates its own record. A reviewed-plan apply instead advances the original planning record and attaches its result artifacts to the same run_id; it does not create a second Sync identity. add_prefect_execution appends stage and retry links as they become known without changing the Sync record's identity.

Mutations against a missing Sync run raise RunNotFoundError. Read operations continue to return LookupResult with reason="run-not-found", so absence remains a normal, non-exceptional lookup result.

create_run accepts an unfinished ProductRun, including initial actor, audit, summary, results, phase, and Prefect-link metadata. It rejects records that already have a finish timestamp, outcome, or artifact reference; those completion fields must be added through the publication and finish operations so their integrity checks cannot be bypassed.

Sync API mutations reserve a MutationReceipt unique by actor and SHA-256 digest of the client idempotency key. A receipt binds the operation, target, request fingerprint, reason, Sync run, opaque Prefect key, state, and exact accepted response. The raw client key is not stored. Run creation commits its receipt and unfinished product run in one relational transaction. AuditEvent records secret-safe actor, reason, operation, and outcome evidence for accepted mutations and refusals.

record_results updates retained result evidence without changing product phase, outcome, or finish time. Service verification uses this operation because verification is read-only for both the destination and product lifecycle.

Artifact publication and lookup

Artifact keys contain their SHA-256 digest and never change. Publication first reserves a non-readable relational reference for the run-owned artifact identity. It then writes artifact data and its manifest, and finally marks that exact reference published in a second relational transaction. The injected local seam commits data and manifest using an atomic directory rename. The service S3-compatible profile copies staged data to its immutable key and uses a create-only manifest put as the object-store commit point. A crash before the final relational mark leaves durable pending evidence rather than exposing the artifact. The run cannot be finished successfully while any pending publication exists. It can be finished with outcome="failed" so publication failure evidence reaches a safe terminal state while the pending artifact remains unavailable.

An S3Client implementation must make put(..., if_absent=True) an atomic, create-only write and translate an existing-key conflict to DuplicateArtifactError. This prevents a racing publisher from replacing the immutable manifest.

To recover from an interrupted publication, retry publish_artifact with the same run and artifact IDs, content, kind, and media type. Matching is performed after secret redaction. If data and manifest are already complete, the retry verifies them and performs only the missing relational mark. If the manifest is absent, the retry resumes object publication and then marks the reference published when needed. On S3-compatible storage, that resume re-uploads the already-redacted data to its same content-addressed object key before the create-only manifest commit. This exact-match repair also covers a relational row already marked published whose manifest is missing. Different content or metadata is rejected without changing the pending reservation or writing to the artifact provider; correct the caller input and retry the original publication. Once published, the artifact remains immutable and any further publication with that identity is rejected as already published.

A lookup returns LookupResult rather than treating absence as an empty record. Reasons include run-not-found, artifact-reference-not-found, manifest-unavailable, artifact-publication-incomplete, data-unavailable, artifact-expired, and integrity failures. Normal run lookup includes only published references. An unavailable or pending artifact does not prevent the remaining product record from being read.

artifact-expired is a forward-compatible read behavior for references that already contain an expiry timestamp. The public publication operation does not set expiry, and the MVP does not expire or delete product records or artifacts automatically. Prefect retention is independent.

Secret boundary

Pass the credential values collected by collect_secret_values to mutation, audit, run, execution-link, result, artifact, and finish operations. Values are redacted from nested record data and raw artifact bytes before either provider is called. Do not put secrets into identifiers; credentials should continue to come from environment variables or a secret manager.

Reproducible sizing evidence

The sizing test repeats VAL-8's fixed-density 88k retained-artifact shape and the smaller 10k workload from the same source. Decimal bytes reproduce the published Parquet sizes: two payload stores, two hash indexes, and one plan. The separate 12.61 MB raw JSONL current-side wire measurement is not a retained artifact in this fixture.

Run:

uv sync --extra dev
uv run pytest -q -s tests/product_store/test_sizing.py

Observed on macOS with Python 3.13.3 on 2026-08-09:

Fixture	Records (baseline/current)	Artifact payload	Manifests	Relational store	Total files	Total logical bytes
VAL-8 88k	88,117 / 87,868	23,015,700 B	2,732 B	40,960 B	11	23,059,392 B
Representative 10k	10,051 / 10,023	2,722,700 B	2,877 B	40,960 B	11	2,766,537 B

The test fixture stores ten object files (data plus manifest) and one SQLite run/reference/link database. Filesystem allocation, database page size, and service object-store metadata can change physical billing; the payload and manifest byte counts are the portable sizing inputs.