Manifests & Frames

Manifests describe how every blob is assembled inside Graviton. They list the ordered block keys, byte ranges, and derived attributes that the runtime needs to rehydrate a stream without re-reading the original upload. Frames wrap those manifests (and optional block payloads) in a versioned, authenticated binary envelope so different backends can share a single durability format.

Manifest schema

graviton.runtime.model.BlockManifest is a thin container over a chunk of BlockManifestEntry values plus the aggregate uncompressed size. Each entry captures the canonical block hash, its ordinal position, and where it lands in the contiguous blob space:

Field	Type	Description
index	BlockIndex	Monotonic counter for the block’s position inside the blob.
offset	Size	Absolute byte offset where the block begins.
key	BinaryKey.Block	Content-addressed key derived from the block payload.
size	BlockSize	Refined size (max bounded by MaxBlockBytes).

Manifests also carry the merged BinaryAttributes map so consumers can see which metadata was advertised by the client and which fields were confirmed by the runtime (BinaryAttributes.confirm*). The manifest’s totalUncompressed value is recomputed from the entries every time BlockManifest.build runs, preventing mismatched attribute sizes from ever reaching disk.

Entry invariants and validation

BlockManifestEntry.make enforces the basic invariants:

• index and offset must be non-negative refined types.
• size is validated via CanonicalBlock.refineBlockSize, guaranteeing it never exceeds MaxBlockBytes.
• BlockManifest.build folds over the entries and ensures the total uncompressed byte count matches the sum of individual sizes.

Writers are expected to append entries in increasing offset order—BlockManifest does not reorder blocks for you. The ingest path maintains a running Size counter while chunking so the offsets line up with the deduplicated stream. Readers can therefore treat manifests as trustable truth: once a manifest loads, offsets and chunk counts already satisfy the runtime’s refined constraints.

Framing pipeline

While manifests are pure data, storage backends persist them inside binary frames generated by BlockWritePlan and FrameSynthesis:

1. Ingest chooses a BlockWritePlan that selects layout (FrameLayout.BlockPerFrame vs aggregate), compression, encryption, and whether duplicate blocks should be forwarded downstream.
2. BlockFramer.synthesizeBlock (or the eventual manifest framer) derives a FrameHeader, builds Additional Authenticated Data (AAD), applies compression/encryption plans, and emits a BlockFrame.
3. The resulting frame contains everything needed to replay writes on another backend: header, serialized AAD, ciphertext/plaintext payload, and optional authentication tags.

Frame header layout

FrameHeader is shared across block, manifest, attribute, and index frames:

• version: current format version (defaults to 1 via BlockFramer.FrameVersion).
• frameType: one of Block, Manifest, Attribute, or Index.
• algorithm: resolved FrameAlgorithm enum (Plain, Compressed, Encrypted, or CompressedThenEncrypted) derived from the chosen compression/encryption plans.
• payloadLength: length of the bytes that follow the header.
• aadLength: length of the serialized AAD blob.
• keyId/nonce: optional encryption metadata for AEAD modes.

Because the header is schema-driven (zio.schema), expanding the enum or adding optional fields does not break binary compatibility—old readers can skip unknown tags.

Additional authenticated data (AAD)

Frames capture structured context without leaking it into the payload:

• FrameContext provides per-upload inputs such as orgId, blobKey, policyTag, and the running block index.
• FrameAadPlan (embedded inside an EncryptionPlan.Aead) toggles which fields should be included and allows arbitrary key/value pairs via extra.
• BlockFramer materializes this plan into a FrameAad, renders it as a UTF-8 JSON object, and authenticates it alongside the payload so tampering is detectable.

AAD lets deployments prove which blob and organization a block belonged to—even when the payload stays encrypted or deduplicated.

Algorithms and layouts

FrameSynthesis combines three orthogonal decisions:

• Layout (FrameLayout): block-per-frame is implemented today; aggregate framing will pack multiple blocks once streaming repair needs it.
• Compression (CompressionPlan): placeholder for Zstd support; the default keeps payloads verbatim so canonical hashes stay stable.
• Encryption (EncryptionPlan): toggles AEAD support and drives the AAD plan, nonce sizing, and key identifiers.

These knobs live entirely in the runtime so storage backends simply persist whatever BlockFrame they receive.

Forward-compatibility guarantees

The manifest + frame format bakes in several durability promises:

• Versioned header – every frame begins with the FrameVersion byte. Future releases can bump this and still parse older frames because the version guards the decoder path.
• Extensible enums – FrameType, FrameAlgorithm, and related enums are schema-based; adding new cases does not change the binary layout of existing ones.
• Length-prefixed sections – both payload and AAD lengths live in the header, allowing readers to skip unfamiliar sections safely.
• Optional metadata – FrameAad.extra and manifest attributes can introduce new keys without invalidating older clients. Unknown keys are ignored while still being authenticated.
• Strict size accounting – BlockManifest.build refuses to produce manifests where totals drift, so deduped replay remains safe even if new attributes appear later.

Together these rules mean a v0.1 runtime can ingest data that future Graviton versions will continue to read, reframe, or mirror without re-uploading source bytes.

Validation and decoding flow

When a backend receives blocks from the ingest pipeline it performs the following steps:

1. Chunk bytes through BlockStore.putBlocks, deriving canonical hashes and building BlockManifestEntry values.
2. Run BlockFramer.synthesizeBlock for each canonical block. This enforces non-negative indexes, renders AAD, and ensures the header reflects the chosen write plan.
3. Persist the resulting BlockFrame alongside the manifest frame. Attributes are confirmed before the manifest is sealed so readers get the finalized metadata map.
4. During reads, load the manifest frame first, verify the header/AAD, and stream block frames via the offsets recorded in the manifest.

Because manifests, frames, and attributes all use the same refined types exposed from graviton-core, errors surface as Either[String, _] values instead of thrown exceptions. This keeps ingestion deterministic and makes it safe to run repair jobs or future frame migrations offline.

Related guides

• Binary Streaming Guide – how chunkers, block stores, and manifests interleave.
• Ingest Chunking – strategies for choosing block boundaries that still satisfy manifest invariants.