Choose the filesystem engine and recoverable mutation contract #2

Closed
opened 2026-10-11 19:07:21 +00:00 by phg · 7 comments
Owner

Parent

IODD Tool specification

Context

The confirmed scope is a native Swift app for macOS 15+ on Apple Silicon, all external FAT32/exFAT disks including fixed media, and IODD mini/newer image-capable devices. Fixed VHD creation requires a valid container and verified complete-file contiguous allocation. Current work is planning and ticket publication only; implementation is a later effort. The public volume-operation API against disposable images is the confirmed main testing boundary.

Type: grilling
Mode: HITL

Question

Which filesystem/access architecture and transaction protocol should the specification require for defrag, compaction, and contiguous VHD creation, including interruption and recovery?

Compare feasible native file APIs with an independent raw-volume engine and decide pure Swift versus a narrowly adapted, independently acquired filesystem component. Choose authorization/process boundaries, stable volume/file identities, snapshot and plan validation, ownership checks, journal location/schema, metadata before/after images, data verification, durable barriers, and source release ordering. Include FAT32 reserved bits/mirrored FAT/FSInfo and exFAT bitmap/NoFatChain/entry-set checksums/DataLength/ValidDataLength in the write contract.

Define busy-volume handling, remount failures, device removal/replacement, cancellation boundaries, helper/UI disconnection, interrupted file creation, incomplete journals, and the limits of recovery on a different Mac. Specify which failures halt a batch and which permit continuation. Decide what is guaranteed under host crashes and what still depends on USB power-loss testing. Ground choices in the reference's observed defects and prerequisite evidence.

Resolve the architecture and invariants with the user, not by turning a research recommendation into an accepted decision. Create an ADR only for choices that meet the domain-modeling skill's tradeoff criteria.

Blocked by

Record ownership

This Forgejo issue governs the live implementation prerequisite. The original local planning question and its history are retained; the implementation publication index links both records. This HITL decision remains open until the user resolves it. Publication does not resolve the original decision or approve production disk writes.

## Parent [IODD Tool specification](https://git.s1q.dev/phg/iodd-tool/issues/7) ## Context The confirmed scope is a native Swift app for macOS 15+ on Apple Silicon, all external FAT32/exFAT disks including fixed media, and IODD mini/newer image-capable devices. Fixed VHD creation requires a valid container and verified complete-file contiguous allocation. Current work is planning and ticket publication only; implementation is a later effort. The public volume-operation API against disposable images is the confirmed main testing boundary. Type: grilling Mode: HITL ## Question Which filesystem/access architecture and transaction protocol should the specification require for defrag, compaction, and contiguous VHD creation, including interruption and recovery? Compare feasible native file APIs with an independent raw-volume engine and decide pure Swift versus a narrowly adapted, independently acquired filesystem component. Choose authorization/process boundaries, stable volume/file identities, snapshot and plan validation, ownership checks, journal location/schema, metadata before/after images, data verification, durable barriers, and source release ordering. Include FAT32 reserved bits/mirrored FAT/FSInfo and exFAT bitmap/NoFatChain/entry-set checksums/DataLength/ValidDataLength in the write contract. Define busy-volume handling, remount failures, device removal/replacement, cancellation boundaries, helper/UI disconnection, interrupted file creation, incomplete journals, and the limits of recovery on a different Mac. Specify which failures halt a batch and which permit continuation. Decide what is guaranteed under host crashes and what still depends on USB power-loss testing. Ground choices in the reference's observed defects and prerequisite evidence. Resolve the architecture and invariants with the user, not by turning a research recommendation into an accepted decision. Create an ADR only for choices that meet the domain-modeling skill's tradeoff criteria. ## Blocked by - [Set the supported volume and device contract](https://git.s1q.dev/phg/iodd-tool/issues/1) ## Record ownership This Forgejo issue governs the live implementation prerequisite. The original local planning question and its history are retained; the implementation publication index links both records. This HITL decision remains open until the user resolves it. Publication does not resolve the original decision or approve production disk writes.
Author
Owner

The prerequisite Set the supported volume and device contract has been resolved with the user and closed. Its resolution comment governs the volume envelope and identity/operation-eligibility requirements; this engine decision is now unblocked and remains open.

Consult Supported volume and device geometry when comparing engines and I/O adapters. In particular, assess the component's advertised exFAT cluster ceiling against the agreed format envelope and preserve separate device/filesystem byte units. The research identifies constraints; it does not choose Swift versus a component, process boundaries, isolation, or recovery semantics.

The prerequisite [Set the supported volume and device contract](https://git.s1q.dev/phg/iodd-tool/issues/1) has been resolved with the user and closed. Its resolution comment governs the volume envelope and identity/operation-eligibility requirements; this engine decision is now unblocked and remains open. Consult [Supported volume and device geometry](https://git.s1q.dev/phg/iodd-tool/src/commit/5cc59dc8e8ea85e077d9dc08434c327a8f694eb9/docs/research/supported-volume-geometry.md) when comparing engines and I/O adapters. In particular, assess the component's advertised exFAT cluster ceiling against the agreed format envelope and preserve separate device/filesystem byte units. The research identifies constraints; it does not choose Swift versus a component, process boundaries, isolation, or recovery semantics.
phg self-assigned this 2026-10-12 10:09:18 +00:00
Author
Owner

Interview checkpoint — access, portability, and verification

phg agreed to all three recommendations in the first interview round on 12 October 2026:

  • Mutations use an independent raw-volume engine while the host volume is unmounted, with verification before remounting. The filesystem implementation and worker/isolation details remain open.
  • Recovery on another Mac may require a separately saved recovery bundle; disk-only recovery is not required. Keep durable evidence on the original Mac and support explicit export/import. Journal storage, discovery, schema, and matching rules remain open.
  • Every moved file requires complete destination readback and streamed content comparison before its original allocation is released. A verification failure stops the operation. Readback is a content check, not proof of physical power-loss durability.

This is an interview checkpoint, not the ticket resolution. The remaining engine, access ownership, transaction, recovery, and failure choices still require the user's answers and final shared-understanding check. Planning-only scope is unchanged.

## Interview checkpoint — access, portability, and verification phg agreed to all three recommendations in the first interview round on 12 October 2026: - Mutations use an independent raw-volume engine while the host volume is unmounted, with verification before remounting. The filesystem implementation and worker/isolation details remain open. - Recovery on another Mac may require a separately saved recovery bundle; disk-only recovery is not required. Keep durable evidence on the original Mac and support explicit export/import. Journal storage, discovery, schema, and matching rules remain open. - Every moved file requires complete destination readback and streamed content comparison before its original allocation is released. A verification failure stops the operation. Readback is a content check, not proof of physical power-loss durability. This is an interview checkpoint, not the ticket resolution. The remaining engine, access ownership, transaction, recovery, and failure choices still require the user's answers and final shared-understanding check. Planning-only scope is unchanged.
Author
Owner

Interview checkpoint — implementation, worker, and isolation

phg agreed to all three recommendations in the second interview round on 12 October 2026:

  • Build the independent filesystem engine in Swift, using a small Darwin/system-call bridge only where required. Do not adopt FatFs as the production filesystem engine.
  • Use an ordinary worker receiving an authorized device descriptor rather than installing a permanent privileged daemon. The worker owns the operation and Disk Arbitration session independently of the UI, reaches a safe boundary on UI disconnection, and retains ownership through cleanup. Its locks and claims do not survive worker death.
  • Temporarily unmount every volume on the same physical device for mutation and require qualified device-level exclusive access. Restrict all filesystem writes to the selected host volume's freshly validated byte range. Refuse when a sibling is busy or exclusivity cannot be established; restore prior mounts only after safe completion/recovery.

Auxiliary evidence: research/engine-recovery-constraints, commit 7160a61012, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; isolated worktree /tmp/fat-defrag-engine-recovery-constraints. The report is integrated as a new file in the main checkout. It uses official FatFs R0.16 source/patches and pinned Apple storage/Disk Arbitration source. No physical-media operations or helper installation occurred.

The report refines the earlier advertised FatFs exFAT limit: the 16-bit sectors-per-cluster representation rejects 32 MiB at 512-byte filesystem sectors; larger filesystem sectors fit the field but are not qualified. Active-FAT, metadata-preservation, and transaction-ownership gaps also argue against describing adoption as a narrow adapter.

This remains an interview checkpoint, not the resolution. Journal, metadata protocol, recovery conflicts/limits, and failure behavior remain open; planning-only authorization is unchanged.

## Interview checkpoint — implementation, worker, and isolation phg agreed to all three recommendations in the second interview round on 12 October 2026: - Build the independent filesystem engine in Swift, using a small Darwin/system-call bridge only where required. Do not adopt FatFs as the production filesystem engine. - Use an ordinary worker receiving an authorized device descriptor rather than installing a permanent privileged daemon. The worker owns the operation and Disk Arbitration session independently of the UI, reaches a safe boundary on UI disconnection, and retains ownership through cleanup. Its locks and claims do not survive worker death. - Temporarily unmount every volume on the same physical device for mutation and require qualified device-level exclusive access. Restrict all filesystem writes to the selected host volume's freshly validated byte range. Refuse when a sibling is busy or exclusivity cannot be established; restore prior mounts only after safe completion/recovery. Auxiliary evidence: research/engine-recovery-constraints, commit 7160a61012aadda8042d29bd2e860536c0bd5089, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; isolated worktree /tmp/fat-defrag-engine-recovery-constraints. The report is integrated as a new file in the main checkout. It uses official FatFs R0.16 source/patches and pinned Apple storage/Disk Arbitration source. No physical-media operations or helper installation occurred. The report refines the earlier advertised FatFs exFAT limit: the 16-bit sectors-per-cluster representation rejects 32 MiB at 512-byte filesystem sectors; larger filesystem sectors fit the field but are not qualified. Active-FAT, metadata-preservation, and transaction-ownership gaps also argue against describing adoption as a narrow adapter. This remains an interview checkpoint, not the resolution. Journal, metadata protocol, recovery conflicts/limits, and failure behavior remain open; planning-only authorization is unchanged.
Author
Owner

Interview checkpoint — journal, transaction boundary, and recovery limits

phg agreed to all three recommendations in the third interview round on 12 October 2026:

  • Store the authoritative recovery journal on the Mac's internal storage, with explicit self-contained recovery-bundle export/import. Do not require an app-specific target journal or marker in the first contract. Persist checksummed metadata before/after images before their writes and use the filesystem's native dirty indicator during mutation. Neither a native dirty indicator nor a target marker enforces a foreign OS mount prohibition.
  • Make each existing-file move a separate transaction: persist recovery evidence; copy into a distinct free allocation and verify the complete destination; persist and verify the metadata switch while retaining source allocation; persist the commit decision; release source allocation and verify the resulting filesystem state. Before commit recovery rolls back; after commit recovery finishes forward. A batch can remain partially complete. Cancellation reaches a safe boundary.
  • Promise recovery only under a validated I/O model, with intact journal evidence and no conflicting external changes. Refuse automatic replay for ambiguous identity, metadata, or journal integrity. Worker death removes process-held protection; physical power-loss claims require actual device/bridge tests.

This remains an interview checkpoint. Preview validity, batch error handling, interrupted creation, filesystem-specific invariants, durable barriers, and the final shared-understanding check remain to be completed. This does not authorize implementation or physical-media writes.

## Interview checkpoint — journal, transaction boundary, and recovery limits phg agreed to all three recommendations in the third interview round on 12 October 2026: - Store the authoritative recovery journal on the Mac's internal storage, with explicit self-contained recovery-bundle export/import. Do not require an app-specific target journal or marker in the first contract. Persist checksummed metadata before/after images before their writes and use the filesystem's native dirty indicator during mutation. Neither a native dirty indicator nor a target marker enforces a foreign OS mount prohibition. - Make each existing-file move a separate transaction: persist recovery evidence; copy into a distinct free allocation and verify the complete destination; persist and verify the metadata switch while retaining source allocation; persist the commit decision; release source allocation and verify the resulting filesystem state. Before commit recovery rolls back; after commit recovery finishes forward. A batch can remain partially complete. Cancellation reaches a safe boundary. - Promise recovery only under a validated I/O model, with intact journal evidence and no conflicting external changes. Refuse automatic replay for ambiguous identity, metadata, or journal integrity. Worker death removes process-held protection; physical power-loss claims require actual device/bridge tests. This remains an interview checkpoint. Preview validity, batch error handling, interrupted creation, filesystem-specific invariants, durable barriers, and the final shared-understanding check remain to be completed. This does not authorize implementation or physical-media writes.
Author
Owner

Interview checkpoint — analysis, stale plans, and batch failures

phg agreed to all three recommendations in the fourth interview round on 12 October 2026:

  • Full Analyze temporarily takes the device's volumes offline and uses read-only access for a stable allocation snapshot. Mounted observations are explicitly preliminary and cannot certify exact free runs or mutation eligibility. Analysis does not request write access or alter dirty state.
  • Reject stale previews rather than silently substituting a new plan. Rescan under exclusive access and renew the preview before materially changed work. Validate each file's expected entry metadata and allocation before starting its transaction; expected changes from this operation's previously verified transactions are tracked explicitly.
  • Continue a batch only for predictable per-file preflight skips such as already contiguous files or insufficient destination space. Halt on any I/O, journal, synchronization, content-verification, ownership, or target-identity failure, even after a successful rollback. Preserve completed outcomes and unresolved evidence.

Cancellation details, unpublished creation, IODD preparation and the final shared-understanding check remain. The selected-volume bounds and planning-only scope remain unchanged.

## Interview checkpoint — analysis, stale plans, and batch failures phg agreed to all three recommendations in the fourth interview round on 12 October 2026: - Full Analyze temporarily takes the device's volumes offline and uses read-only access for a stable allocation snapshot. Mounted observations are explicitly preliminary and cannot certify exact free runs or mutation eligibility. Analysis does not request write access or alter dirty state. - Reject stale previews rather than silently substituting a new plan. Rescan under exclusive access and renew the preview before materially changed work. Validate each file's expected entry metadata and allocation before starting its transaction; expected changes from this operation's previously verified transactions are tracked explicitly. - Continue a batch only for predictable per-file preflight skips such as already contiguous files or insufficient destination space. Halt on any I/O, journal, synchronization, content-verification, ownership, or target-identity failure, even after a successful rollback. Preserve completed outcomes and unresolved evidence. Cancellation details, unpublished creation, IODD preparation and the final shared-understanding check remain. The selected-volume bounds and planning-only scope remain unchanged.
Author
Owner

Interview checkpoint — cancellation, unpublished creation, and IODD preparation

phg agreed to all three recommendations in the fifth interview round on 12 October 2026:

  • Before commit, stop between data-I/O chunks and roll back tentative metadata/allocation changes. After commit, finish source release and verification before stopping. Keep earlier completed files. Use the same policy when the UI disconnects; failed cleanup remains recovery required. Process or OS death instead uses durable recovery evidence.
  • Creation follows the same journal and commit rules: initialize and verify the complete image before linking it into the directory; roll back uncommitted creation, and preserve committed verified output while finishing cleanup. Never report incomplete output as successful. VHD size, guest initialization, filename and overwrite choices remain for the creation-workflow ticket.
  • Before full offline analysis or mutation on IODD hardware, require fresh explicit confirmation that every hardware virtual drive is detached. Check available observations, block known active drives, and distinguish user-declared preparation from verified device state. macOS unmount is insufficient evidence; reconnection requires new preparation.

The interview's fifteen choices are agreed. The final shared-understanding check of the complete engineering contract remains pending; this checkpoint does not close the ticket or authorize implementation.

## Interview checkpoint — cancellation, unpublished creation, and IODD preparation phg agreed to all three recommendations in the fifth interview round on 12 October 2026: - Before commit, stop between data-I/O chunks and roll back tentative metadata/allocation changes. After commit, finish source release and verification before stopping. Keep earlier completed files. Use the same policy when the UI disconnects; failed cleanup remains recovery required. Process or OS death instead uses durable recovery evidence. - Creation follows the same journal and commit rules: initialize and verify the complete image before linking it into the directory; roll back uncommitted creation, and preserve committed verified output while finishing cleanup. Never report incomplete output as successful. VHD size, guest initialization, filename and overwrite choices remain for the creation-workflow ticket. - Before full offline analysis or mutation on IODD hardware, require fresh explicit confirmation that every hardware virtual drive is detached. Check available observations, block known active drives, and distinguish user-declared preparation from verified device state. macOS unmount is insufficient evidence; reconnection requires new preparation. The interview's fifteen choices are agreed. The final shared-understanding check of the complete engineering contract remains pending; this checkpoint does not close the ticket or authorize implementation.
Author
Owner

Resolution — filesystem engine and recoverable mutation contract

Confirmed by phg on 12 October 2026 through fifteen interview choices and the final shared-understanding check.

This records phg's fifteen agreed interview choices on 12 October 2026 and the engineering invariants that make them concrete. It governs the later implementation, not production execution. The supported format, geometry, layout, and identity envelope remains the resolution of Set the supported volume and device contract.

Engine and operation boundary

  • Independently implement the FAT32/exFAT reader, planner, metadata writer, and recovery engine in Swift. Use a small Darwin/C bridge where system calls require one. Do not import recovered proprietary code or adopt FatFs as the production engine.
  • Mounted file APIs may supply explicitly preliminary observations. Full Analyze, defragmentation, compaction, and contiguous fixed-VHD creation use the independent engine while the device's host volumes are offline. Full Analyze opens read-only, does not dirty the filesystem, and does not request write authority. Bounded diagnostics may remain partial when full consistent coverage cannot be established.
  • Keep the public volume-operation API as the primary boundary: analyze, preview, execute, cancel, and recover expose target-bound snapshots, plans, progress, per-file outcomes, and recovery/remount results. Disposable-image and physical-device adapters serve that same boundary. The UI does not own filesystem state or perform sector writes. Exact Swift signatures and private implementation layout are later implementation details.
  • The language choice accepts independent implementation work in exchange for explicit ownership of preservation, placement, and transaction ordering. FatFs has reusable primitives, but its exact source needs substantive changes for the agreed envelope, active FATs, entry preservation, and transaction ownership; generic mounted copy/replace cannot satisfy identity-preserving relocation and chosen placement.

Authorization, worker lifetime, and device isolation

  • Use a bundled ordinary worker, not a permanent privileged daemon. Acquire the selected current device descriptor through Apple's narrowly authorized authopen descriptor-transfer route, using explicit access/open flags and a Unix-domain socket capable of transferring the descriptor. Do not use the convenience truncating open option or deprecated privilege-execution APIs. The worker remains an ordinary process even though its descriptor grants broad device access.
  • Authenticate UI/worker IPC using baseline-compatible signing requirements and a bounded high-level protocol. Worker-side discovery and fresh validation determine the descriptor and allowed volume range; a UI-supplied path, offset, serialized plan, or imported bundle never independently authorizes a raw write. Signing, notarization, standard-user authorization and descriptor-transfer behavior need later platform qualification; no helper is installed in this effort.
  • The worker owns the Disk Arbitration session, claims, mount-approval interlock, device descriptor, journal and transaction independently of the UI. Only one operation owns a physical device at a time, and only one file transaction is active in that operation. UI disconnection requests a safe stop; it does not immediately destroy the storage owner.
  • Capture the original mount state of every sibling volume. Request ordinary unmounts and await success; never force a busy volume offline. Acquire qualified whole-device exclusive access that covers the relevant raw/block/whole/slice aliases. A selected-slice lock or a Disk Arbitration claim alone is insufficient. Failure to obtain this access disables the operation with a specific reason.
  • Access to the whole device is required for isolation; all filesystem reads/writes remain bounded to the selected volume's validated byte range, apart from read-only layout/identity inspection. Partition tables and sibling contents are not mutation targets. Bounds also apply to expanded/aligned read-modify-write ranges; reject an adapter mapping that would write beyond the selected range.
  • Require fresh IODD preparation confirmation before full offline analysis or mutation: every hardware virtual drive detached, known active drives blocked, available observations checked, and user-declared versus verified state recorded separately. An OS unmount is not hardware detachment. A new attachment requires new preparation. Unknown model/firmware qualification alone does not block otherwise eligible ordinary filesystem work.
  • Claims and descriptor exclusivity end after worker/session death. There is no persistent automount quarantine, including across a reboot or transfer to another Mac. While alive, the worker keeps unsafe/recovery-required storage offline and reports that state; it must not present this as protection that survives its death.

Target, snapshot, plan, and ownership invariants

  • Combine current device/partition evidence, validated volume origin/length, geometry, filesystem fingerprints and attachment generation. Validate the opened descriptor against fresh discovery evidence. Labels, mount paths, reused BSD names, or a single clonable UUID cannot establish identity.
  • Full snapshots verify complete allocation ownership, directory coverage, and supported metadata semantics, including hidden/nested entries and filesystem-owned allocations. Loops, cross-links, out-of-range references, missing coverage or unknown allocation ownership prevent mutation. Understood but non-movable structures are pinned; unknown structures are never treated as free space.
  • Bind previews to the snapshot and attachment generation. Before execution, rescan under exclusive access and compare all relevant plan preconditions. Unexpected changes invalidate the plan; materially changed work needs a new preview, not silent replanning. Track expected allocation changes caused by this operation's previously verified transactions explicitly.
  • Identify an existing file through its directory-entry location/set and expected bytes, parent context, name, lengths, allocation form and complete chain/extents. A path alone is insufficient. Preserve its existing logical directory identity, path, names, attributes, timestamps and lengths; this is not a promise of a universal persistent macOS inode identifier that these formats do not provide.
  • Immediately before each transaction, recheck the source entry and complete allocation, candidate destination freedom, disjoint source/destination ranges, selected-volume bounds, and ownership. Budget retained source allocation, complete destination allocation, directory growth and metadata/journal workspace. Do not perform overlapping/in-place copy cycles or move directory structures as compaction targets.

Journal location and semantic schema

  • Keep the authoritative journal in private application-owned storage on the Mac's internal disk, independently of the target. Do not require an app-specific target sidecar/marker or occupy assumed spare/reserved sectors. Use native dirty indicators during mutation; these indicate inconsistency, not a foreign OS mount veto.
  • Before any target change, durably create the operation journal and its discovery entry. Refuse mutation if internal storage, required synchronization or budgeted journal space is unavailable. Persist every metadata patch's full before/after images before issuing that patch; no target metadata write may bypass the journalled path.
  • Use a versioned, bounded, checksummed append-only record format. The semantic header contains format/engine version, operation and transaction identities, device/partition identity evidence, selected range, independent geometry units, filesystem revision/variant, original mount/dirty/error state, and snapshot/plan fingerprints.
  • Transaction records contain operation kind, expected file entry/set and allocation, source/destination extents and lengths, logical content digests, directory/allocation ownership preconditions, and ordered patch records. Each patch has selected-volume-relative byte offset, length, complete preserved write-unit images, image digests, and its phase/sequence. Log successive images where one unit changes in multiple phases. Frame lengths, sequence and integrity checks must be validated before using any record or allocating buffers.
  • Durable phase records distinguish preparation, verified destination, metadata switched, commit decision, source release, final verification, rollback/recovery progress and conclusive completion. The commit decision is the recovery direction boundary, not merely the last attempted write. Interrupted replay is itself logged and idempotent.
  • Hash logical file contents using streamed SHA-256 and identify the digest algorithm in the journal version/schema. The journal does not require a second full copy of existing file payload: source allocation stays intact until the verified new version is durably committed. An integrity checksum is not authority to overwrite an arbitrary volume.
  • Discover unresolved work through the local journal registry or an explicitly imported recovery bundle before starting another mutation. Never reuse/overwrite an unresolved journal for the next file or operation. Archive/retire a journal only after a conclusively verified result; preserve unresolved evidence.
  • Export a coherent self-contained bundle containing the header, records, images, digests, versions and outcomes needed for recovery, not pointers to files on the original Mac. Freeze the relevant operation/evidence during export. Another Mac may require that bundle; disk-alone recovery and knowledge of every foreign host's journals are not promised.

Existing-file transaction and durability ordering

  1. With ownership established, persist the prepared transaction, all required metadata images/preconditions, and the native dirty-state transition before target modifications. Retain the original file's allocation and contents.
  2. Copy to a separate free allocation. The journal/worker tracks reserved destination ownership; no existing file may claim that allocation. Initialize the complete logical destination, synchronize target data, read the complete destination back, and compare it with the source's streamed logical-content digest. Verify the full allocation coverage, not a capped map or payload-only range.
  3. Install the destination allocation metadata and then switch the existing directory entry/set to it, following format-specific ordering. Synchronize and read back the changed metadata; verify the live file, preserved metadata, destination ownership and content. The original allocation remains reserved and unreused even though the live entry now points to the destination.
  4. Persist and synchronize the commit decision on the Mac. No source release is allowed merely because the entry switch was attempted or because a commit frame was buffered.
  5. Release the original allocation through logged metadata patches, update free-space hints/accounting, synchronize and verify the resulting ownership state. Before commit, recovery restores the original live entry/allocation and removes destination reservations; after commit, recovery preserves the new live file and finishes source release. Never replay original allocation back into a cluster already released/reused after commit.
  6. Persist conclusive per-file verification/outcome. Verification uses fresh storage reads and a verifier independent of the placement/write bookkeeping; external test oracles are separately implemented. At the operation's safe end, perform a fresh allocation/consistency scan, durably restore the permitted native clean state, and record completion before releasing access and restoring mounts.

The internal journal and target are separate durability domains. Use checked full-persistence operations for the journal file and its discovery/create/rename lifecycle. For raw target I/O, use the qualified device cache-synchronization adapter and verify errors; raw-character-device fsync is not a durability barrier. An ordering-only barrier is not proof that data has persisted. Short I/O must be completed exactly or reported as failure; no ignored write/flush result or unsupported-success fallback is allowed. Geometry conversions use checked byte arithmetic; physical block size informs alignment without promising atomic writes. A reported content readback does not establish physical power-loss durability.

Filesystem-specific metadata contract

FAT32: interpret allocation in the low 28 FAT bits and preserve each entry's reserved upper bits. Honor mirroring versus the selected active FAT; journal/check every required mirror write and preserve inactive copies in active-only mode. Check the complete old/new chains and directory entry, including entries in later or spanning I/O units; preserve names, lengths, attributes and timestamps. FSInfo signatures and hints are interpreted safely; actual ownership determines free space, and affected valid hints are updated or marked unknown. Log dirty/clean transitions and preserve unrelated error/reserved fields; do not silently repair contradictory allocation or mirror state. Microsoft FAT specification.

exFAT: the allocation bitmap owns allocation; do not treat stale FAT values as ownership for NoFatChain streams. Preserve logical contents, DataLength and ValidDataLength, returning/comparing zeros beyond ValidDataLength. Validate complete entry sets/checksums and preserve supported secondary entries and unrelated fields. Maintain bitmap/FAT allocation forms coherently; initialize/validate a chain before clearing NoFatChain, or establish contiguous ownership before setting it. Log bitmap, FAT and spanning entry-set changes; maintain checksum and dirty state in format order, without clearing unrelated error state or rewriting backup/OEM data unnecessarily. Unknown allocation semantics block mutation. Microsoft exFAT specification.

These rules correct the reference's source-release-before-directory ordering, mishandled FAT bits, missed later directory writes, ignored persistence failures, and overwritten recovery evidence. They are independently specified invariants, not reuse of the reference implementation.

Creation, cancellation, and batch outcomes

  • Contiguous creation is a journalled allocation/publication transaction under the same ownership and commit rules. Initialize and verify the complete image, including its fixed footer and agreed guest contents, before linking its allocation into the directory. Verify the live complete file's one-extent allocation and format before committing/reporting success. Directory growth and other required metadata workspace are also logged.
  • Before commit, recover/abort by removing tentative publication and releasing only allocation owned by that creation. After commit, keep the verified file and finish cleanup. A crash during metadata publication may leave a tentative entry until recovery; no blanket promise that a foreign OS cannot observe it is made. Neither incomplete bytes nor a mere successful reservation is a successful VHD.
  • VHD units, accepted capacity/rounding, guest initialization, final filename/destination, overwrite policy, insufficient-run workflow and any existing-file movement remain for Define fixed VHD creation and allocation failure behavior. This engine contract does not choose those policies or permit silent overwriting.
  • Before commit, Stop/UI disconnection is observed between data-I/O chunks and causes rollback; after commit, finish source release and verification, then stop. Do not abandon an in-progress metadata write/flush at an unsafe point. Earlier committed files remain completed. Failed rollback/cleanup leaves recovery required. Process/host death instead follows durable replay.
  • Continue a batch only after predictable, non-mutating per-file preflight skips. Already contiguous/empty files and insufficient destination space may be reported as skips; selection/priorities remain for Define analysis, defrag, and compaction success criteria. Halt on any actual I/O, journal, synchronization, content-verification, ownership, or identity failure, even after successful rollback. Record completed, skipped, failed, cancelled and recovery-required outcomes individually.

Recovery, removal, and remount outcomes

  • Removal/replacement invalidates the attachment, pending plans and queued writes. Do not reopen a reused pathname or automatically resume on reconnection. Retain evidence, establish a new attachment, repeat identity/geometry/preparation/consistency checks, and require explicit recovery for interrupted work.
  • Recovery acquires the same isolation, validates journal framing/version/integrity and freshly matches the target. Verify relevant ownership, unchanged surrounding metadata, and the original or committed file's required content evidence before patching. Match admissible before/intermediate/after states against the logged sequence. Proven interrupted writes within the qualified fault model may be restored/replayed; conflicting or ambiguous changes are refused, not overwritten with stale sector images.
  • A truncated/corrupt journal is not proof that no commit or write happened. Ignore an incomplete final append only when the intact evidence plus storage state establishes a safe permitted recovery direction; otherwise stop with retained evidence. Missing essential images, ambiguous identity, unsupported journal versions, or missing required content block replay; a user's request to recover does not waive these checks. Repeated recovery must preserve the same direction and completed outcomes.
  • Promise recovery from host/worker interruption only under the validated I/O model, intact durable evidence and nonconflicting target state. A foreign mount/write/repair may invalidate those preconditions; native dirty bits and a user workflow do not enforce persistent quarantine. Loss of the original Mac and all recovery bundles is outside this guarantee. General filesystem repair is not included.
  • Keep unsafe storage offline while the live worker's interlock remains effective. Restore only the previously mounted volumes, after conclusive operation/recovery verification and correct identity. Close raw access and hand back mount ownership in the qualified lifecycle order. Do not force a mount or report a recovery-required target as ready for Finder. Report remount failure separately from the durable file-operation result; a verified file can coexist with a failed remount.

Validation and remaining route

Image-backed public-operation tests with independent content/allocation/VHD oracles must exercise the complete agreed geometry, metadata preservation, each durable boundary, short/failing I/O, lost/reordered/torn writes, cancellation, repeated replay, journal truncation/corruption, target replacement and external modification. Tests may verify safe rejection when recovery preconditions are unavailable; successful filesystem-checker repair is not a passing operation result.

Actual macOS whole/slice/raw/block/sibling exclusion, worker death and mount behavior, standard-user authorization, and device synchronization need platform qualification. Physical power-loss claims require separately authorized disposable-device tests of actual device/bridge configurations. No such row is qualified by this planning decision. Exact release gates, performance expectations and implementation staging remain for Set validation gates and the implementation handoff; recovery UI feedback belongs in Validate the native analysis and VHD creation workflows.

No additional decision ticket is needed: newly precise work fits the existing creation, analysis/compaction, workflow and validation tickets. Persistent foreign-OS quarantine, disk-alone recovery, directory-relocation/in-place algorithms, host repair, production implementation and physical-media trials are outside this chosen first contract/current planning effort.

Evidence and ownership

  • Filesystem engine and recovery constraints: isolated research/engine-recovery-constraints branch, commit 7160a61012, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; worktree /tmp/fat-defrag-engine-recovery-constraints. The research branch is published and the report is integrated into the main checkout; the hosted link is pinned to its evidence commit.
  • Retained evidence: docs/research/macos-volume-access.md, docs/research/supported-volume-geometry.md, docs/research/iodd-compatibility.md, docs/research/fixed-vhd-format.md and docs/reference-baseline.md. Apple source evidence is not certification of every installed driver or macOS build.
  • This hosted resolution is authoritative. Parent/map entries contain a named link and short gist; local planning records preserve their original question/history and point here. Compact ADRs may explain the non-obvious Swift-engine and internal-journal tradeoffs, without becoming competing copies of this contract.
## Resolution — filesystem engine and recoverable mutation contract Confirmed by phg on 12 October 2026 through fifteen interview choices and the final shared-understanding check. This records phg's fifteen agreed interview choices on 12 October 2026 and the engineering invariants that make them concrete. It governs the later implementation, not production execution. The supported format, geometry, layout, and identity envelope remains the resolution of [Set the supported volume and device contract](https://git.s1q.dev/phg/iodd-tool/issues/1). ### Engine and operation boundary - Independently implement the FAT32/exFAT reader, planner, metadata writer, and recovery engine in Swift. Use a small Darwin/C bridge where system calls require one. Do not import recovered proprietary code or adopt FatFs as the production engine. - Mounted file APIs may supply explicitly preliminary observations. Full Analyze, defragmentation, compaction, and contiguous fixed-VHD creation use the independent engine while the device's host volumes are offline. Full Analyze opens read-only, does not dirty the filesystem, and does not request write authority. Bounded diagnostics may remain partial when full consistent coverage cannot be established. - Keep the public volume-operation API as the primary boundary: analyze, preview, execute, cancel, and recover expose target-bound snapshots, plans, progress, per-file outcomes, and recovery/remount results. Disposable-image and physical-device adapters serve that same boundary. The UI does not own filesystem state or perform sector writes. Exact Swift signatures and private implementation layout are later implementation details. - The language choice accepts independent implementation work in exchange for explicit ownership of preservation, placement, and transaction ordering. FatFs has reusable primitives, but its exact source needs substantive changes for the agreed envelope, active FATs, entry preservation, and transaction ownership; generic mounted copy/replace cannot satisfy identity-preserving relocation and chosen placement. ### Authorization, worker lifetime, and device isolation - Use a bundled ordinary worker, not a permanent privileged daemon. Acquire the selected current device descriptor through Apple's narrowly authorized `authopen` descriptor-transfer route, using explicit access/open flags and a Unix-domain socket capable of transferring the descriptor. Do not use the convenience truncating open option or deprecated privilege-execution APIs. The worker remains an ordinary process even though its descriptor grants broad device access. - Authenticate UI/worker IPC using baseline-compatible signing requirements and a bounded high-level protocol. Worker-side discovery and fresh validation determine the descriptor and allowed volume range; a UI-supplied path, offset, serialized plan, or imported bundle never independently authorizes a raw write. Signing, notarization, standard-user authorization and descriptor-transfer behavior need later platform qualification; no helper is installed in this effort. - The worker owns the Disk Arbitration session, claims, mount-approval interlock, device descriptor, journal and transaction independently of the UI. Only one operation owns a physical device at a time, and only one file transaction is active in that operation. UI disconnection requests a safe stop; it does not immediately destroy the storage owner. - Capture the original mount state of every sibling volume. Request ordinary unmounts and await success; never force a busy volume offline. Acquire qualified whole-device exclusive access that covers the relevant raw/block/whole/slice aliases. A selected-slice lock or a Disk Arbitration claim alone is insufficient. Failure to obtain this access disables the operation with a specific reason. - Access to the whole device is required for isolation; all filesystem reads/writes remain bounded to the selected volume's validated byte range, apart from read-only layout/identity inspection. Partition tables and sibling contents are not mutation targets. Bounds also apply to expanded/aligned read-modify-write ranges; reject an adapter mapping that would write beyond the selected range. - Require fresh IODD preparation confirmation before full offline analysis or mutation: every hardware virtual drive detached, known active drives blocked, available observations checked, and user-declared versus verified state recorded separately. An OS unmount is not hardware detachment. A new attachment requires new preparation. Unknown model/firmware qualification alone does not block otherwise eligible ordinary filesystem work. - Claims and descriptor exclusivity end after worker/session death. There is no persistent automount quarantine, including across a reboot or transfer to another Mac. While alive, the worker keeps unsafe/recovery-required storage offline and reports that state; it must not present this as protection that survives its death. ### Target, snapshot, plan, and ownership invariants - Combine current device/partition evidence, validated volume origin/length, geometry, filesystem fingerprints and attachment generation. Validate the opened descriptor against fresh discovery evidence. Labels, mount paths, reused BSD names, or a single clonable UUID cannot establish identity. - Full snapshots verify complete allocation ownership, directory coverage, and supported metadata semantics, including hidden/nested entries and filesystem-owned allocations. Loops, cross-links, out-of-range references, missing coverage or unknown allocation ownership prevent mutation. Understood but non-movable structures are pinned; unknown structures are never treated as free space. - Bind previews to the snapshot and attachment generation. Before execution, rescan under exclusive access and compare all relevant plan preconditions. Unexpected changes invalidate the plan; materially changed work needs a new preview, not silent replanning. Track expected allocation changes caused by this operation's previously verified transactions explicitly. - Identify an existing file through its directory-entry location/set and expected bytes, parent context, name, lengths, allocation form and complete chain/extents. A path alone is insufficient. Preserve its existing logical directory identity, path, names, attributes, timestamps and lengths; this is not a promise of a universal persistent macOS inode identifier that these formats do not provide. - Immediately before each transaction, recheck the source entry and complete allocation, candidate destination freedom, disjoint source/destination ranges, selected-volume bounds, and ownership. Budget retained source allocation, complete destination allocation, directory growth and metadata/journal workspace. Do not perform overlapping/in-place copy cycles or move directory structures as compaction targets. ### Journal location and semantic schema - Keep the authoritative journal in private application-owned storage on the Mac's internal disk, independently of the target. Do not require an app-specific target sidecar/marker or occupy assumed spare/reserved sectors. Use native dirty indicators during mutation; these indicate inconsistency, not a foreign OS mount veto. - Before any target change, durably create the operation journal and its discovery entry. Refuse mutation if internal storage, required synchronization or budgeted journal space is unavailable. Persist every metadata patch's full before/after images before issuing that patch; no target metadata write may bypass the journalled path. - Use a versioned, bounded, checksummed append-only record format. The semantic header contains format/engine version, operation and transaction identities, device/partition identity evidence, selected range, independent geometry units, filesystem revision/variant, original mount/dirty/error state, and snapshot/plan fingerprints. - Transaction records contain operation kind, expected file entry/set and allocation, source/destination extents and lengths, logical content digests, directory/allocation ownership preconditions, and ordered patch records. Each patch has selected-volume-relative byte offset, length, complete preserved write-unit images, image digests, and its phase/sequence. Log successive images where one unit changes in multiple phases. Frame lengths, sequence and integrity checks must be validated before using any record or allocating buffers. - Durable phase records distinguish preparation, verified destination, metadata switched, commit decision, source release, final verification, rollback/recovery progress and conclusive completion. The commit decision is the recovery direction boundary, not merely the last attempted write. Interrupted replay is itself logged and idempotent. - Hash logical file contents using streamed SHA-256 and identify the digest algorithm in the journal version/schema. The journal does not require a second full copy of existing file payload: source allocation stays intact until the verified new version is durably committed. An integrity checksum is not authority to overwrite an arbitrary volume. - Discover unresolved work through the local journal registry or an explicitly imported recovery bundle before starting another mutation. Never reuse/overwrite an unresolved journal for the next file or operation. Archive/retire a journal only after a conclusively verified result; preserve unresolved evidence. - Export a coherent self-contained bundle containing the header, records, images, digests, versions and outcomes needed for recovery, not pointers to files on the original Mac. Freeze the relevant operation/evidence during export. Another Mac may require that bundle; disk-alone recovery and knowledge of every foreign host's journals are not promised. ### Existing-file transaction and durability ordering 1. With ownership established, persist the prepared transaction, all required metadata images/preconditions, and the native dirty-state transition before target modifications. Retain the original file's allocation and contents. 2. Copy to a separate free allocation. The journal/worker tracks reserved destination ownership; no existing file may claim that allocation. Initialize the complete logical destination, synchronize target data, read the complete destination back, and compare it with the source's streamed logical-content digest. Verify the full allocation coverage, not a capped map or payload-only range. 3. Install the destination allocation metadata and then switch the existing directory entry/set to it, following format-specific ordering. Synchronize and read back the changed metadata; verify the live file, preserved metadata, destination ownership and content. The original allocation remains reserved and unreused even though the live entry now points to the destination. 4. Persist and synchronize the commit decision on the Mac. No source release is allowed merely because the entry switch was attempted or because a commit frame was buffered. 5. Release the original allocation through logged metadata patches, update free-space hints/accounting, synchronize and verify the resulting ownership state. Before commit, recovery restores the original live entry/allocation and removes destination reservations; after commit, recovery preserves the new live file and finishes source release. Never replay original allocation back into a cluster already released/reused after commit. 6. Persist conclusive per-file verification/outcome. Verification uses fresh storage reads and a verifier independent of the placement/write bookkeeping; external test oracles are separately implemented. At the operation's safe end, perform a fresh allocation/consistency scan, durably restore the permitted native clean state, and record completion before releasing access and restoring mounts. The internal journal and target are separate durability domains. Use checked full-persistence operations for the journal file and its discovery/create/rename lifecycle. For raw target I/O, use the qualified device cache-synchronization adapter and verify errors; raw-character-device `fsync` is not a durability barrier. An ordering-only barrier is not proof that data has persisted. Short I/O must be completed exactly or reported as failure; no ignored write/flush result or unsupported-success fallback is allowed. Geometry conversions use checked byte arithmetic; physical block size informs alignment without promising atomic writes. A reported content readback does not establish physical power-loss durability. ### Filesystem-specific metadata contract **FAT32:** interpret allocation in the low 28 FAT bits and preserve each entry's reserved upper bits. Honor mirroring versus the selected active FAT; journal/check every required mirror write and preserve inactive copies in active-only mode. Check the complete old/new chains and directory entry, including entries in later or spanning I/O units; preserve names, lengths, attributes and timestamps. FSInfo signatures and hints are interpreted safely; actual ownership determines free space, and affected valid hints are updated or marked unknown. Log dirty/clean transitions and preserve unrelated error/reserved fields; do not silently repair contradictory allocation or mirror state. [Microsoft FAT specification](https://www.cs.fsu.edu/~cop4610t/assignments/project3/spec/fatspec.pdf). **exFAT:** the allocation bitmap owns allocation; do not treat stale FAT values as ownership for `NoFatChain` streams. Preserve logical contents, DataLength and ValidDataLength, returning/comparing zeros beyond ValidDataLength. Validate complete entry sets/checksums and preserve supported secondary entries and unrelated fields. Maintain bitmap/FAT allocation forms coherently; initialize/validate a chain before clearing `NoFatChain`, or establish contiguous ownership before setting it. Log bitmap, FAT and spanning entry-set changes; maintain checksum and dirty state in format order, without clearing unrelated error state or rewriting backup/OEM data unnecessarily. Unknown allocation semantics block mutation. [Microsoft exFAT specification](https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification). These rules correct the reference's source-release-before-directory ordering, mishandled FAT bits, missed later directory writes, ignored persistence failures, and overwritten recovery evidence. They are independently specified invariants, not reuse of the reference implementation. ### Creation, cancellation, and batch outcomes - Contiguous creation is a journalled allocation/publication transaction under the same ownership and commit rules. Initialize and verify the complete image, including its fixed footer and agreed guest contents, before linking its allocation into the directory. Verify the live complete file's one-extent allocation and format before committing/reporting success. Directory growth and other required metadata workspace are also logged. - Before commit, recover/abort by removing tentative publication and releasing only allocation owned by that creation. After commit, keep the verified file and finish cleanup. A crash during metadata publication may leave a tentative entry until recovery; no blanket promise that a foreign OS cannot observe it is made. Neither incomplete bytes nor a mere successful reservation is a successful VHD. - VHD units, accepted capacity/rounding, guest initialization, final filename/destination, overwrite policy, insufficient-run workflow and any existing-file movement remain for [Define fixed VHD creation and allocation failure behavior](https://git.s1q.dev/phg/iodd-tool/issues/3). This engine contract does not choose those policies or permit silent overwriting. - Before commit, Stop/UI disconnection is observed between data-I/O chunks and causes rollback; after commit, finish source release and verification, then stop. Do not abandon an in-progress metadata write/flush at an unsafe point. Earlier committed files remain completed. Failed rollback/cleanup leaves recovery required. Process/host death instead follows durable replay. - Continue a batch only after predictable, non-mutating per-file preflight skips. Already contiguous/empty files and insufficient destination space may be reported as skips; selection/priorities remain for [Define analysis, defrag, and compaction success criteria](https://git.s1q.dev/phg/iodd-tool/issues/4). Halt on any actual I/O, journal, synchronization, content-verification, ownership, or identity failure, even after successful rollback. Record completed, skipped, failed, cancelled and recovery-required outcomes individually. ### Recovery, removal, and remount outcomes - Removal/replacement invalidates the attachment, pending plans and queued writes. Do not reopen a reused pathname or automatically resume on reconnection. Retain evidence, establish a new attachment, repeat identity/geometry/preparation/consistency checks, and require explicit recovery for interrupted work. - Recovery acquires the same isolation, validates journal framing/version/integrity and freshly matches the target. Verify relevant ownership, unchanged surrounding metadata, and the original or committed file's required content evidence before patching. Match admissible before/intermediate/after states against the logged sequence. Proven interrupted writes within the qualified fault model may be restored/replayed; conflicting or ambiguous changes are refused, not overwritten with stale sector images. - A truncated/corrupt journal is not proof that no commit or write happened. Ignore an incomplete final append only when the intact evidence plus storage state establishes a safe permitted recovery direction; otherwise stop with retained evidence. Missing essential images, ambiguous identity, unsupported journal versions, or missing required content block replay; a user's request to recover does not waive these checks. Repeated recovery must preserve the same direction and completed outcomes. - Promise recovery from host/worker interruption only under the validated I/O model, intact durable evidence and nonconflicting target state. A foreign mount/write/repair may invalidate those preconditions; native dirty bits and a user workflow do not enforce persistent quarantine. Loss of the original Mac and all recovery bundles is outside this guarantee. General filesystem repair is not included. - Keep unsafe storage offline while the live worker's interlock remains effective. Restore only the previously mounted volumes, after conclusive operation/recovery verification and correct identity. Close raw access and hand back mount ownership in the qualified lifecycle order. Do not force a mount or report a recovery-required target as ready for Finder. Report remount failure separately from the durable file-operation result; a verified file can coexist with a failed remount. ### Validation and remaining route Image-backed public-operation tests with independent content/allocation/VHD oracles must exercise the complete agreed geometry, metadata preservation, each durable boundary, short/failing I/O, lost/reordered/torn writes, cancellation, repeated replay, journal truncation/corruption, target replacement and external modification. Tests may verify safe rejection when recovery preconditions are unavailable; successful filesystem-checker repair is not a passing operation result. Actual macOS whole/slice/raw/block/sibling exclusion, worker death and mount behavior, standard-user authorization, and device synchronization need platform qualification. Physical power-loss claims require separately authorized disposable-device tests of actual device/bridge configurations. No such row is qualified by this planning decision. Exact release gates, performance expectations and implementation staging remain for [Set validation gates and the implementation handoff](https://git.s1q.dev/phg/iodd-tool/issues/6); recovery UI feedback belongs in [Validate the native analysis and VHD creation workflows](https://git.s1q.dev/phg/iodd-tool/issues/5). No additional decision ticket is needed: newly precise work fits the existing creation, analysis/compaction, workflow and validation tickets. Persistent foreign-OS quarantine, disk-alone recovery, directory-relocation/in-place algorithms, host repair, production implementation and physical-media trials are outside this chosen first contract/current planning effort. ### Evidence and ownership - [Filesystem engine and recovery constraints](https://git.s1q.dev/phg/iodd-tool/src/commit/7160a61012aadda8042d29bd2e860536c0bd5089/docs/research/engine-recovery-constraints.md): isolated research/engine-recovery-constraints branch, commit 7160a61012aadda8042d29bd2e860536c0bd5089, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; worktree /tmp/fat-defrag-engine-recovery-constraints. The research branch is published and the report is integrated into the main checkout; the hosted link is pinned to its evidence commit. - Retained evidence: docs/research/macos-volume-access.md, docs/research/supported-volume-geometry.md, docs/research/iodd-compatibility.md, docs/research/fixed-vhd-format.md and docs/reference-baseline.md. Apple source evidence is not certification of every installed driver or macOS build. - This hosted resolution is authoritative. Parent/map entries contain a named link and short gist; local planning records preserve their original question/history and point here. Compact ADRs may explain the non-obvious Swift-engine and internal-journal tradeoffs, without becoming competing copies of this contract.
phg closed this issue 2026-10-12 11:01:49 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
phg/iodd-tool#2
No description provided.