Choose the filesystem engine and recoverable mutation contract #2
Labels
No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
phg/iodd-tool#2
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent
IODD Tool specification
Context
The confirmed scope is a native Swift app for macOS 15+ on Apple Silicon, all external FAT32/exFAT disks including fixed media, and IODD mini/newer image-capable devices. Fixed VHD creation requires a valid container and verified complete-file contiguous allocation. Current work is planning and ticket publication only; implementation is a later effort. The public volume-operation API against disposable images is the confirmed main testing boundary.
Type: grilling
Mode: HITL
Question
Which filesystem/access architecture and transaction protocol should the specification require for defrag, compaction, and contiguous VHD creation, including interruption and recovery?
Compare feasible native file APIs with an independent raw-volume engine and decide pure Swift versus a narrowly adapted, independently acquired filesystem component. Choose authorization/process boundaries, stable volume/file identities, snapshot and plan validation, ownership checks, journal location/schema, metadata before/after images, data verification, durable barriers, and source release ordering. Include FAT32 reserved bits/mirrored FAT/FSInfo and exFAT bitmap/NoFatChain/entry-set checksums/DataLength/ValidDataLength in the write contract.
Define busy-volume handling, remount failures, device removal/replacement, cancellation boundaries, helper/UI disconnection, interrupted file creation, incomplete journals, and the limits of recovery on a different Mac. Specify which failures halt a batch and which permit continuation. Decide what is guaranteed under host crashes and what still depends on USB power-loss testing. Ground choices in the reference's observed defects and prerequisite evidence.
Resolve the architecture and invariants with the user, not by turning a research recommendation into an accepted decision. Create an ADR only for choices that meet the domain-modeling skill's tradeoff criteria.
Blocked by
Record ownership
This Forgejo issue governs the live implementation prerequisite. The original local planning question and its history are retained; the implementation publication index links both records. This HITL decision remains open until the user resolves it. Publication does not resolve the original decision or approve production disk writes.
The prerequisite Set the supported volume and device contract has been resolved with the user and closed. Its resolution comment governs the volume envelope and identity/operation-eligibility requirements; this engine decision is now unblocked and remains open.
Consult Supported volume and device geometry when comparing engines and I/O adapters. In particular, assess the component's advertised exFAT cluster ceiling against the agreed format envelope and preserve separate device/filesystem byte units. The research identifies constraints; it does not choose Swift versus a component, process boundaries, isolation, or recovery semantics.
Interview checkpoint — access, portability, and verification
phg agreed to all three recommendations in the first interview round on 12 October 2026:
This is an interview checkpoint, not the ticket resolution. The remaining engine, access ownership, transaction, recovery, and failure choices still require the user's answers and final shared-understanding check. Planning-only scope is unchanged.
Interview checkpoint — implementation, worker, and isolation
phg agreed to all three recommendations in the second interview round on 12 October 2026:
Auxiliary evidence: research/engine-recovery-constraints, commit
7160a61012, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; isolated worktree /tmp/fat-defrag-engine-recovery-constraints. The report is integrated as a new file in the main checkout. It uses official FatFs R0.16 source/patches and pinned Apple storage/Disk Arbitration source. No physical-media operations or helper installation occurred.The report refines the earlier advertised FatFs exFAT limit: the 16-bit sectors-per-cluster representation rejects 32 MiB at 512-byte filesystem sectors; larger filesystem sectors fit the field but are not qualified. Active-FAT, metadata-preservation, and transaction-ownership gaps also argue against describing adoption as a narrow adapter.
This remains an interview checkpoint, not the resolution. Journal, metadata protocol, recovery conflicts/limits, and failure behavior remain open; planning-only authorization is unchanged.
Interview checkpoint — journal, transaction boundary, and recovery limits
phg agreed to all three recommendations in the third interview round on 12 October 2026:
This remains an interview checkpoint. Preview validity, batch error handling, interrupted creation, filesystem-specific invariants, durable barriers, and the final shared-understanding check remain to be completed. This does not authorize implementation or physical-media writes.
Interview checkpoint — analysis, stale plans, and batch failures
phg agreed to all three recommendations in the fourth interview round on 12 October 2026:
Cancellation details, unpublished creation, IODD preparation and the final shared-understanding check remain. The selected-volume bounds and planning-only scope remain unchanged.
Interview checkpoint — cancellation, unpublished creation, and IODD preparation
phg agreed to all three recommendations in the fifth interview round on 12 October 2026:
The interview's fifteen choices are agreed. The final shared-understanding check of the complete engineering contract remains pending; this checkpoint does not close the ticket or authorize implementation.
Resolution — filesystem engine and recoverable mutation contract
Confirmed by phg on 12 October 2026 through fifteen interview choices and the final shared-understanding check.
This records phg's fifteen agreed interview choices on 12 October 2026 and the engineering invariants that make them concrete. It governs the later implementation, not production execution. The supported format, geometry, layout, and identity envelope remains the resolution of Set the supported volume and device contract.
Engine and operation boundary
Authorization, worker lifetime, and device isolation
authopendescriptor-transfer route, using explicit access/open flags and a Unix-domain socket capable of transferring the descriptor. Do not use the convenience truncating open option or deprecated privilege-execution APIs. The worker remains an ordinary process even though its descriptor grants broad device access.Target, snapshot, plan, and ownership invariants
Journal location and semantic schema
Existing-file transaction and durability ordering
The internal journal and target are separate durability domains. Use checked full-persistence operations for the journal file and its discovery/create/rename lifecycle. For raw target I/O, use the qualified device cache-synchronization adapter and verify errors; raw-character-device
fsyncis not a durability barrier. An ordering-only barrier is not proof that data has persisted. Short I/O must be completed exactly or reported as failure; no ignored write/flush result or unsupported-success fallback is allowed. Geometry conversions use checked byte arithmetic; physical block size informs alignment without promising atomic writes. A reported content readback does not establish physical power-loss durability.Filesystem-specific metadata contract
FAT32: interpret allocation in the low 28 FAT bits and preserve each entry's reserved upper bits. Honor mirroring versus the selected active FAT; journal/check every required mirror write and preserve inactive copies in active-only mode. Check the complete old/new chains and directory entry, including entries in later or spanning I/O units; preserve names, lengths, attributes and timestamps. FSInfo signatures and hints are interpreted safely; actual ownership determines free space, and affected valid hints are updated or marked unknown. Log dirty/clean transitions and preserve unrelated error/reserved fields; do not silently repair contradictory allocation or mirror state. Microsoft FAT specification.
exFAT: the allocation bitmap owns allocation; do not treat stale FAT values as ownership for
NoFatChainstreams. Preserve logical contents, DataLength and ValidDataLength, returning/comparing zeros beyond ValidDataLength. Validate complete entry sets/checksums and preserve supported secondary entries and unrelated fields. Maintain bitmap/FAT allocation forms coherently; initialize/validate a chain before clearingNoFatChain, or establish contiguous ownership before setting it. Log bitmap, FAT and spanning entry-set changes; maintain checksum and dirty state in format order, without clearing unrelated error state or rewriting backup/OEM data unnecessarily. Unknown allocation semantics block mutation. Microsoft exFAT specification.These rules correct the reference's source-release-before-directory ordering, mishandled FAT bits, missed later directory writes, ignored persistence failures, and overwritten recovery evidence. They are independently specified invariants, not reuse of the reference implementation.
Creation, cancellation, and batch outcomes
Recovery, removal, and remount outcomes
Validation and remaining route
Image-backed public-operation tests with independent content/allocation/VHD oracles must exercise the complete agreed geometry, metadata preservation, each durable boundary, short/failing I/O, lost/reordered/torn writes, cancellation, repeated replay, journal truncation/corruption, target replacement and external modification. Tests may verify safe rejection when recovery preconditions are unavailable; successful filesystem-checker repair is not a passing operation result.
Actual macOS whole/slice/raw/block/sibling exclusion, worker death and mount behavior, standard-user authorization, and device synchronization need platform qualification. Physical power-loss claims require separately authorized disposable-device tests of actual device/bridge configurations. No such row is qualified by this planning decision. Exact release gates, performance expectations and implementation staging remain for Set validation gates and the implementation handoff; recovery UI feedback belongs in Validate the native analysis and VHD creation workflows.
No additional decision ticket is needed: newly precise work fits the existing creation, analysis/compaction, workflow and validation tickets. Persistent foreign-OS quarantine, disk-alone recovery, directory-relocation/in-place algorithms, host repair, production implementation and physical-media trials are outside this chosen first contract/current planning effort.
Evidence and ownership
7160a61012, docs/research/engine-recovery-constraints.md; delegated worker /root/engine_recovery_evidence; worktree /tmp/fat-defrag-engine-recovery-constraints. The research branch is published and the report is integrated into the main checkout; the hosted link is pinned to its evidence commit.