phg/iodd-tool

Fork 0

Defragment files on external storage #22

New issue

Closed

opened 2026-10-11 19:10:49 +00:00 by phg · 3 comments

phg commented

2026-10-11 19:10:49 +00:00

Owner

Parent

IODD Tool specification

What to build

Integrate Defrag File and Defrag All with the production external-storage adapter. Validate preserved data, batch outcomes and recovery/remount handling using disposable virtual volumes and controlled failures.

Acceptance criteria

The selected external target, snapshot, preview, and hardware-detachment preparation are revalidated before every permitted mutation.
Single-file and batch requests reuse the independently verified FAT32/exFAT transaction contracts through the chosen live access route.
Independent hashes/metadata and complete allocation/ownership checks confirm preserved files and actual contiguous results.
Batch error/Stop behavior matches the approved policy and never overwrites unresolved recovery evidence.
Busy unmount, descriptor-isolation failure, changed identity, removal, denied authorization, and unsupported/failed flushes cannot cause stale or falsely successful writes.
A fresh analysis and separate remount result communicate the actual final storage state.
The applicable independent image correctness/recovery evidence and virtual macOS access/lifecycle evidence pass before enabling writes. Tests use disposable virtual targets and retain reproducible software/geometry results.

Blocked by

Validation contract

Completion requires the applicable independently checked software evidence in Set validation gates and the implementation handoff. A closed policy decision does not establish that implementation tests have passed. Use disposable images/virtual volumes and controlled faults; no hardware-trial or performance gate applies.

Scope

This ticket belongs to the later implementation effort. All planning decisions are resolved; current work covers tracker and documentation updates. The parent specification indexes the authoritative product and software-validation contracts. Forgejo governs live updates; retained local files record the handoff.

## Parent [IODD Tool specification](https://git.s1q.dev/phg/iodd-tool/issues/7) ## What to build Integrate Defrag File and Defrag All with the production external-storage adapter. Validate preserved data, batch outcomes and recovery/remount handling using disposable virtual volumes and controlled failures. ## Acceptance criteria - [ ] The selected external target, snapshot, preview, and hardware-detachment preparation are revalidated before every permitted mutation. - [ ] Single-file and batch requests reuse the independently verified FAT32/exFAT transaction contracts through the chosen live access route. - [ ] Independent hashes/metadata and complete allocation/ownership checks confirm preserved files and actual contiguous results. - [ ] Batch error/Stop behavior matches the approved policy and never overwrites unresolved recovery evidence. - [ ] Busy unmount, descriptor-isolation failure, changed identity, removal, denied authorization, and unsupported/failed flushes cannot cause stale or falsely successful writes. - [ ] A fresh analysis and separate remount result communicate the actual final storage state. - [ ] The applicable independent image correctness/recovery evidence and virtual macOS access/lifecycle evidence pass before enabling writes. Tests use disposable virtual targets and retain reproducible software/geometry results. ## Blocked by - [Recover interrupted operations in the app](https://git.s1q.dev/phg/iodd-tool/issues/19) - [Discover and analyze external host volumes](https://git.s1q.dev/phg/iodd-tool/issues/20) - [Set validation gates and the implementation handoff](https://git.s1q.dev/phg/iodd-tool/issues/6) ## Validation contract Completion requires the applicable independently checked software evidence in [Set validation gates and the implementation handoff](https://git.s1q.dev/phg/iodd-tool/issues/6). A closed policy decision does not establish that implementation tests have passed. Use disposable images/virtual volumes and controlled faults; no hardware-trial or performance gate applies. ## Scope This ticket belongs to the later implementation effort. All planning decisions are resolved; current work covers tracker and documentation updates. The parent specification indexes the authoritative product and software-validation contracts. Forgejo governs live updates; retained local files record the handoff.

phg added the

ready-for-agent

label

2026-10-11 19:10:50 +00:00

phg referenced this issue

2026-10-11 19:10:51 +00:00

Qualify and package the native app #24

phg referenced this issue

2026-10-13 19:28:12 +00:00

IODD Tool specification #7

phg self-assigned this

2026-10-13 23:12:58 +00:00

phg referenced this issue

2026-10-13 23:31:19 +00:00

Create a verified VHD on external storage #21

phg referenced this issue from a commit

2026-10-13 23:44:54 +00:00

Merge ticket #22 gated Defrag core milestone

phg commented

2026-10-13 23:45:11 +00:00

Author

Owner

The feature-gated Defrag File/Defrag All core milestone is integrated and pushed on implement/iodd-tool-spec-7 at 8fac4122c0d81ce207942fe9eba7fa125d801af8 (Defrag candidate b3912ca3d5309cdec9bce25dbabd0023a83c9d75, combined with the already validated Compact core). The public retained host session now composes the existing reviewed file and batch defrag engines, with Codable batch progress/results for upcoming worker binding.

New independent tracers cover FAT32/exFAT single-file defrag in MBR/GPT/extended layouts and largest-first three-file batches, verifying complete logical/full allocated contents, exact preserved metadata and entry identity, unchanged sibling/surrounding bytes, retained ownership and explicit close/restoration. Independent combined-tree validation passed 32 test functions / 90 expanded cases, including these tracers, new Compact cases, existing batch ordering/skips/Stop, shared Create/recovery/freshness and history checks. Canonical arm64 app/worker release builds, plist and macOS 15 minimum checks passed. The combined merge preserves each operation milestone’s exact path set.

Ticket #22 remains open; production native Defrag is not enabled or qualified by this interim milestone. Fault/recovery/UI/native qualification continues, coordinated with #21’s shared production worker and per-operation gates.

The feature-gated Defrag File/Defrag All core milestone is integrated and pushed on `implement/iodd-tool-spec-7` at `8fac4122c0d81ce207942fe9eba7fa125d801af8` (Defrag candidate `b3912ca3d5309cdec9bce25dbabd0023a83c9d75`, combined with the already validated Compact core). The public retained host session now composes the existing reviewed file and batch defrag engines, with Codable batch progress/results for upcoming worker binding. New independent tracers cover FAT32/exFAT single-file defrag in MBR/GPT/extended layouts and largest-first three-file batches, verifying complete logical/full allocated contents, exact preserved metadata and entry identity, unchanged sibling/surrounding bytes, retained ownership and explicit close/restoration. Independent combined-tree validation passed 32 test functions / 90 expanded cases, including these tracers, new Compact cases, existing batch ordering/skips/Stop, shared Create/recovery/freshness and history checks. Canonical arm64 app/worker release builds, plist and macOS 15 minimum checks passed. The combined merge preserves each operation milestone’s exact path set. Ticket #22 remains open; production native Defrag is not enabled or qualified by this interim milestone. Fault/recovery/UI/native qualification continues, coordinated with #21’s shared production worker and per-operation gates.

phg referenced this issue from a commit

2026-10-14 16:23:13 +00:00

Merge ticket #22 checker-compatible native Defrag fixture

phg commented

2026-10-14 18:04:20 +00:00

Author

Owner

Progress checkpoint; this is not overall ticket acceptance.

Integration branch implement/iodd-tool-spec-7 is pushed and remote-confirmed at 648020a. Its entire tree equals the reviewed candidate 0dc5522; the delta from 7dbe08 changes only worker RPC failure-message construction.

The signed95 macOS15.6.1/24G90 standard-user collection completed eight scoped Defrag scenarios: File followed by fresh All, public-API single-file Copying Stop, public-API first-commit batch Stop, and modeled report failure followed by actual same-owner GUI rollback, each on FAT32/exFAT disposable virtual media. All eight non-repairing filesystem checks exited0 with unchanged checker inputs. Independent complete-file/allocation/metadata/free/pinned/sibling/outside proofs passed. API Stop rows are not GUI-button evidence; retained per-row oracle timing limits remain explicit. The modeled failures occurred after a real target write and successful native cache synchronization, not an actual failed ioctl. Correct recovered totals are FAT32 two writes/seven successful cache calls and exFAT two writes/six successful cache calls (corrected EX namespace receipt v2; original mispath receipt preserved).

During unsafe Finish, the worker correctly retained ownership but displayed an enum wrapper. The one-file correction now preserves its associated reason through the ordinary signed RPC. Fresh independent Standards/Spec reviews found no issues; canonical arm64/macOS15 app+worker, Developer ID/hardened runtime/timestamps and strict signatures passed. The bounded unchanged public safety cohort passed two functions/three cases. A separate actual signed0dc macOS15 parent Finish displayed the exact plain reason while refusing close/remount and preserving the target, original failed result, regular journal, registry and event bytes. Same-owner fresh renewal plus explicit matched rollback restored the pristine whole backing and passed the six-file independent oracle; native close0, raw filesystem check0, normal Quit/free leases and ordinary detach followed.

New signed qualification archive SHA256: 6eea338f609cdaafa69e1dbf4abcc21ee47022bc40f22a42ef190555707d47bc (2,848,251 bytes). Source/Core and worker binary identities changed; old95/6a binary equivalence is not claimed. Old failures and excluded observations remain preserved. This is a development qualification artifact, not the notarized final release.

Local evidence index: /tmp/iodd-tool-issue7-context/ticket22-t3-current15-matrix-v1.json; new native proof/safe handback: ticket22-0dc-formatter-native-GREEN-safe-handback-v1.json; root independently verified42 earlier matrix references/69 collected records and19 new references/22 collected records. All108 historical static/log/artifact files also retain size/SHA/mtime.

All four production flags remain false. Remaining representative native workflow joins, final frozen acceptance on both supported runtimes, lifecycle/install evidence and final signing/notarization/distribution work remain open under21–24. No physical-media/IODD/power-loss or performance qualification is claimed.

Progress checkpoint; this is not overall ticket acceptance. Integration branch `implement/iodd-tool-spec-7` is pushed and remote-confirmed at [648020a](https://git.s1q.dev/phg/iodd-tool/commit/648020a020ed24dcda8d7d40a21991ffe923175a). Its entire tree equals the reviewed candidate `0dc5522`; the delta from `7dbe08` changes only worker RPC failure-message construction. The signed95 macOS15.6.1/24G90 standard-user collection completed eight scoped Defrag scenarios: File followed by fresh All, public-API single-file Copying Stop, public-API first-commit batch Stop, and modeled report failure followed by actual same-owner GUI rollback, each on FAT32/exFAT disposable virtual media. All eight non-repairing filesystem checks exited0 with unchanged checker inputs. Independent complete-file/allocation/metadata/free/pinned/sibling/outside proofs passed. API Stop rows are not GUI-button evidence; retained per-row oracle timing limits remain explicit. The modeled failures occurred after a real target write and successful native cache synchronization, not an actual failed ioctl. Correct recovered totals are FAT32 two writes/seven successful cache calls and exFAT two writes/six successful cache calls (corrected EX namespace receipt v2; original mispath receipt preserved). During unsafe Finish, the worker correctly retained ownership but displayed an enum wrapper. The one-file correction now preserves its associated reason through the ordinary signed RPC. Fresh independent Standards/Spec reviews found no issues; canonical arm64/macOS15 app+worker, Developer ID/hardened runtime/timestamps and strict signatures passed. The bounded unchanged public safety cohort passed two functions/three cases. A separate actual signed0dc macOS15 parent Finish displayed the exact plain reason while refusing close/remount and preserving the target, original failed result, regular journal, registry and event bytes. Same-owner fresh renewal plus explicit matched rollback restored the pristine whole backing and passed the six-file independent oracle; native close0, raw filesystem check0, normal Quit/free leases and ordinary detach followed. New signed qualification archive SHA256: `6eea338f609cdaafa69e1dbf4abcc21ee47022bc40f22a42ef190555707d47bc` (2,848,251 bytes). Source/Core and worker binary identities changed; old95/6a binary equivalence is not claimed. Old failures and excluded observations remain preserved. This is a development qualification artifact, not the notarized final release. Local evidence index: `/tmp/iodd-tool-issue7-context/ticket22-t3-current15-matrix-v1.json`; new native proof/safe handback: `ticket22-0dc-formatter-native-GREEN-safe-handback-v1.json`; root independently verified42 earlier matrix references/69 collected records and19 new references/22 collected records. All108 historical static/log/artifact files also retain size/SHA/mtime. All four production flags remain false. Remaining representative native workflow joins, final frozen acceptance on both supported runtimes, lifecycle/install evidence and final signing/notarization/distribution work remain open under21–24. No physical-media/IODD/power-loss or performance qualification is claimed.

phg commented

2026-10-15 00:12:18 +00:00

Author

Owner

Resolved on 381442f, pushed and remote-confirmed on implement/iodd-tool-spec-7. Its entire tree matches reviewed candidate d514e5516f; all four native operation flags are enabled.

Independent Standards and Spec reviews found no issues. Canonical arm64 builds, macOS 15 minimum/plist, Developer ID hardening/timestamps and strict component signatures passed. The public safety cohort passed 2 functions / 3 cases (exit 0, 7.268 seconds). The later capture helper's exit 1 remains preserved separately, followed by read-only inventory completion. Signed development archive SHA-256: 5e428352dfc9848836891840f2e6bb6d5eb35030f3025bbd603ba038341afabb (2,847,620 bytes).

The dated ledger and adjacent JSON retain earlier evidence and exact source/artifact bindings. Later d514 observations below supplement that snapshot. The older 95eac full 259-function PASS, fixture union and failed 8cb/7fc full runs retain their separate scope; no binary equivalence, original-cause or flake conclusion is inferred.

Exact signed d514 passed one ordinary FAT32 six-file File → fresh complete analysis → separately reviewed NEW All case on macOS 27.0.1 / 26A434, non-admin UID 502. The unchanged 68,277,248-byte corpus used range 32768 + 34089472 and 512-byte logical/physical sectors. File moved FRAG to clusters 7–9; fresh All moved HIDDEN to 5–6 and Unicode to 12–13, skipping four files including now-contiguous FRAG.

Independent intermediate/final oracles preserved all six files, full allocations/slack, raw entries/metadata, system/reserved/pinned and sibling/outside bytes. Exact free runs 14,21],[36,4],[41,65486 derive from pristine bytes and actual reviewed placements, including pinned directory 40. The same worker/raw descriptor stayed held through complete HOST copies and after oracle verification. Actual visible terminals bound uncapped event counts 27 and 69 with no close. Three committed, complete journals bind the range, geometry and originally empty mount set.

Fresh safe review preceded sole Finish: native close 0 at event 70 and raw descriptor gone. Quit freed both leases and removed owner/endpoint/socket. Ordinary detach, checker-only reattachment, raw msdos check 0 with unchanged whole-image SHA-256 a676de91e68fafe4f523c327a98dbec8b936d0807293b8da839a0ab18d4c53f7, then detach left no owned attachments. ROOT verified all 315 references and seal hashes. No restoration is claimed for this initially unmounted fixture.

Earlier signed macOS 15 FAT32/exFAT File/All, public API Stop and same-owner modeled-report rollback evidence retains scope. API Stops are not GUI-button proof. The old tiny-root attempt never executed and remains unqualified; observer failures and original missing-output limits were preserved.

Final frozen suites on both supported runtimes, whole-integration review, complete app smoke, supported manual/Homebrew lifecycle, notarization/stapling, immutable DMG and clean-user Gatekeeper/cask distribution remain in #24. This closes the operation implementation, not final release. Physical IODD, 4K, power-loss and performance qualification are neither claimed nor added as prerequisites.

Resolved on [381442f](https://git.s1q.dev/phg/iodd-tool/commit/381442f79cd5736fceaffa15d5f1edacb1ca8842), pushed and remote-confirmed on implement/iodd-tool-spec-7. Its entire tree matches reviewed candidate d514e5516fb159af36407eaf699800954a73d9c9; all four native operation flags are enabled. Independent Standards and Spec reviews found no issues. Canonical arm64 builds, macOS 15 minimum/plist, Developer ID hardening/timestamps and strict component signatures passed. The public safety cohort passed 2 functions / 3 cases (exit 0, 7.268 seconds). The later capture helper's exit 1 remains preserved separately, followed by read-only inventory completion. Signed development archive SHA-256: 5e428352dfc9848836891840f2e6bb6d5eb35030f3025bbd603ba038341afabb (2,847,620 bytes). The [dated ledger](https://git.s1q.dev/phg/iodd-tool/src/commit/381442f79cd5736fceaffa15d5f1edacb1ca8842/docs/validation/external-mutations.md) and adjacent JSON retain earlier evidence and exact source/artifact bindings. Later d514 observations below supplement that snapshot. The older 95eac full 259-function PASS, fixture union and failed 8cb/7fc full runs retain their separate scope; no binary equivalence, original-cause or flake conclusion is inferred. Exact signed d514 passed one ordinary FAT32 six-file File → fresh complete analysis → separately reviewed NEW All case on macOS 27.0.1 / 26A434, non-admin UID 502. The unchanged 68,277,248-byte corpus used range 32768 + 34089472 and 512-byte logical/physical sectors. File moved FRAG to clusters 7–9; fresh All moved HIDDEN to 5–6 and Unicode to 12–13, skipping four files including now-contiguous FRAG. Independent intermediate/final oracles preserved all six files, full allocations/slack, raw entries/metadata, system/reserved/pinned and sibling/outside bytes. Exact free runs [[14,21],[36,4],[41,65486]] derive from pristine bytes and actual reviewed placements, including pinned directory 40. The same worker/raw descriptor stayed held through complete HOST copies and after oracle verification. Actual visible terminals bound uncapped event counts 27 and 69 with no close. Three committed, complete journals bind the range, geometry and originally empty mount set. Fresh safe review preceded sole Finish: native close 0 at event 70 and raw descriptor gone. Quit freed both leases and removed owner/endpoint/socket. Ordinary detach, checker-only reattachment, raw msdos check 0 with unchanged whole-image SHA-256 a676de91e68fafe4f523c327a98dbec8b936d0807293b8da839a0ab18d4c53f7, then detach left no owned attachments. ROOT verified all 315 references and seal hashes. No restoration is claimed for this initially unmounted fixture. Earlier signed macOS 15 FAT32/exFAT File/All, public API Stop and same-owner modeled-report rollback evidence retains scope. API Stops are not GUI-button proof. The old tiny-root attempt never executed and remains unqualified; observer failures and original missing-output limits were preserved. Final frozen suites on both supported runtimes, whole-integration review, complete app smoke, supported manual/Homebrew lifecycle, notarization/stapling, immutable DMG and clean-user Gatekeeper/cask distribution remain in [#24](https://git.s1q.dev/phg/iodd-tool/issues/24). This closes the operation implementation, not final release. Physical IODD, 4K, power-loss and performance qualification are neither claimed nor added as prerequisites.

phg closed this issue

2026-10-15 00:12:19 +00:00