diff --git a/docs/superpowers/specs/2026-07-22-ft546-tidy-and-batch-replay-design.md b/docs/superpowers/specs/2026-07-22-ft546-tidy-and-batch-replay-design.md new file mode 100644 index 0000000..a65a02b --- /dev/null +++ b/docs/superpowers/specs/2026-07-22-ft546-tidy-and-batch-replay-design.md @@ -0,0 +1,403 @@ +# FT546 WMS Tidy Workflow and Historical Batch Replay Design + +Date: 2026-07-22 +Status: Approved design, pending implementation plan + +## 1. Purpose + +This change completes the missing tidy trigger and execution workflow, then uses the imported site history to validate the real WMS allocation and execution paths in batches. PDA and WCS interactions are simulated through the existing HTTP interfaces; allocation, workflow persistence, inventory changes, and locking still run through production WMS services. + +The validation must answer four questions with per-record evidence: + +1. Does inbound allocate the correct cargo space and create the correct pallet inventory? +2. Does outbound select the correct source pallet? +3. Does deep-lane outbound identify the correct blockers and naturally create the correct relocation and restoration work? +4. Do automatic and manual tidy modes create safe, executable work without starving or conflicting with inbound and outbound? + +## 2. Scope + +### 2.1 Included + +- Automatic tidy triggered from lanes made dirty by completed inbound, outbound, or outbound relocation. +- Manual tidy controlled by an operator switch. +- A persisted tidy mode state machine and restart recovery. +- A dedicated tidy workflow that reuses the existing `in_or_out=28` move-job execution chain. +- A unified admission gate for every entry point that can allocate or create inbound, outbound, outbound-relocation, or tidy work. +- Dynamic manual-tidy capacity based on online, healthy WCS vehicles. +- Idempotent WCS callbacks and explicit lock/failure handling. +- SQL Server migration for missing outbound algorithm objects and new tidy objects. +- Historical batch replay with simulated PDA/WCS calls and per-record reconciliation reports. + +### 2.2 Excluded + +- Changing the three existing tidy-planning policies or their priority order except where a confirmed defect prevents safe execution. +- Interrupting a pallet already being transported. +- Replacing the existing WCS protocol or PDA pages. +- Running the new algorithm in production `ACTIVE` mode. +- Deleting or modifying imported historical finish tables as part of normal execution. +- Forecasting a fixed number of relocation jobs before an outbound source pallet and its current blockers are known. + +## 3. Confirmed Business Rules + +### 3.1 Outbound relocation is demand-driven + +Relocation is not an independent planned batch. It is created only when a selected outbound pallet is blocked in a deep lane. + +For a selected source pallet with `N` current blocking pallets, one outbound workflow must contain: + +- `N` `MOVE_OUT` steps; +- one formal outbound step; +- `N` `RESTORE` steps. + +Each workflow step creates or resolves exactly one legacy job: every `MOVE_OUT` and `RESTORE` step creates one `in_or_out=28` relocation job, while the `OUTBOUND` step resolves the one formal outbound job created by exact-stock allocation. Counts are scoped to the single owning outbound workflow. Therefore its total step count and linked legacy-job count are both `2N+1`; exactly `2N` of those jobs have origin `OUTBOUND_RELOCATION`. When `N=0`, the shallow path contains only the formal outbound job and no outbound workflow relocation steps. + +### 3.2 Automatic tidy + +- Completed inbound, outbound, and outbound-relocation workflows upsert the affected lane into the existing dirty-lane rebuild queue. Successful state rebuilding then versions the lane in the persistent tidy-lane queue. +- A scheduler scans pending lanes every 10 minutes. +- Automatic tidy is allowed only in `NORMAL` mode. +- At most one automatic tidy workflow may be active across the WMS at a time. +- Existing inbound, outbound, or outbound-relocation execution has priority. If any such workflow is active, the scheduler does not start a new automatic tidy workflow. +- A stale plan or a plan that loses a lock race is discarded and recalculated; stale cargo-space choices are never reused. + +### 3.3 Manual tidy + +- The operator controls manual tidy with an on/off switch. +- Turning the switch on immediately blocks creation of new inbound, outbound, and outbound-relocation execution jobs, but still permits receipt and persistence of business documents. +- Existing inbound, outbound, and outbound-relocation jobs drain normally. In-flight pallets are never cancelled or interrupted. +- After the drain completes, manual tidy becomes exclusive and continuously fills `effectiveTidyCapacity` with safely admitted candidates. +- Capacity is calculated dynamically and is never hard-coded to three. `effectiveTidyCapacity` equals the number of online, healthy vehicles that are either idle or already assigned to a non-terminal manual-tidy workflow; a vehicle busy with any other task does not contribute capacity. `newTidySlots = max(0, effectiveTidyCapacity - activeManualTidyWorkflows)`, where active includes claimed, waiting-dispatch, dispatched, and running manual workflows. +- If three simulated vehicles are online and healthy, the coordinator targets three concurrent manual-tidy workflows. If only two safe candidates exist, it creates two; safety takes priority over vehicle saturation. +- Completing one manual-tidy workflow causes the coordinator to plan a replacement while the switch remains on. +- Turning the switch off stops creation of new tidy workflows. Already-dispatched tidy work drains before inbound and outbound admission resumes. +- A safe candidate must still have the same source pallet and inventory, a free compatible destination, a valid current access path, and a resource set disjoint from every active inbound, outbound, relocation, and tidy workflow. The resource set contains the source/destination spaces, intervening access spaces, touched storage lanes, required buffers, and any floor exclusivity required by deep movement. All predicates and locks are rechecked atomically before admission. +- If no safe candidate exists, manual mode remains enabled and reports `NO_SAFE_CANDIDATE`; it does not create conflicting work merely to fill vehicles. + +## 4. Mode State Machine + +The persisted system mode is one of: + +```text +NORMAL -> ENTERING_TIDY -> TIDY_ACTIVE -> EXITING_TIDY -> NORMAL +``` + +| State | New business documents | New business execution jobs | Existing business execution jobs | New manual tidy jobs | New automatic tidy jobs | +| --- | --- | --- | --- | --- | --- | +| `NORMAL` | Accept | Allow | Continue, including outbound relocation | Deny | Allow under auto rules | +| `ENTERING_TIDY` | Accept | Deny, including new outbound relocation | Drain inbound, outbound, and outbound relocation | Deny | Deny | +| `TIDY_ACTIVE` | Accept | Deny, including new outbound relocation | All three business execution types must be zero | Fill safe WCS capacity | Deny | +| `EXITING_TIDY` | Accept | Deny, including new outbound relocation | All three business execution types must be zero | Deny | Deny | + +Transitions are conditional: + +- `NORMAL -> ENTERING_TIDY`: the operator turns the switch on. +- `ENTERING_TIDY -> TIDY_ACTIVE`: no active or queued execution job remains for inbound, outbound, or outbound relocation. +- `TIDY_ACTIVE -> EXITING_TIDY`: the operator turns the switch off. Claimed/planned work with no WCS task is cancelled and its reservations are released; dispatched or running work drains. +- `EXITING_TIDY -> NORMAL`: all dispatched or running manual-tidy workflows reach a terminal state and their locks are released. + +The requested switch value and effective state are stored separately. Repeated requests to set the same value are idempotent. + +## 5. Architecture + +### 5.1 Components + +#### `TidyModeService` + +Owns the persisted switch and state transitions. It exposes the effective mode, requested switch value, drain counts, active tidy count, vehicle capacity, candidate count, and last blocking reason. + +#### Unified execution admission gate + +Every code path that can allocate or create execution work must call one shared gate before beginning its transaction. This includes ERP/API, PDA, scheduled, automatic-allocation, manual-allocation, and retry paths. + +The gate makes an admission decision for a work origin, not merely for a controller endpoint. This prevents an internal service or retry path from bypassing manual-tidy exclusivity. + +Business documents may be accepted in every mode. In `ENTERING_TIDY`, `TIDY_ACTIVE`, and `EXITING_TIDY`, their allocation/execution state remains pending until the mode returns to `NORMAL`. + +#### `TidyScheduler` + +Performs the 10-minute automatic scan and short-interval mode reconciliation. It must use a database-backed singleton claim so multiple WMS instances cannot start duplicate automatic tidy work. + +#### `TidyWorkflowCoordinator` + +Requests candidates from the existing `TidyPlanner`, validates current capacity and mode, claims non-conflicting candidates, and creates workflows. Manual coordination fills the current capacity; automatic coordination is limited to one workflow. + +#### `TidyWorkflowTransaction` + +Creates and advances the persistent workflow, locks source and destination spaces, creates the legacy `in_or_out=28` move job, handles callbacks, updates lane state, and releases locks only at the defined terminal point. + +#### WCS vehicle-capacity adapter + +Returns a stable vehicle ID plus online, fault, and current-assignment state for each vehicle; aggregate counts are derived from that snapshot. Production uses the WCS/device source; batch testing uses a simulator implementing the same adapter contract. A vehicle is healthy only when it is online and has no active fault. It is idle only when WCS reports no current assignment. Workflow admission uses the `effectiveTidyCapacity` and `newTidySlots` formulas in section 3.3. + +#### Tidy control API and operator view + +The WMS exposes status retrieval and idempotent switch update operations. The view shows the switch, effective state, drain progress, active tidy workflows, online healthy vehicle capacity, safe-candidate count, and the last failure or blocking reason. Authorization and audit fields follow existing WMS management conventions. + +### 5.2 Reused planning behavior + +The existing `TidyPlanner` remains the source of candidates: + +- `GUEST_RETURN`; +- `DOWNGRADE_MERGE`; +- `CONSOLIDATE_INCUBATION`. + +The coordinator consumes planner output but does not trust it as a reservation. All candidate preconditions and locks are checked again inside the workflow-creation transaction. + +### 5.3 Work origin + +Every `in_or_out=28` job must be traceable to exactly one origin: + +- `AUTO_TIDY`; +- `MANUAL_TIDY`; +- `OUTBOUND_RELOCATION`. + +The origin is persisted on the owning algorithm workflow and exposed on job-query DTOs by joining the workflow relationship. The legacy `in_or_out` value is not overloaded with new numeric meanings. + +## 6. Persistence + +The SQL Server migration is idempotent and additive. It creates missing algorithm objects without deleting business data. + +### 6.1 Existing algorithm objects required in the target database + +The migration must install the version-pinned definitions maintained in the same commit under `fortucky-algorithm/src/main/resources/db/sqlserver/install.sql` when absent: + +- `alg.outbound_workflow`; +- `alg.outbound_workflow_step`; +- `alg.outbound_workflow_space`; +- `alg.dirty_lane_queue`. + +These objects are in scope because deep outbound allocation, natural blocker relocation, restart recovery, and the required outbound reconciliation cannot execute without them. `alg.dirty_lane_queue` retains its existing responsibility as a short-lived topology/inventory/lane-state rebuild queue; it is not reused as the persistent tidy work queue. Their existing behavior is not otherwise redesigned by this feature. + +### 6.2 New tidy objects + +#### `alg.tidy_mode` + +One persisted mode row per controlled WMS scope, containing the requested switch value, effective state, optimistic version, status reason, updater, and timestamps. + +#### `alg.tidy_workflow` + +One row per tidy candidate execution, containing warehouse/lane, origin, planner candidate type, source pallet and space, destination space, workflow status, plan version, related move-job identifier, failure reason, and lifecycle timestamps. + +#### `alg.tidy_step` + +Ordered execution steps and callback progress for a tidy workflow. A unique workflow-and-sequence constraint prevents duplicate step creation. + +#### `alg.tidy_lane_queue` + +Persistent, versioned tidy eligibility per warehouse and storage lane. It stores `dirty_version`, `PENDING`/`CLAIMED` state, claim owner/lease, retry time/reason, and latest business-change metadata. It survives consumption of the existing rebuild queue and application restarts. + +#### `alg.tidy_space_lock` + +Database-enforced active resource locks for each tidy workflow. Despite the legacy table name, each row has `resource_type` (`SPACE`, `LANE`, or `FLOOR`) and `resource_key`. The claimed set contains source/destination spaces, intervening access spaces, touched storage lanes, required buffers, and floor exclusivity when a deep workflow uses shared floor resources. An active unique key on warehouse, resource type, and resource key prevents conflicting tidy workflows. + +Cross-origin admission is serialized with a SQL Server transaction-scoped application lock for the warehouse/resource keys. While holding it, the transaction checks both existing outbound workflow locks and tidy resource locks before persisting either workflow. This closes the race between outbound relocation and tidy even though their persisted lock tables differ. + +### 6.3 Required constraints + +- One effective `alg.tidy_mode` row per controlled scope. +- A move job belongs to at most one tidy workflow step. +- A space, lane, or floor resource has at most one active tidy lock, and the cross-origin transaction check prevents overlap with active outbound locks. +- Callback event keys are unique within a workflow step. +- Automatic workflow claim is singleton across application instances. +- Workflow status transitions use optimistic version checks. + +## 7. Runtime Data Flow + +### 7.1 Dirty-lane marking + +After an inbound, outbound, or relocation workflow commits its inventory and cargo-space result, the existing triggers upsert each affected lane in `alg.dirty_lane_queue`. For inbound this is the destination lane; for outbound it is the source lane; for relocation it is every storage lane whose occupancy changed on move-out or restore. Ports and relocation-buffer pseudo-lanes are excluded unless they are configured as storage lanes. + +The existing dirty-lane processor first rebuilds topology, inventory snapshots, and lane state. In the same successful processing transaction, it upserts every rebuilt storage lane into `alg.tidy_lane_queue`; only then may it consume the corresponding `alg.dirty_lane_queue` row. A rebuild failure follows the existing requeue behavior and does not expose stale derived state to the tidy planner. + +The tidy queue key is warehouse plus lane. Each upsert increments `dirty_version` and records latest reason/time. Queue state is `PENDING` or `CLAIMED`, with claim owner and lease expiry. A planner failure or expired claim returns the same version to `PENDING` with `retry_after` and reason. Completion clears the row only when its claimed `dirty_version` still equals the current version; a newer dirty event leaves it `PENDING`. This makes duplicate events, failures, restarts, and concurrent completions deterministic. + +### 7.2 Automatic tidy flow + +1. The scheduler confirms `NORMAL` mode, no active inbound, outbound, or outbound-relocation execution, and no active tidy workflow. +2. In one transaction it locks the mode row, verifies that no manual switch transition has begun, and claims the oldest eligible `PENDING` row from `alg.tidy_lane_queue` by `last_dirty_at`, then warehouse/lane code as the deterministic tie-break. +3. The planner calculates a candidate from current inventory and topology. +4. The transaction revalidates the candidate and claims all required space, lane, and floor resources. +5. The workflow and its `in_or_out=28` move job are committed together. +6. Simulated or real WCS advances the existing movement callback chain. +7. Inventory, cargo-space state, lane queue, workflow state, and locks are reconciled at completion. + +### 7.3 Manual tidy flow + +1. The operator enables the switch; mode becomes `ENTERING_TIDY` and the shared gate blocks new execution jobs. +2. The coordinator waits for existing inbound, outbound, and outbound-relocation work to drain. +3. Mode becomes `TIDY_ACTIVE`. +4. The capacity adapter returns the current per-vehicle snapshot and the coordinator derives `effectiveTidyCapacity`. +5. The coordinator calculates `newTidySlots` and safely claims that many mutually disjoint candidates. A candidate passes only when every safety predicate and resource claim defined in section 3.3 succeeds in the creation transaction. +6. Each completion, capacity change, or periodic reconciliation triggers another fill attempt. +7. On switch-off, mode becomes `EXITING_TIDY`; no replacement tidy work is created. Claimed or waiting-dispatch workflows without a WCS task are cancelled and released immediately; workflows with a WCS task drain. +8. After active tidy workflows drain, all locks are reconciled and mode returns to `NORMAL`, allowing pending business documents to allocate. + +## 8. Concurrency, Idempotency, and Recovery + +### 8.1 Planning and lock races + +A plan has no authority until its workflow transaction acquires all required locks and revalidates inventory, cargo-space status, pallet position, lane geometry, and mode version. On conflict, the plan is discarded and recalculated from current state. + +### 8.2 WCS callback idempotency + +Callbacks are keyed by workflow, step, WCS task, and event stage. A duplicate callback returns the prior successful result and cannot create a second job, move inventory twice, or release a lock early. Out-of-order callbacks are rejected with the current expected stage recorded for diagnosis. + +### 8.3 Vehicle state changes + +An offline or faulty vehicle immediately reduces future manual-tidy capacity. A workflow that has already begun physical movement is not cancelled automatically. A planned but undispatched workflow may be returned to the candidate pool only after its reservations are safely released. + +### 8.4 Application restart + +Mode, workflows, steps, and locks are authoritative in SQL Server. The WCS adapter supplies authoritative task state by WCS task ID and retains that simulated state across a WMS-only restart. Startup reconciliation: + +1. reloads the effective mode and requested switch; +2. compares non-terminal workflows with legacy jobs and WCS task state; +3. resumes a workflow when the database and WCS agree on its non-terminal stage; +4. moves a workflow to non-terminal `PAUSED_RECOVERY` when its database stage has no matching authoritative WCS task, or when pallet/inventory/space evidence permits more than one physical interpretation; +5. repairs a stale lock only when its owner is terminal, or when the owner is missing and there is no active legacy/WCS task and inventory identifies one unambiguous stable pallet space; +6. leaves all locks in place for `PAUSED_RECOVERY` and requires an audited operator reconciliation before retry or cancellation; +7. resumes the appropriate drain or capacity-fill loop only after the preceding reconciliation finishes. + +No in-memory flag is allowed to be the sole source of exclusivity. + +### 8.5 Failed movement + +A movement failure moves its workflow to `PAUSED_RECOVERY` and retains the locks needed to represent the uncertain physical state without a time-based auto-release. An operator or recovery service must reconcile the pallet's physical position with WMS before retry or cancellation. The only successful terminal states are `COMPLETED` and `CANCELLED_RECONCILED`. Cancellation is allowed only when inventory, cargo-space state, legacy job state, and locks can be moved to `CANCELLED_RECONCILED` in one transaction. An unexpected `PAUSED_RECOVERY` row makes the batch run fail; a deliberately injected recovery scenario passes only when the expected pause and retained locks are observed, followed by a successful audited reconciliation. + +## 9. Historical Replay Design + +### 9.1 Fixtures and preservation + +The following imported finish data is the immutable historical fixture source: + +- `in_stockin_finish.sql`; +- `in_stockin_detail_finish.sql`; +- `mydo_job_finish.sql`; +- `mydo_job_detail_finish.sql`; +- `out_stockout_finish.sql`; +- `out_stockout_detail_finish.sql`. + +The replay creates new run-scoped open documents and jobs. It does not turn finish rows back into active rows and does not update the source files. + +Historical warehouse identifiers `07` or blank are mapped to the available topology warehouse `01`; an original value of `01` is retained as `01`. Any other warehouse value is excluded from the eligible set and reported as `UNSUPPORTED_SOURCE_WAREHOUSE`; it is never silently mapped. Every considered row retains its original warehouse value and mapping decision in the replay report. The known orphan outbound detail `CKD202605140015-1` is excluded from executable samples and reported as a fixture-integrity exception. + +An eligible inbound detail has an existing historical header, positive quantity, resolvable product/master data, an original warehouse covered by the closed mapping rule above, and a tray that is unique within the replay run. Each historical detail may be replayed at most once per run. Eligible rows are ordered by historical completion time, document number, and detail number so the 100/300 sample boundary is reproducible. A failed execution counts as an attempted reconciliation row but not as a passed sample. + +### 9.2 Identifier safety + +Before replay, document, job, task, and inventory generators must consider both active and finish tables under a concurrency-safe allocation mechanism. The existing collision around `CK202607210002` is a mandatory regression case. Replay business identifiers also carry a unique run prefix so cleanup and reconciliation cannot select unrelated data. + +### 9.3 Test environment sequence + +1. Confirm the exact SQL Server instance and `FT546-wms` development database. +2. Re-query baseline row counts, cargo-space state, active work, vehicle state, and algorithm schema; do not rely on recorded counts alone. +3. Apply and verify the idempotent algorithm migration. +4. Ensure no active business execution exists and validate the cargo-space baseline. The recorded 692-space count is a reference: if the live configured count differs, preflight stops for explicit operator validation. Consistency means every configured space has one valid state, every occupied storage space has matching pallet inventory, every inventory row resolves to one configured occupied space, and no active workflow/resource lock remains. +5. Create a new restorable pre-test backup after migration and before replay. +6. Start WMS plus PDA/WCS simulators and assign a unique replay run ID. +7. Keep normal deployment default at `SHADOW`, but enable a run-scoped execution override only for the confirmed development database and replay run ID. Execute inbound, outbound, automatic-tidy, and manual-tidy phases through real application services and HTTP callbacks, then remove the override before final restoration. +8. Freeze execution, generate reconciliation reports, and retain logs and failing payloads. +9. Stop writers and restore the pre-test backup to return to the imported-history baseline with the migrated schema. +10. Re-query counts and integrity checks to prove restoration. + +No destructive cleanup or restore runs unless the resolved target is the confirmed development database and the new pre-test backup has been verified. + +## 10. Batch Test Matrix + +### 10.1 Inbound phase + +Replay the first 100 eligible historical inbound details in the deterministic order from section 9.1. Compute outbound strata from a committed database snapshot. If any stratum has fewer than 25 eligible source pallets, replay the next deterministic inbound block and recompute, stopping when every stratum has 25 candidates or 300 unique eligible details have been attempted. Reaching 300, or exhausting the eligible set earlier, without all four strata is a run-level `OUTBOUND_STRATUM_COVERAGE_FAILED`; remaining phases may collect diagnostics but the run cannot pass. + +For every inbound detail, record and assert: + +- original historical document/detail and normalized replay identifiers; +- product, batch, pallet, and requested quantity; +- assigned cargo space exists in warehouse `01` and satisfies planner/topology rules; +- allocation order does not create an invalid inner/outer accessibility state; +- exactly one current inventory aggregate for the replay pallet/product/batch/location key, with no duplicate active stock rows representing the same physical pallet content; +- inventory product, batch, quantity, pallet, and cargo-space code match the completed movement; +- cargo-space occupancy and all workflow/space/floor locks match the terminal state; +- duplicate PDA/WCS completion calls do not duplicate inventory. + +### 10.2 Outbound phase + +An eligible outbound fixture has an existing historical header, positive quantity, resolvable product/master data, a warehouse covered by section 9.1, and sufficient matching replay-created inventory at selection time. The orphan fixture is never eligible. Fixtures are considered by historical completion time, document number, and detail number and used at most once per run. + +Execute exactly 25 successful outbound details in each current-blocker stratum `0`, `1`, `2`, and `3+` for a minimum of 100 successful cases. Selection is just-in-time from a committed snapshot because preceding outbounds can change accessibility. At least 50 cases must omit a tray override and therefore exercise planner source selection. Remaining stratum-coverage cases may request a historical source tray only when the imported finished job/detail relationship proves that tray belonged to the same historical outbound fixture and that tray exists in replay-created inventory. The report labels each case `PLANNER_SELECTED` or `HISTORICAL_TRAY_OVERRIDE`. Failed attempts remain in the report and are replaced from the same stratum, but still make the overall run fail. Do not fabricate blockers or directly edit cargo-space positions. + +The source-pallet oracle uses the same published contract as `OutboundPlanner`, evaluated independently from the committed pre-allocation snapshot: + +- an explicitly requested tray selects that exact eligible tray; +- with FIFO off: lowest blocker count, then lane-release rank, floor number, and cargo-space ID; +- with strict FIFO: oldest inbound timestamp, then blocker count, lane-release rank, efficiency score descending, timestamp, and cargo-space ID; +- with tolerant FIFO: restrict to timestamps no later than oldest plus configured tolerance, then apply the same cost ordering as strict FIFO. + +All comparisons and the final cargo-space-ID tie-break are recorded in the report. The production planner result must equal this independently calculated oracle result. + +For every outbound detail, record and assert: + +- requested product, batch/selection constraints, and quantity; +- selected source pallet existed and held eligible stock; `PLANNER_SELECTED` cases must equal the independent oracle, while `HISTORICAL_TRAY_OVERRIDE` cases must equal the proven historical tray; +- blockers are derived from the committed current topology after source selection; +- for `N` blockers, the owning workflow has exactly `N` `MOVE_OUT`, one `OUTBOUND`, and `N` `RESTORE` steps, each resolving to exactly one distinct linked legacy job; exactly `2N` linked jobs have origin `OUTBOUND_RELOCATION` and one is the formal outbound job; +- each blocker identity and temporary/restore destination is correct and non-conflicting; +- only the formal outbound quantity is deducted from inventory; +- blocking pallets retain correct inventory and end in the planned restored/final spaces; +- source and blocking cargo-space states match physical completion; +- workflow reservations, space locks, and floor locks are fully released at terminal completion; +- repeated and out-of-order WCS callbacks follow the idempotency contract. + +An outbound test cannot pass merely because the document closes; its source pallet, blockers, job counts, inventory delta, final pallet positions, and lock state must all reconcile. + +### 10.3 Automatic tidy phase + +- Create dirty-lane events through completed replay work, not by inserting a tidy workflow directly. +- Verify the 10-minute scan or its test-controlled clock selects pending lanes. +- Keep inbound/outbound work active and prove no automatic tidy starts. +- Drain business work and prove exactly one automatic tidy may run. +- Present multiple safe candidates and prove a second automatic tidy does not start until the first is terminal. +- Force a lock race or stale candidate and prove replanning occurs without using the old destination. +- Verify origin `AUTO_TIDY`, inventory/cargo-space correctness, and lock release. A claimed dirty version becomes clean only after successful tidy; a newer dirty version or failed plan remains `PENDING` under the queue rules in section 7.1. + +### 10.4 Manual tidy phase + +- Simulate three online, healthy vehicles and enough independent candidates; prove three manual-tidy workflows are admitted after the business drain. +- Complete one workflow and prove one replacement is created while the switch remains on. +- Reduce healthy capacity from three to two and prove no new work exceeds two active workflows after current physical work naturally drains. +- Provide only two safe candidates with three healthy vehicles and prove only two workflows are created with `NO_SAFE_CANDIDATE`/capacity status visible for the unfilled slot. +- Turn the switch on while inbound/outbound work exists and prove new execution allocation is blocked while existing work drains. +- Submit new business documents during `TIDY_ACTIVE` and prove they are accepted but remain unallocated. +- Turn the switch off and prove no replacement tidy is created, active tidy drains, mode returns to `NORMAL`, and pending business documents resume allocation. +- Restart WMS at three reproducible non-normal checkpoints: `ENTERING_TIDY` with one held inbound callback, `TIDY_ACTIVE` with one running WCS task plus one waiting-dispatch workflow, and `EXITING_TIDY` with one running tidy task. After restart, verify respectively that admission remains blocked and the drain reaches active mode; WCS-agreed work resumes without duplicate dispatch and safe waiting work is reconciled; and no replacement starts before the running task completes and mode returns to `NORMAL`. +- Add one recovery-negative scenario in which WCS deliberately returns unknown for a database-running task. Verify `PAUSED_RECOVERY`, retained resource locks, no replacement job, then audited reconciliation to `COMPLETED` or `CANCELLED_RECONCILED`. + +## 11. Reporting and Acceptance + +Each run writes artifacts under `outputs/ft546-tidy-batch-replay-/`: + +- `inbound-reconciliation.csv`: one row per inbound detail and each asserted field; +- `outbound-reconciliation.csv`: one row per outbound detail, source pallet, blocker count, expected/actual step counts, inventory delta, and lock result; +- `tidy-reconciliation.csv`: one row per automatic/manual tidy workflow, origin, candidate type, vehicle assignment, positions, inventory, and terminal state; +- `events.json`: ordered request, callback, state-transition, and database evidence with secrets removed; +- `report.md`: environment fingerprint, migration result, sample counts, stratum coverage, pass/fail summary, exceptions, restoration proof, and links to raw artifacts. + +Acceptance requires all of the following: + +- at least 100 inbound details and 100 successful outbound details executed through the real allocation/workflow path, with every attempted eligible inbound required to reconcile successfully; +- exactly 25 successful cases in each of the four outbound blocker strata; `OUTBOUND_STRATUM_COVERAGE_FAILED` is a run failure and is shown with available/required counts by stratum; +- zero unexplained inventory, pallet-position, cargo-space, workflow-step, or lock mismatches; +- automatic tidy never exceeds one active workflow and never starts while business execution is active; +- manual tidy respects effective healthy-vehicle capacity and mode exclusivity; +- duplicate callbacks produce no duplicate side effects; +- service restart recovery passes for all non-normal modes; +- every unexpected failure is represented as a failed reconciliation row with evidence, never omitted from aggregate percentages; controlled negative scenarios are separately labelled and pass only when their exact expected state, locks, and recovery outcome match; +- the database is restored and verified against the pre-test imported-history baseline. + +## 12. Implementation Boundaries + +- Preserve the `fortucky-algorithm` module boundary and existing WMS transaction manager/data source integration. +- Keep the normal deployment default at `SHADOW`. The batch harness may use only the run-scoped development-database execution override defined in section 9.3; it must remove that override before restoration. Promotion to `CANARY` or `ACTIVE` remains a separate decision after SQL Server diagnostics, concurrency, idempotency, recovery, backup, and rollback gates pass. +- Add only focused changes required for admission, tidy orchestration, persistence, UI control/status, simulation, and reconciliation. +- Deliver in three ordered implementation phases: (A) migrations, origin tracing, and the unified admission gate; (B) persisted tidy state/workflow, WCS capacity, minimal operator switch/status, and recovery; (C) simulators, historical replay, reconciliation, and restoration tooling. Each phase has its own automated tests, but the feature is accepted only after the complete batch matrix passes. +- Do not revert or overwrite unrelated uncommitted work already present in the repository. +- Implementation must follow test-driven development: failing unit/integration tests first, then production changes, followed by SQL Server replay verification.