Files
FT546MWS/docs/superpowers/specs/2026-07-22-ft546-tidy-and-batch-replay-design.md

34 KiB

FT546 WMS Tidy Workflow and Historical Batch Replay Design

Date: 2026-07-22 Status: Approved design, pending implementation plan

1. Purpose

This change completes the missing tidy trigger and execution workflow, then uses the imported site history to validate the real WMS allocation and execution paths in batches. PDA and WCS interactions are simulated through the existing HTTP interfaces; allocation, workflow persistence, inventory changes, and locking still run through production WMS services.

The validation must answer four questions with per-record evidence:

  1. Does inbound allocate the correct cargo space and create the correct pallet inventory?
  2. Does outbound select the correct source pallet?
  3. Does deep-lane outbound identify the correct blockers and naturally create the correct relocation and restoration work?
  4. Do automatic and manual tidy modes create safe, executable work without starving or conflicting with inbound and outbound?

2. Scope

2.1 Included

  • Automatic tidy triggered from lanes made dirty by completed inbound, outbound, or outbound relocation.
  • Manual tidy controlled by an operator switch.
  • A persisted tidy mode state machine and restart recovery.
  • A dedicated tidy workflow that reuses the existing in_or_out=28 move-job execution chain.
  • A unified admission gate for every entry point that can allocate or create inbound, outbound, outbound-relocation, or tidy work.
  • Dynamic manual-tidy capacity based on online, healthy WCS vehicles.
  • Idempotent WCS callbacks and explicit lock/failure handling.
  • SQL Server migration for missing outbound algorithm objects and new tidy objects.
  • Historical batch replay with simulated PDA/WCS calls and per-record reconciliation reports.

2.2 Excluded

  • Changing the three existing tidy-planning policies or their priority order except where a confirmed defect prevents safe execution.
  • Interrupting a pallet already being transported.
  • Replacing the existing WCS protocol or PDA pages.
  • Running the new algorithm in production ACTIVE mode.
  • Deleting or modifying imported historical finish tables as part of normal execution.
  • Forecasting a fixed number of relocation jobs before an outbound source pallet and its current blockers are known.

3. Confirmed Business Rules

3.1 Outbound relocation is demand-driven

Relocation is not an independent planned batch. It is created only when a selected outbound pallet is blocked in a deep lane.

For a selected source pallet with N current blocking pallets, one outbound workflow must contain:

  • N MOVE_OUT steps;
  • one formal outbound step;
  • N RESTORE steps.

Each workflow step creates or resolves exactly one legacy job: every MOVE_OUT and RESTORE step creates one in_or_out=28 relocation job, while the OUTBOUND step resolves the one formal outbound job created by exact-stock allocation. Counts are scoped to the single owning outbound workflow. Therefore its total step count and linked legacy-job count are both 2N+1; exactly 2N of those jobs have origin OUTBOUND_RELOCATION. When N=0, the shallow path contains only the formal outbound job and no outbound workflow relocation steps.

3.2 Automatic tidy

  • Completed inbound, outbound, and outbound-relocation workflows upsert the affected lane into the existing dirty-lane rebuild queue. Successful state rebuilding then versions the lane in the persistent tidy-lane queue.
  • A scheduler scans pending lanes every 10 minutes.
  • Automatic tidy is allowed only in NORMAL mode.
  • At most one automatic tidy workflow may be active across the WMS at a time.
  • Existing inbound, outbound, or outbound-relocation execution has priority. If any such workflow is active, the scheduler does not start a new automatic tidy workflow.
  • A stale plan or a plan that loses a lock race is discarded and recalculated; stale cargo-space choices are never reused.

3.3 Manual tidy

  • The operator controls manual tidy with an on/off switch.
  • Turning the switch on immediately blocks creation of new inbound, outbound, and outbound-relocation execution jobs, but still permits receipt and persistence of business documents.
  • Existing inbound, outbound, and outbound-relocation jobs drain normally. In-flight pallets are never cancelled or interrupted.
  • After the drain completes, manual tidy becomes exclusive and continuously fills effectiveTidyCapacity with safely admitted candidates.
  • Capacity is calculated dynamically and is never hard-coded to three. effectiveTidyCapacity equals the number of online, healthy vehicles that are either idle or already assigned to a non-terminal manual-tidy workflow; a vehicle busy with any other task does not contribute capacity. newTidySlots = max(0, effectiveTidyCapacity - activeManualTidyWorkflows), where active includes claimed, waiting-dispatch, dispatched, and running manual workflows.
  • If three simulated vehicles are online and healthy, the coordinator targets three concurrent manual-tidy workflows. If only two safe candidates exist, it creates two; safety takes priority over vehicle saturation.
  • Completing one manual-tidy workflow causes the coordinator to plan a replacement while the switch remains on.
  • Turning the switch off stops creation of new tidy workflows. Already-dispatched tidy work drains before inbound and outbound admission resumes.
  • A safe candidate must still have the same source pallet and inventory, a free compatible destination, a valid current access path, and a resource set disjoint from every active inbound, outbound, relocation, and tidy workflow. The resource set contains the source/destination spaces, intervening access spaces, touched storage lanes, required buffers, and any floor exclusivity required by deep movement. All predicates and locks are rechecked atomically before admission.
  • If no safe candidate exists, manual mode remains enabled and reports NO_SAFE_CANDIDATE; it does not create conflicting work merely to fill vehicles.

4. Mode State Machine

The persisted system mode is one of:

NORMAL -> ENTERING_TIDY -> TIDY_ACTIVE -> EXITING_TIDY -> NORMAL
State New business documents New business execution jobs Existing business execution jobs New manual tidy jobs New automatic tidy jobs
NORMAL Accept Allow Continue, including outbound relocation Deny Allow under auto rules
ENTERING_TIDY Accept Deny, including new outbound relocation Drain inbound, outbound, and outbound relocation Deny Deny
TIDY_ACTIVE Accept Deny, including new outbound relocation All three business execution types must be zero Fill safe WCS capacity Deny
EXITING_TIDY Accept Deny, including new outbound relocation All three business execution types must be zero Deny Deny

Transitions are conditional:

  • NORMAL -> ENTERING_TIDY: the operator turns the switch on.
  • ENTERING_TIDY -> TIDY_ACTIVE: no active or queued execution job remains for inbound, outbound, or outbound relocation.
  • TIDY_ACTIVE -> EXITING_TIDY: the operator turns the switch off. Claimed/planned work with no WCS task is cancelled and its reservations are released; dispatched or running work drains.
  • EXITING_TIDY -> NORMAL: all dispatched or running manual-tidy workflows reach a terminal state and their locks are released.

The requested switch value and effective state are stored separately. Repeated requests to set the same value are idempotent.

5. Architecture

5.1 Components

TidyModeService

Owns the persisted switch and state transitions. It exposes the effective mode, requested switch value, drain counts, active tidy count, vehicle capacity, candidate count, and last blocking reason.

Unified execution admission gate

Every code path that can allocate or create execution work must call one shared gate before beginning its transaction. This includes ERP/API, PDA, scheduled, automatic-allocation, manual-allocation, and retry paths.

The gate makes an admission decision for a work origin, not merely for a controller endpoint. This prevents an internal service or retry path from bypassing manual-tidy exclusivity.

Business documents may be accepted in every mode. In ENTERING_TIDY, TIDY_ACTIVE, and EXITING_TIDY, their allocation/execution state remains pending until the mode returns to NORMAL.

TidyScheduler

Performs the 10-minute automatic scan and short-interval mode reconciliation. It must use a database-backed singleton claim so multiple WMS instances cannot start duplicate automatic tidy work.

TidyWorkflowCoordinator

Requests candidates from the existing TidyPlanner, validates current capacity and mode, claims non-conflicting candidates, and creates workflows. Manual coordination fills the current capacity; automatic coordination is limited to one workflow.

TidyWorkflowTransaction

Creates and advances the persistent workflow, locks source and destination spaces, creates the legacy in_or_out=28 move job, handles callbacks, updates lane state, and releases locks only at the defined terminal point.

WCS vehicle-capacity adapter

Returns a stable vehicle ID plus online, fault, and current-assignment state for each vehicle; aggregate counts are derived from that snapshot. Production uses the WCS/device source; batch testing uses a simulator implementing the same adapter contract. A vehicle is healthy only when it is online and has no active fault. It is idle only when WCS reports no current assignment. Workflow admission uses the effectiveTidyCapacity and newTidySlots formulas in section 3.3.

Tidy control API and operator view

The WMS exposes status retrieval and idempotent switch update operations. The view shows the switch, effective state, drain progress, active tidy workflows, online healthy vehicle capacity, safe-candidate count, and the last failure or blocking reason. Authorization and audit fields follow existing WMS management conventions.

5.2 Reused planning behavior

The existing TidyPlanner remains the source of candidates:

  • GUEST_RETURN;
  • DOWNGRADE_MERGE;
  • CONSOLIDATE_INCUBATION.

The coordinator consumes planner output but does not trust it as a reservation. All candidate preconditions and locks are checked again inside the workflow-creation transaction.

5.3 Work origin

Every in_or_out=28 job must be traceable to exactly one origin:

  • AUTO_TIDY;
  • MANUAL_TIDY;
  • OUTBOUND_RELOCATION.

The origin is persisted on the owning algorithm workflow and exposed on job-query DTOs by joining the workflow relationship. The legacy in_or_out value is not overloaded with new numeric meanings.

6. Persistence

The SQL Server migration is idempotent and additive. It creates missing algorithm objects without deleting business data.

6.1 Existing algorithm objects required in the target database

The migration must install the version-pinned definitions maintained in the same commit under fortucky-algorithm/src/main/resources/db/sqlserver/install.sql when absent:

  • alg.outbound_workflow;
  • alg.outbound_workflow_step;
  • alg.outbound_workflow_space;
  • alg.dirty_lane_queue.

These objects are in scope because deep outbound allocation, natural blocker relocation, restart recovery, and the required outbound reconciliation cannot execute without them. alg.dirty_lane_queue retains its existing responsibility as a short-lived topology/inventory/lane-state rebuild queue; it is not reused as the persistent tidy work queue. Their existing behavior is not otherwise redesigned by this feature.

6.2 New tidy objects

alg.tidy_mode

One persisted mode row per controlled WMS scope, containing the requested switch value, effective state, optimistic version, status reason, updater, and timestamps.

alg.tidy_workflow

One row per tidy candidate execution, containing warehouse/lane, origin, planner candidate type, source pallet and space, destination space, workflow status, plan version, related move-job identifier, failure reason, and lifecycle timestamps.

alg.tidy_step

Ordered execution steps and callback progress for a tidy workflow. A unique workflow-and-sequence constraint prevents duplicate step creation.

alg.tidy_lane_queue

Persistent, versioned tidy eligibility per warehouse and storage lane. It stores dirty_version, PENDING/CLAIMED state, claim owner/lease, retry time/reason, and latest business-change metadata. It survives consumption of the existing rebuild queue and application restarts.

alg.tidy_space_lock

Database-enforced active resource locks for each tidy workflow. Despite the legacy table name, each row has resource_type (SPACE, LANE, or FLOOR) and resource_key. The claimed set contains source/destination spaces, intervening access spaces, touched storage lanes, required buffers, and floor exclusivity when a deep workflow uses shared floor resources. An active unique key on warehouse, resource type, and resource key prevents conflicting tidy workflows.

Cross-origin admission is serialized with a SQL Server transaction-scoped application lock for the warehouse/resource keys. While holding it, the transaction checks both existing outbound workflow locks and tidy resource locks before persisting either workflow. This closes the race between outbound relocation and tidy even though their persisted lock tables differ.

6.3 Required constraints

  • One effective alg.tidy_mode row per controlled scope.
  • A move job belongs to at most one tidy workflow step.
  • A space, lane, or floor resource has at most one active tidy lock, and the cross-origin transaction check prevents overlap with active outbound locks.
  • Callback event keys are unique within a workflow step.
  • Automatic workflow claim is singleton across application instances.
  • Workflow status transitions use optimistic version checks.

7. Runtime Data Flow

7.1 Dirty-lane marking

After an inbound, outbound, or relocation workflow commits its inventory and cargo-space result, the existing triggers upsert each affected lane in alg.dirty_lane_queue. For inbound this is the destination lane; for outbound it is the source lane; for relocation it is every storage lane whose occupancy changed on move-out or restore. Ports and relocation-buffer pseudo-lanes are excluded unless they are configured as storage lanes.

The existing dirty-lane processor first rebuilds topology, inventory snapshots, and lane state. In the same successful processing transaction, it upserts every rebuilt storage lane into alg.tidy_lane_queue; only then may it consume the corresponding alg.dirty_lane_queue row. A rebuild failure follows the existing requeue behavior and does not expose stale derived state to the tidy planner.

The tidy queue key is warehouse plus lane. Each upsert increments dirty_version and records latest reason/time. Queue state is PENDING or CLAIMED, with claim owner and lease expiry. A planner failure or expired claim returns the same version to PENDING with retry_after and reason. Completion clears the row only when its claimed dirty_version still equals the current version; a newer dirty event leaves it PENDING. This makes duplicate events, failures, restarts, and concurrent completions deterministic.

7.2 Automatic tidy flow

  1. The scheduler confirms NORMAL mode, no active inbound, outbound, or outbound-relocation execution, and no active tidy workflow.
  2. In one transaction it locks the mode row, verifies that no manual switch transition has begun, and claims the oldest eligible PENDING row from alg.tidy_lane_queue by last_dirty_at, then warehouse/lane code as the deterministic tie-break.
  3. The planner calculates a candidate from current inventory and topology.
  4. The transaction revalidates the candidate and claims all required space, lane, and floor resources.
  5. The workflow and its in_or_out=28 move job are committed together.
  6. Simulated or real WCS advances the existing movement callback chain.
  7. Inventory, cargo-space state, lane queue, workflow state, and locks are reconciled at completion.

7.3 Manual tidy flow

  1. The operator enables the switch; mode becomes ENTERING_TIDY and the shared gate blocks new execution jobs.
  2. The coordinator waits for existing inbound, outbound, and outbound-relocation work to drain.
  3. Mode becomes TIDY_ACTIVE.
  4. The capacity adapter returns the current per-vehicle snapshot and the coordinator derives effectiveTidyCapacity.
  5. The coordinator calculates newTidySlots and safely claims that many mutually disjoint candidates. A candidate passes only when every safety predicate and resource claim defined in section 3.3 succeeds in the creation transaction.
  6. Each completion, capacity change, or periodic reconciliation triggers another fill attempt.
  7. On switch-off, mode becomes EXITING_TIDY; no replacement tidy work is created. Claimed or waiting-dispatch workflows without a WCS task are cancelled and released immediately; workflows with a WCS task drain.
  8. After active tidy workflows drain, all locks are reconciled and mode returns to NORMAL, allowing pending business documents to allocate.

8. Concurrency, Idempotency, and Recovery

8.1 Planning and lock races

A plan has no authority until its workflow transaction acquires all required locks and revalidates inventory, cargo-space status, pallet position, lane geometry, and mode version. On conflict, the plan is discarded and recalculated from current state.

8.2 WCS callback idempotency

Callbacks are keyed by workflow, step, WCS task, and event stage. A duplicate callback returns the prior successful result and cannot create a second job, move inventory twice, or release a lock early. Out-of-order callbacks are rejected with the current expected stage recorded for diagnosis.

8.3 Vehicle state changes

An offline or faulty vehicle immediately reduces future manual-tidy capacity. A workflow that has already begun physical movement is not cancelled automatically. A planned but undispatched workflow may be returned to the candidate pool only after its reservations are safely released.

Superseded on 2026-07-23: WCS task-state lookup and the UNKNOWN-task replay case are replaced by WMS-native evidence recovery. See docs/superpowers/specs/2026-07-23-wms-native-tidy-recovery-design.md and docs/superpowers/plans/2026-07-23-wms-native-tidy-recovery.md. Vehicle snapshot behavior is not superseded.

8.4 Application restart

Mode, workflows, steps, and locks are authoritative in SQL Server. The WCS adapter supplies authoritative task state by WCS task ID and retains that simulated state across a WMS-only restart. Startup reconciliation:

  1. reloads the effective mode and requested switch;
  2. compares non-terminal workflows with legacy jobs and WCS task state;
  3. resumes a workflow when the database and WCS agree on its non-terminal stage;
  4. moves a workflow to non-terminal PAUSED_RECOVERY when its database stage has no matching authoritative WCS task, or when pallet/inventory/space evidence permits more than one physical interpretation;
  5. repairs a stale lock only when its owner is terminal, or when the owner is missing and there is no active legacy/WCS task and inventory identifies one unambiguous stable pallet space;
  6. leaves all locks in place for PAUSED_RECOVERY and requires an audited operator reconciliation before retry or cancellation;
  7. resumes the appropriate drain or capacity-fill loop only after the preceding reconciliation finishes.

No in-memory flag is allowed to be the sole source of exclusivity.

8.5 Failed movement

A movement failure moves its workflow to PAUSED_RECOVERY and retains the locks needed to represent the uncertain physical state without a time-based auto-release. An operator or recovery service must reconcile the pallet's physical position with WMS before retry or cancellation. The only successful terminal states are COMPLETED and CANCELLED_RECONCILED. Cancellation is allowed only when inventory, cargo-space state, legacy job state, and locks can be moved to CANCELLED_RECONCILED in one transaction. An unexpected PAUSED_RECOVERY row makes the batch run fail; a deliberately injected recovery scenario passes only when the expected pause and retained locks are observed, followed by a successful audited reconciliation.

9. Historical Replay Design

9.1 Fixtures and preservation

The following imported finish data is the immutable historical fixture source:

  • in_stockin_finish.sql;
  • in_stockin_detail_finish.sql;
  • mydo_job_finish.sql;
  • mydo_job_detail_finish.sql;
  • out_stockout_finish.sql;
  • out_stockout_detail_finish.sql.

The replay creates new run-scoped open documents and jobs. It does not turn finish rows back into active rows and does not update the source files.

Historical warehouse identifiers 07 or blank are mapped to the available topology warehouse 01; an original value of 01 is retained as 01. Any other warehouse value is excluded from the eligible set and reported as UNSUPPORTED_SOURCE_WAREHOUSE; it is never silently mapped. Every considered row retains its original warehouse value and mapping decision in the replay report. The known orphan outbound detail CKD202605140015-1 is excluded from executable samples and reported as a fixture-integrity exception.

An eligible inbound detail has an existing historical header, positive quantity, resolvable product/master data, an original warehouse covered by the closed mapping rule above, and a tray that is unique within the replay run. Each historical detail may be replayed at most once per run. Eligible rows are ordered by historical completion time, document number, and detail number so the 100/300 sample boundary is reproducible. A failed execution counts as an attempted reconciliation row but not as a passed sample.

9.2 Identifier safety

Before replay, document, job, task, and inventory generators must consider both active and finish tables under a concurrency-safe allocation mechanism. The existing collision around CK202607210002 is a mandatory regression case. Replay business identifiers also carry a unique run prefix so cleanup and reconciliation cannot select unrelated data.

9.3 Test environment sequence

  1. Confirm the exact SQL Server instance and FT546-wms development database.
  2. Re-query baseline row counts, cargo-space state, active work, vehicle state, and algorithm schema; do not rely on recorded counts alone.
  3. Apply and verify the idempotent algorithm migration.
  4. Ensure no active business execution exists and validate the cargo-space baseline. The recorded 692-space count is a reference: if the live configured count differs, preflight stops for explicit operator validation. Consistency means every configured space has one valid state, every occupied storage space has matching pallet inventory, every inventory row resolves to one configured occupied space, and no active workflow/resource lock remains.
  5. Create a new restorable pre-test backup after migration and before replay.
  6. Start WMS plus PDA/WCS simulators and assign a unique replay run ID.
  7. Keep normal deployment default at SHADOW, but enable a run-scoped execution override only for the confirmed development database and replay run ID. Execute inbound, outbound, automatic-tidy, and manual-tidy phases through real application services and HTTP callbacks, then remove the override before final restoration.
  8. Freeze execution, generate reconciliation reports, and retain logs and failing payloads.
  9. Stop writers and restore the pre-test backup to return to the imported-history baseline with the migrated schema.
  10. Re-query counts and integrity checks to prove restoration.

No destructive cleanup or restore runs unless the resolved target is the confirmed development database and the new pre-test backup has been verified.

10. Batch Test Matrix

10.1 Inbound phase

Replay the first 100 eligible historical inbound details in the deterministic order from section 9.1. Compute outbound strata from a committed database snapshot. If any stratum has fewer than 25 eligible source pallets, replay the next deterministic inbound block and recompute, stopping when every stratum has 25 candidates or 300 unique eligible details have been attempted. Reaching 300, or exhausting the eligible set earlier, without all four strata is a run-level OUTBOUND_STRATUM_COVERAGE_FAILED; remaining phases may collect diagnostics but the run cannot pass.

For every inbound detail, record and assert:

  • original historical document/detail and normalized replay identifiers;
  • product, batch, pallet, and requested quantity;
  • assigned cargo space exists in warehouse 01 and satisfies planner/topology rules;
  • allocation order does not create an invalid inner/outer accessibility state;
  • exactly one current inventory aggregate for the replay pallet/product/batch/location key, with no duplicate active stock rows representing the same physical pallet content;
  • inventory product, batch, quantity, pallet, and cargo-space code match the completed movement;
  • cargo-space occupancy and all workflow/space/floor locks match the terminal state;
  • duplicate PDA/WCS completion calls do not duplicate inventory.

10.2 Outbound phase

An eligible outbound fixture has an existing historical header, positive quantity, resolvable product/master data, a warehouse covered by section 9.1, and sufficient matching replay-created inventory at selection time. The orphan fixture is never eligible. Fixtures are considered by historical completion time, document number, and detail number and used at most once per run.

Execute exactly 25 successful outbound details in each current-blocker stratum 0, 1, 2, and 3+ for a minimum of 100 successful cases. Selection is just-in-time from a committed snapshot because preceding outbounds can change accessibility. At least 50 cases must omit a tray override and therefore exercise planner source selection. Remaining stratum-coverage cases may request a historical source tray only when the imported finished job/detail relationship proves that tray belonged to the same historical outbound fixture and that tray exists in replay-created inventory. The report labels each case PLANNER_SELECTED or HISTORICAL_TRAY_OVERRIDE. Failed attempts remain in the report and are replaced from the same stratum, but still make the overall run fail. Do not fabricate blockers or directly edit cargo-space positions.

The source-pallet oracle uses the same published contract as OutboundPlanner, evaluated independently from the committed pre-allocation snapshot:

  • an explicitly requested tray selects that exact eligible tray;
  • with FIFO off: lowest blocker count, then lane-release rank, floor number, and cargo-space ID;
  • with strict FIFO: oldest inbound timestamp, then blocker count, lane-release rank, efficiency score descending, timestamp, and cargo-space ID;
  • with tolerant FIFO: restrict to timestamps no later than oldest plus configured tolerance, then apply the same cost ordering as strict FIFO.

All comparisons and the final cargo-space-ID tie-break are recorded in the report. The production planner result must equal this independently calculated oracle result.

For every outbound detail, record and assert:

  • requested product, batch/selection constraints, and quantity;
  • selected source pallet existed and held eligible stock; PLANNER_SELECTED cases must equal the independent oracle, while HISTORICAL_TRAY_OVERRIDE cases must equal the proven historical tray;
  • blockers are derived from the committed current topology after source selection;
  • for N blockers, the owning workflow has exactly N MOVE_OUT, one OUTBOUND, and N RESTORE steps, each resolving to exactly one distinct linked legacy job; exactly 2N linked jobs have origin OUTBOUND_RELOCATION and one is the formal outbound job;
  • each blocker identity and temporary/restore destination is correct and non-conflicting;
  • only the formal outbound quantity is deducted from inventory;
  • blocking pallets retain correct inventory and end in the planned restored/final spaces;
  • source and blocking cargo-space states match physical completion;
  • workflow reservations, space locks, and floor locks are fully released at terminal completion;
  • repeated and out-of-order WCS callbacks follow the idempotency contract.

An outbound test cannot pass merely because the document closes; its source pallet, blockers, job counts, inventory delta, final pallet positions, and lock state must all reconcile.

10.3 Automatic tidy phase

  • Create dirty-lane events through completed replay work, not by inserting a tidy workflow directly.
  • Verify the 10-minute scan or its test-controlled clock selects pending lanes.
  • Keep inbound/outbound work active and prove no automatic tidy starts.
  • Drain business work and prove exactly one automatic tidy may run.
  • Present multiple safe candidates and prove a second automatic tidy does not start until the first is terminal.
  • Force a lock race or stale candidate and prove replanning occurs without using the old destination.
  • Verify origin AUTO_TIDY, inventory/cargo-space correctness, and lock release. A claimed dirty version becomes clean only after successful tidy; a newer dirty version or failed plan remains PENDING under the queue rules in section 7.1.

10.4 Manual tidy phase

  • Simulate three online, healthy vehicles and enough independent candidates; prove three manual-tidy workflows are admitted after the business drain.
  • Complete one workflow and prove one replacement is created while the switch remains on.
  • Reduce healthy capacity from three to two and prove no new work exceeds two active workflows after current physical work naturally drains.
  • Provide only two safe candidates with three healthy vehicles and prove only two workflows are created with NO_SAFE_CANDIDATE/capacity status visible for the unfilled slot.
  • Turn the switch on while inbound/outbound work exists and prove new execution allocation is blocked while existing work drains.
  • Submit new business documents during TIDY_ACTIVE and prove they are accepted but remain unallocated.
  • Turn the switch off and prove no replacement tidy is created, active tidy drains, mode returns to NORMAL, and pending business documents resume allocation.
  • Restart WMS at three reproducible non-normal checkpoints: ENTERING_TIDY with one held inbound callback, TIDY_ACTIVE with one running WCS task plus one waiting-dispatch workflow, and EXITING_TIDY with one running tidy task. After restart, verify respectively that admission remains blocked and the drain reaches active mode; WCS-agreed work resumes without duplicate dispatch and safe waiting work is reconciled; and no replacement starts before the running task completes and mode returns to NORMAL.
  • Add one recovery-negative scenario in which WCS deliberately returns unknown for a database-running task. Verify PAUSED_RECOVERY, retained resource locks, no replacement job, then audited reconciliation to COMPLETED or CANCELLED_RECONCILED.

11. Reporting and Acceptance

Each run writes artifacts under outputs/ft546-tidy-batch-replay-<run-id>/:

  • inbound-reconciliation.csv: one row per inbound detail and each asserted field;
  • outbound-reconciliation.csv: one row per outbound detail, source pallet, blocker count, expected/actual step counts, inventory delta, and lock result;
  • tidy-reconciliation.csv: one row per automatic/manual tidy workflow, origin, candidate type, vehicle assignment, positions, inventory, and terminal state;
  • events.json: ordered request, callback, state-transition, and database evidence with secrets removed;
  • report.md: environment fingerprint, migration result, sample counts, stratum coverage, pass/fail summary, exceptions, restoration proof, and links to raw artifacts.

Acceptance requires all of the following:

  • at least 100 inbound details and 100 successful outbound details executed through the real allocation/workflow path, with every attempted eligible inbound required to reconcile successfully;
  • exactly 25 successful cases in each of the four outbound blocker strata; OUTBOUND_STRATUM_COVERAGE_FAILED is a run failure and is shown with available/required counts by stratum;
  • zero unexplained inventory, pallet-position, cargo-space, workflow-step, or lock mismatches;
  • automatic tidy never exceeds one active workflow and never starts while business execution is active;
  • manual tidy respects effective healthy-vehicle capacity and mode exclusivity;
  • duplicate callbacks produce no duplicate side effects;
  • service restart recovery passes for all non-normal modes;
  • every unexpected failure is represented as a failed reconciliation row with evidence, never omitted from aggregate percentages; controlled negative scenarios are separately labelled and pass only when their exact expected state, locks, and recovery outcome match;
  • the database is restored and verified against the pre-test imported-history baseline.

12. Implementation Boundaries

  • Preserve the fortucky-algorithm module boundary and existing WMS transaction manager/data source integration.
  • Keep the normal deployment default at SHADOW. The batch harness may use only the run-scoped development-database execution override defined in section 9.3; it must remove that override before restoration. Promotion to CANARY or ACTIVE remains a separate decision after SQL Server diagnostics, concurrency, idempotency, recovery, backup, and rollback gates pass.
  • Add only focused changes required for admission, tidy orchestration, persistence, UI control/status, simulation, and reconciliation.
  • Deliver in three ordered implementation phases: (A) migrations, origin tracing, and the unified admission gate; (B) persisted tidy state/workflow, WCS capacity, minimal operator switch/status, and recovery; (C) simulators, historical replay, reconciliation, and restoration tooling. Each phase has its own automated tests, but the feature is accepted only after the complete batch matrix passes.
  • Do not revert or overwrite unrelated uncommitted work already present in the repository.
  • Implementation must follow test-driven development: failing unit/integration tests first, then production changes, followed by SQL Server replay verification.