FT546MWS/docs/superpowers/specs/2026-07-23-wms-native-tidy-recovery-design.md

WMS-Native Tidy Recovery Design

Date: 2026-07-23
Status: Approved Supersedes: the WCS task-state recovery portions of 2026-07-22-ft546-tidy-and-batch-replay-design.md

Context

A tidy step is executed as a normal WMS move job with in_or_out=28. The WMS creates that job, completes inventory movement inside alg.usp_complete_move_job, archives it to mydo_job_finish, advances alg.tidy_step, and releases workflow locks through the transactional jobFinish callback path.

The current restart reconciler also polls a WCS tidyTaskState endpoint every five seconds. The production WCS does not implement that endpoint; only the replay simulator does. More importantly, WCS task state is not the authoritative source for WMS job or tidy-workflow completion.

Goals

1. Make WMS persisted state the only authority for tidy completion and restart recovery.
2. Preserve normal in_or_out=28 job creation and callback execution.
3. Repair a locally provable completed step idempotently after restart.
4. Pause a workflow only when WMS-local evidence is missing or contradictory.
5. Retain active locks for PAUSED_RECOVERY workflows.
6. Remove the unused WCS task-state HTTP dependency and configuration.

Non-Goals

• Do not change the WCS vehicle snapshot endpoint; manual capacity still needs live vehicle availability.
• Do not infer physical completion from vehicle telemetry.
• Do not auto-complete a move without a matching WMS finish job and matching inventory/space evidence.
• Do not repair unrelated legacy WMS/WCS endpoint mismatches in this change.
• Do not change outbound-relocation workflow recovery.

Considered Approaches

A. WMS-native evidence recovery (selected)

Use alg.tidy_workflow, alg.tidy_step, mydo_job, mydo_job_finish, bas_stock, bas_cargo_space, and alg.tidy_space_lock. This keeps one source of truth, requires no new WCS API, and can distinguish safe repair from a true local inconsistency.

B. Keep WCS state as advisory data

Query WCS only after a timeout and compare it with WMS state. This improves cross-system diagnostics but still requires a new production WCS endpoint, defines a second state contract, and cannot replace WMS transactional evidence.

C. Callback-only recovery

Keep locks and wait forever for a callback. This is simple but leaves orphaned or partially imported records with no deterministic diagnosis or repair path.

Approach A is selected. WCS task querying can be introduced later as operational telemetry without making it a completion authority.

Authoritative Data

For each active tidy workflow, recovery evaluates its first incomplete step and the deterministic move-job ID:

TDY-<workflowId>-<stepId>

The recovery projection contains:

• workflow ID, workflow status, origin, and active lock count;
• step ID, step number, step status, source space, target space, and tray code;
• matching active mydo_job presence, status, type, source, target, and tray;
• matching mydo_job_finish presence, status, type, source, target, and tray;
• stock location and source/target cargo-space occupancy.

An active or finished job is a match only when all of these values agree with the tidy step:

job_id, in_or_out=28, tray_code, cargo_space_id, target_cargo_space_id

Recovery Decision Table

WMS-local evidence	Action
Workflow already PAUSED_RECOVERY	Leave unchanged; do not poll or release locks.
All workflow steps are COMPLETED, and no deterministic tidy job remains active	Idempotently complete the workflow and release its locks.
Matching active move job has status 1, 2, or 3; stock remains at source; source contains the tray; target does not contain it	Leave the step and locks active; normal dispatch/callback processing continues.
Matching active move job exists but stock or cargo-space evidence is not in the expected pre-move state	Pause with reason RECOVERY_ACTIVE_EVIDENCE_INVALID; retain locks.
No active job; matching finish job has status 4; stock is at target; source is empty; target contains the tray	Replay only the tidy step/workflow advancement using callback stage RECOVERY_FINISHED. Do not execute inventory movement again.
Active or finished job exists but its type, tray, source, or target differs	Pause the workflow and active step with reason RECOVERY_JOB_MISMATCH; retain locks.
Finish job exists but status or inventory/space evidence is not terminal and consistent	Pause with reason RECOVERY_FINISH_EVIDENCE_INVALID; retain locks.
Neither matching active nor matching finish job exists	Pause with reason RECOVERY_JOB_MISSING; retain locks.

Database/query failure is not evidence of inconsistency. The transaction rolls back, the scheduler logs the failure, and the workflow remains unchanged until the next reconciliation cycle.

Components

TidyWorkflowRecoveryService

The service remains the restart and periodic reconciliation boundary, but it depends only on WmsAlgorithmMapper and TidyWorkflowTransaction. It never performs HTTP. It evaluates mapper projections inside the WMS transaction manager and applies exactly one decision per workflow per cycle.

TidyRecoveryEvidence

A new immutable mapper projection carries the authoritative fields listed above. It contains data only; decision logic stays in the recovery service.

TidyWorkflowTransaction

Normal MOVE_FINISHED and OUTBOUND_FINISHED callback behavior remains unchanged. A locally proven finish job is advanced through the same idempotent path with callback stage RECOVERY_FINISHED. The special UNKNOWN callback branch is removed; only the recovery service may enter PAUSED_RECOVERY.

Mapper SQL

Replace the WCS-task-oriented recovery selection with WMS-local evidence SQL. The query must use the deterministic job ID and left joins to both active and finish job tables. Finished evidence must also project stock and cargo-space state. Existing complete-step, complete-workflow, release-lock, pause-workflow, and pause-step operations remain the mutation boundary.

Configuration and deleted types

Delete:

• WcsTaskState;
• WcsTaskStateClient;
• tidy-wcs-task-state-url and tidy-wcs-task-state-timeout-ms properties;
• FORTUCKY_TIDY_WCS_TASK_STATE_URL and timeout overrides.

Keep tidy-wcs-snapshot-url and WcsVehicleSnapshotClient unchanged.

Concurrency and Idempotency

• Recovery runs in one WMS transaction.
• The deterministic job ID prevents a second move job for the same step.
• Callback-event insertion remains idempotent by workflow, step, job, and stage.
• A finish-job repair never calls alg.usp_complete_move_job; inventory is not moved twice.
• Concurrent duplicate callbacks may observe an already completed step, but next-job creation and resource release remain idempotent.
• PAUSED_RECOVERY never releases locks automatically.

Replay Changes

Remove the simulator /sim/task-state and /sim/unknown-task endpoints and the FORTUCKY_TIDY_WCS_TASK_STATE_URL replay override.

The replay will add two WMS-local restart cases while the database is protected by the existing verified backup:

1. Repairable completion: stop WMS after creating a tidy move job, complete the move through alg.usp_complete_move_job without advancing the tidy step, restart WMS, and prove the step/workflow advances without a second inventory move.
2. Missing local job: stop WMS, remove the active replay move job while leaving its workflow/step/locks, restart WMS, and prove PAUSED_RECOVERY, reason RECOVERY_JOB_MISSING, and retained locks.

Both fixture mutations are run-scoped, journaled, and removed by the existing database restore. They are test setup, not evidence patching.

Tests and Acceptance

Unit and mapper tests must prove:

• active matching job remains active and causes no mutation;
• matching terminal finish evidence advances exactly once;
• completed steps with no remaining active tidy job finish the workflow and release locks;
• active jobs with invalid pre-move inventory evidence pause safely;
• missing job pauses workflow and step while retaining locks;
• mismatched or invalid finish evidence pauses with the correct reason;
• an already paused workflow causes no repeated action;
• database/client configuration contains no WCS task-state dependency;
• normal and duplicate completion callbacks still pass.

Replay acceptance must prove:

• normal automatic and manual tidy behavior remains unchanged;
• repairable local completion is recovered after restart;
• missing local job enters PAUSED_RECOVERY with retained locks;
• no task-state HTTP request appears in the event journal;
• database restoration still reports mismatches=[].

Documentation

The existing 2026-07-22 design and plan remain historical records. Add a supersession note to their WCS task-state recovery sections, update the handoff document to describe WMS-native recovery, and generate a new final replay report instead of rewriting old evidence.

Risks

• Legacy or manually edited finish rows may look terminal but disagree with inventory. The design pauses rather than guessing.
• A workflow can remain paused until an operator repairs or reconciles the local inconsistency. Automatic lock release is intentionally prohibited.
• The replay uses controlled SQL mutations to reproduce crash boundaries; every such mutation must be logged and covered by final baseline restoration.