# WMS-Native Tidy Recovery Design

> Date: 2026-07-23  
> Status: Approved
> Supersedes: the WCS task-state recovery portions of `2026-07-22-ft546-tidy-and-batch-replay-design.md`

## Context

A tidy step is executed as a normal WMS move job with `in_or_out=28`. The WMS
creates that job, completes inventory movement inside `alg.usp_complete_move_job`,
archives it to `mydo_job_finish`, advances `alg.tidy_step`, and releases workflow
locks through the transactional `jobFinish` callback path.

The current restart reconciler also polls a WCS `tidyTaskState` endpoint every
five seconds. The production WCS does not implement that endpoint; only the
replay simulator does. More importantly, WCS task state is not the authoritative
source for WMS job or tidy-workflow completion.

## Goals

1. Make WMS persisted state the only authority for tidy completion and restart
   recovery.
2. Preserve normal `in_or_out=28` job creation and callback execution.
3. Repair a locally provable completed step idempotently after restart.
4. Pause a workflow only when WMS-local evidence is missing or contradictory.
5. Retain active locks for `PAUSED_RECOVERY` workflows.
6. Remove the unused WCS task-state HTTP dependency and configuration.

## Non-Goals

- Do not change the WCS vehicle snapshot endpoint; manual capacity still needs
  live vehicle availability.
- Do not infer physical completion from vehicle telemetry.
- Do not auto-complete a move without a matching WMS finish job and matching
  inventory/space evidence.
- Do not repair unrelated legacy WMS/WCS endpoint mismatches in this change.
- Do not change outbound-relocation workflow recovery.

## Considered Approaches

### A. WMS-native evidence recovery (selected)

Use `alg.tidy_workflow`, `alg.tidy_step`, `mydo_job`, `mydo_job_finish`,
`bas_stock`, `bas_cargo_space`, and `alg.tidy_space_lock`. This keeps one source
of truth, requires no new WCS API, and can distinguish safe repair from a true
local inconsistency.

### B. Keep WCS state as advisory data

Query WCS only after a timeout and compare it with WMS state. This improves
cross-system diagnostics but still requires a new production WCS endpoint,
defines a second state contract, and cannot replace WMS transactional evidence.

### C. Callback-only recovery

Keep locks and wait forever for a callback. This is simple but leaves orphaned
or partially imported records with no deterministic diagnosis or repair path.

Approach A is selected. WCS task querying can be introduced later as operational
telemetry without making it a completion authority.

## Authoritative Data

For each active tidy workflow, recovery evaluates its first incomplete step and
the deterministic move-job ID:

```text
TDY-<workflowId>-<stepId>
```

The recovery projection contains:

- workflow ID, workflow status, origin, and active lock count;
- step ID, step number, step status, source space, target space, and tray code;
- matching active `mydo_job` presence, status, type, source, target, and tray;
- matching `mydo_job_finish` presence, status, type, source, target, and tray;
- stock location and source/target cargo-space occupancy.

An active or finished job is a match only when all of these values agree with
the tidy step:

```text
job_id, in_or_out=28, tray_code, cargo_space_id, target_cargo_space_id
```

## Recovery Decision Table

| WMS-local evidence | Action |
| --- | --- |
| Workflow already `PAUSED_RECOVERY` | Leave unchanged; do not poll or release locks. |
| All workflow steps are `COMPLETED`, and no deterministic tidy job remains active | Idempotently complete the workflow and release its locks. |
| Matching active move job has status `1`, `2`, or `3`; stock remains at source; source contains the tray; target does not contain it | Leave the step and locks active; normal dispatch/callback processing continues. |
| Matching active move job exists but stock or cargo-space evidence is not in the expected pre-move state | Pause with reason `RECOVERY_ACTIVE_EVIDENCE_INVALID`; retain locks. |
| No active job; matching finish job has status `4`; stock is at target; source is empty; target contains the tray | Replay only the tidy step/workflow advancement using callback stage `RECOVERY_FINISHED`. Do not execute inventory movement again. |
| Active or finished job exists but its type, tray, source, or target differs | Pause the workflow and active step with reason `RECOVERY_JOB_MISMATCH`; retain locks. |
| Finish job exists but status or inventory/space evidence is not terminal and consistent | Pause with reason `RECOVERY_FINISH_EVIDENCE_INVALID`; retain locks. |
| Neither matching active nor matching finish job exists | Pause with reason `RECOVERY_JOB_MISSING`; retain locks. |

Database/query failure is not evidence of inconsistency. The transaction rolls
back, the scheduler logs the failure, and the workflow remains unchanged until
the next reconciliation cycle.

## Components

### `TidyWorkflowRecoveryService`

The service remains the restart and periodic reconciliation boundary, but it
depends only on `WmsAlgorithmMapper` and `TidyWorkflowTransaction`. It never
performs HTTP. It evaluates mapper projections inside the WMS transaction
manager and applies exactly one decision per workflow per cycle.

### `TidyRecoveryEvidence`

A new immutable mapper projection carries the authoritative fields listed
above. It contains data only; decision logic stays in the recovery service.

### `TidyWorkflowTransaction`

Normal `MOVE_FINISHED` and `OUTBOUND_FINISHED` callback behavior remains
unchanged. A locally proven finish job is advanced through the same idempotent
path with callback stage `RECOVERY_FINISHED`. The special `UNKNOWN` callback
branch is removed; only the recovery service may enter `PAUSED_RECOVERY`.

### Mapper SQL

Replace the WCS-task-oriented recovery selection with WMS-local evidence SQL.
The query must use the deterministic job ID and left joins to both active and
finish job tables. Finished evidence must also project stock and cargo-space
state. Existing complete-step, complete-workflow, release-lock, pause-workflow,
and pause-step operations remain the mutation boundary.

### Configuration and deleted types

Delete:

- `WcsTaskState`;
- `WcsTaskStateClient`;
- `tidy-wcs-task-state-url` and `tidy-wcs-task-state-timeout-ms` properties;
- `FORTUCKY_TIDY_WCS_TASK_STATE_URL` and timeout overrides.

Keep `tidy-wcs-snapshot-url` and `WcsVehicleSnapshotClient` unchanged.

## Concurrency and Idempotency

- Recovery runs in one WMS transaction.
- The deterministic job ID prevents a second move job for the same step.
- Callback-event insertion remains idempotent by workflow, step, job, and stage.
- A finish-job repair never calls `alg.usp_complete_move_job`; inventory is not
  moved twice.
- Concurrent duplicate callbacks may observe an already completed step, but
  next-job creation and resource release remain idempotent.
- `PAUSED_RECOVERY` never releases locks automatically.

## Replay Changes

Remove the simulator `/sim/task-state` and `/sim/unknown-task` endpoints and the
`FORTUCKY_TIDY_WCS_TASK_STATE_URL` replay override.

The replay will add two WMS-local restart cases while the database is protected
by the existing verified backup:

1. **Repairable completion:** stop WMS after creating a tidy move job, complete
   the move through `alg.usp_complete_move_job` without advancing the tidy step,
   restart WMS, and prove the step/workflow advances without a second inventory
   move.
2. **Missing local job:** stop WMS, remove the active replay move job while
   leaving its workflow/step/locks, restart WMS, and prove
   `PAUSED_RECOVERY`, reason `RECOVERY_JOB_MISSING`, and retained locks.

Both fixture mutations are run-scoped, journaled, and removed by the existing
database restore. They are test setup, not evidence patching.

## Tests and Acceptance

Unit and mapper tests must prove:

- active matching job remains active and causes no mutation;
- matching terminal finish evidence advances exactly once;
- completed steps with no remaining active tidy job finish the workflow and
  release locks;
- active jobs with invalid pre-move inventory evidence pause safely;
- missing job pauses workflow and step while retaining locks;
- mismatched or invalid finish evidence pauses with the correct reason;
- an already paused workflow causes no repeated action;
- database/client configuration contains no WCS task-state dependency;
- normal and duplicate completion callbacks still pass.

Replay acceptance must prove:

- normal automatic and manual tidy behavior remains unchanged;
- repairable local completion is recovered after restart;
- missing local job enters `PAUSED_RECOVERY` with retained locks;
- no task-state HTTP request appears in the event journal;
- database restoration still reports `mismatches=[]`.

## Documentation

The existing 2026-07-22 design and plan remain historical records. Add a
supersession note to their WCS task-state recovery sections, update the handoff
document to describe WMS-native recovery, and generate a new final replay report
instead of rewriting old evidence.

## Risks

- Legacy or manually edited finish rows may look terminal but disagree with
  inventory. The design pauses rather than guessing.
- A workflow can remain paused until an operator repairs or reconciles the local
  inconsistency. Automatic lock release is intentionally prohibited.
- The replay uses controlled SQL mutations to reproduce crash boundaries; every
  such mutation must be logged and covered by final baseline restoration.