docs: define WMS-native tidy recovery
This commit is contained in:
@@ -0,0 +1,210 @@
|
||||
# WMS-Native Tidy Recovery Design
|
||||
|
||||
> Date: 2026-07-23
|
||||
> Status: Approved direction, awaiting written-spec review
|
||||
> Supersedes: the WCS task-state recovery portions of `2026-07-22-ft546-tidy-and-batch-replay-design.md`
|
||||
|
||||
## Context
|
||||
|
||||
A tidy step is executed as a normal WMS move job with `in_or_out=28`. The WMS
|
||||
creates that job, completes inventory movement inside `alg.usp_complete_move_job`,
|
||||
archives it to `mydo_job_finish`, advances `alg.tidy_step`, and releases workflow
|
||||
locks through the transactional `jobFinish` callback path.
|
||||
|
||||
The current restart reconciler also polls a WCS `tidyTaskState` endpoint every
|
||||
five seconds. The production WCS does not implement that endpoint; only the
|
||||
replay simulator does. More importantly, WCS task state is not the authoritative
|
||||
source for WMS job or tidy-workflow completion.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Make WMS persisted state the only authority for tidy completion and restart
|
||||
recovery.
|
||||
2. Preserve normal `in_or_out=28` job creation and callback execution.
|
||||
3. Repair a locally provable completed step idempotently after restart.
|
||||
4. Pause a workflow only when WMS-local evidence is missing or contradictory.
|
||||
5. Retain active locks for `PAUSED_RECOVERY` workflows.
|
||||
6. Remove the unused WCS task-state HTTP dependency and configuration.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not change the WCS vehicle snapshot endpoint; manual capacity still needs
|
||||
live vehicle availability.
|
||||
- Do not infer physical completion from vehicle telemetry.
|
||||
- Do not auto-complete a move without a matching WMS finish job and matching
|
||||
inventory/space evidence.
|
||||
- Do not repair unrelated legacy WMS/WCS endpoint mismatches in this change.
|
||||
- Do not change outbound-relocation workflow recovery.
|
||||
|
||||
## Considered Approaches
|
||||
|
||||
### A. WMS-native evidence recovery (selected)
|
||||
|
||||
Use `alg.tidy_workflow`, `alg.tidy_step`, `mydo_job`, `mydo_job_finish`,
|
||||
`bas_stock`, `bas_cargo_space`, and `alg.tidy_space_lock`. This keeps one source
|
||||
of truth, requires no new WCS API, and can distinguish safe repair from a true
|
||||
local inconsistency.
|
||||
|
||||
### B. Keep WCS state as advisory data
|
||||
|
||||
Query WCS only after a timeout and compare it with WMS state. This improves
|
||||
cross-system diagnostics but still requires a new production WCS endpoint,
|
||||
defines a second state contract, and cannot replace WMS transactional evidence.
|
||||
|
||||
### C. Callback-only recovery
|
||||
|
||||
Keep locks and wait forever for a callback. This is simple but leaves orphaned
|
||||
or partially imported records with no deterministic diagnosis or repair path.
|
||||
|
||||
Approach A is selected. WCS task querying can be introduced later as operational
|
||||
telemetry without making it a completion authority.
|
||||
|
||||
## Authoritative Data
|
||||
|
||||
For each active tidy workflow, recovery evaluates its first incomplete step and
|
||||
the deterministic move-job ID:
|
||||
|
||||
```text
|
||||
TDY-<workflowId>-<stepId>
|
||||
```
|
||||
|
||||
The recovery projection contains:
|
||||
|
||||
- workflow ID, workflow status, origin, and active lock count;
|
||||
- step ID, step number, step status, source space, target space, and tray code;
|
||||
- matching active `mydo_job` presence, status, type, source, target, and tray;
|
||||
- matching `mydo_job_finish` presence, status, type, source, target, and tray;
|
||||
- stock location and source/target cargo-space occupancy.
|
||||
|
||||
An active or finished job is a match only when all of these values agree with
|
||||
the tidy step:
|
||||
|
||||
```text
|
||||
job_id, in_or_out=28, tray_code, cargo_space_id, target_cargo_space_id
|
||||
```
|
||||
|
||||
## Recovery Decision Table
|
||||
|
||||
| WMS-local evidence | Action |
|
||||
| --- | --- |
|
||||
| Workflow already `PAUSED_RECOVERY` | Leave unchanged; do not poll or release locks. |
|
||||
| All workflow steps are `COMPLETED`, and no deterministic tidy job remains active | Idempotently complete the workflow and release its locks. |
|
||||
| Matching active move job has status `1`, `2`, or `3`; stock remains at source; source contains the tray; target does not contain it | Leave the step and locks active; normal dispatch/callback processing continues. |
|
||||
| Matching active move job exists but stock or cargo-space evidence is not in the expected pre-move state | Pause with reason `RECOVERY_ACTIVE_EVIDENCE_INVALID`; retain locks. |
|
||||
| No active job; matching finish job has status `4`; stock is at target; source is empty; target contains the tray | Replay only the tidy step/workflow advancement using callback stage `RECOVERY_FINISHED`. Do not execute inventory movement again. |
|
||||
| Active or finished job exists but its type, tray, source, or target differs | Pause the workflow and active step with reason `RECOVERY_JOB_MISMATCH`; retain locks. |
|
||||
| Finish job exists but status or inventory/space evidence is not terminal and consistent | Pause with reason `RECOVERY_FINISH_EVIDENCE_INVALID`; retain locks. |
|
||||
| Neither matching active nor matching finish job exists | Pause with reason `RECOVERY_JOB_MISSING`; retain locks. |
|
||||
|
||||
Database/query failure is not evidence of inconsistency. The transaction rolls
|
||||
back, the scheduler logs the failure, and the workflow remains unchanged until
|
||||
the next reconciliation cycle.
|
||||
|
||||
## Components
|
||||
|
||||
### `TidyWorkflowRecoveryService`
|
||||
|
||||
The service remains the restart and periodic reconciliation boundary, but it
|
||||
depends only on `WmsAlgorithmMapper` and `TidyWorkflowTransaction`. It never
|
||||
performs HTTP. It evaluates mapper projections inside the WMS transaction
|
||||
manager and applies exactly one decision per workflow per cycle.
|
||||
|
||||
### `TidyRecoveryEvidence`
|
||||
|
||||
A new immutable mapper projection carries the authoritative fields listed
|
||||
above. It contains data only; decision logic stays in the recovery service.
|
||||
|
||||
### `TidyWorkflowTransaction`
|
||||
|
||||
Normal `MOVE_FINISHED` and `OUTBOUND_FINISHED` callback behavior remains
|
||||
unchanged. A locally proven finish job is advanced through the same idempotent
|
||||
path with callback stage `RECOVERY_FINISHED`. The special `UNKNOWN` callback
|
||||
branch is removed; only the recovery service may enter `PAUSED_RECOVERY`.
|
||||
|
||||
### Mapper SQL
|
||||
|
||||
Replace the WCS-task-oriented recovery selection with WMS-local evidence SQL.
|
||||
The query must use the deterministic job ID and left joins to both active and
|
||||
finish job tables. Finished evidence must also project stock and cargo-space
|
||||
state. Existing complete-step, complete-workflow, release-lock, pause-workflow,
|
||||
and pause-step operations remain the mutation boundary.
|
||||
|
||||
### Configuration and deleted types
|
||||
|
||||
Delete:
|
||||
|
||||
- `WcsTaskState`;
|
||||
- `WcsTaskStateClient`;
|
||||
- `tidy-wcs-task-state-url` and `tidy-wcs-task-state-timeout-ms` properties;
|
||||
- `FORTUCKY_TIDY_WCS_TASK_STATE_URL` and timeout overrides.
|
||||
|
||||
Keep `tidy-wcs-snapshot-url` and `WcsVehicleSnapshotClient` unchanged.
|
||||
|
||||
## Concurrency and Idempotency
|
||||
|
||||
- Recovery runs in one WMS transaction.
|
||||
- The deterministic job ID prevents a second move job for the same step.
|
||||
- Callback-event insertion remains idempotent by workflow, step, job, and stage.
|
||||
- A finish-job repair never calls `alg.usp_complete_move_job`; inventory is not
|
||||
moved twice.
|
||||
- Concurrent duplicate callbacks may observe an already completed step, but
|
||||
next-job creation and resource release remain idempotent.
|
||||
- `PAUSED_RECOVERY` never releases locks automatically.
|
||||
|
||||
## Replay Changes
|
||||
|
||||
Remove the simulator `/sim/task-state` and `/sim/unknown-task` endpoints and the
|
||||
`FORTUCKY_TIDY_WCS_TASK_STATE_URL` replay override.
|
||||
|
||||
The replay will add two WMS-local restart cases while the database is protected
|
||||
by the existing verified backup:
|
||||
|
||||
1. **Repairable completion:** stop WMS after creating a tidy move job, complete
|
||||
the move through `alg.usp_complete_move_job` without advancing the tidy step,
|
||||
restart WMS, and prove the step/workflow advances without a second inventory
|
||||
move.
|
||||
2. **Missing local job:** stop WMS, remove the active replay move job while
|
||||
leaving its workflow/step/locks, restart WMS, and prove
|
||||
`PAUSED_RECOVERY`, reason `RECOVERY_JOB_MISSING`, and retained locks.
|
||||
|
||||
Both fixture mutations are run-scoped, journaled, and removed by the existing
|
||||
database restore. They are test setup, not evidence patching.
|
||||
|
||||
## Tests and Acceptance
|
||||
|
||||
Unit and mapper tests must prove:
|
||||
|
||||
- active matching job remains active and causes no mutation;
|
||||
- matching terminal finish evidence advances exactly once;
|
||||
- completed steps with no remaining active tidy job finish the workflow and
|
||||
release locks;
|
||||
- active jobs with invalid pre-move inventory evidence pause safely;
|
||||
- missing job pauses workflow and step while retaining locks;
|
||||
- mismatched or invalid finish evidence pauses with the correct reason;
|
||||
- an already paused workflow causes no repeated action;
|
||||
- database/client configuration contains no WCS task-state dependency;
|
||||
- normal and duplicate completion callbacks still pass.
|
||||
|
||||
Replay acceptance must prove:
|
||||
|
||||
- normal automatic and manual tidy behavior remains unchanged;
|
||||
- repairable local completion is recovered after restart;
|
||||
- missing local job enters `PAUSED_RECOVERY` with retained locks;
|
||||
- no task-state HTTP request appears in the event journal;
|
||||
- database restoration still reports `mismatches=[]`.
|
||||
|
||||
## Documentation
|
||||
|
||||
The existing 2026-07-22 design and plan remain historical records. Add a
|
||||
supersession note to their WCS task-state recovery sections, update the handoff
|
||||
document to describe WMS-native recovery, and generate a new final replay report
|
||||
instead of rewriting old evidence.
|
||||
|
||||
## Risks
|
||||
|
||||
- Legacy or manually edited finish rows may look terminal but disagree with
|
||||
inventory. The design pauses rather than guessing.
|
||||
- A workflow can remain paused until an operator repairs or reconciles the local
|
||||
inconsistency. Automatic lock release is intentionally prohibited.
|
||||
- The replay uses controlled SQL mutations to reproduce crash boundaries; every
|
||||
such mutation must be logged and covered by final baseline restoration.
|
||||
Reference in New Issue
Block a user