Files
FT546-WCS/docs/superpowers/plans/2026-08-04-standard-wcs-execution-recovery.md

499 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Standard WCS Execution Recovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 把已授权的调度结果转换为可靠设备命令,并实现设备协同、事实对账、Outbox、异常隔离和重启恢复。
**Architecture:** 业务事务先保存 `DeviceCommand` 和 Outbox,再由发送器调用项目 Adapter。设备事件进入标准快照后由 `FeedbackReconciler` 推进命令、步骤和资源;结果不确定时创建 `RecoveryCase`,而不是由 Adapter 重试或改状态。
**Tech Stack:** Java 8, Spring transactions, MyBatis, SQL Server, JUnit 5, scheduled Outbox polling, SLF4J MDC, Micrometer.
## Global Constraints
- 依赖 Plan 2 已交付的车辆绑定、路线、资源预约和 `DispatchCycleResult`。
- 网络 I/O 不在业务数据库事务中执行。
- `send` 成功只表示通信层受理,不表示设备完成。
- `UNKNOWN` 期间只允许快照读取和诊断,不允许并行运动命令。
- 适配器不得访问 WCS Mapper/Repository。
- 设备事件允许重复和乱序,核心必须幂等处理。
- 资源释放必须引用 `releaseEvidenceId`。
- 恢复和人工动作必须保留操作人、原因、前后状态和时间。
---
### Task 1: Implement DeviceCommand and transactional Outbox
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/DeviceCommandState.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/DeviceCommandType.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/DeviceCommand.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/CommandOutboxEntry.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/WmsNotificationOutboxEntry.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/DeviceCommandRepository.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/CommandOutboxRepository.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/WmsNotificationOutboxRepository.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/domain/execution/DeviceCommandTest.java`
**Interfaces:**
- Produces: `DeviceCommand.pending(commandKey,deviceId,type,payload,attempt,createdAt)`.
- Produces: `markSent`, `acknowledge`, `succeed`, `reject`, `failBeforeSend`, `markUnknown`.
- Produces: `CommandOutboxEntry.pending(commandId,commandKey,payload,nextAttemptAt)`.
- Produces: `WmsNotificationOutboxEntry.pending(eventId,notification,nextAttemptAt)`; `eventId` is the callback idempotency key.
- [ ] **Step 1: Write failing UNKNOWN and terminal-state tests**
```java
@Test
void sentCommandBecomesUnknownInsteadOfRetryableFailure() {
DeviceCommand command = pendingCommand();
command.markSent("SEND-1", Instant.EPOCH);
command.markUnknown("ACK_TIMEOUT", Instant.EPOCH.plusSeconds(5));
assertEquals(DeviceCommandState.UNKNOWN, command.getState());
assertFalse(command.canSendMotionCommandAfterIt());
}
@Test
void terminalCommandCannotBeAcknowledgedAgain() {
DeviceCommand command = succeededCommand();
assertThrows(DomainRuleViolation.class, () -> command.acknowledge("ACK-2", Instant.now()));
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=DeviceCommandTest test`
- [ ] **Step 3: Implement explicit command transitions**
```java
public void markUnknown(String reason, Instant at) {
requireAnyState(DeviceCommandState.SENT, DeviceCommandState.ACKNOWLEDGED);
this.state = DeviceCommandState.UNKNOWN;
this.statusReason = requireText(reason, "reason");
this.updatedAt = at;
this.stateVersion++;
}
```
Implement only these transitions: `PENDING -> SENT/FAILED`; `SENT -> ACKNOWLEDGED/SUCCEEDED/REJECTED/UNKNOWN`; `ACKNOWLEDGED -> SUCCEEDED/UNKNOWN`; `UNKNOWN -> SUCCEEDED/FAILED` after matching physical evidence or controlled recovery. `failBeforeSend` is legal only from `PENDING`; `SUCCEEDED/REJECTED/FAILED` are terminal. No transition from `UNKNOWN` may create a new motion command or release an unproven resource.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest=DeviceCommandTest test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/DeviceCommandRepository.java ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/CommandOutboxRepository.java ruoyi-wcs/src/test/java/com/ruoyi/wcs/domain/execution/DeviceCommandTest.java
git commit -m "feat(wcs): add device command and outbox lifecycle"
```
### Task 2: Define device gateway, events, snapshots, and reconciliation
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/DeviceCommandGateway.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/DeviceSnapshotGateway.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/RecoveryCaseOpener.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/in/DeviceEventUseCase.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/DeviceCommandEnvelope.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/SendReceipt.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/EventAcceptanceResult.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/ReconcileResult.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/ReconcileStatus.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/DeviceEvent.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution/DeviceSnapshot.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/FeedbackReconciler.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/FeedbackReconcilerTest.java`
**Interfaces:**
- `DeviceCommandGateway.send(DeviceCommandEnvelope): SendReceipt` returns exactly `ACCEPTED`, `ACCEPTED_UNKNOWN`, `NOT_SENT`, or `REJECTED`.
- `SendReceipt.Status.NOT_SENT` proves no device-side effect and is the only send outcome eligible for automatic retry.
- `SendReceipt.Status.ACCEPTED_UNKNOWN` means a device-side effect may have occurred; the command enters `UNKNOWN` and opens recovery without automatic resend.
- `DeviceSnapshotGateway.readSnapshot(String deviceId): DeviceSnapshot` performs read-only diagnostics.
- `DeviceEventUseCase.accept(DeviceEvent): EventAcceptanceResult` is idempotent by `eventId`; the enum values are `APPLIED`, `DUPLICATE`, `STALE`, and `RECOVERY_REQUIRED`.
- `FeedbackReconciler` implements `DeviceEventUseCase.accept(DeviceEvent)` and also exposes `reconcile(DeviceSnapshot): ReconcileResult`; `ReconcileStatus` values are `CONSISTENT`, `ADVANCED`, `IGNORED_STALE`, and `RECOVERY_REQUIRED`.
- `RecoveryCaseOpener.open(deviceId,commandKey,reason,evidenceIds)` is the only dependency used by reconciliation and Outbox delivery to request isolation; Task 6 supplies its production implementation.
- [ ] **Step 1: Write failing duplicate, stale, and release-evidence tests**
```java
@Test
void duplicateEventDoesNotAdvanceTheCommandTwice() {
EventAcceptanceResult first = reconciler.accept(completedEvent("E-1", "CMD-1"));
EventAcceptanceResult duplicate = reconciler.accept(completedEvent("E-1", "CMD-1"));
assertEquals(EventAcceptanceResult.APPLIED, first);
assertEquals(EventAcceptanceResult.DUPLICATE, duplicate);
assertEquals(1, repository.updateCount("CMD-1"));
}
@Test
void taskNumberMismatchCreatesRecoveryInsteadOfReleasing() {
ReconcileResult result = reconciler.reconcile(snapshot("VEH-1", "WRONG-TASK", "B"));
assertEquals(ReconcileStatus.RECOVERY_REQUIRED, result.getStatus());
assertEquals(0, reservationRepository.releaseCount());
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=FeedbackReconcilerTest test`
- [ ] **Step 3: Implement standard event envelopes and release predicates**
Each event contains `eventId`, `deviceId`, `commandKey`, `deviceSequence`, `deviceTime`, `receivedTime`, `eventType`, and typed facts. Reconcile movement only when the node is a legal non-regressing suffix of the committed route; release each passed resource with `releaseEvidenceId=eventId`.
Implement `SendReceipt` as an immutable class with nested `Status` enum, `receiptId`, and `detail`. `DeviceCommandEnvelope` carries `commandId`, stable `commandKey`, `deviceId`, typed command payload, attempt, topology version, and fencing token. `ReconcileResult` carries `ReconcileStatus`, affected command key, released resource keys, and optional recovery reason.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest=FeedbackReconcilerTest,DeviceCommandTest,ResourceLedgerTest test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/port ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/execution ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device
git commit -m "feat(wcs): reconcile device facts with execution state"
```
### Task 3: Implement command creation and Outbox delivery service
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/DeviceExecutionService.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/CommandOutboxDispatcher.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/OutboxDispatchResult.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/WmsNotificationOutboxProjector.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/WmsNotificationOutboxDispatcher.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/CommandOutboxDispatcherTest.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/WmsNotificationOutboxDispatcherTest.java`
**Interfaces:**
- Consumes: granted assignment and protected route from Plan 2.
- Produces: `createCommand(GrantedDispatch): DeviceCommand` and one Outbox entry in the same transaction.
- Produces: `dispatchDue(int limit,Instant now): OutboxDispatchResult`.
- Produces: one `WmsNotificationOutboxEntry` for each terminal or suspension business event; `WmsNotificationOutboxDispatcher.dispatchDue` calls `WmsNotificationGateway` outside the business transaction.
- [ ] **Step 1: Write failing transaction and retry-boundary tests**
```java
@Test
void persistsCommandAndOutboxBeforeCallingTheGateway() {
service.createCommand(grantedDispatch());
assertTrue(commandRepository.exists("CMD-1"));
assertTrue(outboxRepository.existsForCommand("CMD-1"));
assertEquals(0, gateway.sendCount());
}
@Test
void acceptedSendIsNotMarkedCompleted() {
dispatcher.dispatchDue(10, Instant.EPOCH);
assertEquals(DeviceCommandState.SENT, commandRepository.get("CMD-1").getState());
}
@Test
void acceptedUnknownIsIsolatedAndNeverAutomaticallyRetried() {
gateway.returnReceipt(SendReceipt.acceptedUnknown("WRITE_RESULT_AMBIGUOUS"));
dispatcher.dispatchDue(10, Instant.EPOCH);
assertEquals(DeviceCommandState.UNKNOWN, commandRepository.get("CMD-1").getState());
assertEquals(0, outboxRepository.retryableCount("CMD-1"));
}
@Test
void duplicateBusinessEventCreatesOneWmsOutboxEntry() {
projector.on(orderCompletedEvent("EVENT-1"));
projector.on(orderCompletedEvent("EVENT-1"));
assertEquals(1, wmsOutboxRepository.countByEventId("EVENT-1"));
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=CommandOutboxDispatcherTest test`
- [ ] **Step 3: Implement delivery outcomes**
`NOT_SENT` increments attempt and schedules exponential backoff; `ACCEPTED` marks SENT and completes the device Outbox row; `ACCEPTED_UNKNOWN` marks `UNKNOWN`, isolates the command, and opens recovery; `REJECTED` marks the command rejected. A WMS notification transport failure remains pending with backoff, while a successful callback records the external receipt against the original `eventId` and never changes the already-completed physical task.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest=CommandOutboxDispatcherTest,WmsNotificationOutboxDispatcherTest,FeedbackReconcilerTest test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/CommandOutboxDispatcherTest.java
git commit -m "feat(wcs): deliver device commands through outbox"
```
### Task 4: Coordinate lift transfers as joint resources
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/LiftTransferState.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/LiftTransfer.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/LiftDecision.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/LiftDecisionType.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/LiftCoordinator.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/LiftCoordinatorTest.java`
**Interfaces:**
- Produces: `prepare(String stepId,String vehicleId,String liftId,String sourcePort,String targetPort)`.
- Produces: `onVehicleLeftTarget(LiftTransfer,DeviceSnapshot): LiftDecision` after validating combined vehicle, lift, door, and port facts.
- `LiftDecisionType` values are `READY`, `BLOCKED`, `ADVANCED`, `COMPLETED`, and `RECOVERY_REQUIRED`; every decision carries one stable reason code.
- Produces states `WAITING_SOURCE`, `SOURCE_ENTRY_COMMITTED`, `VEHICLE_IN_LIFT`, `LIFT_MOVING`, `AT_TARGET`, `TARGET_EXIT_COMMITTED`, `VEHICLE_LEFT_LIFT`, `COMPLETED`.
- [ ] **Step 1: Write failing joint-reservation tests**
```java
@Test
void doesNotEnterLiftWhenTargetExitCannotBeReserved() {
ledger.occupyResource("LIFT_PORT:TARGET", "OTHER");
LiftDecision decision = coordinator.prepare("STEP-1", "V1", "L1", "P1", "P2");
assertEquals(LiftDecisionType.BLOCKED, decision.getType());
assertEquals(0, commandRepository.count());
}
@Test
void releasesLiftOnlyAfterVehicleLeavesTargetPort() {
LiftTransfer transfer = transferAtTarget();
coordinator.onVehicleLeftTarget(transfer, matchingFacts());
assertEquals(LiftTransferState.COMPLETED, transfer.getState());
assertTrue(ledger.isAvailable("LIFT:L1"));
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=LiftCoordinatorTest test`
- [ ] **Step 3: Implement the state machine and UNKNOWN branch**
Reserve source port, lift body, and target port atomically. Any unknown entry/movement/exit fact suspends the step and isolates all already committed lift resources until combined vehicle, floor, door, and port facts reconcile.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest=LiftCoordinatorTest,ResourceLedgerTest test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/LiftCoordinator.java ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/LiftCoordinatorTest.java
git commit -m "feat(wcs): coordinate lift joint resources"
```
### Task 5: Coordinate conveyor ports and charging
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/PortMode.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/ConveyorPortState.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/ConveyorCoordinator.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/ChargingPolicyDecision.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/PendingWorkload.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment/AvailableChargers.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device/ChargingCoordinator.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/PortSelectionPolicy.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/ChargingPolicy.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/ConveyorAndChargingCoordinatorTest.java`
**Interfaces:**
- Produces: `requestModeSwitch(portId,targetMode)` and `startTransfer(stepId,portId,direction)`.
- Produces: `ChargingPolicy.decide(VehicleSnapshot,PendingWorkload,AvailableChargers)`.
- `PendingWorkload` is an immutable summary of ready urgent/ordinary work and oldest wait time; `AvailableChargers` contains only healthy, reachable, unreserved charger descriptors and topology distance.
- [ ] **Step 1: Write failing mode-lock and charging-priority tests**
```java
@Test
void locksPortAgainstNewOutboundAssignmentDuringInboundSwitchPermission() {
coordinator.grantModeSwitch("PORT-1", PortMode.INBOUND);
assertFalse(coordinator.canAssignOutbound("PORT-1"));
}
@Test
void forcedChargeWinsOverOrdinaryReadyTask() {
ChargingPolicyDecision decision = charging.decide(vehicleWithBattery(15), ordinaryWork(), chargers());
assertEquals(ChargingPolicyDecision.Type.FORCED_CHARGE, decision.getType());
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=ConveyorAndChargingCoordinatorTest test`
- [ ] **Step 3: Implement port and charging state**
Port mode changes require zero remaining bound jobs and a switch lock that blocks new opposite-direction assignments. Charging reserves one charger resource and creates a normal workflow step; it never interrupts an active physical task solely because battery crossed a threshold.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest=ConveyorAndChargingCoordinatorTest,LiftCoordinatorTest test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/equipment ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/device ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/PortSelectionPolicy.java ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/ChargingPolicy.java ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/device/ConveyorAndChargingCoordinatorTest.java
git commit -m "feat(wcs): coordinate conveyor ports and charging"
```
### Task 6: Implement RecoveryCase and startup reconciliation
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery/RecoveryCaseState.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery/RecoveryCase.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery/RecoveryEvidence.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery/ControlStatus.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery/StartupRecoveryResult.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/in/RecoveryUseCase.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/RecoveryCaseRepository.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/recovery/StartupRecoveryService.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/recovery/RecoveryApplicationService.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/recovery/StartupRecoveryServiceTest.java`
**Interfaces:**
- Produces: `recover(): StartupRecoveryResult`.
- Produces: `confirmFacts(caseId,operatorId,RecoveryEvidence)` and `abort(caseId,operatorId,reason)`.
- Consumes: active orders, reservations, commands, Outbox entries, last snapshots, and fresh `DeviceSnapshotGateway` reads.
- `ControlStatus` values are `PAUSED`, `PARTIALLY_ENABLED`, and `ENABLED`; `StartupRecoveryResult` returns control status, isolated device IDs, isolated resource keys, resumed order IDs, and unresolved recovery case IDs.
- `RecoveryEvidence` is a typed immutable DTO containing observed site, load state, task number, device mode, evidence time, source, and operator reason; it does not accept arbitrary state overrides.
- `RecoveryApplicationService` implements both `RecoveryUseCase` and `RecoveryCaseOpener`; automatic callers may open/isolate a case but only `RecoveryUseCase` can confirm facts or abort it with an authenticated operator.
- [ ] **Step 1: Write failing restart and operator-evidence tests**
```java
@Test
void startupKeepsDispatchPausedUntilAllActiveDevicesAreReconciled() {
StartupRecoveryResult result = recovery.recover();
assertEquals(ControlStatus.PAUSED, result.getControlStatus());
assertEquals(Collections.singleton("VEH-2"), result.getIsolatedDeviceIds());
}
@Test
void operatorCannotResolveCaseWithoutMatchingEvidence() {
assertThrows(DomainRuleViolation.class, () ->
service.confirmFacts("RC-1", "OP-1", mismatchingEvidence()));
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=StartupRecoveryServiceTest test`
- [ ] **Step 3: Implement ordered startup recovery**
Acquire an instance lease and new fencing token, load all active state, query devices, reconcile, resume matched work, isolate mismatches, restore callback Outbox, and enable dispatch only after no unexplained global resource remains.
- [ ] **Step 4: Run GREEN**
Run: `mvn -pl ruoyi-wcs -am -Dtest='*Recovery*Test,*FeedbackReconcilerTest,*CommandOutboxDispatcherTest' test`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/domain/recovery ruoyi-wcs/src/main/java/com/ruoyi/wcs/application/recovery ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/in/RecoveryUseCase.java ruoyi-wcs/src/main/java/com/ruoyi/wcs/port/out/RecoveryCaseRepository.java ruoyi-wcs/src/test/java/com/ruoyi/wcs/application/recovery
git commit -m "feat(wcs): recover execution state after restart"
```
### Task 7: Persist execution state and expose stable operations queries
**Files:**
- Create: `sql/wcs/V003__wcs_execution_and_recovery.sql`
- Create: `sql/wcs/verify_V003__wcs_execution_and_recovery.sql`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/query/WcsOperationsQueryService.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/query/OrderExecutionView.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/query/DeviceOperationsView.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/query/ResourceOperationsView.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/query/RecoveryCaseView.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/query/WcsOperationsQueryServiceTest.java`
**Interfaces:**
- Persists: `wcs_device_command`, `wcs_command_outbox`, `wcs_wms_notification_outbox`, `wcs_device_snapshot`, `wcs_alarm`, `wcs_recovery_case`, `wcs_manual_action_audit`.
- Produces read-only page queries by order, device, resource, command, alarm, and recovery status.
- `wcs_wms_notification_outbox.event_id` has a unique constraint and stores notification type, canonical payload, delivery state, attempt, next attempt time, external receipt, last error, created time, and delivered time.
- [ ] **Step 1: Write failing query-projection tests**
```java
@Test
void returnsCommandAndRecoveryCorrelationForAnOrder() {
OrderExecutionView view = queries.order("ORDER-1");
assertEquals("CMD-1", view.getCurrentCommandKey());
assertEquals("RC-1", view.getRecoveryCaseId());
assertEquals("UNKNOWN", view.getCommandState());
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=WcsOperationsQueryServiceTest test`
- [ ] **Step 3: Add tables, indexes, and read models**
Index `command_key`, `device_id + command_state`, `order_id + step_id`, `resource_key + reservation_state`, and `recovery_state + created_time`. Query DTOs contain no Mapper rows and expose explicit strings/timestamps only.
- [ ] **Step 4: Run full Plan 3 verification**
Run: `mvn -pl ruoyi-wcs -am -Dtest='*Command*,*Reconciler*,*Lift*,*Conveyor*,*Charging*,*Recovery*,*OperationsQuery*' test`
Run: `git diff --check`
- [ ] **Step 5: Commit**
```bash
git add sql/wcs ruoyi-wcs/src/main/java/com/ruoyi/wcs/query ruoyi-wcs/src/test/java/com/ruoyi/wcs/query
git commit -m "feat(wcs): persist and query execution recovery state"
```
### Task 8: Add correlated logs and product metrics
**Files:**
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/observability/CorrelationContext.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/observability/WcsMetricNames.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/observability/WcsTelemetry.java`
- Create: `ruoyi-wcs/src/main/java/com/ruoyi/wcs/observability/MicrometerWcsTelemetry.java`
- Create: `ruoyi-wcs/src/test/java/com/ruoyi/wcs/observability/WcsObservabilityTest.java`
**Interfaces:**
- `CorrelationContext` carries `traceId`, `externalRequestId`, `orderId`, `workflowId`, `stepId`, `deviceId`, `commandKey`, `resourceId`, `eventId`, and `recoveryCaseId` into SLF4J MDC for one application operation and clears it in `close()`.
- `WcsTelemetry` records gauges/counters/timers named `wcs.task.backlog`, `wcs.dispatch.duration`, `wcs.traffic.wait.duration`, `wcs.command.ack.duration`, `wcs.command.complete.duration`, `wcs.device.utilization`, `wcs.device.battery`, `wcs.lift.wait.duration`, `wcs.outbox.retry.count`, `wcs.device.fault.count`, and `wcs.recovery.duration`.
- Metric labels are bounded to warehouse, device type, command type, order type, outcome, and recovery level; IDs stay in logs and never become metric labels.
- [ ] **Step 1: Write failing correlation and metric tests**
```java
@Test
void writesAllCorrelationFieldsAndClearsMdcAfterTheOperation() {
try (CorrelationContext ignored = CorrelationContext.open(fullCorrelation())) {
assertEquals("ORDER-1", MDC.get("orderId"));
assertEquals("RC-1", MDC.get("recoveryCaseId"));
}
assertNull(MDC.get("orderId"));
}
@Test
void recordsTheRequiredBoundedProductMetrics() {
telemetry.recordDispatch(Duration.ofMillis(80), "INBOUND", "GRANTED");
telemetry.incrementOutboxRetry("DEVICE_COMMAND", "NOT_SENT");
assertEquals(1L, registry.counter("wcs.outbox.retry.count",
"outboxType", "DEVICE_COMMAND", "outcome", "NOT_SENT").count(), 0.0D);
}
```
- [ ] **Step 2: Run RED**
Run: `mvn -pl ruoyi-wcs -am -Dtest=WcsObservabilityTest test`
- [ ] **Step 3: Instrument application boundaries**
Open one `CorrelationContext` at WMS command acceptance, dispatch, device event acceptance, Outbox delivery, recovery, and operator action boundaries. Record timers only around the owned operation; publish backlog, device utilization/battery, and isolated recovery counts from read-only repository snapshots. Never log PLC credentials, WMS signatures, or complete raw payloads.
- [ ] **Step 4: Run full Plan 3 verification**
Run: `mvn -pl ruoyi-wcs -am -Dtest='*Command*,*Reconciler*,*Lift*,*Conveyor*,*Charging*,*Recovery*,*OperationsQuery*,*Observability*' test`
Run: `git diff --check`
- [ ] **Step 5: Commit**
```bash
git add ruoyi-wcs/src/main/java/com/ruoyi/wcs/observability ruoyi-wcs/src/test/java/com/ruoyi/wcs/observability
git commit -m "feat(wcs): add correlated logs and product metrics"
```