zyr/FT546MWS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: observe legacy failures in shadow worker

Browse Source
This commit is contained in:
zgw
2026-09-03 16:32:31 +08:00
parent 87700246d8
commit ffd3cc04f3
3 changed files with 117 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
.superpowers/sdd/2026-09-03-ft546wms-shadow-v3/task-6-report.md Normal file
View File

@@ -0,0 +1,69 @@
# Task 6 Report
## Status
GREEN, pending independent review.
## RED Evidence
- The initial Task 6 focused command failed test compilation because
`ShadowRuntimeMetrics`, `ShadowFailureStage`, tracked completion factories, the injected
`Sleeper`, the v3 gateway constructor, and the metric mapper operation did not exist.
- After the initial coordinator recovery implementation, the new
`staleRefreshTicketCannotCancelANewerRefreshLease` test failed with expected `BUSY` but actual
`UNAVAILABLE`, proving that a late old ticket could cancel a newer refresh lease.
## Implementation
- Inbound and outbound completion tokens now support idempotent `complete` and non-throwing
`fail(Throwable)`, close caller-owned Legacy activity exactly once, and record Legacy duration
without mapper access.
- `ShadowRuntimeMetrics` uses fixed-cardinality atomic counters and histogram buckets. `drain()`
returns an immutable bounded batch for the worker.
- `ShadowObservationGateway` now performs asynchronous startup refresh, coordinator admission,
snapshot-qualified planning, worker-only Legacy waiting/comparison, concurrency invalidation,
one-thread/capacity-one execution, quiet-window refresh, 1/2/4-second retry, one-minute recovery
coalescing, worker-only structured warnings, and worker-side metric persistence.
- Snapshot generations continue above the currently stored database generation.
- Refresh leases are unique; stale ticket completion cannot clear a later refresh.
- The mapper upserts only fixed metric rows into `alg.shadow_runtime_metric`.
## GREEN Evidence
Task 6 focused suite was run twice with Maven 3.9.11 after implementation:
& 'D:\BDevTools\maven\apache-maven-3.9.11\bin\mvn.cmd' -pl 'ruoyi-admin' -am '-Dtest=ShadowObservationGatewayTest,ShadowRuntimeMetricsTest,ShadowSnapshotCoordinatorTest' '-Dsurefire.failIfNoSpecifiedTests=false' test
Both runs completed with 21 tests, 0 failures, 0 errors, and `BUILD SUCCESS`.
The final Task 5+6 compatibility command was then run after the metric SQL boundary assertions:
& 'D:\BDevTools\maven\apache-maven-3.9.11\bin\mvn.cmd' -pl 'ruoyi-admin' -am '-Dtest=ShadowPlanningEngineTest,ShadowObservationGatewayTest,ShadowRuntimeMetricsTest,ShadowSnapshotCoordinatorTest' '-Dsurefire.failIfNoSpecifiedTests=false' test
Result: 36 tests, 0 failures, 0 errors, `BUILD SUCCESS`.
## Concerns
- Tests intentionally emit structured warning lines for simulated plan, wait, compare, and refresh
failures. They contain only request id, configured warehouse, generation, fixed stage, exception
class, and elapsed microseconds; exception messages are not logged.
- Existing unrelated Lombok superclass and JVM class-data-sharing warnings remain unchanged.
## Fix Round 1
### RED Evidence
- Command: `mvn -pl ruoyi-admin -am -Dtest=ShadowObservationGatewayTest#inboundLegacyFailureIsRecordedAtLegacyWaitStage+outboundLegacyFailureIsRecordedAtLegacyWaitStage -Dsurefire.failIfNoSpecifiedTests=false test`
- Result: 2 tests run, 2 failures, 0 errors. Both latch assertions timed out because inbound and outbound workers returned immediately when `LegacyResult.failure()` was non-null.
### GREEN Evidence
- Command: `mvn -pl ruoyi-admin -am -Dtest=ShadowObservationGatewayTest,ShadowRuntimeMetricsTest,ShadowSnapshotCoordinatorTest -Dsurefire.failIfNoSpecifiedTests=false test`
- Result: `ShadowObservationGatewayTest` 13 tests, `ShadowRuntimeMetricsTest` 3 tests, and `ShadowSnapshotCoordinatorTest` passed with 0 failures and 0 errors; Maven build succeeded.
### Changes and Self-review
- Files: `ruoyi-admin/src/main/java/com/ruoyi/algorithm/shadow/ShadowObservationGateway.java`, `ruoyi-admin/src/test/java/com/ruoyi/algorithm/shadow/ShadowObservationGatewayTest.java`, and this report.
- Inbound/outbound worker `LEGACY_WAIT` failures now use the existing fixed-stage metric and structured warning path. Warning data remains request id, warehouse, generation, fixed stage, exception class, and elapsed time; failure messages and payloads are not logged.
- Added direct inbound/outbound `fail(Throwable)` regression tests, including duplicate `fail`/`complete` assertions; Legacy admission remains closed exactly once by the completion token.
- Review scope stayed limited to the required Important; asynchronous timing Minors were not changed.

4
ruoyi-admin/src/main/java/com/ruoyi/algorithm/shadow/ShadowObservationGateway.java
View File

@@ -156,6 +156,7 @@ public class ShadowObservationGateway {
stage = ShadowFailureStage.LEGACY_WAIT;
InboundShadowCompletion.LegacyResult legacy = completion.awaitLegacyResult();
if (legacy.failure() != null) {
recordWorkerFailure(taskNo, admission, stage, legacy.failure(), startedNanos);
return;
}
if (!coordinator.isComparable(admission)) {
@@ -185,6 +186,7 @@ public class ShadowObservationGateway {
stage = ShadowFailureStage.LEGACY_WAIT;
OutboundShadowCompletion.LegacyResult legacy = completion.awaitLegacyResult();
if (legacy.failure() != null) {
recordWorkerFailure(stockoutDetailId, admission, stage, legacy.failure(), startedNanos);
return;
}
if (!coordinator.isComparable(admission)) {
@@ -334,7 +336,7 @@ public class ShadowObservationGateway {
private void recordWorkerFailure(String requestId,
ShadowSnapshotCoordinator.Admission admission, ShadowFailureStage stage,
Exception failure, long startedNanos) {
Throwable failure, long startedNanos) {
metrics.recordError(stage);
warnFailure(requestId, admission.snapshot().generation(), stage, failure, startedNanos);
}

45
ruoyi-admin/src/test/java/com/ruoyi/algorithm/shadow/ShadowObservationGatewayTest.java
View File

@@ -255,6 +255,51 @@ class ShadowObservationGatewayTest {
assertThat(coordinator.activeLegacyCount()).isZero();
}
@Test
void inboundLegacyFailureIsRecordedAtLegacyWaitStage() throws Exception {
ShadowPlanningEngine engine = mock(ShadowPlanningEngine.class);
ShadowInputMapper mapper = mock(ShadowInputMapper.class);
ShadowSnapshotCoordinator coordinator = coordinatorWithSnapshot();
when(engine.planInbound(SNAPSHOT, "TASK-FAIL", "TRAY-FAIL")).thenReturn(inboundObservation());
CountDownLatch waitMetric = metricLatch(mapper, "ERROR_LEGACY_WAIT");
stubSuccessfulRecovery(engine, mapper, 43L);
ShadowObservationGateway gateway = gateway(engine, mapper, coordinator,
new ShadowRuntimeMetrics(), Clock.fixed(NOW, ZoneOffset.UTC), delay -> { });
InboundShadowCompletion completion = gateway.observeInbound("TASK-FAIL", "TRAY-FAIL");
completion.fail(new IllegalStateException("business payload must not be logged"));
assertThat(waitMetric.await(1, TimeUnit.SECONDS)).isTrue();
assertThat(coordinator.activeLegacyCount()).isZero();
completion.fail(new IllegalStateException("duplicate"));
completion.complete("10", "SPACE-FAIL");
assertThat(coordinator.activeLegacyCount()).isZero();
}
@Test
void outboundLegacyFailureIsRecordedAtLegacyWaitStage() throws Exception {
ShadowPlanningEngine engine = mock(ShadowPlanningEngine.class);
ShadowInputMapper mapper = mock(ShadowInputMapper.class);
ShadowSnapshotCoordinator coordinator = coordinatorWithSnapshot();
OutboundShadowObservation observation = new OutboundShadowObservation(
true, 42L, "01", NOW, "ABC123", "DETAIL-FAIL", "READY",
"STOCK-FAIL", "SPACE-FAIL", "diagnostic");
when(engine.planProjectOutbound(SNAPSHOT, "DETAIL-FAIL")).thenReturn(observation);
CountDownLatch waitMetric = metricLatch(mapper, "ERROR_LEGACY_WAIT");
stubSuccessfulRecovery(engine, mapper, 43L);
ShadowObservationGateway gateway = gateway(engine, mapper, coordinator,
new ShadowRuntimeMetrics(), Clock.fixed(NOW, ZoneOffset.UTC), delay -> { });
OutboundShadowCompletion completion = gateway.observeProjectOutbound("DETAIL-FAIL");
completion.fail(new IllegalStateException("business payload must not be logged"));
assertThat(waitMetric.await(1, TimeUnit.SECONDS)).isTrue();
assertThat(coordinator.activeLegacyCount()).isZero();
completion.fail(new IllegalStateException("duplicate"));
completion.complete("10");
assertThat(coordinator.activeLegacyCount()).isZero();
}
@Test
void projectAndExactStockOutboundUseTheLeasedGeneration() throws Exception {
assertOutboundUsesLease(false);