fix: cap concurrent shadow planning

This commit is contained in:
zgw
2026-08-11 20:04:31 +08:00
parent 6a870601b5
commit 6738eff609
4 changed files with 76 additions and 3 deletions

View File

@@ -17,6 +17,7 @@ origin/main = 916ac6bda32a5b102a3181cc8c5dbd5319c5656d
- 默认配置 `FORTUCKY_SHADOW_ENABLED=false`,未显式开启时不执行影子规划。
- 启用时只允许 `FORTUCKY_ALGORITHM_MODE=SHADOW`;误设 `ACTIVE` 或 `LEGACY` 会导致应用拒绝启动。
- 影子事务使用 `REQUIRES_NEW`、`READ_UNCOMMITTED` 和 3 秒超时。
- 单个 JVM 同时最多执行一个影子规划;并发到达的其他观察会立即跳过,不排队、不等待,也不延迟 Legacy 分配。
- 影子异常、超时、输入缺失和比较日志写入失败都会 fail-open,不跳过 Legacy 流程。
- 安装包不在 `dbo.bas_cargo_space`、`dbo.bas_stock`、`dbo.mydo_job` 或其他 Legacy 表上创建触发器。
- 安装包不对 Legacy 业务表执行 `INSERT`、`UPDATE`、`DELETE` 或 `MERGE`。
@@ -174,6 +175,8 @@ FORTUCKY_ALGORITHM_LEGACY_FALLBACK_ON_TECHNICAL_ERROR=false
6. 确认 Legacy 返回码、库位和现场设备动作与稳定流程一致。
7. 确认 `alg.decision_log` 出现 `INBOUND`/`OUTBOUND` 及相应的 `*_COMPARE` 记录。
并发业务触发单 JVM 规划门时,被跳过的请求不会调用算法,也可能没有对应的计划/比较决策日志;Legacy 存储过程仍按原流程立即执行。这是保护现场数据库负载的预期行为,不要据此重放业务请求。
## 8. 观察与停止条件
应用日志中的以下文本表示影子已失败开放,Legacy 仍继续执行:
@@ -195,7 +198,7 @@ Shadow outbound comparison failed open
- 同一 Legacy 存储过程疑似被重复调用;
- WMS、WCS、PDA 或现场操作出现异常。
影子规划每次都会按仓库全量刷新读模型。`READ_UNCOMMITTED` 降低了对 Legacy 写事务的锁干扰,但无法消除全表扫描带来的 CPU 和 I/O,所以必须从少量受控业务开始,不能直接长时间全量开放。
实际进入影子规划的请求每次都会按仓库全量刷新读模型。应用在单个 JVM 内将并发规划上限限制为 1,其他并发观察立即跳过;这能限制重叠刷新,但 `READ_UNCOMMITTED` 和单并发都无法消除全表扫描带来的 CPU 和 I/O,所以必须从少量受控业务开始,不能直接长时间全量开放。若部署多个影子 JVM,该上限分别生效,现场仍只允许启动一个 WMS 实例。
## 9. 数据采集与交付
@@ -278,6 +281,7 @@ SELECT SCHEMA_ID(N'alg') AS alg_schema_id;
- `READ_UNCOMMITTED` 可能读到业务事务的中间状态;该结果只用于影子比较,不参与 Legacy 决策。
- 3 秒事务超时会触发 fail-open,但 JDBC 取消和 SQL Server 回滚完成时间不保证严格等于 3 秒。
- 每次影子规划前的全量刷新可能增加数据库 CPU 和 I/O;现场必须监控并设置短测试窗口。
- JVM 内单并发门只避免同一实例的重叠刷新,不限制其他进程或数据库会话;禁止同时运行稳定版和影子版,也禁止运行多个影子实例。
- 应用级 fail-open 能保护 Legacy 控制流,但不能消除数据库、网络或主机资源争用。
这些残余风险是采用“零 Legacy 触发器、每次规划前全量刷新”的代价,也是现场必须先小流量验证、随时可切回稳定源码的原因。

View File

@@ -13,6 +13,7 @@ Create a clean field-test branch directly from `origin/main` that runs the Java
5. No ACTIVE entry point is exposed by `ruoyi-admin`. Startup rejects any algorithm mode other than `SHADOW` when shadow observation is enabled.
6. Shadow mode writes only `alg.*` read-model and diagnostic objects. It never calls reservation, tidy, deep-workflow, replay, move-completion, or other procedures that write WMS business tables.
7. Shadow observation is disabled by default and must be enabled explicitly in the IDEA Run Configuration.
8. At most one shadow planning operation runs per JVM. Concurrent observations are skipped immediately without queueing or delaying Legacy allocation.
## Architecture
@@ -20,7 +21,7 @@ The existing `fortucky-algorithm` planner and JDBC read model are reused as an i
- `ShadowObservationProperties` controls enablement, warehouse fallback, and timeout.
- `ShadowSafetyGuard` validates the effective mode at startup.
- `ShadowPlanningObserver` performs fail-open inbound/outbound planning in an independent transaction and returns an immutable observation token.
- `ShadowPlanningObserver` performs fail-open inbound/outbound planning in an independent transaction, admits at most one plan per JVM, and returns an immutable observation token.
- Legacy services invoke the observer immediately before their existing stored procedure, execute the original Legacy code unchanged, then submit the Legacy result for comparison logging.
There are no algorithm controllers, ACTIVE schedulers, workflow callbacks, reservation calls, or changes to the frontend.
@@ -45,7 +46,7 @@ There are no algorithm controllers, ACTIVE schedulers, workflow callbacks, reser
## Transaction and failure isolation
Shadow planning uses `TransactionTemplate` with `PROPAGATION_REQUIRES_NEW`, `ISOLATION_READ_UNCOMMITTED`, read-model writes enabled, and a short timeout. The transaction completes or rolls back before Legacy allocation starts, so algorithm locks and rollback state cannot leak into the Legacy transaction. The adapter catches runtime failures at its public boundary and returns an empty observation.
Shadow planning uses `TransactionTemplate` with `PROPAGATION_REQUIRES_NEW`, `ISOLATION_READ_UNCOMMITTED`, read-model writes enabled, and a short timeout. The transaction completes or rolls back before Legacy allocation starts, so algorithm locks and rollback state cannot leak into the Legacy transaction. A JVM-local nonblocking gate permits one planning call at a time; a concurrent call returns an empty observation immediately, while successful and failed calls both release the gate in `finally`. Comparison logging is not gated. The adapter catches runtime failures at its public boundary and returns an empty observation.
The integration never queries `alg.tidy_mode` and never uses an update lock for admission control.

View File

@@ -5,6 +5,7 @@ import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicBoolean;
@Component
public class ShadowObservationGateway {
@@ -12,6 +13,7 @@ public class ShadowObservationGateway {
private final ShadowObservationProperties properties;
private final ShadowPlanningEngine engine;
private final AtomicBoolean planning = new AtomicBoolean();
public ShadowObservationGateway(ShadowObservationProperties properties, ShadowPlanningEngine engine) {
this.properties = Objects.requireNonNull(properties, "properties");
@@ -22,11 +24,16 @@ public class ShadowObservationGateway {
if (!properties.isEnabled() || engine == null) {
return InboundShadowObservation.empty();
}
if (!planning.compareAndSet(false, true)) {
return InboundShadowObservation.empty();
}
try {
return engine.planInbound(taskNo, trayNo);
} catch (RuntimeException exception) {
LOG.warn("Shadow inbound planning failed open for task {}: {}", taskNo, exception.getMessage());
return InboundShadowObservation.empty();
} finally {
planning.set(false);
}
}
@@ -34,12 +41,17 @@ public class ShadowObservationGateway {
if (!properties.isEnabled() || engine == null) {
return OutboundShadowObservation.empty();
}
if (!planning.compareAndSet(false, true)) {
return OutboundShadowObservation.empty();
}
try {
return engine.planProjectOutbound(stockoutDetailId);
} catch (RuntimeException exception) {
LOG.warn("Shadow outbound planning failed open for detail {}: {}",
stockoutDetailId, exception.getMessage());
return OutboundShadowObservation.empty();
} finally {
planning.set(false);
}
}
@@ -47,12 +59,17 @@ public class ShadowObservationGateway {
if (!properties.isEnabled() || engine == null) {
return OutboundShadowObservation.empty();
}
if (!planning.compareAndSet(false, true)) {
return OutboundShadowObservation.empty();
}
try {
return engine.planExactStockOutbound(stockoutDetailId, stockId);
} catch (RuntimeException exception) {
LOG.warn("Shadow exact-stock planning failed open for detail {} stock {}: {}",
stockoutDetailId, stockId, exception.getMessage());
return OutboundShadowObservation.empty();
} finally {
planning.set(false);
}
}

View File

@@ -2,6 +2,12 @@ package com.ruoyi.algorithm.shadow;
import org.junit.jupiter.api.Test;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
@@ -29,11 +35,15 @@ class ShadowObservationGatewayTest {
ShadowPlanningEngine engine = mock(ShadowPlanningEngine.class);
when(engine.planInbound("TASK-2", "TRAY-2"))
.thenThrow(new IllegalStateException("algorithm objects are unavailable"));
when(engine.planProjectOutbound("DETAIL-2"))
.thenReturn(OutboundShadowObservation.unavailable("W01", "DETAIL-2", "TEST", "completed"));
ShadowObservationGateway gateway = new ShadowObservationGateway(properties, engine);
InboundShadowObservation observation = gateway.planInbound("TASK-2", "TRAY-2");
OutboundShadowObservation followingObservation = gateway.planProjectOutbound("DETAIL-2");
assertThat(observation.observed()).isFalse();
assertThat(followingObservation.observed()).isTrue();
}
@Test
@@ -46,4 +56,45 @@ class ShadowObservationGatewayTest {
assertThat(observation.observed()).isFalse();
}
@Test
void concurrentPlanningIsSkippedInsteadOfAddingDatabaseLoad() throws Exception {
ShadowObservationProperties properties = new ShadowObservationProperties();
properties.setEnabled(true);
ShadowPlanningEngine engine = mock(ShadowPlanningEngine.class);
CountDownLatch planningStarted = new CountDownLatch(1);
CountDownLatch releasePlanning = new CountDownLatch(1);
when(engine.planInbound("TASK-3", "TRAY-3")).thenAnswer(invocation -> {
planningStarted.countDown();
if (!releasePlanning.await(5, TimeUnit.SECONDS)) {
throw new IllegalStateException("test planning was not released");
}
return InboundShadowObservation.unavailable("W01", "TASK-3", "TEST", "completed");
});
when(engine.planProjectOutbound("DETAIL-3"))
.thenReturn(OutboundShadowObservation.unavailable("W01", "DETAIL-3", "TEST", "completed"));
ShadowObservationGateway gateway = new ShadowObservationGateway(properties, engine);
ExecutorService executor = Executors.newSingleThreadExecutor();
try {
Future<InboundShadowObservation> first =
executor.submit(() -> gateway.planInbound("TASK-3", "TRAY-3"));
assertThat(planningStarted.await(5, TimeUnit.SECONDS)).isTrue();
OutboundShadowObservation concurrent = gateway.planProjectOutbound("DETAIL-3");
assertThat(concurrent.observed()).isFalse();
verify(engine, never()).planProjectOutbound("DETAIL-3");
releasePlanning.countDown();
assertThat(first.get(5, TimeUnit.SECONDS).observed()).isTrue();
OutboundShadowObservation afterRelease = gateway.planProjectOutbound("DETAIL-3");
assertThat(afterRelease.observed()).isTrue();
verify(engine).planProjectOutbound("DETAIL-3");
} finally {
releasePlanning.countDown();
executor.shutdownNow();
}
}
}