test(ft546): verify read-only shadow operation
This commit is contained in:
@@ -13,6 +13,12 @@
|
||||
|
||||
若依是一套全部开源的快速开发平台，毫无保留给个人及企业免费使用。
|
||||
|
||||
## FT546 只读影子调度
|
||||
|
||||
FT546 新调度算法通过独立 `ruoyi-wcs-shadow` 进程进行只读影子验证。`LEGACY` 仍是唯一生产调度者和 PLC/WMS 写入者；当前工作不授权 `STANDARD` 切换或生产上线。
|
||||
|
||||
现场准备、预检、精确启停、立即停止条件和证据导出见 [FT546 实时只读影子运行手册](docs/runbooks/ft546-realtime-shadow-runbook.md)。
|
||||
|
||||
* 前端采用Vue、Element UI。
|
||||
* 后端采用Spring Boot、Spring Security、Redis & Jwt。
|
||||
* 权限认证使用Jwt，支持多终端认证系统。
|
||||
|
||||
252
docs/runbooks/ft546-realtime-shadow-runbook.md
Normal file
252
docs/runbooks/ft546-realtime-shadow-runbook.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# FT546 实时只读影子运行手册
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本手册只用于运行独立的 `ruoyi-wcs-shadow` 进程，采集新调度算法的只读影子证据。
|
||||
|
||||
- 现场 `LEGACY` WCS 始终是唯一生产调度者、PLC 写入者和 WMS 回调者。
|
||||
- 本阶段不切换 `STANDARD`，不允许 shadow 连接 PLC/WMS，也不允许写生产业务表。
|
||||
- 停止或移除 shadow 进程就是本阶段的回滚；不得停止或切换稳定 WCS。
|
||||
- 本地自动化测试不能替代 SQL Server、网络隔离或现场观察门禁。
|
||||
|
||||
## 2. 当前验证边界
|
||||
|
||||
| 门禁 | 当前状态 | 允许的结论 |
|
||||
| --- | --- | --- |
|
||||
| Java 单元、Spring 上下文、H2 E2E | 可在本地执行 | 代码级只读链路和协议证据可验证 |
|
||||
| PowerShell 脚本生命周期 | 可用无网络 sleeper jar 本地执行 | 确认、哈希、PID 精确启停和离线导出逻辑可验证 |
|
||||
| SQL Server 集成 | `NOT_RUN` | 不得宣称数据库集成或现场就绪 |
|
||||
| PLC/WMS 网络 deny | `NOT_RUN` | 必须由目标 shadow 主机的新鲜审计报告证明 |
|
||||
| HIL 与现场影子窗口 | `NOT_RUN` | 不得部署或开始采集 |
|
||||
|
||||
`application-shadow-local.yml` 是外部配置模板，不是可直接运行的 H2 profile。H2 仍是 test-scope 依赖，且测试资源不会进入打包 jar；没有审批过的独立 SQL Server 环境时，不运行 jar 本地 smoke。
|
||||
|
||||
当前 `shadow_hold_history` 能跨重启恢复 wait edge 和连续 HOLD 计数，测试覆盖第三次相同 HOLD 后计数为 3。生产调度内核尚无消费该计数并触发 reroute 的策略，因此不得把“计数恢复”报告成“第三次 HOLD 已重规划”。该语义应在取得影子证据并明确路线/阻塞资源状态后单独设计。
|
||||
|
||||
## 3. 前置条件
|
||||
|
||||
开始任何现场操作前，逐项确认：
|
||||
|
||||
1. 稳定 WCS 健康，实际制品、配置、PID、命令行和监听端口可读取。
|
||||
2. source 登录只能读取 `shadow_ro.*` 批准视图，无生产表 `INSERT`、`UPDATE`、`DELETE`、`EXECUTE`、`ALTER`、`CONTROL` 或所有权权限。
|
||||
3. evidence 数据库与 source 数据库物理分离，evidence 登录只在 evidence 数据库具备所需写权限。
|
||||
4. shadow 部署目录、日志目录、端口和 PID 文件均与稳定 WCS 分离。
|
||||
5. shadow 主机到 PLC 写端点和生产 WMS 回调端点已在网络层拒绝，并生成不超过 60 分钟的 JSON 审计报告。
|
||||
6. 已约定生产 DB 延迟、shadow CPU/内存、source lag 和 evidence 磁盘阈值。
|
||||
7. 已准备立即停止 shadow 的值守人员；不安排任何稳定 WCS 切换动作。
|
||||
|
||||
权限报告最小结构：
|
||||
|
||||
```json
|
||||
{
|
||||
"generatedAtUtc": "2026-08-14T10:00:00Z",
|
||||
"status": "PASS",
|
||||
"source": {
|
||||
"server": "source-host",
|
||||
"database": "FT546-wcs",
|
||||
"login": "ft546_shadow_reader",
|
||||
"approvedViewsReadable": true,
|
||||
"productionWritePermissionCount": 0
|
||||
},
|
||||
"evidence": {
|
||||
"server": "evidence-host",
|
||||
"database": "FT546-shadow",
|
||||
"login": "ft546_shadow_writer",
|
||||
"requiredWritesGranted": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
网络 deny 报告最小结构：
|
||||
|
||||
```json
|
||||
{
|
||||
"generatedAtUtc": "2026-08-14T10:00:00Z",
|
||||
"status": "PASS",
|
||||
"plcEgressDenied": true,
|
||||
"wmsEgressDenied": true
|
||||
}
|
||||
```
|
||||
|
||||
报告必须来自已批准环境的实际检查。不得手工伪造 `PASS`。
|
||||
|
||||
## 4. 构建与本地验证
|
||||
|
||||
在隔离工作树执行：
|
||||
|
||||
```powershell
|
||||
mvn clean test
|
||||
mvn -pl ruoyi-wcs-shadow -am -DskipTests package
|
||||
powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass `
|
||||
-File .\tools\shadow\tests\Test-Ft546ShadowScripts.ps1
|
||||
```
|
||||
|
||||
打包后确认 shadow jar 不含 H2 或测试 profile：
|
||||
|
||||
```powershell
|
||||
jar tf .\ruoyi-wcs-shadow\target\ruoyi-wcs-shadow-3.8.9.jar |
|
||||
Select-String 'h2|application-shadow-test|application-shadow-local'
|
||||
```
|
||||
|
||||
预期没有匹配项。该检查证明打包边界，不证明现场可运行。
|
||||
|
||||
## 5. 捕获稳定 WCS 基线
|
||||
|
||||
使用现场实际路径和 PID，不用 Git 分支或远端引用代替已部署制品：
|
||||
|
||||
```powershell
|
||||
.\tools\shadow\Capture-StableWcsManifest.ps1 `
|
||||
-StableArtifact 'D:\wcs-stable\ruoyi-admin.jar' `
|
||||
-StablePid 1234 `
|
||||
-Configuration 'D:\wcs-stable\application.yml' `
|
||||
-OutputJson 'D:\shadow-evidence\stable-manifest.json'
|
||||
```
|
||||
|
||||
归档输出 JSON。确认其中的 artifact/config SHA-256、PID、启动时间、命令行和监听端口与现场一致。
|
||||
|
||||
## 6. 准备 shadow 目录和配置
|
||||
|
||||
将打包的 shadow jar 和一份外部 site 配置放在独立部署目录。配置只允许以下网络相关内容：
|
||||
|
||||
- loopback 状态端点；
|
||||
- source SQL Server 只读连接；
|
||||
- 独立 evidence SQL Server 写连接。
|
||||
|
||||
配置不得包含 PLC IP/port 或 WMS URL。密码使用受控环境变量或主机密钥设施，不写入 Git、预检 JSON、PID 文件或命令行。
|
||||
|
||||
`Start-Ft546Shadow.ps1` 会从成功预检向进程注入 artifact/config SHA-256、source server/database、loopback address/port 和磁盘阈值，供应用内 startup gate 复核。
|
||||
|
||||
## 7. 运行预检
|
||||
|
||||
```powershell
|
||||
.\tools\shadow\Test-Ft546ShadowPreflight.ps1 `
|
||||
-StableManifest 'D:\shadow-evidence\stable-manifest.json' `
|
||||
-ShadowArtifact 'D:\ft546-shadow\ruoyi-wcs-shadow.jar' `
|
||||
-Config 'D:\ft546-shadow\application-shadow-site.yml' `
|
||||
-DeploymentDirectory 'D:\ft546-shadow' `
|
||||
-PermissionReport 'D:\shadow-evidence\permission-report.json' `
|
||||
-NetworkDenyReport 'D:\shadow-evidence\network-deny-report.json' `
|
||||
-StatusUri 'http://127.0.0.1:18091/shadow/status' `
|
||||
-MinimumFreeBytes 10737418240 `
|
||||
-OutputJson 'D:\shadow-evidence\shadow-preflight.json'
|
||||
```
|
||||
|
||||
预检会失败关闭并留下 JSON，检查内容包括：
|
||||
|
||||
- 稳定 PID 健康且 artifact/config 哈希未变；
|
||||
- shadow jar/config 位于独立部署目录；
|
||||
- YAML 中 source/evidence server、database、username 与权限报告一致；
|
||||
- source/evidence 数据库不同；
|
||||
- 权限和网络 deny 报告为新鲜 `PASS`；
|
||||
- 状态端点为 loopback，端口不与稳定 WCS 冲突；
|
||||
- 配置无 PLC/WMS 端点；
|
||||
- evidence 磁盘空间达到阈值；
|
||||
- shadow 部署目录中不存在当前 `shadow.pid.json`。即使其中记录的进程已退出，陈旧 PID 文件也会使预检失败；必须先核验并按停机/归档流程处理，禁止覆盖后启动。
|
||||
|
||||
任何一项失败都不得启动。
|
||||
|
||||
### 7.1 处理陈旧 PID 记录
|
||||
|
||||
不得删除或覆盖 `shadow.pid.json`。先读取其中的 `pid`：如果该 PID 仍存在，只能执行第 10 节的 `Stop-Ft546Shadow.ps1`，由脚本校验启动时间、Java 可执行文件、shadow jar 哈希和 `-jar` 命令行；校验不一致时停止操作并升级处理。
|
||||
|
||||
只有在 `Get-Process` 连续两次（间隔至少 5 秒）均确认该 PID 不存在后，才可由两名复核人员将原记录改名为 `shadow.stale.<UTC timestamp>.json`。变更单必须保留原记录 SHA-256、两次查询输出、归档路径、操作者和复核人。归档后重新执行完整预检，旧的预检 JSON 不得复用。
|
||||
|
||||
## 8. 启动与初始观察
|
||||
|
||||
只有经过双人复核的成功预检才能启动：
|
||||
|
||||
```powershell
|
||||
$started = .\tools\shadow\Start-Ft546Shadow.ps1 `
|
||||
-ConfirmReadOnlyShadow `
|
||||
-PreflightJson 'D:\shadow-evidence\shadow-preflight.json'
|
||||
|
||||
$started
|
||||
Invoke-RestMethod 'http://127.0.0.1:18091/shadow/status'
|
||||
```
|
||||
|
||||
启动后立即重新核对：
|
||||
|
||||
1. 稳定 PID、制品哈希、健康状态和 PLC writer 所有权均未变化。
|
||||
2. shadow PID 与稳定 PID 不同；PID 文件位于 shadow 部署目录。
|
||||
3. shadow 只监听预检中的 loopback 端口。
|
||||
4. source 审计只有批准视图的 `SELECT`；生产业务表写审计为零。
|
||||
5. PLC/WMS 网络监控没有 shadow 连接尝试。
|
||||
|
||||
先以低频轮询观察 15 分钟。只有稳定 WCS 指标不变、source 查询延迟在阈值内、source lag 正常、evidence 持续增长且磁盘安全时，才可延长观察窗口。
|
||||
|
||||
`STARTING` 表示尚无完整 cycle；`READY` 表示最近 cycle 完成；`DEGRADED` 必须按 reason code 调查。证据存储失败或碰撞会永久停止该进程继续读取，需停止并保留现场。
|
||||
|
||||
## 9. 立即停止条件
|
||||
|
||||
出现任一条件，立即停止 shadow，不等待当前观察窗口结束：
|
||||
|
||||
- 稳定 WCS 健康、CPU、内存、线程、PID 或 PLC writer 所有权变化；
|
||||
- 生产 DB 延迟超过约定阈值，或出现锁等待/连接池压力；
|
||||
- source 权限、数据库身份或批准视图发生变化；
|
||||
- source lag 超出事实新鲜度，或连续 query timeout；
|
||||
- 任意 PLC/WMS 连接尝试或任意生产表写审计；
|
||||
- evidence 磁盘低于阈值、写失败、哈希碰撞或 evidence backlog 异常；
|
||||
- shadow CPU/内存超过约定上限；
|
||||
- 状态为 `DEGRADED` 且原因不能立即解释为短暂 source 故障。
|
||||
|
||||
## 10. 精确停止与回滚
|
||||
|
||||
```powershell
|
||||
.\tools\shadow\Stop-Ft546Shadow.ps1 `
|
||||
-PidFile 'D:\ft546-shadow\shadow.pid.json'
|
||||
```
|
||||
|
||||
停止脚本只接受该 PID 文件记录的进程，并逐项验证 PID、启动时间、Java 可执行文件、shadow jar 哈希和 `-jar` 命令行。任何不一致都拒绝停止，不会按 `java` 进程名批量查杀。成功后 PID 记录改名归档为 `shadow.stopped.<timestamp>.json`。
|
||||
|
||||
回滚完成条件：shadow PID 不存在，稳定 PID/健康/PLC 所有权保持原值，网络与数据库审计无 shadow 副作用。稳定 WCS 从未停止或切换。
|
||||
|
||||
## 11. 导出证据
|
||||
|
||||
现场 SQL Server 模式使用集成认证，或把 SQL 密码短暂放入当前进程环境变量 `FT546_SHADOW_EVIDENCE_PASSWORD`：
|
||||
|
||||
```powershell
|
||||
.\tools\shadow\Export-Ft546ShadowEvidence.ps1 `
|
||||
-EvidenceServer 'evidence-host' `
|
||||
-EvidenceDatabase 'FT546-shadow' `
|
||||
-EvidenceUsername 'ft546_shadow_writer' `
|
||||
-RunId '00000000-0000-0000-0000-000000000000' `
|
||||
-OutputDirectory 'D:\shadow-evidence\export-20260814'
|
||||
```
|
||||
|
||||
离线复核模式接受已审批工具生成的快照目录：
|
||||
|
||||
```powershell
|
||||
.\tools\shadow\Export-Ft546ShadowEvidence.ps1 `
|
||||
-SnapshotDirectory 'D:\shadow-evidence\approved-snapshot' `
|
||||
-RunId '00000000-0000-0000-0000-000000000000' `
|
||||
-OutputDirectory 'D:\shadow-evidence\export-20260814'
|
||||
```
|
||||
|
||||
输出固定包含：
|
||||
|
||||
- `cycles.csv`、`cycles.json`；
|
||||
- `protocol-differences.csv`；
|
||||
- `algorithm-differences.csv`；
|
||||
- `insufficient-facts.csv`；
|
||||
- `startup-checks.json`；
|
||||
- `manifest.json`、`manifest.sha256`。
|
||||
|
||||
校验：
|
||||
|
||||
```powershell
|
||||
$expected = (Get-Content 'D:\shadow-evidence\export-20260814\manifest.sha256').Split()[0]
|
||||
$actual = (Get-FileHash 'D:\shadow-evidence\export-20260814\manifest.json' -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
if ($expected -ne $actual) { throw 'EVIDENCE_MANIFEST_HASH_MISMATCH' }
|
||||
```
|
||||
|
||||
将稳定 manifest、preflight JSON、权限报告、网络 deny 报告、启动/停止 PID 记录、状态响应、现场监控截图和导出目录作为同一批次归档。
|
||||
|
||||
## 12. 禁止事项
|
||||
|
||||
- 不在生产数据库内创建 evidence schema 代替独立 evidence 数据库。
|
||||
- 不给 source 登录任何生产写权限。
|
||||
- 不把 `ruoyi-admin` 以 `SHADOW` 模式启动。
|
||||
- 不允许 shadow 配置 PLC/WMS endpoint。
|
||||
- 不用宽泛 Java 进程查杀命令代替 `Stop-Ft546Shadow.ps1`。
|
||||
- 不因为本地/H2 测试通过就跳过 SQL Server、网络或现场门禁。
|
||||
- 不把本阶段结果解释为 `STANDARD` 上线许可。
|
||||
@@ -10,11 +10,13 @@ import com.ruoyi.shadow.evidence.ShadowEvidenceStorageHealth;
|
||||
import com.ruoyi.shadow.evidence.ShadowRunEvidence;
|
||||
import com.ruoyi.shadow.facts.ShadowFactBatchValidator;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowCycle;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowScheduler;
|
||||
import com.ruoyi.shadow.kernel.JdbcShadowTrafficStateRepository;
|
||||
import com.ruoyi.shadow.kernel.RecordingFt546CommandGateway;
|
||||
import com.ruoyi.shadow.kernel.ShadowCommandIntentFactory;
|
||||
import com.ruoyi.shadow.kernel.ShadowDispatchSnapshotAssembler;
|
||||
import com.ruoyi.shadow.startup.ShadowStartupReport;
|
||||
import com.ruoyi.shadow.facts.SqlServerShadowFactReader;
|
||||
import com.ruoyi.wcs.application.ReadOnlyDispatchCycle;
|
||||
import com.ruoyi.wcs.application.dispatch.DispatchCoordinator;
|
||||
import com.ruoyi.wcs.domain.dispatch.DefaultVehicleAssignmentPolicy;
|
||||
@@ -176,4 +178,14 @@ public class ShadowKernelConfiguration {
|
||||
shadowEvidenceRepository, shadowCycleEvidenceCollector,
|
||||
shadowStartupReport);
|
||||
}
|
||||
|
||||
@Bean(destroyMethod = "close")
|
||||
public Ft546ShadowScheduler ft546ShadowScheduler(
|
||||
SqlServerShadowFactReader factReader,
|
||||
Ft546ShadowCycle shadowCycle,
|
||||
ShadowStartupReport shadowStartupReport,
|
||||
@Qualifier("shadowClock") Clock clock) {
|
||||
return new Ft546ShadowScheduler(factReader, shadowCycle,
|
||||
shadowStartupReport, clock);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
package com.ruoyi.shadow.kernel;
|
||||
|
||||
import com.ruoyi.shadow.facts.ShadowFactBatch;
|
||||
import com.ruoyi.shadow.facts.ShadowFactReadException;
|
||||
import com.ruoyi.shadow.facts.ShadowQueueFact;
|
||||
import com.ruoyi.shadow.facts.ShadowReadCursor;
|
||||
import com.ruoyi.shadow.facts.SqlServerShadowFactReader;
|
||||
import com.ruoyi.shadow.startup.ShadowStartupReport;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.scheduling.annotation.Scheduled;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.ThreadFactory;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
public final class Ft546ShadowScheduler implements AutoCloseable {
|
||||
private static final Logger LOGGER =
|
||||
LoggerFactory.getLogger(Ft546ShadowScheduler.class);
|
||||
private static final Duration CYCLE_DEADLINE = Duration.ofSeconds(4);
|
||||
private static final Duration CRASH_BACKOFF = Duration.ofSeconds(1);
|
||||
private static final String FINISH_QUEUE_VIEW =
|
||||
"shadow_ro.v_wcs_srgv_queue_finish";
|
||||
|
||||
private final SqlServerShadowFactReader factReader;
|
||||
private final Ft546ShadowCycle shadowCycle;
|
||||
private final ShadowStartupReport statusReport;
|
||||
private final Clock clock;
|
||||
private final Duration cycleDeadline;
|
||||
private final ExecutorService cycleExecutor;
|
||||
private final AtomicBoolean inFlight = new AtomicBoolean();
|
||||
private final AtomicBoolean permanentlyStopped = new AtomicBoolean();
|
||||
private volatile ShadowReadCursor cursor = ShadowReadCursor.initial();
|
||||
private volatile Instant nextAttemptAt = Instant.EPOCH;
|
||||
|
||||
public Ft546ShadowScheduler(SqlServerShadowFactReader factReader,
|
||||
Ft546ShadowCycle shadowCycle,
|
||||
ShadowStartupReport statusReport,
|
||||
Clock clock) {
|
||||
this(factReader, shadowCycle, statusReport, clock, CYCLE_DEADLINE);
|
||||
}
|
||||
|
||||
public Ft546ShadowScheduler(SqlServerShadowFactReader factReader,
|
||||
Ft546ShadowCycle shadowCycle,
|
||||
ShadowStartupReport statusReport,
|
||||
Clock clock,
|
||||
Duration cycleDeadline) {
|
||||
this.factReader = Objects.requireNonNull(factReader, "factReader");
|
||||
this.shadowCycle = Objects.requireNonNull(shadowCycle, "shadowCycle");
|
||||
this.statusReport = Objects.requireNonNull(statusReport,
|
||||
"statusReport");
|
||||
this.clock = Objects.requireNonNull(clock, "clock");
|
||||
this.cycleDeadline = requirePositive(cycleDeadline);
|
||||
this.cycleExecutor = Executors.newSingleThreadExecutor(
|
||||
shadowCycleThreadFactory());
|
||||
}
|
||||
|
||||
@Scheduled(
|
||||
initialDelayString = "${shadow.poll.initial-delay-ms:5000}",
|
||||
fixedDelayString = "${shadow.poll.fixed-delay-ms:5000}")
|
||||
public void runScheduledCycle() {
|
||||
runOnce();
|
||||
}
|
||||
|
||||
public boolean runOnce() {
|
||||
Instant now = clock.instant();
|
||||
if (permanentlyStopped.get() || now.isBefore(nextAttemptAt)
|
||||
|| !inFlight.compareAndSet(false, true)) {
|
||||
return false;
|
||||
}
|
||||
Future<CycleAttempt> future = cycleExecutor.submit(
|
||||
() -> executeCycle(now));
|
||||
try {
|
||||
CycleAttempt attempt = future.get(cycleDeadline.toMillis(),
|
||||
TimeUnit.MILLISECONDS);
|
||||
if ("DEGRADED".equals(attempt.result.getStatusCode())) {
|
||||
permanentlyStopped.set(true);
|
||||
statusReport.markPermanentlyDegraded(
|
||||
attempt.result.getReasonCode(), clock.instant());
|
||||
LOGGER.error("FT546 shadow collection stopped: {}",
|
||||
attempt.result.getReasonCode());
|
||||
} else {
|
||||
cursor = advanceFinishCursor(cursor, attempt.batch);
|
||||
}
|
||||
nextAttemptAt = Instant.EPOCH;
|
||||
return true;
|
||||
} catch (TimeoutException timeout) {
|
||||
future.cancel(true);
|
||||
return stopPermanently("SHADOW_CYCLE_TIMEOUT", timeout);
|
||||
} catch (InterruptedException interrupted) {
|
||||
future.cancel(true);
|
||||
Thread.currentThread().interrupt();
|
||||
return stopPermanently("SHADOW_CYCLE_INTERRUPTED", interrupted);
|
||||
} catch (ExecutionException wrapped) {
|
||||
return handleFailure(wrapped.getCause());
|
||||
} finally {
|
||||
inFlight.set(false);
|
||||
}
|
||||
}
|
||||
|
||||
public boolean isPermanentlyStopped() {
|
||||
return permanentlyStopped.get();
|
||||
}
|
||||
|
||||
public Duration getRemainingBackoff() {
|
||||
Duration remaining = Duration.between(clock.instant(), nextAttemptAt);
|
||||
return remaining.isNegative() ? Duration.ZERO : remaining;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
permanentlyStopped.set(true);
|
||||
cycleExecutor.shutdownNow();
|
||||
}
|
||||
|
||||
private CycleAttempt executeCycle(Instant startedAt) {
|
||||
ShadowFactBatch batch = factReader.read(cursor,
|
||||
startedAt.plus(cycleDeadline));
|
||||
return new CycleAttempt(batch, shadowCycle.run(batch, clock.instant()));
|
||||
}
|
||||
|
||||
private boolean handleFailure(Throwable failure) {
|
||||
if (failure instanceof ShadowFactReadException) {
|
||||
ShadowFactReadException readFailure =
|
||||
(ShadowFactReadException) failure;
|
||||
Instant failedAt = clock.instant();
|
||||
statusReport.markDegraded(readFailure.getReasonCode(), failedAt);
|
||||
nextAttemptAt = failedAt.plus(readFailure.getRetryAfter());
|
||||
LOGGER.warn("FT546 shadow source read failed: {}; retry after {} ms",
|
||||
readFailure.getReasonCode(),
|
||||
readFailure.getRetryAfter().toMillis());
|
||||
return false;
|
||||
}
|
||||
if (failure instanceof Error) {
|
||||
throw (Error) failure;
|
||||
}
|
||||
RuntimeException runtimeFailure = failure instanceof RuntimeException
|
||||
? (RuntimeException) failure
|
||||
: new IllegalStateException("shadow cycle failed", failure);
|
||||
Instant failedAt = clock.instant();
|
||||
statusReport.markDegraded("SHADOW_CYCLE_FAILED", failedAt);
|
||||
nextAttemptAt = failedAt.plus(CRASH_BACKOFF);
|
||||
LOGGER.error("FT546 shadow cycle failed", runtimeFailure);
|
||||
return false;
|
||||
}
|
||||
|
||||
private boolean stopPermanently(String reasonCode, Exception failure) {
|
||||
permanentlyStopped.set(true);
|
||||
nextAttemptAt = Instant.EPOCH;
|
||||
statusReport.markPermanentlyDegraded(reasonCode, clock.instant());
|
||||
LOGGER.error("FT546 shadow collection stopped: {}", reasonCode,
|
||||
failure);
|
||||
return false;
|
||||
}
|
||||
|
||||
private static Duration requirePositive(Duration value) {
|
||||
Duration duration = Objects.requireNonNull(value, "cycleDeadline");
|
||||
if (duration.isZero() || duration.isNegative()
|
||||
|| duration.toMillis() == 0L) {
|
||||
throw new IllegalArgumentException(
|
||||
"cycleDeadline must be at least one millisecond");
|
||||
}
|
||||
return duration;
|
||||
}
|
||||
|
||||
private static ThreadFactory shadowCycleThreadFactory() {
|
||||
return runnable -> {
|
||||
Thread thread = new Thread(runnable, "ft546-shadow-cycle");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
};
|
||||
}
|
||||
|
||||
private static ShadowReadCursor advanceFinishCursor(
|
||||
ShadowReadCursor current, ShadowFactBatch batch) {
|
||||
Instant latestDate = current.getLastCreateDate();
|
||||
String latestGuid = current.getLastGuid();
|
||||
for (ShadowQueueFact fact : batch.getSourceQueues()) {
|
||||
if (!FINISH_QUEUE_VIEW.equals(fact.getTableName())
|
||||
|| fact.getCreatedAt() == null) {
|
||||
continue;
|
||||
}
|
||||
int dateOrder = fact.getCreatedAt().compareTo(latestDate);
|
||||
if (dateOrder > 0 || (dateOrder == 0
|
||||
&& fact.getGuid().compareTo(latestGuid) > 0)) {
|
||||
latestDate = fact.getCreatedAt();
|
||||
latestGuid = fact.getGuid();
|
||||
}
|
||||
}
|
||||
return new ShadowReadCursor(latestDate, latestGuid);
|
||||
}
|
||||
|
||||
private static final class CycleAttempt {
|
||||
private final ShadowFactBatch batch;
|
||||
private final ShadowCycleResult result;
|
||||
|
||||
private CycleAttempt(ShadowFactBatch batch, ShadowCycleResult result) {
|
||||
this.batch = batch;
|
||||
this.result = result;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ public final class ShadowStartupReport {
|
||||
private volatile Long sourceLagMillis;
|
||||
private volatile long evidenceBacklog;
|
||||
private volatile String reasonCode;
|
||||
private volatile String permanentDegradedReason;
|
||||
private final Map<String, String> startupChecks;
|
||||
|
||||
private ShadowStartupReport(Status status, Instant checkedAt,
|
||||
@@ -46,6 +47,9 @@ public final class ShadowStartupReport {
|
||||
|
||||
public synchronized void markReady(Instant completedAt,
|
||||
Long sourceLagMillis) {
|
||||
if (permanentDegradedReason != null) {
|
||||
return;
|
||||
}
|
||||
this.status = Status.READY;
|
||||
this.checkedAt = completedAt;
|
||||
this.lastCompletedCycle = completedAt;
|
||||
@@ -55,12 +59,31 @@ public final class ShadowStartupReport {
|
||||
|
||||
public synchronized void markDegraded(String reasonCode,
|
||||
Instant checkedAt) {
|
||||
String normalized = requireReasonCode(reasonCode);
|
||||
if (permanentDegradedReason != null) {
|
||||
return;
|
||||
}
|
||||
this.status = Status.DEGRADED;
|
||||
this.checkedAt = checkedAt;
|
||||
this.reasonCode = normalized;
|
||||
}
|
||||
|
||||
public synchronized void markPermanentlyDegraded(String reasonCode,
|
||||
Instant checkedAt) {
|
||||
String normalized = requireReasonCode(reasonCode);
|
||||
if (permanentDegradedReason == null) {
|
||||
permanentDegradedReason = normalized;
|
||||
this.checkedAt = checkedAt;
|
||||
}
|
||||
this.status = Status.DEGRADED;
|
||||
this.reasonCode = permanentDegradedReason;
|
||||
}
|
||||
|
||||
private static String requireReasonCode(String reasonCode) {
|
||||
if (reasonCode == null || reasonCode.trim().isEmpty()) {
|
||||
throw new IllegalArgumentException(
|
||||
"reasonCode must not be blank");
|
||||
}
|
||||
this.status = Status.DEGRADED;
|
||||
this.checkedAt = checkedAt;
|
||||
this.reasonCode = reasonCode.trim();
|
||||
return reasonCode.trim();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,9 @@ spring:
|
||||
banner-mode: "off"
|
||||
shadow:
|
||||
enabled: false
|
||||
poll:
|
||||
initial-delay-ms: 5000
|
||||
fixed-delay-ms: 5000
|
||||
source:
|
||||
jdbc-url:
|
||||
username:
|
||||
|
||||
@@ -9,6 +9,7 @@ import com.ruoyi.shadow.startup.ShadowDatabaseGate;
|
||||
import com.ruoyi.shadow.startup.ShadowStartupReport;
|
||||
import com.ruoyi.shadow.web.ShadowStatusController;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowCycle;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowScheduler;
|
||||
import com.ruoyi.shadow.kernel.JdbcShadowTrafficStateRepository;
|
||||
import com.ruoyi.shadow.kernel.RecordingFt546CommandGateway;
|
||||
import com.ruoyi.shadow.kernel.ShadowDispatchSnapshotAssembler;
|
||||
@@ -82,6 +83,7 @@ class Ft546ShadowApplicationTest {
|
||||
assertNotNull(context.getBean(DispatchCoordinator.class));
|
||||
assertNotNull(context.getBean(ReadOnlyDispatchCycle.class));
|
||||
assertNotNull(context.getBean(Ft546ShadowCycle.class));
|
||||
assertNotNull(context.getBean(Ft546ShadowScheduler.class));
|
||||
assertNotNull(context.getBean(ShadowDispatchSnapshotAssembler.class));
|
||||
assertNotNull(context.getBean(RecordingFt546CommandGateway.class));
|
||||
assertNotNull(context.getBean(
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
package com.ruoyi.shadow;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.ruoyi.ft546.protocol.Ft546DeviceProfile;
|
||||
import com.ruoyi.ft546.protocol.Ft546EncodedCommand;
|
||||
import com.ruoyi.ft546.protocol.Ft546SrgvCommandEncoder;
|
||||
import com.ruoyi.ft546.protocol.Ft546SrgvCommandIntent;
|
||||
import com.ruoyi.shadow.evidence.JdbcShadowEvidenceRepository;
|
||||
import com.ruoyi.shadow.evidence.ShadowCycleEvidenceCollector;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceRepository;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceStorageHealth;
|
||||
import com.ruoyi.shadow.evidence.ShadowRunEvidence;
|
||||
import com.ruoyi.shadow.facts.ShadowFactBatch;
|
||||
import com.ruoyi.shadow.facts.ShadowFactBatchValidator;
|
||||
import com.ruoyi.shadow.facts.ShadowMapFact;
|
||||
import com.ruoyi.shadow.facts.ShadowQueueFact;
|
||||
import com.ruoyi.shadow.facts.ShadowVehicleFact;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowCycle;
|
||||
import com.ruoyi.shadow.kernel.JdbcShadowTrafficStateRepository;
|
||||
import com.ruoyi.shadow.kernel.RecordingFt546CommandGateway;
|
||||
import com.ruoyi.shadow.kernel.ShadowCommandIntentFactory;
|
||||
import com.ruoyi.shadow.kernel.ShadowCycleResult;
|
||||
import com.ruoyi.shadow.kernel.ShadowDispatchSnapshotAssembler;
|
||||
import com.ruoyi.shadow.startup.ShadowStartupReport;
|
||||
import com.ruoyi.wcs.application.ReadOnlyDispatchCycle;
|
||||
import com.ruoyi.wcs.application.dispatch.DispatchCommit;
|
||||
import com.ruoyi.wcs.application.dispatch.DispatchCoordinator;
|
||||
import com.ruoyi.wcs.domain.dispatch.DefaultVehicleAssignmentPolicy;
|
||||
import com.ruoyi.wcs.domain.traffic.AStarRoutePlanner;
|
||||
import com.ruoyi.wcs.domain.traffic.DeadlockGuard;
|
||||
import com.ruoyi.wcs.domain.traffic.ReservationDecision;
|
||||
import com.ruoyi.wcs.domain.traffic.ReservationRequest;
|
||||
import com.ruoyi.wcs.domain.traffic.ResourceLedger;
|
||||
import com.ruoyi.wcs.domain.traffic.RoutePlanner;
|
||||
import com.ruoyi.wcs.domain.traffic.WaitForGraph;
|
||||
import com.ruoyi.wcs.port.out.DispatchPolicy;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class Ft546ShadowEndToEndTest {
|
||||
private static final String FIXTURES = "/fixtures/";
|
||||
|
||||
@Test
|
||||
void ordinaryCrossFloorAndTwoStageFixturesReachVirtualPlcEvidence()
|
||||
throws Exception {
|
||||
ScenarioResult ordinary = run("ordinary-cycle.json");
|
||||
ScenarioResult crossFloor = run("cross-floor-cycle.json");
|
||||
ScenarioResult twoStage = run("two-stage-cycle.json");
|
||||
|
||||
assertComparable(ordinary, "4004", "4000", "4002", "4021");
|
||||
assertComparable(crossFloor, "4004", "4000", "4002", "4021",
|
||||
"4020", "4022");
|
||||
assertComparable(twoStage, "4404", "4422", "4423", "4402",
|
||||
"4400");
|
||||
assertEquals(3, ordinary.persistedCycles + crossFloor.persistedCycles
|
||||
+ twoStage.persistedCycles);
|
||||
}
|
||||
|
||||
@Test
|
||||
void avoidanceFixtureProducesHoldWithoutVirtualWrites() throws Exception {
|
||||
ScenarioResult avoidance = run("hold-avoidance-cycle.json");
|
||||
|
||||
assertEquals("COMPARABLE", avoidance.cycle.getStatusCode());
|
||||
assertEquals(1, avoidance.cycle.getDecisions().getHeld().size(),
|
||||
"granted=" + avoidance.cycle.getDecisions().getGranted().size()
|
||||
+ ", yielded=" + avoidance.cycle.getDecisions()
|
||||
.getYielded().size());
|
||||
assertEquals("HOLD", avoidance.cycle.getDecisions().getHeld().get(0)
|
||||
.getDecisionType().name());
|
||||
assertEquals(Collections.emptyList(),
|
||||
avoidance.cycle.getEncodedCommands());
|
||||
assertEquals(1, avoidance.persistedCycles);
|
||||
}
|
||||
|
||||
private static ScenarioResult run(String resource) throws Exception {
|
||||
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||
Scenario fixture;
|
||||
try (InputStream input = Ft546ShadowEndToEndTest.class
|
||||
.getResourceAsStream(FIXTURES + resource)) {
|
||||
assertNotNull(input, "missing fixture " + resource);
|
||||
fixture = mapper.readValue(input, Scenario.class);
|
||||
}
|
||||
Instant now = Instant.parse(fixture.capturedAt);
|
||||
DriverManagerDataSource dataSource = new DriverManagerDataSource(
|
||||
"jdbc:h2:mem:" + UUID.randomUUID()
|
||||
+ ";MODE=MSSQLServer;DB_CLOSE_DELAY=-1", "sa", "");
|
||||
JdbcTemplate jdbc = new JdbcTemplate(dataSource);
|
||||
createSchema(jdbc);
|
||||
TransactionTemplate transaction = new TransactionTemplate(
|
||||
new DataSourceTransactionManager(dataSource));
|
||||
RecordingFt546CommandGateway gateway = gateway();
|
||||
ShadowCommandIntentFactory intentFactory =
|
||||
new ShadowCommandIntentFactory();
|
||||
JdbcShadowTrafficStateRepository traffic =
|
||||
new JdbcShadowTrafficStateRepository(jdbc, transaction,
|
||||
mapper, Clock.fixed(now, ZoneOffset.UTC),
|
||||
intentFactory, gateway);
|
||||
AtomicInteger ids = new AtomicInteger();
|
||||
RoutePlanner routePlanner = new AStarRoutePlanner();
|
||||
DispatchCoordinator coordinator = new DispatchCoordinator(
|
||||
new DefaultVehicleAssignmentPolicy(routePlanner,
|
||||
DispatchPolicy.NONE),
|
||||
new DeadlockGuard(new WaitForGraph(), routePlanner), traffic,
|
||||
() -> "SHADOW-E2E-" + ids.incrementAndGet());
|
||||
ReadOnlyDispatchCycle<Ft546SrgvCommandIntent> dispatch =
|
||||
new ReadOnlyDispatchCycle<Ft546SrgvCommandIntent>(
|
||||
coordinator::dispatch, intentFactory::create);
|
||||
ShadowDispatchSnapshotAssembler assembler =
|
||||
new ShadowDispatchSnapshotAssembler("FT546", 3L,
|
||||
topology -> {
|
||||
com.ruoyi.shadow.kernel.ShadowTrafficState state =
|
||||
traffic.load(topology);
|
||||
if (fixture.blockedNode != null
|
||||
&& state.getSnapshotVersion() == 0L) {
|
||||
ResourceLedger blockedLedger =
|
||||
state.getLedger();
|
||||
ReservationDecision blocked = blockedLedger
|
||||
.reserve(new ReservationRequest(
|
||||
"BLOCKER-RES", "BLOCKER-ORDER",
|
||||
"BLOCKER-STEP", "VEH-BLOCKER",
|
||||
Collections.singletonList(
|
||||
"NODE:" + fixture.blockedNode)));
|
||||
assertTrue(blocked.isGranted());
|
||||
assertTrue(traffic.commit(new DispatchCommit(
|
||||
"FT546", topology.version(), 0L,
|
||||
Collections.emptyList(),
|
||||
Collections.emptyList(),
|
||||
Collections.emptyList(),
|
||||
Collections.emptyList(),
|
||||
blockedLedger,
|
||||
state.getWaitForGraph())));
|
||||
state = traffic.load(topology);
|
||||
assertEquals(0, state.getLedger()
|
||||
.availableCapacity("NODE:"
|
||||
+ fixture.blockedNode));
|
||||
}
|
||||
return state;
|
||||
}, Clock.fixed(now, ZoneOffset.UTC));
|
||||
UUID runId = UUID.randomUUID();
|
||||
ShadowRunEvidence run = new ShadowRunEvidence(runId, now,
|
||||
repeat('a'), repeat('b'), "LOCAL", "SHADOW_E2E", "sa",
|
||||
"RUNNING");
|
||||
ShadowEvidenceRepository evidence =
|
||||
new JdbcShadowEvidenceRepository(jdbc, transaction, mapper,
|
||||
run);
|
||||
ShadowStartupReport report = ShadowStartupReport.starting(now,
|
||||
Collections.emptyMap());
|
||||
Ft546ShadowCycle cycle = new Ft546ShadowCycle(
|
||||
new ShadowFactBatchValidator(Duration.ofSeconds(4),
|
||||
Duration.ofSeconds(30)),
|
||||
assembler::assemble, dispatch, gateway,
|
||||
() -> ShadowEvidenceStorageHealth.Check.available(
|
||||
Long.MAX_VALUE),
|
||||
evidence, new ShadowCycleEvidenceCollector(runId, 3L, mapper),
|
||||
report);
|
||||
|
||||
ShadowCycleResult result = cycle.run(batch(fixture, now), now);
|
||||
int persisted = jdbc.queryForObject("SELECT COUNT(*) FROM shadow_cycle",
|
||||
Integer.class).intValue();
|
||||
return new ScenarioResult(result, persisted);
|
||||
}
|
||||
|
||||
private static ShadowFactBatch batch(Scenario fixture, Instant now) {
|
||||
ShadowQueueFact.Builder queue = ShadowQueueFact.builder(
|
||||
"shadow_ro.v_wcs_srgv_queue", fixture.queueGuid)
|
||||
.taskNo(fixture.taskNo)
|
||||
.taskType(fixture.taskType)
|
||||
.priority(Long.valueOf(fixture.priority))
|
||||
.cargoPickSite(fixture.startSite)
|
||||
.cargoPlaceSite(fixture.endSite)
|
||||
.startSite(fixture.startSite)
|
||||
.endSite(fixture.endSite)
|
||||
.createdAt(now.minusSeconds(10))
|
||||
.default02(fixture.protocolTaskType);
|
||||
if (fixture.crossFloorTarget != null) {
|
||||
queue.default07(fixture.crossFloorTarget);
|
||||
}
|
||||
if (fixture.stage != null) {
|
||||
queue.default08(fixture.stage);
|
||||
}
|
||||
ShadowVehicleFact vehicle = ShadowVehicleFact.builder(
|
||||
fixture.vehicleGuid, fixture.vehicleId)
|
||||
.agvStatus("1")
|
||||
.currentSite(fixture.vehicleSite)
|
||||
.battery(80)
|
||||
.workMode(5)
|
||||
.alarm(0)
|
||||
.status(1)
|
||||
.errorCode(0)
|
||||
.allowSend(1)
|
||||
.currentFloor(fixture.vehicleFloor)
|
||||
.capturedAt(now)
|
||||
.build();
|
||||
List<ShadowMapFact> maps = new ArrayList<ShadowMapFact>();
|
||||
for (MapNode node : fixture.map) {
|
||||
maps.add(new ShadowMapFact(node.guid, node.siteId, node.x,
|
||||
node.y, node.z, node.value));
|
||||
}
|
||||
return ShadowFactBatch.create(now.minusSeconds(1), now,
|
||||
Collections.singletonList(queue.build()),
|
||||
Collections.singletonList(vehicle), maps);
|
||||
}
|
||||
|
||||
private static void assertComparable(ScenarioResult result,
|
||||
String... addresses) {
|
||||
assertEquals("COMPARABLE", result.cycle.getStatusCode());
|
||||
assertEquals(1, result.cycle.getDecisions().getGranted().size());
|
||||
assertEquals(1, result.cycle.getEncodedCommands().size());
|
||||
Ft546EncodedCommand encoded = result.cycle.getEncodedCommands().get(0);
|
||||
List<String> actual = new ArrayList<String>();
|
||||
encoded.getWrites().forEach(write -> actual.add(write.getAddress()));
|
||||
assertEquals(Arrays.asList(addresses), actual);
|
||||
}
|
||||
|
||||
private static RecordingFt546CommandGateway gateway() {
|
||||
Ft546DeviceProfile profile = new Ft546DeviceProfile();
|
||||
Map<String, String> registers = new LinkedHashMap<String, String>();
|
||||
registers.put("taskNo", "4000");
|
||||
registers.put("taskType", "4002");
|
||||
registers.put("path", "4004");
|
||||
registers.put("trigger", "4021");
|
||||
registers.put("crossFloorFlag", "4020");
|
||||
registers.put("crossFloorTarget", "4022");
|
||||
registers.put("secondTaskNo", "4400");
|
||||
registers.put("secondTaskType", "4402");
|
||||
registers.put("secondPath", "4404");
|
||||
registers.put("secondStageFlag", "4422");
|
||||
registers.put("secondComplete", "4423");
|
||||
profile.setRegisters(registers);
|
||||
Set<String> capabilities = new LinkedHashSet<String>(Arrays.asList(
|
||||
"POINT_TO_POINT_COMMAND", "TWO_STAGE_ATOMIC_TASK"));
|
||||
profile.setCapabilities(capabilities);
|
||||
return new RecordingFt546CommandGateway(
|
||||
new Ft546SrgvCommandEncoder(), profile, capabilities);
|
||||
}
|
||||
|
||||
private static void createSchema(JdbcTemplate jdbc) {
|
||||
jdbc.execute("CREATE TABLE shadow_traffic_state (state_key nvarchar(256) PRIMARY KEY,snapshot_version bigint NOT NULL,state_json nvarchar(max) NOT NULL,updated_at timestamp NOT NULL)");
|
||||
jdbc.execute("CREATE TABLE shadow_hold_history (parent_task_id nvarchar(128) NOT NULL,vehicle_id nvarchar(64) NOT NULL,resource_key nvarchar(256) NOT NULL,reason_code nvarchar(512) NOT NULL,consecutive_count int NOT NULL,updated_at timestamp NOT NULL,PRIMARY KEY(parent_task_id,vehicle_id,resource_key,reason_code))");
|
||||
jdbc.execute("CREATE TABLE shadow_run (run_id UUID PRIMARY KEY,started_at TIMESTAMP NOT NULL,artifact_sha256 CHAR(64) NOT NULL,configuration_sha256 CHAR(64) NOT NULL,source_server VARCHAR(256) NOT NULL,source_database VARCHAR(128) NOT NULL,source_login VARCHAR(128) NOT NULL,status VARCHAR(32) NOT NULL)");
|
||||
jdbc.execute("CREATE TABLE shadow_cycle (cycle_id UUID PRIMARY KEY,run_id UUID NOT NULL,input_batch_hash CHAR(64) NOT NULL,parent_task_id VARCHAR(128) NOT NULL,phase VARCHAR(64) NOT NULL,route_version BIGINT NOT NULL,status VARCHAR(64) NOT NULL,reason_code VARCHAR(512),database_captured_at TIMESTAMP,completed_at TIMESTAMP NOT NULL,evidence_hash CHAR(64) NOT NULL,FOREIGN KEY (run_id) REFERENCES shadow_run(run_id),UNIQUE (run_id,input_batch_hash,parent_task_id,phase,route_version))");
|
||||
jdbc.execute("CREATE TABLE shadow_input_batch (cycle_id UUID PRIMARY KEY,content_hash CHAR(64) NOT NULL,captured_json CLOB NOT NULL,FOREIGN KEY (cycle_id) REFERENCES shadow_cycle(cycle_id))");
|
||||
jdbc.execute("CREATE TABLE shadow_legacy_queue (cycle_id UUID PRIMARY KEY,vehicle_id VARCHAR(64),path_info CLOB,result_status VARCHAR(64) NOT NULL,result_json CLOB NOT NULL,content_hash CHAR(64) NOT NULL,FOREIGN KEY (cycle_id) REFERENCES shadow_cycle(cycle_id))");
|
||||
jdbc.execute("CREATE TABLE shadow_decision (cycle_id UUID PRIMARY KEY,decision VARCHAR(64) NOT NULL,vehicle_id VARCHAR(64),reason_code VARCHAR(512),decision_json CLOB NOT NULL,content_hash CHAR(64) NOT NULL,FOREIGN KEY (cycle_id) REFERENCES shadow_cycle(cycle_id))");
|
||||
jdbc.execute("CREATE TABLE shadow_encoded_write (cycle_id UUID NOT NULL,write_sequence INT NOT NULL,register_address VARCHAR(16) NOT NULL,data_type VARCHAR(16) NOT NULL,values_json CLOB NOT NULL,trigger_value BIGINT,PRIMARY KEY (cycle_id,write_sequence),FOREIGN KEY (cycle_id) REFERENCES shadow_cycle(cycle_id))");
|
||||
jdbc.execute("CREATE TABLE shadow_difference (difference_id BIGINT IDENTITY PRIMARY KEY,cycle_id UUID NOT NULL,category VARCHAR(64) NOT NULL,reason_code VARCHAR(512) NOT NULL,expected_json CLOB,actual_json CLOB,FOREIGN KEY (cycle_id) REFERENCES shadow_cycle(cycle_id))");
|
||||
}
|
||||
|
||||
private static String repeat(char value) {
|
||||
char[] chars = new char[64];
|
||||
Arrays.fill(chars, value);
|
||||
return new String(chars);
|
||||
}
|
||||
|
||||
public static final class Scenario {
|
||||
public String capturedAt;
|
||||
public String queueGuid;
|
||||
public String taskNo;
|
||||
public String taskType;
|
||||
public long priority;
|
||||
public String protocolTaskType;
|
||||
public String crossFloorTarget;
|
||||
public String stage;
|
||||
public String vehicleGuid;
|
||||
public String vehicleId;
|
||||
public String vehicleSite;
|
||||
public int vehicleFloor;
|
||||
public String startSite;
|
||||
public String endSite;
|
||||
public String blockedNode;
|
||||
public List<MapNode> map;
|
||||
}
|
||||
|
||||
public static final class MapNode {
|
||||
public String guid;
|
||||
public String siteId;
|
||||
public int x;
|
||||
public int y;
|
||||
public int z;
|
||||
public int value;
|
||||
}
|
||||
|
||||
private static final class ScenarioResult {
|
||||
private final ShadowCycleResult cycle;
|
||||
private final int persistedCycles;
|
||||
|
||||
private ScenarioResult(ShadowCycleResult cycle, int persistedCycles) {
|
||||
this.cycle = cycle;
|
||||
this.persistedCycles = persistedCycles;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,440 @@
|
||||
package com.ruoyi.shadow;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.ruoyi.ft546.protocol.Ft546DeviceProfile;
|
||||
import com.ruoyi.ft546.protocol.Ft546SrgvCommandEncoder;
|
||||
import com.ruoyi.ft546.protocol.Ft546SrgvCommandIntent;
|
||||
import com.ruoyi.shadow.evidence.ShadowCycleEvidence;
|
||||
import com.ruoyi.shadow.evidence.ShadowCycleEvidenceCollector;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceDigest;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceException;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceRepository;
|
||||
import com.ruoyi.shadow.evidence.ShadowEvidenceStorageHealth;
|
||||
import com.ruoyi.shadow.facts.ShadowFactBatchValidator;
|
||||
import com.ruoyi.shadow.facts.ShadowMapFact;
|
||||
import com.ruoyi.shadow.facts.ShadowQueueFact;
|
||||
import com.ruoyi.shadow.facts.ShadowReadCursor;
|
||||
import com.ruoyi.shadow.facts.ShadowVehicleFact;
|
||||
import com.ruoyi.shadow.facts.SqlServerShadowFactReader;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowCycle;
|
||||
import com.ruoyi.shadow.kernel.Ft546ShadowScheduler;
|
||||
import com.ruoyi.shadow.kernel.RecordingFt546CommandGateway;
|
||||
import com.ruoyi.shadow.kernel.ShadowCommandIntentFactory;
|
||||
import com.ruoyi.shadow.startup.ShadowStartupReport;
|
||||
import com.ruoyi.wcs.application.ReadOnlyDispatchCycle;
|
||||
import com.ruoyi.wcs.application.dispatch.DispatchCycleResult;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
import org.springframework.core.env.EnumerablePropertySource;
|
||||
import org.springframework.core.env.PropertySource;
|
||||
import org.springframework.boot.env.YamlPropertySourceLoader;
|
||||
import org.springframework.dao.QueryTimeoutException;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
import org.springframework.transaction.support.TransactionTemplate;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.LocalDate;
|
||||
import java.time.ZoneId;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class Ft546ShadowFaultIsolationTest {
|
||||
private static final Instant NOW = Instant.parse("2026-08-14T06:00:00Z");
|
||||
|
||||
@Test
|
||||
void sourceTimeoutNeverEntersCycleOrRecordsACommand() {
|
||||
MutableClock clock = new MutableClock(NOW);
|
||||
AtomicInteger sourceCalls = new AtomicInteger();
|
||||
SqlServerShadowFactReader reader = reader(timeoutSource(sourceCalls), clock);
|
||||
RecordingEvidenceRepository evidence = new RecordingEvidenceRepository();
|
||||
RecordingFt546CommandGateway gateway = gateway();
|
||||
ShadowStartupReport report = report();
|
||||
Ft546ShadowScheduler scheduler = new Ft546ShadowScheduler(reader,
|
||||
evidenceCycle(report, evidence, gateway,
|
||||
() -> ShadowEvidenceStorageHealth.Check.available(
|
||||
Long.MAX_VALUE)), report, clock);
|
||||
|
||||
assertFalse(scheduler.runOnce());
|
||||
|
||||
assertEquals(1, sourceCalls.get());
|
||||
assertEquals(0, evidence.transactionCalls.get());
|
||||
assertEquals(Collections.emptyList(), gateway.getRecorded());
|
||||
assertEquals(ShadowStartupReport.Status.DEGRADED, report.getStatus());
|
||||
assertEquals("SOURCE_QUERY_TIMEOUT", report.getReasonCode());
|
||||
assertEquals(Duration.ofSeconds(1), scheduler.getRemainingBackoff());
|
||||
}
|
||||
|
||||
@Test
|
||||
void evidenceFailurePermanentlyStopsLaterSourceReads() {
|
||||
MutableClock clock = new MutableClock(NOW);
|
||||
AtomicInteger sourceCalls = new AtomicInteger();
|
||||
RecordingEvidenceRepository evidence = new RecordingEvidenceRepository();
|
||||
evidence.saveFailure = new ShadowEvidenceException(
|
||||
"EVIDENCE_WRITE_FAILED");
|
||||
RecordingFt546CommandGateway gateway = gateway();
|
||||
ShadowStartupReport report = report();
|
||||
Ft546ShadowScheduler scheduler = new Ft546ShadowScheduler(
|
||||
reader(emptySource(sourceCalls), clock),
|
||||
evidenceCycle(report, evidence, gateway,
|
||||
() -> ShadowEvidenceStorageHealth.Check.available(
|
||||
Long.MAX_VALUE)), report, clock);
|
||||
|
||||
assertTrue(scheduler.runOnce());
|
||||
int callsAfterFailure = sourceCalls.get();
|
||||
clock.advance(Duration.ofMinutes(1));
|
||||
|
||||
assertFalse(scheduler.runOnce());
|
||||
assertEquals(callsAfterFailure, sourceCalls.get());
|
||||
assertEquals(1, evidence.transactionCalls.get());
|
||||
assertEquals(Collections.emptyList(), gateway.getRecorded());
|
||||
assertEquals("EVIDENCE_STORAGE_UNAVAILABLE", report.getReasonCode());
|
||||
assertTrue(scheduler.isPermanentlyStopped());
|
||||
}
|
||||
|
||||
@Test
|
||||
void shadowCrashBacksOffWithoutRecordingCommands() {
|
||||
MutableClock clock = new MutableClock(NOW);
|
||||
AtomicInteger sourceCalls = new AtomicInteger();
|
||||
RecordingEvidenceRepository evidence = new RecordingEvidenceRepository();
|
||||
RecordingFt546CommandGateway gateway = gateway();
|
||||
ShadowStartupReport report = report();
|
||||
Ft546ShadowScheduler scheduler = new Ft546ShadowScheduler(
|
||||
reader(emptySource(sourceCalls), clock),
|
||||
evidenceCycle(report, evidence, gateway,
|
||||
() -> { throw new IllegalStateException("disk probe crash"); }),
|
||||
report, clock);
|
||||
|
||||
assertFalse(scheduler.runOnce());
|
||||
int callsAfterCrash = sourceCalls.get();
|
||||
assertFalse(scheduler.runOnce());
|
||||
|
||||
assertEquals(callsAfterCrash, sourceCalls.get());
|
||||
assertEquals(Collections.emptyList(), gateway.getRecorded());
|
||||
assertEquals("SHADOW_CYCLE_FAILED", report.getReasonCode());
|
||||
assertEquals(Duration.ofSeconds(1), scheduler.getRemainingBackoff());
|
||||
}
|
||||
|
||||
@Test
|
||||
void blockingShadowCycleTimesOutAndPermanentlyStopsCollection()
|
||||
throws Exception {
|
||||
MutableClock clock = new MutableClock(NOW);
|
||||
AtomicInteger sourceCalls = new AtomicInteger();
|
||||
RecordingEvidenceRepository evidence = new RecordingEvidenceRepository();
|
||||
RecordingFt546CommandGateway gateway = gateway();
|
||||
ShadowStartupReport report = report();
|
||||
CountDownLatch cycleEntered = new CountDownLatch(1);
|
||||
CountDownLatch releaseCycle = new CountDownLatch(1);
|
||||
ShadowEvidenceStorageHealth blockingHealth = () -> {
|
||||
cycleEntered.countDown();
|
||||
boolean interrupted = false;
|
||||
while (true) {
|
||||
try {
|
||||
releaseCycle.await();
|
||||
break;
|
||||
} catch (InterruptedException ignored) {
|
||||
interrupted = true;
|
||||
}
|
||||
}
|
||||
if (interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
return ShadowEvidenceStorageHealth.Check.available(Long.MAX_VALUE);
|
||||
};
|
||||
Ft546ShadowScheduler scheduler = new Ft546ShadowScheduler(
|
||||
reader(emptySource(sourceCalls), clock),
|
||||
evidenceCycle(report, evidence, gateway, blockingHealth),
|
||||
report, clock, Duration.ofSeconds(1));
|
||||
Thread delayedRelease = new Thread(() -> {
|
||||
try {
|
||||
Thread.sleep(2500L);
|
||||
} catch (InterruptedException ignored) {
|
||||
Thread.currentThread().interrupt();
|
||||
} finally {
|
||||
releaseCycle.countDown();
|
||||
}
|
||||
}, "shadow-cycle-test-release");
|
||||
delayedRelease.setDaemon(true);
|
||||
delayedRelease.start();
|
||||
|
||||
long startedAt = System.nanoTime();
|
||||
try {
|
||||
assertFalse(scheduler.runOnce());
|
||||
long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(
|
||||
System.nanoTime() - startedAt);
|
||||
int callsAfterTimeout = sourceCalls.get();
|
||||
|
||||
assertTrue(cycleEntered.await(0L, TimeUnit.MILLISECONDS));
|
||||
assertTrue(elapsedMillis < 1800L,
|
||||
"runOnce must not wait for the blocked cycle");
|
||||
assertTrue(scheduler.isPermanentlyStopped());
|
||||
assertEquals(ShadowStartupReport.Status.DEGRADED,
|
||||
report.getStatus());
|
||||
assertEquals("SHADOW_CYCLE_TIMEOUT", report.getReasonCode());
|
||||
assertFalse(scheduler.runOnce());
|
||||
assertEquals(callsAfterTimeout, sourceCalls.get());
|
||||
assertEquals(Collections.emptyList(), gateway.getRecorded());
|
||||
|
||||
report.markReady(NOW.plusSeconds(1), Long.valueOf(0L));
|
||||
assertEquals(ShadowStartupReport.Status.DEGRADED,
|
||||
report.getStatus());
|
||||
assertEquals("SHADOW_CYCLE_TIMEOUT", report.getReasonCode());
|
||||
} finally {
|
||||
releaseCycle.countDown();
|
||||
scheduler.close();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void shippedShadowConfigurationContainsNoPlcOrWmsEndpoint()
|
||||
throws Exception {
|
||||
List<PropertySource<?>> sources = new YamlPropertySourceLoader().load(
|
||||
"shadow", new ClassPathResource("application-shadow.yml"));
|
||||
List<String> configuredKeys = new ArrayList<String>();
|
||||
for (PropertySource<?> source : sources) {
|
||||
if (source instanceof EnumerablePropertySource) {
|
||||
configuredKeys.addAll(Arrays.asList(
|
||||
((EnumerablePropertySource<?>) source)
|
||||
.getPropertyNames()));
|
||||
}
|
||||
}
|
||||
|
||||
assertTrue(configuredKeys.contains("shadow.source.jdbc-url"));
|
||||
assertTrue(configuredKeys.contains("shadow.evidence.jdbc-url"));
|
||||
assertFalse(configuredKeys.stream().anyMatch(
|
||||
Ft546ShadowFaultIsolationTest::isOutboundEndpointKey));
|
||||
}
|
||||
|
||||
@Test
|
||||
void activeQueueCannotAdvanceTheFinishHistoryCursor() {
|
||||
MutableClock clock = new MutableClock(NOW);
|
||||
Instant finishCreatedAt = NOW.minusSeconds(30);
|
||||
ShadowQueueFact active = queue(
|
||||
"shadow_ro.v_wcs_srgv_queue", "ACTIVE-NEW", NOW);
|
||||
ShadowQueueFact finished = queue(
|
||||
"shadow_ro.v_wcs_srgv_queue_finish", "FINISH-NEW",
|
||||
finishCreatedAt);
|
||||
List<ShadowReadCursor> observed = new ArrayList<ShadowReadCursor>();
|
||||
SqlServerShadowFactReader.FactSource source = new FactSourceAdapter() {
|
||||
@Override
|
||||
public Instant readDatabaseClock(Instant deadline) { return NOW; }
|
||||
|
||||
@Override
|
||||
public List<ShadowQueueFact> readQueues(ShadowReadCursor cursor,
|
||||
Instant deadline) {
|
||||
observed.add(cursor);
|
||||
return Arrays.asList(active, finished);
|
||||
}
|
||||
};
|
||||
RecordingEvidenceRepository evidence = new RecordingEvidenceRepository();
|
||||
ShadowStartupReport report = report();
|
||||
Ft546ShadowScheduler scheduler = new Ft546ShadowScheduler(
|
||||
reader(source, clock),
|
||||
evidenceCycle(report, evidence, gateway(),
|
||||
() -> ShadowEvidenceStorageHealth.Check.available(
|
||||
Long.MAX_VALUE)), report, clock);
|
||||
|
||||
assertTrue(scheduler.runOnce());
|
||||
assertTrue(scheduler.runOnce());
|
||||
|
||||
assertEquals(2, observed.size());
|
||||
assertEquals(finishCreatedAt.minusSeconds(5),
|
||||
observed.get(1).getLastCreateDate());
|
||||
assertEquals("", observed.get(1).getLastGuid());
|
||||
}
|
||||
|
||||
private static SqlServerShadowFactReader reader(
|
||||
SqlServerShadowFactReader.FactSource source, Clock clock) {
|
||||
DriverManagerDataSource dataSource = new DriverManagerDataSource(
|
||||
"jdbc:h2:mem:" + UUID.randomUUID()
|
||||
+ ";MODE=MSSQLServer;DB_CLOSE_DELAY=-1", "sa", "");
|
||||
return new SqlServerShadowFactReader(source,
|
||||
new TransactionTemplate(
|
||||
new DataSourceTransactionManager(dataSource)), clock);
|
||||
}
|
||||
|
||||
private static SqlServerShadowFactReader.FactSource timeoutSource(
|
||||
AtomicInteger calls) {
|
||||
return new FactSourceAdapter() {
|
||||
@Override
|
||||
public Instant readDatabaseClock(Instant deadline) {
|
||||
calls.incrementAndGet();
|
||||
throw new QueryTimeoutException("simulated source timeout");
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
private static SqlServerShadowFactReader.FactSource emptySource(
|
||||
AtomicInteger calls) {
|
||||
return new FactSourceAdapter() {
|
||||
@Override
|
||||
public Instant readDatabaseClock(Instant deadline) {
|
||||
calls.incrementAndGet();
|
||||
return NOW;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
private static Ft546ShadowCycle evidenceCycle(
|
||||
ShadowStartupReport report, RecordingEvidenceRepository evidence,
|
||||
RecordingFt546CommandGateway gateway,
|
||||
ShadowEvidenceStorageHealth storageHealth) {
|
||||
return new Ft546ShadowCycle(
|
||||
new ShadowFactBatchValidator(Duration.ofSeconds(4),
|
||||
Duration.ofSeconds(30)),
|
||||
batch -> { throw new AssertionError("empty facts must not assemble"); },
|
||||
new ReadOnlyDispatchCycle<Ft546SrgvCommandIntent>(
|
||||
snapshot -> DispatchCycleResult.retrySnapshotChanged(),
|
||||
new ShadowCommandIntentFactory()::create),
|
||||
gateway, storageHealth, evidence,
|
||||
new ShadowCycleEvidenceCollector(evidence.runId, 1L,
|
||||
new ObjectMapper().findAndRegisterModules()), report);
|
||||
}
|
||||
|
||||
private static RecordingFt546CommandGateway gateway() {
|
||||
Ft546DeviceProfile profile = new Ft546DeviceProfile();
|
||||
Map<String, String> registers = new LinkedHashMap<String, String>();
|
||||
registers.put("taskNo", "4000");
|
||||
registers.put("taskType", "4002");
|
||||
registers.put("path", "4004");
|
||||
registers.put("trigger", "4021");
|
||||
profile.setRegisters(registers);
|
||||
Set<String> capabilities = new LinkedHashSet<String>(
|
||||
Collections.singletonList("POINT_TO_POINT_COMMAND"));
|
||||
profile.setCapabilities(capabilities);
|
||||
return new RecordingFt546CommandGateway(
|
||||
new Ft546SrgvCommandEncoder(), profile, capabilities);
|
||||
}
|
||||
|
||||
private static ShadowStartupReport report() {
|
||||
return ShadowStartupReport.starting(NOW, Collections.emptyMap());
|
||||
}
|
||||
|
||||
private static boolean isOutboundEndpointKey(String key) {
|
||||
String normalized = key.toLowerCase(java.util.Locale.ROOT)
|
||||
.replace('-', '.').replace('_', '.');
|
||||
boolean networkOwner = normalized.contains(".plc.")
|
||||
|| normalized.startsWith("plc.")
|
||||
|| normalized.contains(".wms.")
|
||||
|| normalized.startsWith("wms.");
|
||||
return networkOwner && (normalized.endsWith(".ip")
|
||||
|| normalized.endsWith(".host")
|
||||
|| normalized.endsWith(".port")
|
||||
|| normalized.endsWith(".url"));
|
||||
}
|
||||
|
||||
private static ShadowQueueFact queue(String table, String guid,
|
||||
Instant createdAt) {
|
||||
return ShadowQueueFact.builder(table, guid)
|
||||
.taskNo("30001")
|
||||
.taskType("2")
|
||||
.priority(1L)
|
||||
.startSite("70501")
|
||||
.endSite("70502")
|
||||
.createdAt(createdAt)
|
||||
.default02("1")
|
||||
.build();
|
||||
}
|
||||
|
||||
private abstract static class FactSourceAdapter
|
||||
implements SqlServerShadowFactReader.FactSource {
|
||||
@Override
|
||||
public List<ShadowQueueFact> readQueues(ShadowReadCursor cursor,
|
||||
Instant deadline) {
|
||||
return Collections.emptyList();
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ShadowVehicleFact> readVehicles(Instant deadline) {
|
||||
return Collections.emptyList();
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ShadowMapFact> readMaps(Instant deadline) {
|
||||
return Collections.emptyList();
|
||||
}
|
||||
}
|
||||
|
||||
private static final class MutableClock extends Clock {
|
||||
private Instant now;
|
||||
|
||||
private MutableClock(Instant now) { this.now = now; }
|
||||
|
||||
private void advance(Duration duration) { now = now.plus(duration); }
|
||||
|
||||
@Override
|
||||
public ZoneId getZone() { return ZoneOffset.UTC; }
|
||||
|
||||
@Override
|
||||
public Clock withZone(ZoneId zone) { return this; }
|
||||
|
||||
@Override
|
||||
public Instant instant() { return now; }
|
||||
}
|
||||
|
||||
private static final class RecordingEvidenceRepository
|
||||
implements ShadowEvidenceRepository {
|
||||
private final UUID runId = UUID.randomUUID();
|
||||
private final AtomicInteger transactionCalls = new AtomicInteger();
|
||||
private RuntimeException saveFailure;
|
||||
|
||||
@Override
|
||||
public SavedCycle save(ShadowCycleEvidence evidence) {
|
||||
return saveAll(Collections.singletonList(evidence)).get(0);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<SavedCycle> saveAll(List<ShadowCycleEvidence> evidence) {
|
||||
if (saveFailure != null) {
|
||||
throw saveFailure;
|
||||
}
|
||||
List<SavedCycle> saved = new ArrayList<SavedCycle>();
|
||||
for (ShadowCycleEvidence value : evidence) {
|
||||
saved.add(new SavedCycle(value.getCycleId(),
|
||||
value.getEvidenceHash(), value.getStatus(),
|
||||
value.getReasonCode()));
|
||||
}
|
||||
return saved;
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T inTransaction(Supplier<T> work) {
|
||||
transactionCalls.incrementAndGet();
|
||||
return work.get();
|
||||
}
|
||||
|
||||
@Override
|
||||
public SavedCycle findCycle(UUID runId, String inputBatchHash,
|
||||
String parentTaskId, String phase,
|
||||
long routeVersion) {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public ShadowEvidenceDigest dailyDigest(LocalDate date,
|
||||
ZoneId zoneId) {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isHealthy() { return true; }
|
||||
}
|
||||
}
|
||||
@@ -73,7 +73,7 @@ class JdbcShadowTrafficStateRepositoryTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void restartReloadPreservesReservationsWaitEdgesHoldsAndFencing() {
|
||||
void restartReloadPreservesReservationsWaitEdgesAndThreeHoldCount() {
|
||||
JdbcShadowTrafficStateRepository first = repository();
|
||||
DispatchCommit initial = commit(0L, protectedLedger(), waitGraph(),
|
||||
held("CAPACITY_CONFLICT"));
|
||||
@@ -102,7 +102,17 @@ class JdbcShadowTrafficStateRepositoryTest {
|
||||
Arrays.asList(twice.getConsecutiveHoldCounts().values()
|
||||
.toArray(new Integer[0])));
|
||||
|
||||
ReservationDecision next = twice.getLedger().reserve(
|
||||
assertTrue(repository().commit(commit(2L, twice.getLedger(),
|
||||
twice.getWaitForGraph(), held("CAPACITY_CONFLICT"))));
|
||||
ShadowTrafficState threeTimes = repository().load(topology);
|
||||
assertEquals(3L, threeTimes.getSnapshotVersion());
|
||||
assertEquals(Collections.singleton("VEH-2"),
|
||||
threeTimes.getWaitForGraph().holdersWaitedOnBy("VEH-1"));
|
||||
assertEquals(Collections.singletonList(Integer.valueOf(3)),
|
||||
Arrays.asList(threeTimes.getConsecutiveHoldCounts().values()
|
||||
.toArray(new Integer[0])));
|
||||
|
||||
ReservationDecision next = threeTimes.getLedger().reserve(
|
||||
new ReservationRequest("RES-2", "ORDER-2", "STEP-2",
|
||||
"VEH-2", Collections.singletonList("NODE:B")));
|
||||
assertTrue(next.isGranted());
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
spring:
|
||||
profiles:
|
||||
active: shadow
|
||||
server:
|
||||
address: 127.0.0.1
|
||||
port: 18091
|
||||
shadow:
|
||||
enabled: true
|
||||
poll:
|
||||
initial-delay-ms: 5000
|
||||
fixed-delay-ms: 5000
|
||||
source:
|
||||
jdbc-url: ${SHADOW_LOCAL_SOURCE_JDBC_URL:}
|
||||
username: ${SHADOW_LOCAL_SOURCE_USERNAME:}
|
||||
password: ${SHADOW_LOCAL_SOURCE_PASSWORD:}
|
||||
evidence:
|
||||
jdbc-url: ${SHADOW_LOCAL_EVIDENCE_JDBC_URL:}
|
||||
username: ${SHADOW_LOCAL_EVIDENCE_USERNAME:}
|
||||
password: ${SHADOW_LOCAL_EVIDENCE_PASSWORD:}
|
||||
output-directory: ${SHADOW_LOCAL_OUTPUT_DIRECTORY:./outputs/shadow-local}
|
||||
minimum-free-bytes: ${SHADOW_LOCAL_MINIMUM_FREE_BYTES:1073741824}
|
||||
run:
|
||||
artifact-sha256:
|
||||
configuration-sha256:
|
||||
source-server:
|
||||
source-database:
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"capturedAt": "2026-08-14T06:00:00Z",
|
||||
"queueGuid": "CROSS_FLOOR",
|
||||
"taskNo": "30002",
|
||||
"taskType": "2",
|
||||
"priority": 10,
|
||||
"protocolTaskType": "8",
|
||||
"crossFloorTarget": "8",
|
||||
"vehicleGuid": "VEH-GUID-2",
|
||||
"vehicleId": "VEH-2",
|
||||
"vehicleSite": "70501",
|
||||
"vehicleFloor": 1,
|
||||
"startSite": "70501",
|
||||
"endSite": "70502",
|
||||
"map": [
|
||||
{"guid": "MAP-1", "siteId": "70501", "x": 0, "y": 0, "z": 1, "value": 1},
|
||||
{"guid": "MAP-2", "siteId": "70502", "x": 1, "y": 0, "z": 1, "value": 1}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"capturedAt": "2026-08-14T06:00:00Z",
|
||||
"queueGuid": "AVOIDANCE",
|
||||
"taskNo": "30004",
|
||||
"taskType": "2",
|
||||
"priority": 10,
|
||||
"protocolTaskType": "1",
|
||||
"vehicleGuid": "VEH-GUID-4",
|
||||
"vehicleId": "VEH-4",
|
||||
"vehicleSite": "70501",
|
||||
"vehicleFloor": 1,
|
||||
"startSite": "70501",
|
||||
"endSite": "70502",
|
||||
"blockedNode": "70502",
|
||||
"map": [
|
||||
{"guid": "MAP-1", "siteId": "70501", "x": 0, "y": 0, "z": 1, "value": 1},
|
||||
{"guid": "MAP-2", "siteId": "70502", "x": 1, "y": 0, "z": 1, "value": 1}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"capturedAt": "2026-08-14T06:00:00Z",
|
||||
"queueGuid": "ORDINARY",
|
||||
"taskNo": "30001",
|
||||
"taskType": "2",
|
||||
"priority": 10,
|
||||
"protocolTaskType": "1",
|
||||
"vehicleGuid": "VEH-GUID-1",
|
||||
"vehicleId": "VEH-1",
|
||||
"vehicleSite": "70501",
|
||||
"vehicleFloor": 1,
|
||||
"startSite": "70501",
|
||||
"endSite": "70502",
|
||||
"map": [
|
||||
{"guid": "MAP-1", "siteId": "70501", "x": 0, "y": 0, "z": 1, "value": 1},
|
||||
{"guid": "MAP-2", "siteId": "70502", "x": 1, "y": 0, "z": 1, "value": 1}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"capturedAt": "2026-08-14T06:00:00Z",
|
||||
"queueGuid": "STAGE_2",
|
||||
"taskNo": "30003",
|
||||
"taskType": "2",
|
||||
"priority": 10,
|
||||
"protocolTaskType": "1",
|
||||
"stage": "2",
|
||||
"vehicleGuid": "VEH-GUID-3",
|
||||
"vehicleId": "VEH-3",
|
||||
"vehicleSite": "70501",
|
||||
"vehicleFloor": 1,
|
||||
"startSite": "70501",
|
||||
"endSite": "70502",
|
||||
"map": [
|
||||
{"guid": "MAP-1", "siteId": "70501", "x": 0, "y": 0, "z": 1, "value": 1},
|
||||
{"guid": "MAP-2", "siteId": "70502", "x": 1, "y": 0, "z": 1, "value": 1}
|
||||
]
|
||||
}
|
||||
54
tools/shadow/Capture-StableWcsManifest.ps1
Normal file
54
tools/shadow/Capture-StableWcsManifest.ps1
Normal file
@@ -0,0 +1,54 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$StableArtifact,
|
||||
[int]$StablePid = 0,
|
||||
[string]$ServiceName,
|
||||
[string[]]$Configuration = @(),
|
||||
[Parameter(Mandatory = $true)][string]$OutputJson
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
. (Join-Path $PSScriptRoot 'ShadowScriptCommon.ps1')
|
||||
|
||||
$artifact = Resolve-ShadowFile $StableArtifact
|
||||
$service = $null
|
||||
if ($ServiceName) {
|
||||
$service = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" `
|
||||
-ErrorAction Stop
|
||||
if ($StablePid -eq 0) { $StablePid = [int]$service.ProcessId }
|
||||
}
|
||||
elseif ($StablePid -gt 0) {
|
||||
try {
|
||||
$service = Get-CimInstance Win32_Service -Filter "ProcessId=$StablePid" `
|
||||
-ErrorAction Stop | Select-Object -First 1
|
||||
}
|
||||
catch { $service = $null }
|
||||
}
|
||||
if ($StablePid -le 0) { throw 'STABLE_PID_OR_SERVICE_REQUIRED' }
|
||||
|
||||
$process = Get-Process -Id $StablePid -ErrorAction Stop
|
||||
$configurationEvidence = @($Configuration | ForEach-Object {
|
||||
$path = Resolve-ShadowFile $_
|
||||
[pscustomobject]@{
|
||||
path = $path
|
||||
sha256 = Get-ShadowSha256 $path
|
||||
}
|
||||
})
|
||||
$manifest = [pscustomobject]@{
|
||||
schemaVersion = 1
|
||||
capturedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
status = 'PASS'
|
||||
artifactPath = $artifact
|
||||
artifactSha256 = Get-ShadowSha256 $artifact
|
||||
pid = [int]$process.Id
|
||||
processStartTimeUtc = $process.StartTime.ToUniversalTime().ToString('o')
|
||||
executablePath = $process.Path
|
||||
commandLine = Get-ShadowCommandLine $process.Id
|
||||
serviceName = if ($null -eq $service) { $null } else { $service.Name }
|
||||
serviceState = if ($null -eq $service) { $null } else { $service.State }
|
||||
configuration = $configurationEvidence
|
||||
listeningPorts = @(Get-ShadowListeningPorts $process.Id)
|
||||
}
|
||||
Write-ShadowJson -Value $manifest -Path $OutputJson
|
||||
return $manifest
|
||||
167
tools/shadow/Export-Ft546ShadowEvidence.ps1
Normal file
167
tools/shadow/Export-Ft546ShadowEvidence.ps1
Normal file
@@ -0,0 +1,167 @@
|
||||
[CmdletBinding(DefaultParameterSetName = 'Snapshot')]
|
||||
param(
|
||||
[Parameter(Mandatory = $true, ParameterSetName = 'Snapshot')]
|
||||
[string]$SnapshotDirectory,
|
||||
[Parameter(Mandatory = $true, ParameterSetName = 'Database')]
|
||||
[string]$EvidenceServer,
|
||||
[Parameter(Mandatory = $true, ParameterSetName = 'Database')]
|
||||
[string]$EvidenceDatabase,
|
||||
[Parameter(ParameterSetName = 'Database')][string]$EvidenceUsername,
|
||||
[Parameter(ParameterSetName = 'Database')][switch]$IntegratedSecurity,
|
||||
[Parameter(Mandatory = $true)][guid]$RunId,
|
||||
[Parameter(Mandatory = $true)][string]$OutputDirectory
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
. (Join-Path $PSScriptRoot 'ShadowScriptCommon.ps1')
|
||||
|
||||
$requiredFiles = @(
|
||||
'cycles.csv', 'cycles.json', 'protocol-differences.csv',
|
||||
'algorithm-differences.csv', 'insufficient-facts.csv',
|
||||
'startup-checks.json'
|
||||
)
|
||||
$output = Resolve-ShadowDirectory $OutputDirectory -Create
|
||||
if (@(Get-ChildItem -LiteralPath $output -Force).Count -gt 0) {
|
||||
throw 'EVIDENCE_EXPORT_DIRECTORY_MUST_BE_EMPTY'
|
||||
}
|
||||
|
||||
function Convert-DataTableRows {
|
||||
param([System.Data.DataTable]$Table)
|
||||
return @($Table.Rows | ForEach-Object {
|
||||
$value = [ordered]@{}
|
||||
foreach ($column in $Table.Columns) {
|
||||
$cell = $_[$column.ColumnName]
|
||||
$value[$column.ColumnName] = if ($cell -is [DBNull]) { $null } else { $cell }
|
||||
}
|
||||
[pscustomobject]$value
|
||||
})
|
||||
}
|
||||
|
||||
if ($PSCmdlet.ParameterSetName -eq 'Snapshot') {
|
||||
$snapshot = Resolve-ShadowDirectory $SnapshotDirectory
|
||||
foreach ($name in $requiredFiles) {
|
||||
$source = Resolve-ShadowFile (Join-Path $snapshot $name)
|
||||
Copy-Item -LiteralPath $source -Destination (Join-Path $output $name)
|
||||
}
|
||||
}
|
||||
else {
|
||||
$builder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder
|
||||
$builder.DataSource = $EvidenceServer
|
||||
$builder.InitialCatalog = $EvidenceDatabase
|
||||
$builder.Encrypt = $true
|
||||
$builder.TrustServerCertificate = $false
|
||||
$builder.ConnectTimeout = 5
|
||||
if ($IntegratedSecurity.IsPresent) {
|
||||
$builder.IntegratedSecurity = $true
|
||||
}
|
||||
else {
|
||||
if ([string]::IsNullOrWhiteSpace($EvidenceUsername)) {
|
||||
throw 'EVIDENCE_USERNAME_REQUIRED'
|
||||
}
|
||||
$password = [Environment]::GetEnvironmentVariable(
|
||||
'FT546_SHADOW_EVIDENCE_PASSWORD', 'Process')
|
||||
if ([string]::IsNullOrWhiteSpace($password)) {
|
||||
throw 'FT546_SHADOW_EVIDENCE_PASSWORD_REQUIRED'
|
||||
}
|
||||
$builder.UserID = $EvidenceUsername
|
||||
$builder.Password = $password
|
||||
}
|
||||
$connection = New-Object System.Data.SqlClient.SqlConnection $builder.ConnectionString
|
||||
function Invoke-EvidenceQuery {
|
||||
param([string]$Sql)
|
||||
$command = $connection.CreateCommand()
|
||||
$command.CommandTimeout = 10
|
||||
$command.CommandText = $Sql
|
||||
$parameter = $command.Parameters.Add('@run_id',
|
||||
[System.Data.SqlDbType]::UniqueIdentifier)
|
||||
$parameter.Value = $RunId
|
||||
$adapter = New-Object System.Data.SqlClient.SqlDataAdapter $command
|
||||
$table = New-Object System.Data.DataTable
|
||||
[void]$adapter.Fill($table)
|
||||
return $table
|
||||
}
|
||||
try {
|
||||
$connection.Open()
|
||||
$cycles = Invoke-EvidenceQuery @'
|
||||
SELECT cycle_id,run_id,input_batch_hash,parent_task_id,phase,route_version,
|
||||
status,reason_code,database_captured_at,completed_at,evidence_hash
|
||||
FROM dbo.shadow_cycle WHERE run_id=@run_id ORDER BY completed_at,cycle_id
|
||||
'@
|
||||
Convert-DataTableRows $cycles | Export-Csv -NoTypeInformation `
|
||||
-Encoding UTF8 -LiteralPath (Join-Path $output 'cycles.csv')
|
||||
$protocol = Invoke-EvidenceQuery @'
|
||||
SELECT d.difference_id,d.cycle_id,d.category,d.reason_code,
|
||||
d.expected_json,d.actual_json
|
||||
FROM dbo.shadow_difference d JOIN dbo.shadow_cycle c ON c.cycle_id=d.cycle_id
|
||||
WHERE c.run_id=@run_id AND d.category LIKE 'PROTOCOL_%'
|
||||
ORDER BY d.difference_id
|
||||
'@
|
||||
Convert-DataTableRows $protocol | Export-Csv -NoTypeInformation `
|
||||
-Encoding UTF8 -LiteralPath (Join-Path $output 'protocol-differences.csv')
|
||||
$algorithm = Invoke-EvidenceQuery @'
|
||||
SELECT d.difference_id,d.cycle_id,d.category,d.reason_code,
|
||||
d.expected_json,d.actual_json
|
||||
FROM dbo.shadow_difference d JOIN dbo.shadow_cycle c ON c.cycle_id=d.cycle_id
|
||||
WHERE c.run_id=@run_id AND d.category LIKE 'ALGORITHM_%'
|
||||
ORDER BY d.difference_id
|
||||
'@
|
||||
Convert-DataTableRows $algorithm | Export-Csv -NoTypeInformation `
|
||||
-Encoding UTF8 -LiteralPath (Join-Path $output 'algorithm-differences.csv')
|
||||
$insufficient = Invoke-EvidenceQuery @'
|
||||
SELECT cycle_id,parent_task_id,phase,status,reason_code,database_captured_at,
|
||||
completed_at,evidence_hash
|
||||
FROM dbo.shadow_cycle
|
||||
WHERE run_id=@run_id AND status='INSUFFICIENT_FACTS'
|
||||
ORDER BY completed_at,cycle_id
|
||||
'@
|
||||
Convert-DataTableRows $insufficient | Export-Csv -NoTypeInformation `
|
||||
-Encoding UTF8 -LiteralPath (Join-Path $output 'insufficient-facts.csv')
|
||||
$startup = Invoke-EvidenceQuery @'
|
||||
SELECT check_id,check_name,status,checked_at FROM dbo.shadow_startup_check
|
||||
WHERE checked_at >= (SELECT started_at FROM dbo.shadow_run WHERE run_id=@run_id)
|
||||
ORDER BY check_id
|
||||
'@
|
||||
Convert-DataTableRows $startup | ConvertTo-Json -Depth 6 | `
|
||||
Set-Content -LiteralPath (Join-Path $output 'startup-checks.json') `
|
||||
-Encoding UTF8
|
||||
$cycleJson = [ordered]@{
|
||||
cycles = Convert-DataTableRows $cycles
|
||||
decisions = Convert-DataTableRows (Invoke-EvidenceQuery `
|
||||
'SELECT d.* FROM dbo.shadow_decision d JOIN dbo.shadow_cycle c ON c.cycle_id=d.cycle_id WHERE c.run_id=@run_id ORDER BY d.cycle_id')
|
||||
encodedWrites = Convert-DataTableRows (Invoke-EvidenceQuery `
|
||||
'SELECT w.* FROM dbo.shadow_encoded_write w JOIN dbo.shadow_cycle c ON c.cycle_id=w.cycle_id WHERE c.run_id=@run_id ORDER BY w.cycle_id,w.write_sequence')
|
||||
differences = Convert-DataTableRows (Invoke-EvidenceQuery `
|
||||
'SELECT d.* FROM dbo.shadow_difference d JOIN dbo.shadow_cycle c ON c.cycle_id=d.cycle_id WHERE c.run_id=@run_id ORDER BY d.difference_id')
|
||||
}
|
||||
$cycleJson | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath `
|
||||
(Join-Path $output 'cycles.json') -Encoding UTF8
|
||||
}
|
||||
finally {
|
||||
if ($null -ne $connection) { $connection.Dispose() }
|
||||
[Environment]::SetEnvironmentVariable(
|
||||
'FT546_SHADOW_EVIDENCE_PASSWORD', $null, 'Process')
|
||||
}
|
||||
}
|
||||
|
||||
$files = @($requiredFiles | Sort-Object | ForEach-Object {
|
||||
$path = Resolve-ShadowFile (Join-Path $output $_)
|
||||
[pscustomobject]@{
|
||||
name = $_
|
||||
length = (Get-Item -LiteralPath $path).Length
|
||||
sha256 = Get-ShadowSha256 $path
|
||||
}
|
||||
})
|
||||
$manifest = [pscustomobject]@{
|
||||
schemaVersion = 1
|
||||
runId = $RunId.ToString()
|
||||
exportedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
sourceMode = $PSCmdlet.ParameterSetName
|
||||
files = $files
|
||||
}
|
||||
$manifestPath = Join-Path $output 'manifest.json'
|
||||
Write-ShadowJson -Value $manifest -Path $manifestPath
|
||||
$manifestHash = Get-ShadowSha256 $manifestPath
|
||||
"$manifestHash manifest.json" | Set-Content -LiteralPath `
|
||||
(Join-Path $output 'manifest.sha256') -Encoding ASCII
|
||||
return $manifest
|
||||
157
tools/shadow/ShadowScriptCommon.ps1
Normal file
157
tools/shadow/ShadowScriptCommon.ps1
Normal file
@@ -0,0 +1,157 @@
|
||||
function Resolve-ShadowFile {
|
||||
param([Parameter(Mandatory = $true)][string]$Path)
|
||||
$item = Get-Item -LiteralPath $Path -ErrorAction Stop
|
||||
if ($item.PSIsContainer) { throw "FILE_REQUIRED:$Path" }
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Resolve-ShadowDirectory {
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$Path,
|
||||
[switch]$Create
|
||||
)
|
||||
if ($Create -and -not (Test-Path -LiteralPath $Path)) {
|
||||
New-Item -ItemType Directory -Path $Path -Force | Out-Null
|
||||
}
|
||||
$item = Get-Item -LiteralPath $Path -ErrorAction Stop
|
||||
if (-not $item.PSIsContainer) { throw "DIRECTORY_REQUIRED:$Path" }
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Write-ShadowJson {
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]$Value,
|
||||
[Parameter(Mandatory = $true)][string]$Path,
|
||||
[int]$Depth = 12
|
||||
)
|
||||
$parent = Split-Path -Parent ([IO.Path]::GetFullPath($Path))
|
||||
if (-not (Test-Path -LiteralPath $parent)) {
|
||||
New-Item -ItemType Directory -Path $parent -Force | Out-Null
|
||||
}
|
||||
$Value | ConvertTo-Json -Depth $Depth | Set-Content -LiteralPath $Path `
|
||||
-Encoding UTF8
|
||||
}
|
||||
|
||||
function Read-ShadowJson {
|
||||
param([Parameter(Mandatory = $true)][string]$Path)
|
||||
$resolved = Resolve-ShadowFile $Path
|
||||
return Get-Content -LiteralPath $resolved -Raw | ConvertFrom-Json
|
||||
}
|
||||
|
||||
function Get-ShadowSha256 {
|
||||
param([Parameter(Mandatory = $true)][string]$Path)
|
||||
return (Get-FileHash -LiteralPath (Resolve-ShadowFile $Path) `
|
||||
-Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
}
|
||||
|
||||
function Get-ShadowCommandLine {
|
||||
param([Parameter(Mandatory = $true)][int]$ProcessId)
|
||||
try {
|
||||
$row = Get-CimInstance Win32_Process -Filter "ProcessId=$ProcessId" `
|
||||
-ErrorAction Stop
|
||||
return $row.CommandLine
|
||||
}
|
||||
catch { return $null }
|
||||
}
|
||||
|
||||
function Get-ShadowListeningPorts {
|
||||
param([Parameter(Mandatory = $true)][int]$ProcessId)
|
||||
try {
|
||||
return @(Get-NetTCPConnection -OwningProcess $ProcessId -State Listen `
|
||||
-ErrorAction Stop | Sort-Object LocalPort | ForEach-Object {
|
||||
[pscustomobject]@{
|
||||
address = $_.LocalAddress
|
||||
port = [int]$_.LocalPort
|
||||
state = $_.State.ToString()
|
||||
}
|
||||
})
|
||||
}
|
||||
catch { return @() }
|
||||
}
|
||||
|
||||
function Test-ShadowPathWithin {
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$Candidate,
|
||||
[Parameter(Mandatory = $true)][string]$Directory
|
||||
)
|
||||
$candidatePath = [IO.Path]::GetFullPath($Candidate)
|
||||
$directoryPath = [IO.Path]::GetFullPath($Directory).TrimEnd('\', '/')
|
||||
if ($candidatePath.Equals($directoryPath,
|
||||
[StringComparison]::OrdinalIgnoreCase)) { return $true }
|
||||
$prefix = $directoryPath + [IO.Path]::DirectorySeparatorChar
|
||||
return $candidatePath.StartsWith($prefix,
|
||||
[StringComparison]::OrdinalIgnoreCase)
|
||||
}
|
||||
|
||||
function Get-ShadowFreeBytes {
|
||||
param([Parameter(Mandatory = $true)][string]$Path)
|
||||
$root = [IO.Path]::GetPathRoot([IO.Path]::GetFullPath($Path))
|
||||
return ([IO.DriveInfo]::new($root)).AvailableFreeSpace
|
||||
}
|
||||
|
||||
function Resolve-ShadowScalarPlaceholder {
|
||||
param([AllowNull()][string]$Value)
|
||||
if ($null -eq $Value) { return $null }
|
||||
if ($Value -match '^\$\{([A-Za-z_][A-Za-z0-9_]*):(.*)\}$') {
|
||||
$environmentValue = [Environment]::GetEnvironmentVariable($matches[1])
|
||||
return if ($null -eq $environmentValue) { $matches[2] } `
|
||||
else { $environmentValue }
|
||||
}
|
||||
return $Value
|
||||
}
|
||||
|
||||
function Read-ShadowYamlScalars {
|
||||
param([Parameter(Mandatory = $true)][string]$Path)
|
||||
$resolved = Resolve-ShadowFile $Path
|
||||
$parents = @{}
|
||||
$values = @{}
|
||||
foreach ($line in Get-Content -LiteralPath $resolved) {
|
||||
if ($line -match '^\s*(#.*)?$') { continue }
|
||||
if ($line -notmatch '^(\s*)([A-Za-z0-9_-]+):(?:\s*(.*))?$') {
|
||||
continue
|
||||
}
|
||||
$indent = $matches[1].Length
|
||||
$key = $matches[2]
|
||||
$raw = if ($null -eq $matches[3]) { '' } else { $matches[3].Trim() }
|
||||
foreach ($level in @($parents.Keys | Where-Object { [int]$_ -ge $indent })) {
|
||||
$parents.Remove($level)
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($raw)) {
|
||||
$parents[$indent] = $key
|
||||
continue
|
||||
}
|
||||
$segments = @($parents.Keys | Sort-Object { [int]$_ } | ForEach-Object {
|
||||
$parents[$_]
|
||||
}) + $key
|
||||
if (($raw.StartsWith('"') -and $raw.EndsWith('"')) -or `
|
||||
($raw.StartsWith("'") -and $raw.EndsWith("'"))) {
|
||||
$raw = $raw.Substring(1, $raw.Length - 2)
|
||||
}
|
||||
$values[($segments -join '.')] = Resolve-ShadowScalarPlaceholder $raw
|
||||
}
|
||||
return $values
|
||||
}
|
||||
|
||||
function Get-ShadowSqlServerJdbcIdentity {
|
||||
param([Parameter(Mandatory = $true)][string]$JdbcUrl)
|
||||
if ($JdbcUrl -notmatch '(?i)^jdbc:sqlserver://([^;]+)(?:;(.*))?$') {
|
||||
throw 'SQLSERVER_JDBC_URL_REQUIRED'
|
||||
}
|
||||
$server = $matches[1].Trim()
|
||||
$properties = @{}
|
||||
foreach ($part in @($matches[2] -split ';')) {
|
||||
if ($part -match '^\s*([^=]+)=(.*)$') {
|
||||
$properties[$matches[1].Trim().ToLowerInvariant()] =
|
||||
$matches[2].Trim()
|
||||
}
|
||||
}
|
||||
$database = $properties['databasename']
|
||||
if ([string]::IsNullOrWhiteSpace($database)) {
|
||||
$database = $properties['database']
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($server) -or `
|
||||
[string]::IsNullOrWhiteSpace($database)) {
|
||||
throw 'SQLSERVER_JDBC_IDENTITY_INCOMPLETE'
|
||||
}
|
||||
return [pscustomobject]@{ server = $server; database = $database }
|
||||
}
|
||||
112
tools/shadow/Start-Ft546Shadow.ps1
Normal file
112
tools/shadow/Start-Ft546Shadow.ps1
Normal file
@@ -0,0 +1,112 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[switch]$ConfirmReadOnlyShadow,
|
||||
[Parameter(Mandatory = $true)][string]$PreflightJson,
|
||||
[string]$JavaExecutable = 'java',
|
||||
[int]$MaximumPreflightAgeMinutes = 30
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
. (Join-Path $PSScriptRoot 'ShadowScriptCommon.ps1')
|
||||
|
||||
if (-not $ConfirmReadOnlyShadow.IsPresent) {
|
||||
throw 'CONFIRM_READ_ONLY_SHADOW_REQUIRED'
|
||||
}
|
||||
$preflightPath = Resolve-ShadowFile $PreflightJson
|
||||
$preflight = Read-ShadowJson $preflightPath
|
||||
if (-not [bool]$preflight.success) { throw 'SUCCESSFUL_PREFLIGHT_REQUIRED' }
|
||||
$generatedAt = [DateTime]::Parse($preflight.generatedAtUtc).ToUniversalTime()
|
||||
if (([DateTime]::UtcNow - $generatedAt).TotalMinutes -gt `
|
||||
$MaximumPreflightAgeMinutes) {
|
||||
throw 'PREFLIGHT_EXPIRED'
|
||||
}
|
||||
if ($null -eq (Get-Process -Id ([int]$preflight.stable.pid) `
|
||||
-ErrorAction SilentlyContinue)) {
|
||||
throw 'STABLE_PROCESS_NOT_HEALTHY'
|
||||
}
|
||||
|
||||
$artifact = Resolve-ShadowFile $preflight.shadowArtifact.path
|
||||
$config = Resolve-ShadowFile $preflight.config.path
|
||||
$deployment = Resolve-ShadowDirectory $preflight.deploymentDirectory
|
||||
if (-not (Test-ShadowPathWithin $artifact $deployment) -or `
|
||||
-not (Test-ShadowPathWithin $config $deployment)) {
|
||||
throw 'SHADOW_FILES_MUST_BE_INSIDE_DEPLOYMENT_DIRECTORY'
|
||||
}
|
||||
if ((Get-ShadowSha256 $artifact) -ne $preflight.shadowArtifact.sha256) {
|
||||
throw 'SHADOW_ARTIFACT_CHANGED_AFTER_PREFLIGHT'
|
||||
}
|
||||
if ((Get-ShadowSha256 $config) -ne $preflight.config.sha256) {
|
||||
throw 'SHADOW_CONFIG_CHANGED_AFTER_PREFLIGHT'
|
||||
}
|
||||
if ((Get-ShadowSha256 $preflight.permissionReport.path) -ne `
|
||||
$preflight.permissionReport.sha256) {
|
||||
throw 'PERMISSION_REPORT_CHANGED_AFTER_PREFLIGHT'
|
||||
}
|
||||
if ((Get-ShadowSha256 $preflight.networkDenyReport.path) -ne `
|
||||
$preflight.networkDenyReport.sha256) {
|
||||
throw 'NETWORK_DENY_REPORT_CHANGED_AFTER_PREFLIGHT'
|
||||
}
|
||||
|
||||
$pidFile = [IO.Path]::GetFullPath($preflight.pidFile)
|
||||
if (-not (Test-ShadowPathWithin $pidFile $deployment)) {
|
||||
throw 'PID_FILE_MUST_BE_INSIDE_DEPLOYMENT_DIRECTORY'
|
||||
}
|
||||
if (Test-Path -LiteralPath $pidFile) { throw 'SHADOW_PID_FILE_ALREADY_EXISTS' }
|
||||
$java = (Get-Command $JavaExecutable -ErrorAction Stop).Source
|
||||
$logs = Join-Path $deployment 'logs'
|
||||
New-Item -ItemType Directory -Path $logs -Force | Out-Null
|
||||
$stdout = Join-Path $logs 'shadow-stdout.log'
|
||||
$stderr = Join-Path $logs 'shadow-stderr.log'
|
||||
$configUri = ([uri]$config).AbsoluteUri
|
||||
$statusUri = [uri]$preflight.statusUri
|
||||
$runtimeProperties = @(
|
||||
'--spring.profiles.active=shadow',
|
||||
'--shadow.enabled=true',
|
||||
"--spring.config.additional-location=$configUri",
|
||||
"--server.address=$($statusUri.Host)",
|
||||
"--server.port=$($statusUri.Port)",
|
||||
"--shadow.evidence.run.artifact-sha256=$($preflight.shadowArtifact.sha256)",
|
||||
"--shadow.evidence.run.configuration-sha256=$($preflight.config.sha256)",
|
||||
"--shadow.evidence.run.source-server=$($preflight.sourceIdentity.server)",
|
||||
"--shadow.evidence.run.source-database=$($preflight.sourceIdentity.database)",
|
||||
"--shadow.evidence.minimum-free-bytes=$($preflight.minimumFreeBytes)"
|
||||
)
|
||||
$quotedProperties = @($runtimeProperties | ForEach-Object {
|
||||
'"{0}"' -f $_.Replace('"', '\"')
|
||||
}) -join ' '
|
||||
$arguments = '-jar "{0}" {1}' -f $artifact.Replace('"', '\"'), `
|
||||
$quotedProperties
|
||||
|
||||
$process = $null
|
||||
try {
|
||||
$process = Start-Process -FilePath $java -ArgumentList $arguments `
|
||||
-WorkingDirectory $deployment -WindowStyle Hidden -PassThru `
|
||||
-RedirectStandardOutput $stdout -RedirectStandardError $stderr
|
||||
Start-Sleep -Milliseconds 300
|
||||
if ($process.HasExited) {
|
||||
throw "SHADOW_PROCESS_EXITED_EARLY:$($process.ExitCode)"
|
||||
}
|
||||
$record = [pscustomobject]@{
|
||||
schemaVersion = 1
|
||||
pid = [int]$process.Id
|
||||
processStartTimeUtc = $process.StartTime.ToUniversalTime().ToString('o')
|
||||
javaExecutable = [IO.Path]::GetFullPath($java)
|
||||
shadowArtifact = $artifact
|
||||
shadowArtifactSha256 = Get-ShadowSha256 $artifact
|
||||
config = $config
|
||||
configSha256 = Get-ShadowSha256 $config
|
||||
deploymentDirectory = $deployment
|
||||
preflightJson = $preflightPath
|
||||
preflightSha256 = Get-ShadowSha256 $preflightPath
|
||||
startedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
}
|
||||
Write-ShadowJson -Value $record -Path $pidFile
|
||||
return $record
|
||||
}
|
||||
catch {
|
||||
if ($null -ne $process -and -not $process.HasExited) {
|
||||
Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
throw
|
||||
}
|
||||
58
tools/shadow/Stop-Ft546Shadow.ps1
Normal file
58
tools/shadow/Stop-Ft546Shadow.ps1
Normal file
@@ -0,0 +1,58 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$PidFile,
|
||||
[int]$WaitSeconds = 10
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
. (Join-Path $PSScriptRoot 'ShadowScriptCommon.ps1')
|
||||
|
||||
$pidPath = Resolve-ShadowFile $PidFile
|
||||
$record = Read-ShadowJson $pidPath
|
||||
$deployment = Resolve-ShadowDirectory $record.deploymentDirectory
|
||||
if (-not (Test-ShadowPathWithin $pidPath $deployment)) {
|
||||
throw 'PID_FILE_OUTSIDE_RECORDED_DEPLOYMENT'
|
||||
}
|
||||
$artifact = Resolve-ShadowFile $record.shadowArtifact
|
||||
if ((Get-ShadowSha256 $artifact) -ne $record.shadowArtifactSha256) {
|
||||
throw 'RECORDED_SHADOW_ARTIFACT_HASH_MISMATCH'
|
||||
}
|
||||
$process = Get-Process -Id ([int]$record.pid) -ErrorAction Stop
|
||||
$actualExecutable = [IO.Path]::GetFullPath($process.Path)
|
||||
$recordedExecutable = [IO.Path]::GetFullPath($record.javaExecutable)
|
||||
if (-not $actualExecutable.Equals($recordedExecutable,
|
||||
[StringComparison]::OrdinalIgnoreCase)) {
|
||||
throw 'RECORDED_PROCESS_EXECUTABLE_MISMATCH'
|
||||
}
|
||||
$recordedStart = [DateTime]::Parse($record.processStartTimeUtc).ToUniversalTime()
|
||||
$actualStart = $process.StartTime.ToUniversalTime()
|
||||
if ([Math]::Abs(($actualStart - $recordedStart).TotalSeconds) -gt 1) {
|
||||
throw 'RECORDED_PROCESS_START_TIME_MISMATCH'
|
||||
}
|
||||
$commandLine = Get-ShadowCommandLine $process.Id
|
||||
if ([string]::IsNullOrWhiteSpace($commandLine) -or `
|
||||
$commandLine -notlike '*-jar*' -or `
|
||||
($commandLine -notlike "*$artifact*" -and `
|
||||
$commandLine -notlike "*$([IO.Path]::GetFileName($artifact))*")) {
|
||||
throw 'RECORDED_PROCESS_COMMAND_MISMATCH'
|
||||
}
|
||||
|
||||
Stop-Process -Id $process.Id -Force -ErrorAction Stop
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds($WaitSeconds)
|
||||
while ([DateTime]::UtcNow -lt $deadline -and `
|
||||
$null -ne (Get-Process -Id $process.Id -ErrorAction SilentlyContinue)) {
|
||||
Start-Sleep -Milliseconds 100
|
||||
}
|
||||
if ($null -ne (Get-Process -Id $process.Id -ErrorAction SilentlyContinue)) {
|
||||
throw 'SHADOW_PROCESS_DID_NOT_STOP'
|
||||
}
|
||||
$archive = Join-Path $deployment `
|
||||
("shadow.stopped.{0}.json" -f [DateTime]::UtcNow.ToString('yyyyMMddHHmmssfff'))
|
||||
Move-Item -LiteralPath $pidPath -Destination $archive
|
||||
return [pscustomobject]@{
|
||||
status = 'STOPPED'
|
||||
pid = [int]$record.pid
|
||||
stoppedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
archivedPidRecord = $archive
|
||||
}
|
||||
194
tools/shadow/Test-Ft546ShadowPreflight.ps1
Normal file
194
tools/shadow/Test-Ft546ShadowPreflight.ps1
Normal file
@@ -0,0 +1,194 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$StableManifest,
|
||||
[Parameter(Mandatory = $true)][string]$ShadowArtifact,
|
||||
[Parameter(Mandatory = $true)][string]$Config,
|
||||
[Parameter(Mandatory = $true)][string]$DeploymentDirectory,
|
||||
[Parameter(Mandatory = $true)][string]$PermissionReport,
|
||||
[Parameter(Mandatory = $true)][string]$NetworkDenyReport,
|
||||
[uri]$StatusUri = 'http://127.0.0.1:18091/shadow/status',
|
||||
[long]$MinimumFreeBytes = 1073741824,
|
||||
[int]$MaximumEvidenceAgeMinutes = 60,
|
||||
[Parameter(Mandatory = $true)][string]$OutputJson
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
. (Join-Path $PSScriptRoot 'ShadowScriptCommon.ps1')
|
||||
|
||||
$checks = New-Object System.Collections.ArrayList
|
||||
function Add-PreflightCheck {
|
||||
param([string]$Name, [bool]$Passed, [string]$Detail)
|
||||
[void]$checks.Add([pscustomobject]@{
|
||||
name = $Name
|
||||
status = if ($Passed) { 'PASS' } else { 'FAIL' }
|
||||
detail = $Detail
|
||||
})
|
||||
}
|
||||
|
||||
$stablePath = Resolve-ShadowFile $StableManifest
|
||||
$stable = Read-ShadowJson $stablePath
|
||||
$artifact = Resolve-ShadowFile $ShadowArtifact
|
||||
$configPath = Resolve-ShadowFile $Config
|
||||
$deployment = Resolve-ShadowDirectory $DeploymentDirectory -Create
|
||||
$permissions = Read-ShadowJson $PermissionReport
|
||||
$network = Read-ShadowJson $NetworkDenyReport
|
||||
$configValues = Read-ShadowYamlScalars $configPath
|
||||
|
||||
$stableAlive = $null -ne (Get-Process -Id ([int]$stable.pid) `
|
||||
-ErrorAction SilentlyContinue)
|
||||
Add-PreflightCheck 'STABLE_PROCESS_HEALTHY' $stableAlive `
|
||||
"pid=$($stable.pid)"
|
||||
$stableArtifactExists = Test-Path -LiteralPath $stable.artifactPath -PathType Leaf
|
||||
$stableHashMatches = $stableArtifactExists -and `
|
||||
((Get-ShadowSha256 $stable.artifactPath) -eq $stable.artifactSha256)
|
||||
Add-PreflightCheck 'STABLE_ARTIFACT_HASH_UNCHANGED' $stableHashMatches `
|
||||
$stable.artifactPath
|
||||
$stableConfigMatches = $true
|
||||
foreach ($stableConfig in @($stable.configuration)) {
|
||||
$stableConfigMatches = $stableConfigMatches -and `
|
||||
(Test-Path -LiteralPath $stableConfig.path -PathType Leaf) -and `
|
||||
((Get-ShadowSha256 $stableConfig.path) -eq $stableConfig.sha256)
|
||||
}
|
||||
Add-PreflightCheck 'STABLE_CONFIG_HASHES_UNCHANGED' $stableConfigMatches `
|
||||
"count=$(@($stable.configuration).Count)"
|
||||
|
||||
$artifactHash = Get-ShadowSha256 $artifact
|
||||
$configHash = Get-ShadowSha256 $configPath
|
||||
$artifactInDeployment = Test-ShadowPathWithin $artifact $deployment
|
||||
$configInDeployment = Test-ShadowPathWithin $configPath $deployment
|
||||
Add-PreflightCheck 'SHADOW_ARTIFACT_IN_DEPLOYMENT' $artifactInDeployment $artifact
|
||||
Add-PreflightCheck 'SHADOW_CONFIG_IN_DEPLOYMENT' $configInDeployment $configPath
|
||||
|
||||
$stableDirectory = Split-Path -Parent $stable.artifactPath
|
||||
$deploymentDistinct = -not ([IO.Path]::GetFullPath($deployment).Equals(
|
||||
[IO.Path]::GetFullPath($stableDirectory),
|
||||
[StringComparison]::OrdinalIgnoreCase))
|
||||
Add-PreflightCheck 'DEPLOYMENT_DISTINCT_FROM_STABLE' $deploymentDistinct `
|
||||
"stable=$stableDirectory;shadow=$deployment"
|
||||
|
||||
$permissionPassed = $permissions.status -eq 'PASS' -and `
|
||||
[bool]$permissions.source.approvedViewsReadable -and `
|
||||
([int]$permissions.source.productionWritePermissionCount -eq 0) -and `
|
||||
[bool]$permissions.evidence.requiredWritesGranted
|
||||
Add-PreflightCheck 'DATABASE_PERMISSIONS_APPROVED' $permissionPassed `
|
||||
"source=$($permissions.source.login);evidence=$($permissions.evidence.login)"
|
||||
$permissionFresh = $false
|
||||
try {
|
||||
$permissionAge = ([DateTime]::UtcNow -
|
||||
[DateTime]::Parse($permissions.generatedAtUtc).ToUniversalTime()).TotalMinutes
|
||||
$permissionFresh = $permissionAge -ge 0 -and `
|
||||
$permissionAge -le $MaximumEvidenceAgeMinutes
|
||||
}
|
||||
catch { $permissionFresh = $false }
|
||||
Add-PreflightCheck 'DATABASE_PERMISSION_REPORT_FRESH' $permissionFresh `
|
||||
"maximumMinutes=$MaximumEvidenceAgeMinutes"
|
||||
|
||||
$configSource = $null
|
||||
$configEvidence = $null
|
||||
$configIdentityError = $null
|
||||
try {
|
||||
$configSource = Get-ShadowSqlServerJdbcIdentity `
|
||||
$configValues['shadow.source.jdbc-url']
|
||||
$configEvidence = Get-ShadowSqlServerJdbcIdentity `
|
||||
$configValues['shadow.evidence.jdbc-url']
|
||||
}
|
||||
catch { $configIdentityError = $_.Exception.Message }
|
||||
$sourceConfigMatches = $null -ne $configSource -and `
|
||||
$configSource.server.Equals([string]$permissions.source.server,
|
||||
[StringComparison]::OrdinalIgnoreCase) -and `
|
||||
$configSource.database.Equals([string]$permissions.source.database,
|
||||
[StringComparison]::OrdinalIgnoreCase) -and `
|
||||
([string]$configValues['shadow.source.username']).Equals(
|
||||
[string]$permissions.source.login,
|
||||
[StringComparison]::OrdinalIgnoreCase)
|
||||
Add-PreflightCheck 'CONFIG_SOURCE_IDENTITY_MATCHES_PERMISSION_REPORT' `
|
||||
$sourceConfigMatches $(if ($configIdentityError) { $configIdentityError } `
|
||||
else { "config=$($configSource.server)|$($configSource.database)" })
|
||||
$evidenceConfigMatches = $null -ne $configEvidence -and `
|
||||
$configEvidence.server.Equals([string]$permissions.evidence.server,
|
||||
[StringComparison]::OrdinalIgnoreCase) -and `
|
||||
$configEvidence.database.Equals([string]$permissions.evidence.database,
|
||||
[StringComparison]::OrdinalIgnoreCase) -and `
|
||||
([string]$configValues['shadow.evidence.username']).Equals(
|
||||
[string]$permissions.evidence.login,
|
||||
[StringComparison]::OrdinalIgnoreCase)
|
||||
Add-PreflightCheck 'CONFIG_EVIDENCE_IDENTITY_MATCHES_PERMISSION_REPORT' `
|
||||
$evidenceConfigMatches $(if ($configIdentityError) { $configIdentityError } `
|
||||
else { "config=$($configEvidence.server)|$($configEvidence.database)" })
|
||||
$sourceIdentity = ("{0}|{1}" -f $permissions.source.server,
|
||||
$permissions.source.database).ToLowerInvariant()
|
||||
$evidenceIdentity = ("{0}|{1}" -f $permissions.evidence.server,
|
||||
$permissions.evidence.database).ToLowerInvariant()
|
||||
$databasesDistinct = $sourceIdentity -ne $evidenceIdentity
|
||||
Add-PreflightCheck 'SOURCE_EVIDENCE_DATABASES_DISTINCT' $databasesDistinct `
|
||||
"source=$sourceIdentity;evidence=$evidenceIdentity"
|
||||
|
||||
$networkPassed = $network.status -eq 'PASS' -and `
|
||||
[bool]$network.plcEgressDenied -and [bool]$network.wmsEgressDenied
|
||||
Add-PreflightCheck 'PLC_WMS_EGRESS_DENIED' $networkPassed `
|
||||
"plc=$($network.plcEgressDenied);wms=$($network.wmsEgressDenied)"
|
||||
$networkFresh = $false
|
||||
try {
|
||||
$networkAge = ([DateTime]::UtcNow -
|
||||
[DateTime]::Parse($network.generatedAtUtc).ToUniversalTime()).TotalMinutes
|
||||
$networkFresh = $networkAge -ge 0 -and `
|
||||
$networkAge -le $MaximumEvidenceAgeMinutes
|
||||
}
|
||||
catch { $networkFresh = $false }
|
||||
Add-PreflightCheck 'NETWORK_DENY_REPORT_FRESH' $networkFresh `
|
||||
"maximumMinutes=$MaximumEvidenceAgeMinutes"
|
||||
|
||||
$loopback = $StatusUri.Host -in @('127.0.0.1', 'localhost', '::1')
|
||||
Add-PreflightCheck 'STATUS_ENDPOINT_LOOPBACK' $loopback $StatusUri.AbsoluteUri
|
||||
$stablePorts = @($stable.listeningPorts | ForEach-Object { [int]$_.port })
|
||||
$portDistinct = $stablePorts -notcontains [int]$StatusUri.Port
|
||||
Add-PreflightCheck 'SHADOW_PORT_DISTINCT_FROM_STABLE' $portDistinct `
|
||||
"shadow=$($StatusUri.Port);stable=$($stablePorts -join ',')"
|
||||
|
||||
$configText = Get-Content -LiteralPath $configPath -Raw
|
||||
$outboundKeyPattern = '(?im)^\s*[^#\r\n]*(plc|wms)[^:\r\n]*(ip|host|port|url)\s*:'
|
||||
$noOutboundEndpoints = -not [regex]::IsMatch($configText, $outboundKeyPattern)
|
||||
Add-PreflightCheck 'NO_PLC_WMS_ENDPOINT_IN_CONFIG' $noOutboundEndpoints $configPath
|
||||
|
||||
$freeBytes = Get-ShadowFreeBytes $deployment
|
||||
$spaceAvailable = $MinimumFreeBytes -ge 0 -and $freeBytes -ge $MinimumFreeBytes
|
||||
Add-PreflightCheck 'OUTPUT_FREE_SPACE' $spaceAvailable `
|
||||
"available=$freeBytes;required=$MinimumFreeBytes"
|
||||
|
||||
$pidFile = Join-Path $deployment 'shadow.pid.json'
|
||||
$noShadowPidFile = -not (Test-Path -LiteralPath $pidFile)
|
||||
Add-PreflightCheck 'NO_EXISTING_SHADOW_PID_FILE' $noShadowPidFile $pidFile
|
||||
|
||||
$failed = @($checks | Where-Object { $_.status -ne 'PASS' })
|
||||
$result = [pscustomobject]@{
|
||||
schemaVersion = 1
|
||||
generatedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
success = $failed.Count -eq 0
|
||||
checks = @($checks)
|
||||
stable = $stable
|
||||
shadowArtifact = [pscustomobject]@{ path = $artifact; sha256 = $artifactHash }
|
||||
config = [pscustomobject]@{ path = $configPath; sha256 = $configHash }
|
||||
deploymentDirectory = $deployment
|
||||
pidFile = $pidFile
|
||||
statusUri = $StatusUri.AbsoluteUri
|
||||
sourceIdentity = $permissions.source
|
||||
evidenceIdentity = $permissions.evidence
|
||||
configuredSourceIdentity = $configSource
|
||||
configuredEvidenceIdentity = $configEvidence
|
||||
minimumFreeBytes = $MinimumFreeBytes
|
||||
permissionReport = [pscustomobject]@{
|
||||
path = Resolve-ShadowFile $PermissionReport
|
||||
sha256 = Get-ShadowSha256 $PermissionReport
|
||||
}
|
||||
networkDenyReport = [pscustomobject]@{
|
||||
path = Resolve-ShadowFile $NetworkDenyReport
|
||||
sha256 = Get-ShadowSha256 $NetworkDenyReport
|
||||
}
|
||||
freeBytes = $freeBytes
|
||||
}
|
||||
Write-ShadowJson -Value $result -Path $OutputJson
|
||||
if ($failed.Count -gt 0) {
|
||||
throw "SHADOW_PREFLIGHT_FAILED:$($failed.name -join ',')"
|
||||
}
|
||||
return $result
|
||||
254
tools/shadow/tests/Test-Ft546ShadowScripts.ps1
Normal file
254
tools/shadow/tests/Test-Ft546ShadowScripts.ps1
Normal file
@@ -0,0 +1,254 @@
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
|
||||
function Assert-True {
|
||||
param([bool]$Condition, [string]$Message)
|
||||
if (-not $Condition) { throw "ASSERTION_FAILED: $Message" }
|
||||
}
|
||||
|
||||
function Assert-Throws {
|
||||
param([scriptblock]$Action, [string]$Reason)
|
||||
try {
|
||||
& $Action
|
||||
}
|
||||
catch {
|
||||
if ($_.Exception.Message -like "*$Reason*") { return }
|
||||
throw "EXPECTED_$Reason, ACTUAL=$($_.Exception.Message)"
|
||||
}
|
||||
throw "EXPECTED_$Reason, BUT_COMMAND_SUCCEEDED"
|
||||
}
|
||||
|
||||
$toolsDirectory = Split-Path -Parent $PSScriptRoot
|
||||
$capture = Join-Path $toolsDirectory 'Capture-StableWcsManifest.ps1'
|
||||
$preflight = Join-Path $toolsDirectory 'Test-Ft546ShadowPreflight.ps1'
|
||||
$start = Join-Path $toolsDirectory 'Start-Ft546Shadow.ps1'
|
||||
$stop = Join-Path $toolsDirectory 'Stop-Ft546Shadow.ps1'
|
||||
$export = Join-Path $toolsDirectory 'Export-Ft546ShadowEvidence.ps1'
|
||||
|
||||
foreach ($script in @($capture, $preflight, $start, $stop, $export)) {
|
||||
Assert-True (Test-Path -LiteralPath $script -PathType Leaf) `
|
||||
"missing script $script"
|
||||
}
|
||||
|
||||
$testRoot = Join-Path ([IO.Path]::GetTempPath()) `
|
||||
("ft546-shadow-script-test-" + [guid]::NewGuid().ToString('N'))
|
||||
$deployment = Join-Path $testRoot 'shadow-deployment'
|
||||
$stableDirectory = Join-Path $testRoot 'stable-deployment'
|
||||
$snapshot = Join-Path $testRoot 'evidence-snapshot'
|
||||
$exportDirectory = Join-Path $testRoot 'evidence-export'
|
||||
$startedPid = $null
|
||||
|
||||
try {
|
||||
New-Item -ItemType Directory -Path $deployment, $stableDirectory, $snapshot `
|
||||
-Force | Out-Null
|
||||
|
||||
$sourceFile = Join-Path $testRoot 'ShadowSleeper.java'
|
||||
@'
|
||||
public final class ShadowSleeper {
|
||||
public static void main(String[] args) throws Exception {
|
||||
Thread.sleep(300000L);
|
||||
}
|
||||
}
|
||||
'@ | Set-Content -LiteralPath $sourceFile -Encoding ASCII
|
||||
& javac -d $testRoot $sourceFile
|
||||
if ($LASTEXITCODE -ne 0) { throw 'JAVAC_FAILED' }
|
||||
$shadowArtifact = Join-Path $deployment 'ruoyi-wcs-shadow.jar'
|
||||
& jar cfe $shadowArtifact ShadowSleeper -C $testRoot 'ShadowSleeper.class'
|
||||
if ($LASTEXITCODE -ne 0) { throw 'JAR_FAILED' }
|
||||
$stableArtifact = Join-Path $stableDirectory 'ruoyi-admin.jar'
|
||||
Copy-Item -LiteralPath $shadowArtifact -Destination $stableArtifact
|
||||
|
||||
$stableConfig = Join-Path $stableDirectory 'application.yml'
|
||||
'spring.profiles.active: stable' | Set-Content -LiteralPath $stableConfig `
|
||||
-Encoding ASCII
|
||||
$shadowConfig = Join-Path $deployment 'application-shadow-site.yml'
|
||||
@'
|
||||
server:
|
||||
address: 127.0.0.1
|
||||
port: 18091
|
||||
shadow:
|
||||
enabled: true
|
||||
source:
|
||||
jdbc-url: jdbc:sqlserver://source.example.invalid;databaseName=FT546-wcs
|
||||
username: shadow_reader
|
||||
evidence:
|
||||
jdbc-url: jdbc:sqlserver://evidence.example.invalid;databaseName=FT546-shadow
|
||||
username: shadow_writer
|
||||
'@ | Set-Content -LiteralPath $shadowConfig -Encoding ASCII
|
||||
|
||||
$stableManifest = Join-Path $testRoot 'stable-manifest.json'
|
||||
& $capture -StableArtifact $stableArtifact -StablePid $PID `
|
||||
-Configuration $stableConfig -OutputJson $stableManifest | Out-Null
|
||||
$stable = Get-Content -LiteralPath $stableManifest -Raw | ConvertFrom-Json
|
||||
Assert-True ($stable.status -eq 'PASS') 'stable manifest must pass'
|
||||
Assert-True ($stable.pid -eq $PID) 'stable PID must be captured exactly'
|
||||
|
||||
$permissionReport = Join-Path $testRoot 'permission-report.json'
|
||||
[ordered]@{
|
||||
generatedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
status = 'PASS'
|
||||
source = [ordered]@{
|
||||
server = 'source.example.invalid'; database = 'FT546-wcs'
|
||||
login = 'shadow_reader'; approvedViewsReadable = $true
|
||||
productionWritePermissionCount = 0
|
||||
}
|
||||
evidence = [ordered]@{
|
||||
server = 'evidence.example.invalid'; database = 'FT546-shadow'
|
||||
login = 'shadow_writer'; requiredWritesGranted = $true
|
||||
}
|
||||
} | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $permissionReport `
|
||||
-Encoding UTF8
|
||||
$networkReport = Join-Path $testRoot 'network-deny-report.json'
|
||||
[ordered]@{
|
||||
generatedAtUtc = [DateTime]::UtcNow.ToString('o')
|
||||
status = 'PASS'; plcEgressDenied = $true; wmsEgressDenied = $true
|
||||
} | ConvertTo-Json | Set-Content -LiteralPath $networkReport -Encoding UTF8
|
||||
|
||||
$wrongPermissionReport = Join-Path $testRoot 'wrong-permission-report.json'
|
||||
$wrongPermissions = Get-Content -LiteralPath $permissionReport -Raw | `
|
||||
ConvertFrom-Json
|
||||
$wrongPermissions.source.database = 'WRONG_DATABASE'
|
||||
$wrongPermissions | ConvertTo-Json -Depth 5 | Set-Content `
|
||||
-LiteralPath $wrongPermissionReport -Encoding UTF8
|
||||
Assert-Throws {
|
||||
& $preflight -StableManifest $stableManifest `
|
||||
-ShadowArtifact $shadowArtifact -Config $shadowConfig `
|
||||
-DeploymentDirectory $deployment `
|
||||
-PermissionReport $wrongPermissionReport `
|
||||
-NetworkDenyReport $networkReport `
|
||||
-StatusUri 'http://127.0.0.1:18091/shadow/status' `
|
||||
-MinimumFreeBytes 0 `
|
||||
-OutputJson (Join-Path $testRoot 'wrong-preflight.json') | Out-Null
|
||||
} 'CONFIG_SOURCE_IDENTITY_MATCHES_PERMISSION_REPORT'
|
||||
|
||||
$stalePidFile = Join-Path $deployment 'shadow.pid.json'
|
||||
[ordered]@{ pid = 2147483000 } | ConvertTo-Json | Set-Content `
|
||||
-LiteralPath $stalePidFile -Encoding UTF8
|
||||
Assert-Throws {
|
||||
& $preflight -StableManifest $stableManifest `
|
||||
-ShadowArtifact $shadowArtifact -Config $shadowConfig `
|
||||
-DeploymentDirectory $deployment `
|
||||
-PermissionReport $permissionReport `
|
||||
-NetworkDenyReport $networkReport `
|
||||
-StatusUri 'http://127.0.0.1:18091/shadow/status' `
|
||||
-MinimumFreeBytes 0 `
|
||||
-OutputJson (Join-Path $testRoot 'stale-pid-preflight.json') | Out-Null
|
||||
} 'NO_EXISTING_SHADOW_PID_FILE'
|
||||
Remove-Item -LiteralPath $stalePidFile
|
||||
|
||||
$preflightJson = Join-Path $testRoot 'shadow-preflight.json'
|
||||
& $preflight -StableManifest $stableManifest `
|
||||
-ShadowArtifact $shadowArtifact -Config $shadowConfig `
|
||||
-DeploymentDirectory $deployment -PermissionReport $permissionReport `
|
||||
-NetworkDenyReport $networkReport -StatusUri 'http://127.0.0.1:18091/shadow/status' `
|
||||
-MinimumFreeBytes 0 -OutputJson $preflightJson | Out-Null
|
||||
$preflightResult = Get-Content -LiteralPath $preflightJson -Raw | ConvertFrom-Json
|
||||
Assert-True ([bool]$preflightResult.success) 'preflight must pass'
|
||||
Assert-True ($preflightResult.stable.pid -eq $PID) `
|
||||
'preflight must preserve stable PID'
|
||||
|
||||
Assert-Throws {
|
||||
& $start -PreflightJson $preflightJson | Out-Null
|
||||
} 'CONFIRM_READ_ONLY_SHADOW_REQUIRED'
|
||||
|
||||
$configBytes = [IO.File]::ReadAllBytes($shadowConfig)
|
||||
Add-Content -LiteralPath $shadowConfig -Value '# changed after preflight' `
|
||||
-Encoding ASCII
|
||||
Assert-Throws {
|
||||
& $start -ConfirmReadOnlyShadow -PreflightJson $preflightJson | Out-Null
|
||||
} 'SHADOW_CONFIG_CHANGED_AFTER_PREFLIGHT'
|
||||
[IO.File]::WriteAllBytes($shadowConfig, $configBytes)
|
||||
|
||||
$startResult = & $start -ConfirmReadOnlyShadow `
|
||||
-PreflightJson $preflightJson
|
||||
$startedPid = [int]$startResult.pid
|
||||
Assert-True ($startedPid -ne $PID) 'shadow PID must differ from stable PID'
|
||||
Assert-True ($null -ne (Get-Process -Id $startedPid -ErrorAction Stop)) `
|
||||
'shadow process must be alive after start'
|
||||
Assert-True ($null -ne (Get-Process -Id $PID -ErrorAction Stop)) `
|
||||
'stable process must remain alive after shadow start'
|
||||
$shadowCommandLine = (Get-CimInstance Win32_Process `
|
||||
-Filter "ProcessId=$startedPid").CommandLine
|
||||
Assert-True ($shadowCommandLine -like `
|
||||
'*--shadow.evidence.run.artifact-sha256=*') `
|
||||
'start must pass the verified artifact hash'
|
||||
Assert-True ($shadowCommandLine -like `
|
||||
'*--shadow.evidence.run.configuration-sha256=*') `
|
||||
'start must pass the verified configuration hash'
|
||||
Assert-True ($shadowCommandLine -like `
|
||||
'*--shadow.evidence.run.source-database=FT546-wcs*') `
|
||||
'start must pass the verified source database'
|
||||
Assert-True ($shadowCommandLine -like '*--server.address=127.0.0.1*') `
|
||||
'start must force the preflight loopback address'
|
||||
Assert-True ($shadowCommandLine -like '*--server.port=18091*') `
|
||||
'start must force the preflight status port'
|
||||
|
||||
$pidFile = Join-Path $deployment 'shadow.pid.json'
|
||||
& $stop -PidFile $pidFile | Out-Null
|
||||
Assert-True ($null -eq (Get-Process -Id $startedPid -ErrorAction SilentlyContinue)) `
|
||||
'recorded shadow PID must stop'
|
||||
$startedPid = $null
|
||||
Assert-True ($null -ne (Get-Process -Id $PID -ErrorAction Stop)) `
|
||||
'stable process must remain alive after shadow stop'
|
||||
|
||||
$stableProcess = Get-Process -Id $PID
|
||||
[ordered]@{
|
||||
pid = $PID
|
||||
processStartTimeUtc = $stableProcess.StartTime.ToUniversalTime().ToString('o')
|
||||
javaExecutable = $stableProcess.Path
|
||||
shadowArtifact = $shadowArtifact
|
||||
shadowArtifactSha256 = (Get-FileHash -LiteralPath $shadowArtifact -Algorithm SHA256).Hash
|
||||
deploymentDirectory = $deployment
|
||||
} | ConvertTo-Json | Set-Content -LiteralPath $pidFile -Encoding UTF8
|
||||
Assert-Throws { & $stop -PidFile $pidFile | Out-Null } `
|
||||
'RECORDED_PROCESS_COMMAND_MISMATCH'
|
||||
Assert-True ($null -ne (Get-Process -Id $PID -ErrorAction Stop)) `
|
||||
'mismatched PID must never be stopped'
|
||||
|
||||
$requiredEvidence = @(
|
||||
'cycles.csv', 'cycles.json', 'protocol-differences.csv',
|
||||
'algorithm-differences.csv', 'insufficient-facts.csv',
|
||||
'startup-checks.json'
|
||||
)
|
||||
foreach ($name in $requiredEvidence) {
|
||||
"fixture:$name" | Set-Content -LiteralPath (Join-Path $snapshot $name) `
|
||||
-Encoding UTF8
|
||||
}
|
||||
& $export -SnapshotDirectory $snapshot -OutputDirectory $exportDirectory `
|
||||
-RunId ([guid]::NewGuid()) | Out-Null
|
||||
Assert-True (Test-Path -LiteralPath (Join-Path $exportDirectory 'manifest.json')) `
|
||||
'evidence manifest must exist'
|
||||
Assert-True (Test-Path -LiteralPath (Join-Path $exportDirectory 'manifest.sha256')) `
|
||||
'evidence manifest hash must exist'
|
||||
$expectedManifestHash = ((Get-Content -LiteralPath `
|
||||
(Join-Path $exportDirectory 'manifest.sha256')) -split '\s+')[0]
|
||||
$actualManifestHash = (Get-FileHash -LiteralPath `
|
||||
(Join-Path $exportDirectory 'manifest.json') -Algorithm SHA256).Hash
|
||||
Assert-True ($expectedManifestHash -eq $actualManifestHash.ToLowerInvariant()) `
|
||||
'evidence manifest hash must verify'
|
||||
foreach ($name in $requiredEvidence) {
|
||||
Assert-True (Test-Path -LiteralPath (Join-Path $exportDirectory $name)) `
|
||||
"evidence export missing $name"
|
||||
}
|
||||
|
||||
[pscustomobject]@{
|
||||
status = 'PASS'
|
||||
stablePid = $PID
|
||||
preflight = $preflightJson
|
||||
exportManifest = (Join-Path $exportDirectory 'manifest.json')
|
||||
}
|
||||
}
|
||||
finally {
|
||||
if ($null -ne $startedPid) {
|
||||
Stop-Process -Id $startedPid -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
$resolvedTemp = [IO.Path]::GetFullPath($testRoot)
|
||||
$tempRoot = [IO.Path]::GetFullPath([IO.Path]::GetTempPath())
|
||||
if ($resolvedTemp.StartsWith($tempRoot, [StringComparison]::OrdinalIgnoreCase) `
|
||||
-and (Test-Path -LiteralPath $resolvedTemp)) {
|
||||
Remove-Item -LiteralPath $resolvedTemp -Recurse -Force
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user