fix(shadow): repair field routing and harden evidence collection
This commit is contained in:
108
docs/runbooks/ft546-shadow-20260907-repair-notes.md
Normal file
108
docs/runbooks/ft546-shadow-20260907-repair-notes.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# FT546 Shadow Repair: 2026-09-07
|
||||
|
||||
## Delivery Boundary
|
||||
|
||||
Source changes in the existing `optimize/wcs` worktree, based on `9829b9c0fa71d8a4623a9f968178183217724db1`.
|
||||
Prior uncommitted field fixes are preserved. During the repair/validation stage, no commit,
|
||||
push, deployment, live SQL write, schema migration, production application startup,
|
||||
PLC command or WMS call was performed. The user subsequently authorized committing and
|
||||
pushing the repaired source to `origin/optimize/wcs`; this does not authorize deployment.
|
||||
No deployment scripts or JAR deliverables are included.
|
||||
|
||||
This is a shadow repair, not approval to replace the production scheduler.
|
||||
|
||||
## Repairs
|
||||
|
||||
- Keep virtual TD/WY transit nodes in route planning and resource protection. Generate
|
||||
PLC preview points from numeric start/end/turn nodes, rather than every transit node.
|
||||
Missing geometry, nonnumeric command points and more than six turns remain rejected.
|
||||
- Deduplicate protected resource keys when approach and execution routes overlap.
|
||||
Preserve ordered route nodes, including repeated visits and turns.
|
||||
- Tag new hypothetical reservations with `predictionReservationIds`. On the next poll
|
||||
or restart, remove only tagged predictions from the working ledger. Preserve untagged
|
||||
legacy reservations and all OCCUPIED, command-unknown or recovery-unresolved protection.
|
||||
Keep snapshot CAS, fencing tokens, HOLD counters and transactional evidence rollback.
|
||||
- Use a UUID startup prefix for new reservation IDs, preventing reuse after restart.
|
||||
- Bound evidence growth and persist current run state, including empty polls and shutdown.
|
||||
|
||||
## Capacity Settings
|
||||
|
||||
Existing local external configuration remains usable; omitted new settings use these defaults:
|
||||
|
||||
```yaml
|
||||
shadow:
|
||||
evidence:
|
||||
maximum-database-bytes: 17179869184
|
||||
maximum-batch-payload-bytes: 33554432
|
||||
```
|
||||
|
||||
The database limit is 16 GiB of allocated data plus log files, read from the evidence
|
||||
database's `sys.database_files`. It is not a table row-count limit, used-page measurement,
|
||||
SQL Server MAXSIZE setting or exact physical disk quota. A batch/file-growth increment can
|
||||
overshoot the threshold before the next poll. The existing local `minimum-free-bytes`
|
||||
check still applies to `output-directory`; it does not measure every remote SQL volume.
|
||||
|
||||
The 32 MiB batch limit counts newly inserted large text payloads as UTF-16 bytes.
|
||||
It excludes database page, index and transaction-log overhead. Existing identical replays
|
||||
do not consume the insert budget. Over-budget batches roll back with their traffic state.
|
||||
|
||||
Collection stops with a specific reason: `EVIDENCE_DATABASE_SIZE_LIMIT`,
|
||||
`EVIDENCE_DATABASE_SIZE_CHECK_FAILED`, `EVIDENCE_BATCH_PAYLOAD_LIMIT`, or the existing
|
||||
storage availability/free-space reason. It does not automatically resume or delete rows.
|
||||
Raw `captured_json`, its content hash, evidence hashes and existing replay identity remain
|
||||
unchanged. Lossless deduplication, archival and retention require a separate reviewed
|
||||
migration/operational decision. Existing evidence is not compacted by this repair.
|
||||
|
||||
## Durable Runtime State
|
||||
|
||||
Each new process uses its own run UUID. In the already-authorized evidence tables:
|
||||
|
||||
- `shadow_run.status` records STARTING, RUNNING (status endpoint READY), DEGRADED or STOPPED.
|
||||
- `shadow_traffic_state.state_key = FT546:RUN:<runId>` contains a `RUN_HEARTBEAT` JSON record.
|
||||
- `heartbeatAt`, `lastAttemptAt`, `lastCompletedCycle` and `checkedAt` use UTC ISO timestamps.
|
||||
- `collectionStopped`, `reasonCode` and `cycleDiagnostics` distinguish process liveness,
|
||||
attempted collection, completed collection and permanent collection failure.
|
||||
- Pre/post-poll writes and scheduled backoff/stopped ticks refresh the same bounded row.
|
||||
Runtime JSON is limited to 32,768 characters, independently of business evidence size.
|
||||
- Runtime persistence failure stops further source reads. Failed writes cannot guarantee
|
||||
that the database contains the latest stop reason; check logs and heartbeat freshness.
|
||||
- Normal close records STOPPED. Forced termination/power loss can only be detected by a
|
||||
stale heartbeat. A RUNNING row alone is not proof of a live process or successful work.
|
||||
|
||||
Traffic-ledger readers should select `FT546:TOPOLOGY:<version>` rows; runtime rows have
|
||||
their own key namespace and `kind`. Historical runs without a heartbeat are not rewritten
|
||||
or retrospectively marked stopped. With default polling, investigate a heartbeat older
|
||||
than 60 seconds; adjust this operational threshold when poll/deadline settings change.
|
||||
|
||||
## Local Verification
|
||||
|
||||
Final offline Maven command used Maven 3.6.0 and the supplied complete-v2 capture ZIP:
|
||||
`mvn -o -pl ruoyi-wcs-shadow -am test -Dshadow.replay.zip=<complete-v2-zip>`.
|
||||
|
||||
- 298 tests passed: 128 WCS, 10 protocol, 160 shadow; zero failures, errors or skipped tests.
|
||||
- 300 independent captured inputs: 600 available legacy projections, 300 normal
|
||||
NO_ELIGIBLE_VEHICLE holds, zero new grants. This sample does not prove live dispatch readiness.
|
||||
- 10 additional captured field batches: 9 encodable grants, 9 normal holds,
|
||||
zero input or grant-projection errors. Independent empty initial traffic state per batch.
|
||||
- 6 historical queue paths: zero legacy, routing or command-point projection errors.
|
||||
These checks are protocol-only, not a reconstruction of historical vehicle eligibility.
|
||||
- Focused regressions cover overlap, virtual transit nodes, invalid command points,
|
||||
prediction expiry, protected state, ID restart, storage limits, atomic rollback,
|
||||
heartbeat, retries, timeout permanence, normal stop and runtime write failure.
|
||||
- Independent read-only review found no blocking issues. `git diff --check` passed.
|
||||
Legacy Admin and existing PLC/WMS implementation files have no changes.
|
||||
|
||||
Detailed local logs are under
|
||||
`D:/workSpace/PROJECT_SPACE/FT546/outputs/wcs-shadow-review-20260907/`:
|
||||
`repair-full-with-captured-replay.log` and `live-database-repair-final-replay.log`.
|
||||
|
||||
## Remaining Field Validation
|
||||
|
||||
SQL persistence tests use isolated H2 databases. The new metadata query and runtime writes
|
||||
have not been exercised against SQL Server 2016 under the deployed shadow login in this
|
||||
repair turn. No physical PLC/HIL test, forced network-stall shutdown test or sustained
|
||||
production-load test was run. Query/pool timeouts are not an OS-level hard shutdown bound.
|
||||
|
||||
After an explicitly authorized shadow update, verify the new run's heartbeat, collection
|
||||
progress, capacity reasons, source latency and evidence completeness under real load.
|
||||
Review new shadow evidence before deciding on any production scheduler cutover.
|
||||
Reference in New Issue
Block a user