FT546MWS/docs/superpowers/specs/2026-09-03-ft546wms-shadow-v3-design.md

FT546 WMS Shadow v3 Stable-Snapshot Design

Goal

Produce trustworthy pre-Legacy inbound and outbound comparison samples while preserving the production behavior of the current Legacy WMS. Shadow v3 remains observation-only: it cannot select a business location, change a Legacy result, or write a Legacy business table.

The field-test branch is test/ft546wms-shadow-v3-20260903, based on the deployed shadow v2 commit 3f7ed3718defef2c4ab37e1deb2534d2329747e4. At branch creation, verified main commit 916ac6b is an ancestor of that v2 commit. Deployment tooling must verify the expected branch, commit, clean worktree, and main ancestry again instead of relying on a moving origin/main name.

Non-negotiable safety invariants

1. Legacy stored procedures remain the sole owners of inbound and outbound decisions and business writes. Each procedure is invoked exactly once with the same inputs and its result remains authoritative.
2. The Legacy caller never waits for shadow planning, comparison, snapshot refresh, retry, or logging. Submission performs only bounded in-memory bookkeeping; a busy, unavailable, or failed shadow path returns a no-op completion token immediately.
3. No shadow exception may escape into a Legacy request, participate in its transaction, change its response, or cause its transaction to roll back.
4. Shadow code writes only package-owned alg.* read-model and diagnostic objects. It performs no insert, update, delete, reservation, callback, or workflow action against Legacy WMS business tables.
5. Shadow v3 exposes no ACTIVE routing flag or controller. Turning shadow off restores the application to Legacy-only behavior without a database rollback.
6. Data quality is conservative: an observation is skipped when its preconditions cannot be proven. An uncertain result is never reported as a valid comparison.
7. This v3 field design supports one FT546WMS JVM connected to the shadow package. The deployment check must refuse to enable shadow when another FT546WMS process or recent shadow-instance heartbeat is present. Multi-instance support requires a later cross-instance coordination design.

Why v3 is required

Shadow v2 refreshes its SQL read model inside every planning call. Field refreshes took about one second. Although the task is submitted before the Legacy procedure, thread scheduling commonly lets Legacy mutate the source rows before the background refresh reads them. This produced post-Legacy or incomplete observations, including outbound stock-not-found and waiting-channel results with missing snapshot diagnostics.

Shadow v3 separates snapshot refresh from planning. A planner can read only a previously completed, immutable stable snapshot; it never refreshes from live Legacy tables as part of the plan.

Architecture

Snapshot coordinator

A JVM-local coordinator owns the lifecycle of the single installed warehouse snapshot. Its states are:

• INITIALIZING: startup refresh has not produced a usable snapshot.
• READY: a completed stable snapshot may be leased by one observation.
• BUSY: a request has leased the snapshot, Legacy activity is in progress, or refresh is pending.
• UNAVAILABLE: refresh failed or the snapshot cannot be proven stable.

The coordinator stores a monotonically increasing Legacy activity epoch, active-Legacy count, snapshot generation, warehouse id, completion time, and last failure. Admission, activity changes, final epoch validation, and generation publication use one lock-free, stamped state transition protocol. A new Legacy entry changes the stamp, so a concurrent publish compare-and-set must fail. The Legacy path performs no database access, file access, synchronous logging, monitor acquisition, sleep, or wait.

Exactly one request may atomically enter Legacy activity, change READY to BUSY, and lease the current generation. While v3 shadow is enabled, every inbound/outbound Legacy invocation enters the activity protocol, including requests whose observation is stale, busy, or rejected. The matching exit is guaranteed by a finally block. Requests arriving in every other state receive a no-op observation token while still contributing to activity invalidation. With shadow disabled the gateway remains an immediate no-op. This sacrifices sample volume during bursts in exchange for trustworthy samples and zero Legacy waiting.

The caller-side path has a one-millisecond p99 engineering budget and uses only fixed-size immutable metadata plus non-waiting atomic operations. Worker admission uses a bounded queue with immediate rejection; it never runs work on the caller. Tests enforce structurally that admission contains no blocking API and record timing under contention without treating JVM pauses as functional failures. The field export reports admission maximum and percentile timings.

Planning engine

Inbound and outbound planning receive an explicit snapshot generation and use that generation without calling refreshChangedLanes. The application service gains a stable-snapshot planning entry point; existing non-shadow behavior is left unchanged.

The SQL read model publishes a new generation only after a complete refresh succeeds. The refresh of topology, inventory, lane state, preserved inbound/outbound planning inputs, and metadata runs in one shadow-owned SQL transaction. The final metadata update is the publish marker and rolls back with every component on failure. Because the one worker serializes planning and refresh, no planner runs while those tables are being replaced; a planner also verifies the published marker before and after its reads. Published data is logically immutable until the lease completes. Planning is rejected when the requested generation, configured warehouse, or snapshot metadata does not match the published generation.

The refresh reads Legacy sources with READ UNCOMMITTED semantics so it does not hold shared locks that delay Legacy writers. Consistency is guarded by the application activity stamp for the supported single-JVM allocation entry points plus pre/post source row-count and checksum signatures over the planning fields. If the stamp or signature changes during refresh, the new generation is discarded and never becomes a valid sample source. This adds background reads but replaces v2's per-request refreshes, so total refresh frequency is bounded and expected to decrease. The design does not enable a database-wide isolation option.

Background worker

One daemon worker continues to own all shadow work. The executor has one worker and at most one pending control task; rejection is immediate and never uses the Legacy caller as a fallback executor. Only one refresh may be scheduled at a time. For an admitted observation it:

1. plans against the leased stable snapshot;
2. waits in the background for the Legacy completion token;
3. writes the comparison and complete diagnostics;
4. waits for a quiet Legacy window;
5. refreshes and validates the next snapshot generation;
6. publishes READY only when the refresh is stable.

The Legacy completion wait is bounded by FORTUCKY_SHADOW_TIMEOUT_SECONDS, which applies only on the worker. Timeout, interruption, planning failure, comparison failure, and diagnostic failure release the observation lease in finally, invalidate the consumed generation, and continue to asynchronous refresh/recovery. If Legacy itself remains active beyond the deadline, shadow remains unavailable until the activity count returns to zero; it does not accumulate request payloads or additional observation tasks.

Request data flow

Common admission flow

1. At the existing pre-Legacy integration point, atomically enter Legacy activity, advance the epoch, and attempt to lease the current snapshot.
2. If shadow is disabled, the snapshot is not READY, it is stale, the warehouse is wrong, or another observation owns it, return a no-op token immediately. Skip counts are updated in fixed-size in-memory counters and flushed by the worker; the caller performs no database or synchronous log write.
3. If admitted, copy only the immutable request identifiers already in memory and submit the worker task.
4. Execute the original Legacy procedure immediately, regardless of the shadow admission result.
5. In a finally block, complete the token with either the Legacy outcome or failure marker and exit Legacy activity. Completion never waits for the worker.

Inbound

The admitted worker resolves and plans the inbound request exclusively from the leased snapshot. It records the shadow-selected space, the Legacy-selected space, snapshot generation and time, warehouse, candidate counts, outcome, timings, and skip/error reason.

Outbound

The admitted worker resolves project, product, requested quantity, and stock candidates from the leased snapshot before recording the comparison. The diagnostic record contains an immutable copy of these inputs, so deletion or mutation of live outbound detail rows by Legacy cannot erase the evidence used by shadow planning.

Refresh sequencing and concurrency

Snapshot refresh is asynchronous and can never delay a Legacy request.

1. Startup begins in INITIALIZING; all requests remain Legacy-only until the first refresh is validated.
2. After an admitted Legacy operation completes, the coordinator remains BUSY. Requests during this period run Legacy normally and are logged as skipped.
3. The worker waits until the local active-Legacy count reaches zero and remains quiet for the configured quiet interval.
4. It captures the activity epoch, refreshes the complete SQL read model in one shadow-owned transaction, then atomically publishes only if the active count is still zero, the epoch is unchanged, and the stamped coordinator state still represents that refresh. The same protocol prevents a request that starts between validation and publication from being missed.
5. If activity occurred during refresh, that generation is not published. The worker retries only in the background after another quiet interval.
6. Refresh is single-flight. It retries at most three times with one-, two-, and four-second background backoff. If refresh fails or the retry budget is exhausted, state becomes UNAVAILABLE; the last completed generation is retained for diagnosis but cannot be leased. Recovery is rate-limited to one cycle per minute. Business requests continue through Legacy throughout.

This rule also covers skipped requests that execute while an earlier sample owns the snapshot: the next snapshot cannot become READY until those Legacy calls have completed and a quiet refresh has succeeded.

Readiness and staleness

A snapshot is usable only when all of the following are true:

• its warehouse matches the configured algorithm warehouse;
• its refresh completed successfully and is the currently published generation;
• its completion time is no older than the configured maximum age;
• no Legacy activity has invalidated the coordinator state;
• all required topology, inventory, lane-state, and outbound-input read-model components belong to the same generation.

The default maximum age is 60 seconds and accepts values from 5 through 300 seconds. An expired snapshot causes the current request to skip and raises one coalesced asynchronous refresh signal. The default quiet interval is one second and accepts values from 100 through 10,000 milliseconds. Invalid values fail shadow initialization closed while Legacy remains available. Configuration is read at startup only. These are operational controls and never enable ACTIVE behavior.

Warehouse normalization

FORTUCKY_ALGORITHM_WAREHOUSE_ID=01 is authoritative for this field test. A raw request value such as W02 is retained in diagnostics but cannot override the configured algorithm warehouse.

Purely numeric warehouse identifiers are compared after removing leading zeroes, so 1 and 01 are equivalent. Non-numeric identifiers are compared as exact trimmed values; W02 is not equal to 01.

Failure handling

Every shadow-only failure has one of these outcomes:

• SKIP_NOT_READY: startup or recovery has no published snapshot.
• SKIP_BUSY: another observation or refresh owns the coordinator.
• SKIP_STALE: the published snapshot exceeded its maximum age.
• SKIP_WAREHOUSE: request diagnostics do not match the configured warehouse; Legacy still runs normally.
• SKIP_CONCURRENT: Legacy activity changed during refresh or invalidated the sample window.
• ERROR_PLAN or ERROR_COMPARE: the current observation is invalid, its consumed generation cannot return to READY, the lease is released in finally, and asynchronous refresh begins.
• ERROR_REFRESH: the candidate generation is rolled back; after bounded retries the coordinator becomes UNAVAILABLE and retains the previous generation for diagnosis only.

Errors are caught at the shadow boundary and logged with request id, warehouse, snapshot generation, stage, exception class, and elapsed time. Logs must not contain credentials or full payloads. A planning or comparison error leads to refresh/recovery; it never retries the Legacy procedure.

SQL package changes

The shadow installer upgrades only package-owned alg.* objects. It adds snapshot-generation metadata and the read-model fields required to preserve inbound and outbound planning inputs. Refresh replaces the read model and publish marker in one transaction and publishes only after all components succeed. SQL remains compatible with SQL Server 2016 RTM.

The installer is idempotent for the recognized shadow package version and refuses to take ownership of unknown alg objects. The installer and uninstaller share one explicit allow-list of tables, views, procedures, functions, constraints, and indexes; they verify package id/version and object ownership before mutation. Installation is transactional and rolls back partial upgrades. The uninstaller removes only allow-listed shadow objects after manifest verification. Static SQL checks reject triggers and reject INSERT, UPDATE, DELETE, MERGE, or procedure calls targeting dbo.* or any non-alg schema. The Java shadow repository exposes only fixed, reviewed SQL statements for those allow-listed alg.* writes and required Legacy reads; it cannot accept a dynamic table, procedure, or SQL fragment.

Configuration

Existing v2 variables remain valid. v3 adds bounded snapshot controls with these defaults:

FORTUCKY_SHADOW_ENABLED=true
FORTUCKY_ALGORITHM_MODE=SHADOW
FORTUCKY_ALGORITHM_DATABASE_WRITES_ENABLED=true
FORTUCKY_ALGORITHM_WAREHOUSE_ID=01
FORTUCKY_SHADOW_TIMEOUT_SECONDS=3
FORTUCKY_SHADOW_SNAPSHOT_MAX_AGE_SECONDS=60
FORTUCKY_SHADOW_QUIET_INTERVAL_MILLIS=1000

FORTUCKY_SHADOW_TIMEOUT_SECONDS accepts 1 through 30 seconds and limits only background worker stages; it is never a caller wait timeout. FORTUCKY_ALGORITHM_DATABASE_WRITES_ENABLED=true authorizes the existing shadow repository only when mode is exactly SHADOW; the startup guard rejects every other mode, and the module exposes no active business writer entry point. SQL allow-list and integration tests prove that these writes target only package-owned alg.* objects. No formal-cutover or ACTIVE variable is introduced.

Verification strategy

Automated tests must prove:

1. disabled, busy, stale, unavailable, rejected, timed-out, and failing shadow paths still invoke Legacy exactly once and preserve its response;
2. no Legacy test thread waits for planning, comparison, refresh, or recovery, and the admission path meets its one-millisecond budget under contention;
3. planning never calls live refresh and accepts only its leased generation;
4. all admitted and skipped Legacy requests enter and exit activity in finally; concurrent starts, including a start between final validation and publication, prevent publication of an invalid refresh;
5. a refresh generation is published only when all components succeed, the Legacy epoch stays unchanged, and the pre/post source signatures match;
6. inbound/outbound input rows carry the same generation and outbound inputs remain in diagnostics after simulated deletion of live detail rows;
7. 1 equals 01, while W02 does not override configured warehouse 01;
8. SQL install/uninstall ownership checks remain symmetrical and contain no Legacy business-table DML;
9. the full Maven test suite, packaging, and existing Node script checks pass.

Field acceptance criteria

Shadow v3 is not sufficient by itself to authorize formal cutover. A new field run must meet all of these conditions:

• at least seven consecutive days with zero exception whose stack originates in the shadow package and escapes into a Legacy request, and zero difference in Legacy response codes or response fields compared with the approved Legacy baseline;
• inventory parity has zero missing/extra warehouse 01 rows and exact equality for space, product, project, quantity, lock/reservation, availability, lane, and occupancy fields included by the export;
• at least 100 valid inbound and 100 valid outbound comparisons, excluding explicitly skipped requests;
• every valid comparison has snapshot generation/time, immutable input diagnostics, candidate counts, and both outcomes;
• zero valid sample has a post-Legacy snapshot, unexplained stock-not-found, unexplained waiting-channel result, or cross-warehouse input;
• the export records zero Legacy request blocked by an alg.* session; compared with the approved Legacy-only baseline under the same shift/load class, Legacy p95 latency increases by no more than 5% and p99 by no more than 10%; caller-side shadow admission p99 remains at or below one millisecond;
• differing recommended locations are reviewed as algorithm choices rather than treated as automatic failures.

An unexplained result is any valid sample whose reason cannot be reproduced from its recorded snapshot generation, immutable inputs, candidate counts, and policy trace. If volume cannot reach the sample minimum, the observation period is extended. Formal routing is a separate change with a separate approval, design, deployment, and rollback decision.

Deployment and rollback boundary

The v3 source replaces the running v2 shadow source only during an agreed IDEA restart window. Before enabling shadow, the deployment script verifies exactly one FT546WMS JVM and no second recent shadow-instance heartbeat. The stable Legacy behavior inside the application remains authoritative after restart. Operators retain the current known-good source directory and IDEA Run Configuration.

Rollback is application-only: stop v3 and restart the known-good v2 or stable-main configuration. If immediate isolation is preferred, restart v3 with FORTUCKY_SHADOW_ENABLED=false. Because v3 never writes Legacy business tables, no business-data rollback is required. Shadow SQL objects may remain installed for diagnosis and should be removed only with the verified v3 uninstaller after exporting evidence.