The business problem
An approximately four-hour end-of-day batch kept dependent operational work waiting until late at night. Further analytical workloads could begin only after those processes finished, leaving a compressed window before the next day’s client delivery deadlines.
The batch consumed time that every dependent stage needed. The modernization needed to make upstream data ready earlier while preserving the correctness of results as inputs changed.
Sorrell Consulting was engaged to modernize the batch-processing workload. Joshua Sorrell designed the incremental solution and led its implementation, including the migration to containerized services in Go.
Before
- Upstream updates gathered for processing
- Approximately four-hour end-of-day batch
- Dependent operational work, then further analytics
After
- Incremental preparation and correction as inputs change
- Required upstream processing typically complete within minutes of the daily cutoff
- Dependent workflows proceed without the late-night batch wait
Preserving what the ordered batch did well
The scheduled workflow gathered updates and processed interacting information in an order. That ordering was useful: inputs could conflict, supersede one another, or change the interpretation of the same derived state.
If one input changes the meaning of another, applying each independently and keeping both contributions can produce the wrong answer. The batch supplied a practical way to manage those relationships within a scheduled cycle.
Removing the overnight dependency required making those responsibilities explicit in the incremental design: determine which information applied, identify results affected by a change, and revise those results safely.
Advancing work as information changed
Incoming notifications prompted the processor to evaluate relevant upstream changes. Deterministic revision and authority rules selected the applicable information; arrival order did not decide which revision was current. A delayed older message was not equivalent to a genuine correction.
Message versioning distinguished revised and superseded inputs. Relationships between inputs and computed results provided dependency tracking, locating earlier work that might need to change.
These mechanisms answered two questions: which information applies now, and which results relied on an earlier interpretation? Repairing those results required an additional step.
Revising a prior result
Consider a synthetic example: input A contributes to result R. A later revision changes how A should be understood. Treating that revision as another independent addition can leave the old contribution in R. Merely marking the first version of A as superseded can leave R unchanged.
The processor used the input-to-result relationships to locate where the earlier interpretation first affected derived state. It then re-evaluated the affected work from an unaffected starting point.
That re-evaluation could alter several intermediate conclusions. Comparing revised results with previous results identified what needed to reach downstream consumers, without sending every prior output again. The relevant scope was the work that depended on the interpretation that had changed.
Keeping corrections trustworthy in operation
A correction could arrive while computation based on an older interpretation was still running. Early cancellation saved effort, but a guarded commit supplied the correctness check: it rejected a result when the state it relied on had changed since computation began. If the earlier computation committed first, the correction still had to proceed against the resulting state.
Publishing the correction was a separate responsibility. The state change and the intention to deliver its update were recorded together, preserving the update for continued or retried delivery after an interruption.
Recoverable delivery allowed downstream views to lag committed state, and consumers still needed to handle repeated notifications. The design made interrupted publication recoverable without requiring every participant to change state in a single operation.
Validation before changing production
Regression testing in the user acceptance testing (UAT) environment covered several years of historical data, comparing the new system’s outputs against the previously computed results.
The new event-driven system then ran in parallel with production for one month. Each day, its outputs were compared with the results of the existing nightly batch. Those daily comparisons showed no differences. The production process was switched to the new system after that validation period.
This provided both historical regression coverage and a comparison against the batch operating on current production inputs before the new system took over.
Earlier readiness for dependent workflows
After the change, the upstream processing required by dependent workloads typically completed within minutes of the daily cutoff. Preparing and revising state as relevant changes arrived made required downstream data available when scheduled end-of-day workflows began.
The approximately four-hour batch dependency was removed from the overnight schedule. Dependent operational work and subsequent analytics no longer had to wait behind that late-night run.
The modernization lesson
The business value depended on changing when downstream work could proceed while preserving the correctness responsibilities previously handled by the batch. Explicit revision handling, scoped re-evaluation, guarded commits, and recoverable publication made that change possible.
For other deadline-sensitive financial, logistics, or readiness workflows, the corresponding design question is:
When new information changes an earlier answer, how will the system find and correct the work that depended on it without making every dependent workflow wait for another batch cycle?