In short: A “live” system that quietly falls behind is worse than no system, because people trust it. Separating how often data is read from how often it’s analysed keeps it genuinely live.
It’s common for a monitoring system to start life reading historical CSV exports. It’s easy to prototype against, easy to debug, and doesn’t require touching production infrastructure. The real test comes when that prototype needs to become a live service, reading data directly off a PLC as it’s produced. That transition breaks more assumptions than it seems like it should.
Why “just poll faster” doesn’t work
The naive path from batch to real-time looks like this: take the loop that used to read a whole CSV file, and instead have it read one new value on each iteration, on a timer. In practice this runs into two problems immediately:
- Read cadence and compute cadence are different, and coupling them is wasteful. You often want to read a fast-changing value every second, but only need to recompute a statistical model every thirty seconds or a minute. Forcing both onto the same loop means either reading too slowly (missing data) or recomputing too often (wasting CPU on a result that hasn’t meaningfully changed).
- A single loop can’t tolerate slow steps. If the scoring step takes longer than the read interval, which becomes more likely as models get more sophisticated, the whole pipeline falls behind, and the “live” system silently isn’t live anymore.
The two-loop pattern
The fix is to decouple reading from computing into two concurrent loops, connected by an in-memory buffer:
async def _poll_loop(client, buffer, nodes, interval):
while True:
values = await client.read_values(nodes)
buffer.append(values)
await asyncio.sleep(interval)
async def _score_loop(buffer, interval):
while True:
snapshot = list(buffer) # copy, don't mutate live buffer
if len(snapshot) >= MIN_REQUIRED:
result = run_scoring(snapshot)
await write_result(result)
await asyncio.sleep(interval)- The poll loop runs fast and does the minimum possible work: read values off the PLC and push them into a buffer.
- The score loop runs on its own, slower cadence, takes a snapshot of the buffer, and runs the (potentially expensive) statistical computation.
Because they’re separate asyncio tasks running concurrently, a slow scoring pass doesn’t block reads from continuing to come in, and a burst of new data doesn’t force scoring to run more often than intended.
Buffer design matters
The buffer connecting the two loops is a bounded structure: a fixed-size deque, not an ever-growing list. This is a small detail with an outsized effect on a long-running service. An unbounded buffer in a process meant to run for weeks or months is a slow memory leak waiting to happen. A bounded deque naturally discards the oldest data as new data arrives, which also happens to match the statistical need, since most of these methods only care about a recent window, not the entire history since the service started.
Practical OPC UA notes
A few details that matter once you’re reading real tags instead of CSV columns:
- Node addressing conventions should be consistent and derivable. A predictable, zero-based indexing pattern across tags makes it possible to generate node lists programmatically instead of hardcoding each one.
- Tag types matter for both reads and writes. Count-based results (like out-of-spec counts) are naturally integers; most computed statistical outputs should write as floats; alarm and trigger flags should write as booleans. Mismatching these causes silent truncation or type errors in the PLC that are easy to miss until they cause a real problem downstream.
- Buffer historian writes. Tuning their scheduled cadence protects the SQL backend from being overwhelmed while still keeping trend data intact through brief network interruptions.
Making it visible
A timestamp-freshness indicator next to each live value, showing not just the value but how old it is, makes stale data visually distinct from current data instead of indistinguishable from it. A frozen tag and a genuinely flat process reading look identical on a chart; only a visible “last updated” timestamp, or a staleness flag comparing the tag’s timestamp to now, tells them apart.
The broader lesson
Real-time doesn’t mean “everything runs as fast as possible”. It means decoupling read cadence from compute cadence, so each can run at the rate that actually makes sense for it. That distinction is easy to state and easy to skip under deadline pressure, but it’s the difference between a service that degrades gracefully under load and one that quietly stops being live the moment a computation takes a little longer than expected.
This is Part 4 of a 7-part series on building a real-time process monitoring system. Part 5 covers the statistics problem that matters most once the data really is live: autocorrelation.