Real-time systems are built around uncertainty

[ 01 ]

A message is not a state

A dashboard may show a valve as open, but that is only a claim based on the most recent information it received. The actual device may be offline, a command may still be in transit, or the reported state may be older than the operator expects. Real-time interfaces have to expose that uncertainty instead of quietly replacing it with a reassuring icon.

In my work on connected monitoring and control across more than 100 active edge devices, the interface and the communication path have to be considered together. Telemetry, requested actions and confirmed device state are different things. Keeping them separate makes it possible to show pending, confirmed, failed and stale conditions honestly.

[ 02 ]

Separate telemetry from control

A common topology has field devices publishing telemetry to a broker, a backend validating and persisting relevant events, and browser or mobile clients receiving authorized updates through WebSocket connections. MQTT is useful on constrained or intermittent links because it has lightweight publish/subscribe semantics. WebSockets give an application a bidirectional channel; they do not themselves define delivery guarantees or authorization.

Topic structure should reflect tenancy and device identity without relying on an opaque topic name as the only access control. Broker credentials and ACLs restrict publication and subscription, while the application checks that a user may view or operate the specific device. Avoid exposing a wildcard subscription that crosses organizations.

Persist the current state needed for a fast initial screen separately from the history needed for investigation. Subscribe, then reconcile with a snapshot or a versioned stream so the client does not miss an update during initialization. A timestamp from a device with an untrusted clock is not sufficient as the only ordering mechanism.

[ 03 ]

Design commands as a lifecycle

Remote control should have a command identifier, target, requested operation and expiry. The server verifies permission, records the request and publishes it. The device acknowledges the command and reports its resulting state. The user interface displays those phases separately; a broker acknowledgment is not proof that a physical actuator moved.

A timeout means the result is unknown, not necessarily that the operation failed. Before retrying, reconcile the device state and consider whether the operation is safe to repeat. Prefer idempotent desired-state commands such as “set closed” to toggle commands, where a duplicate can reverse the intended action. For safety-critical operations, local interlocks and an independent risk assessment remain necessary.

type CommandStatus = 'pending' | 'acknowledged' | 'confirmed' | 'expired';

type Command = {
  id: string;
  deviceId: string;
  desiredState: 'open' | 'closed';
  expiresAt: string;
  status: CommandStatus;
};

[ 04 ]

Reconnect without inventing continuity

Connections disappear during mobile network changes, broker restarts and browser sleep. Use bounded exponential backoff with jitter to avoid synchronized reconnect storms. On reconnection, authenticate again, restore only permitted subscriptions and refresh the latest snapshot. Stop retrying when credentials are invalid until they are renewed.

MQTT quality-of-service levels describe protocol delivery behavior, not end-to-end business completion. Even when a message reaches the broker, the device or backend may not have processed it. Retained telemetry can help a new subscriber find the latest value, but it must include freshness information; retained commands require particular care because a delayed command may no longer be safe.

[ 05 ]

Control the cost of updates

Rendering every measurement individually is rarely necessary. Batch UI updates, aggregate chart data at a resolution appropriate to the viewport, and cap retained client-side history. On the backend, partition hot paths by device or organization only after measuring the load and preserving the ordering guarantees required by that stream.

A slow consumer needs a defined policy: coalesce replaceable telemetry, apply backpressure or disconnect and resynchronize. Do not silently drop control acknowledgments along with disposable chart points. Track connection state, message age, queue depth, retries and command latency as separate signals.

Test intermittent connectivity, duplicate messages, delayed acknowledgments and restart recovery. Use a simulator and fault injection before involving physical devices. Protect device credentials, rotate them where feasible, and ensure logs do not disclose secrets or unrestricted location data.

An honest “last reported five minutes ago” is safer than an apparently live state that nobody can verify.

[ 06 ]

A device identity has a lifecycle

Provisioning is where a device becomes part of an organization, not merely where it receives a password. Associate a durable device identity with an installation, define who can claim or reassign it, and keep the authorization record separate from a mutable display name. Provisioning mistakes are particularly expensive when hardware is already installed in the field.

Credentials should be unique to a device or a deliberately managed identity, transported over an authenticated encrypted connection, and rotated or revoked when the device is retired or compromised. A shared credential is operationally convenient at first but makes selective revocation and attribution difficult. Device replacement also requires a policy for historical telemetry: transferring the installation does not necessarily mean transferring the old device identity.

Configuration changes need versions and acknowledgments. If a device reconnects with an older configuration, the system should know whether to deliver a newer desired version, wait for a compatible firmware release or request intervention. Do not equate configuration publication with successful application on the hardware.

[ 07 ]

Ordering has a scope and a cost

Events can arrive late or more than once. A practical telemetry envelope can include a device identifier, event identifier, device sequence or boot identifier, device-observed time and server-received time. A sequence number helps identify gaps within the scope in which it is generated; a reboot or counter reset needs an explicit rule rather than an assumption that all sequence values are globally increasing.

Use server receipt time to reason about transport delay, while retaining device time as an observation whose accuracy may vary. A late measurement may belong in historical charts without overwriting a newer current-state projection. Define that comparison in one place and test clock drift, reconnect bursts and out-of-order arrival.

Retention is another architectural decision. High-resolution readings may be useful for diagnosing a recent fault but costly to retain indefinitely. Downsample for longer periods only when the aggregation preserves the statistics that matter, such as extrema as well as averages. Document what was discarded and keep incident evidence according to the actual operating policy.

[ 08 ]

Design the operator’s recovery path

An operator needs to understand whether a device is unreachable, reporting old data or refusing a command. Those states should not collapse into the same red badge. Show the last confirmed observation, the time it was received and the command history; provide a clear path to retry or escalate only when the operation is safe to repeat.

Alerting deserves the same care. A single missed reading can reflect a weak network rather than a failure, while averaging can conceal a short but important excursion. Use thresholds and persistence windows appropriate to the equipment, and include recovery conditions so an alert does not oscillate indefinitely. Human acknowledgement is distinct from the underlying condition becoming healthy.

Exercise the workflow with the network deliberately disconnected, the broker restarted and messages replayed. A useful test checks not only that the connection eventually returns, but that the operator sees an honest intermediate state and that pending commands are not executed after their expiry.

[ 09 ]

Scale the path that is actually constrained

More devices can stress very different parts of the system: broker connections, write throughput, retained history, browser rendering or support operations. Establish the arrival rate, message size and required retention before deciding whether to shard or introduce a stream-processing platform. A fleet count alone does not describe the workload.

Separate a replaceable latest-value update from a business event that must be retained. This distinction supports coalescing for dashboards while protecting command acknowledgments and audit records. Set queue limits and monitor oldest-message age; a growing queue is evidence that the system is falling behind even when no requests have failed yet.

Recovery capacity matters too. After an outage, thousands of devices may reconnect and publish buffered data at once. Rate limits, jitter and bounded processing queues protect the service, but the recovery plan must also say what happens when the backlog exceeds available storage or the device can no longer retain its own events.

Practice this recovery under load rather than waiting for a real outage. Observe how quickly the oldest queued event is processed and whether interactive control remains responsive while history is catching up. Separate capacity for those paths may be warranted when bulk telemetry would otherwise delay time-sensitive operator feedback.

[ 10 ]

Make uncertainty visible

The purpose of a live interface is not to maximize animation. It is to help an operator decide what is known, what was requested and what still needs confirmation. Clear state transitions and recovery paths are a more useful measure of quality than how quickly a number changes.

The architecture can grow from a small broker and service to more specialized processing, but the fundamentals remain: scoped identity, explicit acknowledgments, bounded retries and a record that explains what happened.