Distributed and Networked Asynchronous Systems

Inside one computer, a function call either returns or crashes the program.

Learning Objectives

Contrast local and distributed concurrency: latency, partial failure, async network I/O, request lifecycles, connection pools and concurrent request handling.

Apply timeouts and deadlines, retries and idempotency; handle duplicate delivery and delivery semantics (at-most-once, at-least-once, "effectively-once").

Use message queues and brokers, consumer groups, ordering guarantees, backpressure, flow control and rate limiting; circuit breakers and bulkheads; manage fan-out and tail latency; propagate cancellation across services.

Reason about distributed locks, leases and fencing tokens, leader election, optimistic concurrency, compare-and-swap at the service/database level, transactions and isolation levels.

Design event-driven architectures, actor systems with remote messaging and idempotent consumers; degrade gracefully; observe across asynchronous boundaries with correlation IDs; avoid unbounded task creation; test resilience; understand consistency and concurrency control.

Plain-Language Explanation

Inside one computer, a function call either returns or crashes the program. Across a network, a request can vanish, arrive twice, arrive late, or succeed on the server while the reply is lost — and you cannot tell which. Distributed concurrency is concurrency where messages are slow and unreliable and any participant can fail independently. Every pattern in this module exists to make that uncertainty manageable.

Precise Technical Explanation

Local concurrency versus distributed concurrency shared memory, nanoseconds messages, micro- to milliseconds whole process (usually) partial: some nodes, some links no global clock; messages reorder consensus, transactions, conditional writes Network latency and partial failure

A timeout tells you nothing about whether the remote operation happened. Nodes pause (GC, VM migration), networks partition, clocks drift.

Async network I/O Event loops and async clients (Module 11) let one process hold thousands of outstanding requests.

Request/response lifecycles Accept → parse → authenticate → handle (often fanning out) → respond → clean up; each stage needs timeouts and cancellation on client disconnect.

Reuse TCP/TLS connections to databases and services; pool size bounds concurrency to the downstream (a bulkhead); waiting for a connection needs a timeout (demonstrated in Module 21).

Concurrent request handling Thread-per-request, event-driven, or virtual threads (Modules 11, 15, 21).

Timeouts and deadlines Module 14; propagate the remaining budget in headers (grpc-timeout).

Retries and idempotency Module 14; retries + timeouts imply duplicates, so handlers must be idempotent.

Retries, broker redelivery after a consumer crash, and network replays all produce duplicates.

At-most-once, at-least-once and effectively-once processing terminology

At-most-once: never retried; messages may be lost.

At-least-once: retried until acknowledged; duplicates possible — the common default.

"Exactly-once"/effectively-once: at-least-once delivery + idempotent or transactional processing (deduplication by message ID, atomic "process and record offset") so the *effect* happens once. True exactly-once *delivery* is impossible in general.

Message queues and brokers Kafka (partitioned logs), RabbitMQ (queues/exchanges), SQS, NATS, Pulsar: decouple producers and consumers, buffer bursts, redeliver on failure.

Consumers in a group share partitions/queues so each message is processed by one member; rebalancing on membership change can cause duplicates.

Usually per partition/key only (Kafka), or none (SQS standard queues); retries can reorder. Design consumers to tolerate reordering or route related messages to one partition.

Backpressure and flow control TCP windows, HTTP/2 flow control, gRPC streaming with demand, broker consumer prefetch limits, reactive streams (Module 26).

Per client/tenant at the edge (token buckets in API gateways); distributed limits need shared counters (Redis) or per-node shares.

Module 14, per downstream dependency.

Module 14: separate pools per dependency.

Fan-out requests and tail latency If one request fans out to N backends and the request waits for all, the probability that at least one is slow grows quickly: with a 1% chance of a slow response per backend, 100 backends make 63% of requests slow (computed below). Mitigations: hedging, partial results, tighter per-backend timeouts.

Cancellation propagation across service calls When the client disconnects or a deadline expires, cancel downstream calls (gRPC propagates cancellation; HTTP needs explicit handling) to stop wasted work.

Distributed locks: use cases and risks Locks across machines (ZooKeeper, etcd, Redis-based) can be lost without the holder noticing (GC pause, partition): two nodes may both believe they hold the lock. Use them for efficiency (avoid duplicate work), not for correctness — unless combined with fencing.

Leases and fencing tokens A lease is a lock with an expiry; the holder must renew it. A fencing token is a monotonically increasing number issued with each lease; the protected resource rejects requests carrying an older token than it has seen, so a stale holder cannot corrupt data (simulated below).

Leader election overview Choose one node to coordinate (Raft/Paxos-based systems, ZooKeeper/etcd elections); leaders hold leases; fencing protects against old leaders.

Concurrent updates and optimistic concurrency Read a version; update only if the version is unchanged (UPDATE … WHERE id = ? AND version = ?, HTTP If-Match with ETags); retry or report a conflict on failure (demonstrated with SQLite below).

Compare-and-swap at service/database level Conditional writes: DynamoDB condition expressions, etcd transactions, S3 conditional puts, Redis WATCH/MULTI — the distributed analogue of CAS.

Transactions and isolation levels overview Read uncommitted, read committed, repeatable read, snapshot isolation, serializable; weaker levels allow anomalies (lost updates, write skew, phantoms). Know your database's default (PostgreSQL: read committed; MySQL InnoDB: repeatable read).

Services publish events; others react asynchronously — loose coupling and scalability at the cost of eventual consistency and harder debugging (outbox pattern for atomic "update + publish").

Actor systems and remote messaging Akka Cluster, Orleans, Erlang distribution: location-transparent actors with at-most-once messaging by default; supervision handles failures.

Record processed message IDs (or use naturally idempotent operations) inside the same transaction as the effect (demonstrated below).

Module 14: serve partial or cached results when dependencies fail.

Observability across asynchronous boundaries Distributed tracing (OpenTelemetry) propagates trace context through HTTP headers, message metadata and async context.

Tracing correlation IDs A request ID generated at the edge and attached to every log line and downstream call ties together work done on many threads, tasks and machines (demonstrated with contextvars below).

Avoiding unbounded task/request creation Bound concurrency per request (semaphores around fan-out), per service (pools) and per client (rate limits).

Fault injection (latency, errors, partitions), chaos experiments, Jepsen-style tests for consistency claims.

Consistency and concurrency control in distributed systems Linearizable stores (via consensus), causal consistency, eventual consistency (with CRDTs, Module 26); choose per operation, and make concurrency control explicit (versions, conditional writes, transactions).

Why It Matters in Real Systems

Every cloud application is distributed: double charges, lost messages, split-brain leaders and fan-out latency are the everyday failures these patterns prevent.

Key Terms

Some components fail while others continue Retried delivery with possible duplicates

Processing a duplicate has no additional effect Time-limited lock that must be renewed

Monotonic token that lets resources reject stale holders Conditional update on an unchanged version High-percentile response time

Identifier carried through all work for one request

Diagram or Execution Timeline

flowchart LR
  C[client: request id r-17] --> G[gateway: deadline 2s]
  G -->|remaining 1.9s| A[service A]
  G -->|remaining 1.9s| B[service B]
  A -->|event + id| Q[(broker)]
  Q -->|at-least-once| W[idempotent consumer]
  W --> D[(database: dedupe table + effect in one transaction)]

Syntax and API Overview

This is a concurrency concept, not a language construct. The building blocks are HTTP/gRPC clients with deadlines, broker client libraries, conditional database writes and tracing libraries.

Complete Code Example

Optimistic concurrency, an idempotent consumer, fencing tokens, fan-out tail latency and correlation IDs (CPython 3.13.1 with the standard-library sqlite3; deterministic):

import asyncio
import contextvars
import random
import sqlite3

db = sqlite3.connect(":memory:", isolation_level=None)
db.executescript("""
CREATE TABLE stock(item TEXT PRIMARY KEY, qty INT, version INT);
INSERT INTO stock VALUES ('widget', 10, 1);
CREATE TABLE processed(msg_id TEXT PRIMARY KEY);
CREATE TABLE ledger(account TEXT, amount INT);
""")

# 1. optimistic concurrency: compare-and-swap on a version column
def reserve(read_version, read_qty, n):
    cur = db.execute("UPDATE stock SET qty = ?, version = version + 1 WHERE item = 'widget' AND version = ?",
                     (read_qty - n, read_version))
    return cur.rowcount == 1                             # 0 rows: someone else updated first

qty, ver = db.execute("SELECT qty, version FROM stock").fetchone()
client_a = reserve(ver, qty, 3)                          # both clients read version 1 ...
client_b = reserve(ver, qty, 4)                          # ... the second CAS must fail
print(f"optimistic concurrency: client A {'committed' if client_a else 'conflict'}, client B {'committed' if client_b else 'conflict -> re-read and retry'}; stock now {db.execute('SELECT qty, version FROM stock').fetchone()}")

# 2. idempote
…

optimistic concurrency: client A committed, client B conflict -> re-read and retry; stock now (7, 2)
idempotent consumer: ['applied', 'applied', 'duplicate ignored', 'duplicate ignored'] | ledger total: 70
fencing: accepted token 34 | rejected write from stale token 33 | stored: from node 2
fan-out: P(at least one of N backends slow) with 1% slow per backend
  N=  1: model  1.0%, simulated  1.0%
  N= 10: model  9.6%, simulated  9.6%
  N=100: model 63.4%, simulated 63.4%
correlated logs (interleaved requests, each line tagged):
  [r-17] request received
  [r-17] users responded
  [r-17] orders responded
  [r-17] response sent
  [r-18] request received
  [r-18] users responded
  [r-18] orders responded
  [r-18] response sent

Line-by-Line Explanation

Optimistic concurrency: both clients read version 1; the first UPDATE … WHERE version = 1 matches and bumps the version; the second matches 0 rows — a detected conflict, resolved by re-reading and retrying.

Idempotent consumer: the primary key on processed makes the dedupe insert fail for a redelivered message, and because it shares a transaction with the ledger insert, the effect is applied exactly once.

Fencing: the storage remembers the highest token; node 1's late write with token 33 is rejected after node 2 wrote with 34 — even though node 1 still *believes* it holds the lease.

Fan-out: the closed form 1 − 0.99^N and a seeded simulation agree: at N = 100 most requests hit a slow backend.

Correlation IDs: contextvars give each request handler (and its child tasks) its own request_id, so interleaved log lines remain attributable.

Output

Shown above; deterministic (seeded simulation, sorted logs).

Interleaving or State Trace

Fencing timeline: t1 node 1 lease (token 33) → t2 node 1 pauses → t3 lease expires → t4 node 2 lease (34) and writes → t5 node 1 resumes and writes with 33 → storage rejects.

Correctness and Safety Analysis

Correctness comes from making each state change conditional and recorded atomically: version checks for concurrent updates, dedupe records in the same transaction as effects, fencing tokens at the resource. Locks and timeouts alone cannot provide safety under partial failure.

Common Mistakes and Misconceptions

"Exactly-once delivery" promises without idempotent processing.

Distributed locks without fencing protecting correctness-critical data.

Retries without idempotency keys.

Unbounded fan-out and no per-request deadlines.

Recording the dedupe ID in a different transaction than the effect.

Failure Modes and Debugging Steps

Duplicates → check idempotency keys and transaction boundaries; split-brain → leases without fencing; latency spikes → fan-out tail and missing hedging/timeouts; lost messages → acknowledgements before processing (at-most-once by accident). Use distributed traces with correlation IDs to reconstruct the path.

Performance and Scalability Implications

Batching and pipelining amortise round-trips; connection pools bound concurrency; partitioning (by key) scales brokers and databases while preserving per-key order; tail latency dominates user experience in fan-out architectures.

Security Considerations

Idempotency keys and request IDs must not be guessable if they authorise anything; replayed messages are an attack vector (sign and timestamp them); rate limits and quotas protect shared services; trace context must not carry secrets.

Platform and Runtime Caveats

SQLite (used here) has database-level locking; server databases provide row-level locking and different default isolation levels. Broker guarantees vary by configuration (Kafka idempotent producers and transactions, SQS FIFO deduplication windows).

Real-World Applications

Payment processing (idempotency keys), inventory reservation (optimistic locking), Kafka consumers (offset + effect in one transaction), Kubernetes leader election (leases), search fan-out (hedging, partial results).

Best Practices

Assume duplicates, reordering and partial failure.

Make every handler idempotent; store dedupe IDs atomically with effects.

Use conditional writes / versions for concurrent updates.

Fence anything protected by leases.

Propagate deadlines, cancellation and correlation IDs end to end.

Exercises

E1 · Beginner. Why can't a client distinguish "the server never got my request" from "the server processed it but the reply was lost" after a timeout?

Answer: In both cases no response arrives; the network gives no further information. Hence idempotent retries.

E2 · Intermediate. Design deduplication for a consumer that charges cards from a Kafka topic.

Answer: Include a unique payment ID in each message; in one database transaction insert the ID into a processed table (unique constraint) and record the charge (or call the payment provider with the ID as its idempotency key); on a duplicate, skip. Commit the Kafka offset after the transaction.

E3 · Advanced challenge. Why is a Redis lock with a TTL insufficient to protect a file in object storage, and how do fencing tokens fix it?

Answer: A holder can pause beyond the TTL, the lock is granted to another node, and the first holder resumes and writes, corrupting data. With fencing, every write carries the lock's monotonically increasing token and the storage rejects writes with a lower token than it has seen — which requires the storage (or a proxy) to check tokens.

Quiz

? What does "effectively-once" processing require?
+ At-least-once delivery plus idempotent or transactional processing. :: Duplicates are delivered but have no extra effect.
- A network that never duplicates messages. :: No such network exists.
- At-most-once delivery. :: That loses messages.
- A distributed lock around every message. :: Locks can be lost under partitions.

? With a 1% chance of slowness per backend, roughly what fraction of requests that wait for 100 backends will be slow?
+ About 63% :: 1 - 0.99^100.
- About 1% :: That is per backend.
- 100% :: Not certain.
- About 10% :: Too low.

? Which make concurrent updates safe in a distributed system? (select all that apply)
+ Conditional writes on a version number :: Optimistic concurrency.
+ Fencing tokens checked by the resource :: Reject stale lock holders.
- A plain read-then-write from two services :: Lost updates.
+ Unique constraints for deduplication within the same transaction as the effect :: Idempotent consumers.

Summary

Distributed concurrency adds latency, partial failure and duplicates. Deadlines, idempotency, conditional writes, deduplication in the same transaction as effects, leases with fencing, bounded fan-out with hedging, graceful degradation and end-to-end correlation make systems correct and observable despite them.