Lessons from My First Distributed System
The small surprises that appear when a program stops living in one process.
The first surprise was how many assumptions came from single-process programming. Function calls either returned or threw. State changed in one place. Time felt more like a sequence than a negotiation.
Once the system crossed process boundaries, every interaction needed a failure model. Requests could be duplicated, delayed, reordered, or lost. A node could be alive but useless. A timeout could mean failure, overload, or simply a slow network.
The biggest lesson was to design for uncertainty directly. Idempotency, explicit ownership, health checks, backpressure, and boring operational visibility matter because the system will eventually enter a state your local tests did not imagine.