Distributed systems
The key problem in distributed systems is getting the system to converge toward consistency as it runs. There are three core factors: the expected calibration point you set, observation, and error. What we want is to shrink the error between what actually gets carried out and the calibration point, so each measurement gives you the direction to correct in, and you keep iterating and converging. That’s the “coordination control loop”. But there’s obviously some craft to this. Where should the observation anchor go, for instance? What should the controller be set to?
Think about the distributed problem from the angle of complex systems, because its problem is plain enough: coordination and crossover, essentially the same as the relationships between human departments, misalignment, timing that doesn’t line up. Our society, or any subsystem inside a complex system, only has local information, just like an agent in ABM. It can only lean on the others around it as signal reference, and order emerges from the whole in the end. We’re not talking about emergence, we’re talking about a controllable non-complex system (at the current scale, at least), so its situation is a lot simpler, but the local information problem is still there. You solve local information either with a common standard, ISO or a broadcast like periodic time calibration, or by having some coordination signal propagate through the system and bring it closer to convergence as it runs.
The first one is really the classic contract alignment: everyone acts by one common standard and so reaches coordinated consistency under asynchrony, and the core of it is “agreement”. For the second, the key isn’t only propagation but also “merging”. The direction of convergence has to be deterministic, and every subsystem, on receiving that signal, should converge in the same direction, otherwise the oscillation goes on and on. Which shows it still needs “agreement”, since the direction of convergence is obviously something agreed on. So the two don’t contradict each other. Inside an engineered distributed subsystem, how each component converges is something I defined from the start, and our purpose is to make the subsystems rush toward that “fixed point” once they get a particular signal, converging as close as possible to a designated global state. My picture here is cells: a hormone signal arrives and triggers a particular behavior. Each cell only attends to itself and doesn’t consider the outside, while still being regulated by signals, and that signal doesn’t have to be sent from a center, it can come from other cells, forming a feedback loop that relies only on itself.
In engineering this is called the Actor model / reactive systems, that is, treating each subsystem as a cell that actively senses external state at all times and regulates itself. A cell’s broadcast outward is pure sending, with no concern for whether the other side received it, and the actual handling depends on its own control, which is fundamentally what being external-independent looks like. So message passing under this model is a graded concentration (level-triggered, continually resampling the outside, level trigger), firing on its own once it takes the information in, with its own gatekeeping, say, flatly refusing a given external operation. But reactive systems are steady at the macro level and don’t suit “hard operations” that have to hold up in one shot. What they rely on is the same “self-healing” as cells, and the redundant stability that “irons out” error oscillations through feedback. For a “hard operation” you can keep the concentration-sensing resample, but you also need guarantees hard enough that the behavior holds, atomicity guards on the distributed operation and a checking fallback, rather than leaning only on redundant “self-healing”.
A lot of distributed problems of this kind come down to “local information” in the end. Put bluntly, the magistrate counts for less than the clerk on the spot. The top and the bottom fall out of step, and one side’s information doesn’t match the other’s, which produces the misalignment. Borrowing the fix from real-world bureaucracy, one effective move against this thing is “authorization”. Who has the authority to act, that decides the range of what the clerk on the spot can do. Permission control is also a line that syncs the magistrate with the clerk, making sure everything the clerk does is within the magistrate’s knowledge, so that even in the worst case the boundary of the clerk’s behavior is explicit. In reality it’s hard to actually keep the clerk in check this way, but a program isn’t a person. A one-way lock is a one-way lock; if you’ve set it so only one behavior has permission, then permission exists only there.
Translation note. I wrote this in Chinese. This English version is an LLM translation, so the wording is not mine even though the thinking is. Original: 分布式系统.