Werner Heisenberg (1901 – 1976) defined the uncertainty principle, which states that the more you know about the location of a particle, the less you know about its movement. Basically, you can’t know everything about anything.
In a distributed system you have a gaggle of servers, each of which lives in various states of health, death, or garbage collection. The vast majority of the time you can chat with a server and get a crisp and timely result. Other times you don’t get a prompt answer and it’s tough to know if you should abandon the slacker or wait patiently. Furthermore, you don’t know if the server got the request, did the work, and just hasn’t answered. Any time a request goes to a single system, you don’t know when the request will be delayed.2,6
In some distributed systems, it is essential to have an extremely consistent and fast response time for online users. To accomplish this, multiple requests must be issued, and the completion of a subset of the requests is accepted as happiness.
In a distributed system, you can know where the work is done or you can know when the work is done but you can’t know both.
To know when a request is done within a statistical SLA (service-level agreement), you need to accept that you don’t know where the work will be done. Retries of the request are the only option to get a timely answer often enough. Hence, the requests had better be idempotent.