> MTBF is a factor of the number of parts in your system.
> A single machine will have substantially less risk
> of breaking down than a complex setup
A. The probability that a single component in your system fails increases as the number of components increases.
B. The probability of the entire system failing decreases as the number of components increases.
Where (B) goes wrong is if the system is designed in such a way that components are dependent on each other.
Imagine you have a system containing 4 parts, all of which have to have at least one operating component for the system to remain operational. The components are:
WS = Web Server
DB = Database Server
AS = Application Server (executing long-running tasks)
LB = Load Balancer
Each component has a different probability of failure on a given day, given here:
WS, AS = 0.001
DB = 0.002
LB = 0.000001
If you do this:
10 x WS = (0.001)^10
10 x AS = (0.001)^10
1 x DB = (0.002)^1
10 x LB = (0.000001)^2
Then the probability of failure is 0.002, because if the database fails then the system fails. To increase redundancy you need to increase the number of DB servers too. If you have two DB servers, then the probability is 0.000004, 500 times lower.
I believe that you really did experience a problem with your setup: and I'll hazard a guess that the root cause is nothing to do with the architecture of your system but everything to do with the exponential increase in SNAFUs cause by the extra complexity.
Hardware failures are rare, people failures are common.
> I'll hazard a guess that the root cause is nothing to do with the architecture of your system but everything to do with the exponential increase in SNAFUs cause by the extra complexity.
Almost :) It has more to do with the fact that testing such a setup under realistic conditions modelling all the potential failure modes is no substitute for the variety of ways in which a distributed system can fail. Network cards that still send but don't receive? Check (heartbeat thinks you're doing a-ok). Link between to DCs down, DCs themselves still up and running? Check... and so on.
Doing this right is extremely hard, and even the best of the best still get caught out (witness Amazon and Google outages, and I refuse to believe they don't know their stuff).
Hardware failures are rare, people failures are common, distributed systems are hard.
A. The probability that a single component in your system fails increases as the number of components increases.
B. The probability of the entire system failing decreases as the number of components increases.
Where (B) goes wrong is if the system is designed in such a way that components are dependent on each other.
Imagine you have a system containing 4 parts, all of which have to have at least one operating component for the system to remain operational. The components are:
WS = Web Server DB = Database Server AS = Application Server (executing long-running tasks) LB = Load Balancer
Each component has a different probability of failure on a given day, given here:
WS, AS = 0.001 DB = 0.002 LB = 0.000001
If you do this:
10 x WS = (0.001)^10 10 x AS = (0.001)^10 1 x DB = (0.002)^1 10 x LB = (0.000001)^2
Then the probability of failure is 0.002, because if the database fails then the system fails. To increase redundancy you need to increase the number of DB servers too. If you have two DB servers, then the probability is 0.000004, 500 times lower.
I believe that you really did experience a problem with your setup: and I'll hazard a guess that the root cause is nothing to do with the architecture of your system but everything to do with the exponential increase in SNAFUs cause by the extra complexity.
Hardware failures are rare, people failures are common.