Description:
Fabric uses threads to process XML-RPC requests (Session Threads),
update the state store (Executor Threads) and periodically check
servers' availability (Failure Detector Threads).
Each thread gets assigned its own database connection when it
starts.
If we configure too many executor threads and session threads,
the maximum number of database connections might be reached and
the failure detector will not be able to fetch information on
servers and monitor them. So if there is a failure, servers will
not be set to faulty and a new master will not be elected.
How to repeat:
1. Set the max_connections so that:
(#executor_threads + #session_threads >= #max_connections)
2. Create a group, add servers to it and promote a server to master.
3. Enable the built-in failure detector.
4. Kill the master.
5. Check that a new master is not elected.
Suggested fix:
Check whether (#executor_threads + #session_threads >= #max_connections) is true and raise an exception if this happens. This must be done when Fabric starts and when the built-in failure detector is enabled.