Make RabbitMQ OCF script tolerate rabbitmqctl timeouts
Make RabbitMQ OCF script tolerate rabbitmqctl timeouts

The change makes OCF script ignore small number of timeouts of rabbitmqctl
for 'heavy' operations: list_channels, get_alarms and list_queues.
Number of tolerated timeouts in a row is configured through a new variable
'max_rabbitmqctl_timeouts'. By default it is set to 1, i.e. rabbitmqctl
timeouts are not tolerated at all.

Bug #1487517 is fixed by extracting declaration of local variables
'rc_alarms' and 'rc_queues' from assignment operations.


Text for Operations Guide:

If on node where RabbitMQ is deployed
other processes consume significant part of CPU, RabbitMQ starts
responding slow to queries by 'rabbitmqctl' utility. The utility is
used by RabbitMQ's OCF script to monitor state of the RabbitMQ.
When utility fails to return in pre-defined timeout, OCF script
considers RabbitMQ to be down and restarts it, which might lead to
a limited (several minutes) OpenStack downtime. Such restarts
are undesirable as they cause downtime without benefit. To
mitigate the issue, the OCF script might be told to tolerate
certain amount of rabbitmqctl timeouts in a row using the following
command:
  crm_resource --resource p_rabbitmq-server --set-parameter \
      max_rabbitmqctl_timeouts --parameter-value N

Here N should be replaced with the number of timeouts. For instance,
if it is set to 3, the OCF script will tolerate two rabbitmqctl
timeouts in a row, but fail if the third one occurs.

By default the parameter is set to 1, i.e. rabbitmqctl timeout is not
tolerated at all. The downside of increasing the parameter is that
if a real issue occurs which causes rabbitmqctl timeout, OCF script
will detect that only after N monitor runs and so the restart, which
might fix the issue, will be delayed.

To understand that RabbitMQ's restart was caused by rabbitmqctl timeout
you should examine lrmd.log of the corresponding controller on Fuel
master node in /var/log/docker-logs/remote/ directory. Here lines like
"the invoked command exited 137: /usr/sbin/rabbitmqctl list_channels ..."

indicate rabbitmqctl timeout. The next line will explain if it
caused restart or not. For example:
"rabbitmqctl timed out 2 of max. 3 time(s) in a row. Doing nothing for now."

DocImpact: user-guide, operations-guide

Closes-Bug: #1479815
Closes-Bug: #1487517
Change-Id: I9dec06fc08dbeefbc67249b9e9633c8aab5e09ca

stackforge/fuel-library,ddepaoli3/fuel-library-dev,ddepaoli3/fuel-library-dev,ddepaoli3/fuel-library-dev,xarses/fuel-library,ddepaoli3/fuel-library-dev,eayunstack/fuel-library,eayunstack/fuel-library,stackforge/fuel-library,huntxu/fuel-library,xarses/fuel-library,huntxu/fuel-library,eayunstack/fuel-library,huntxu/fuel-library,eayunstack/fuel-library,xarses/fuel-library,eayunstack/fuel-library,stackforge/fuel-library,xarses/fuel-library,ddepaoli3/fuel-library-dev,huntxu/fuel-library,stackforge/fuel-library,huntxu/fuel-library