Do not waste CPU cycles on stat(2) on each _auth
Do not waste CPU cycles on stat(2) on each _auth

Currently servers that handle many thousands of minions spend measurable time
doing only stat(2) calls.

In strace it looks like::

     0.000093 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
     0.000236 open("/etc/salt/pki/master/minions", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 154
     0.011412 stat("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", {st_mode=S_IFREG|0644, st_size=800, ...}) = 0
     0.000102 stat("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", {st_mode=S_IFREG|0644, st_size=800, ...}) = 0
     ...many thousands lines...
     0.000064 stat("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", {st_mode=S_IFREG|0644, st_size=800, ...}) = 0
     0.000062 stat("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", {st_mode=S_IFREG|0644, st_size=796, ...}) = 0
     0.000485 stat("/export/apps/salt/log/master", {st_mode=S_IFREG|0644, st_size=37769598988, ...}) = 0
     0.000065 stat("/export/apps/salt/log/master", {st_mode=S_IFREG|0644, st_size=37769598988, ...}) = 0
     0.000184 stat("/etc/salt/pki/master/minions_rejected/hostXXXX.linkedin.com", 0x7fff28209f40) = -1 ENOENT (No such file or directory)
     0.000071 stat("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", {st_mode=S_IFREG|0644, st_size=800, ...}) = 0
     0.000074 open("/etc/salt/pki/master/minions/hostXXXX.linkedin.com", O_RDONLY) = 154

Happens that on each _auth() call salt is counting files in pki/master/minions
and checks whenever those are regular files by applying os.path.isfile()
function to each which considerably slows things down.

This patch removes isfile() call that effectively makes check_max_open_files()
check less precise (but since it's already subject to external race conditions
like creation/removal of files by another process it should be OK) in exchange
for making it much faster.

PS. Note that calling salt.utils.verify.check_max_open_files() on each _auth is
not really efficient, it probably should be done periodically in a background
thread.

Sponsored by: LinkedIn
Signed-off-by: Alexey Ivanov <1b3c8565341e01839efcfcc8c00d738e60059dd6@GMail.com>

saltstack/salt,saltstack/salt,saltstack/salt,saltstack/salt,saltstack/salt