ensure-dirs: Speed up when using big queues
ensure-dirs: Speed up when using big queues

The “ensure-dirs” script as included in Ganeti 2.3 is very slow when
working with big queues requiring a change of permissions on many or all
files.

$ find /var/lib/ganeti/queue/ | wc -l
52354

Before this change:
$ time /usr/local/lib/ganeti/ensure-dirs -f
real    16m4.739s

While not adressed in this patch, I'd like to record the overall
ineffiency of the “ensure-dirs” script, even after this change:

$ time /usr/local/lib/ganeti/ensure-dirs -f
real    5m57.362s
[…]
$ strace -e clone,execve -f -c /usr/local/lib/ganeti/ensure-dirs -f
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 50.08    5.147090          49    104774           clone
 49.92    5.131094          49    104739           execve

More changes will be needed. Just for comparision, a small Python
snippet changing permissions on all files (“ensure-dirs” changes the
owner too):

$ time python -c 'import os; from ganeti import utils;
[os.chmod(i, 0644) for i in
utils.ListVisibleFiles("/var/lib/ganeti/queue/archive/big")]'
real    0m0.605s
[…]

Signed-off-by: Michael Hanselmann <68b67885ff74281219b4a4be33cc20fd45078d7c@google.com>
Reviewed-by: Iustin Pop <acf83a5274f31a43af6b914614365999469c1a31@google.com>

yiannist/ganeti,apyrgio/ganeti,leshchevds/ganeti,andir/ganeti,dimara/ganeti,leshchevds/ganeti,ganeti/ganeti,ganeti/ganeti,ganeti-github-testing/ganeti-test-1,apyrgio/ganeti,ganeti/ganeti,onponomarev/ganeti,bitemyapp/ganeti,yiannist/ganeti,ganeti-github-testing/ganeti-test-1,apyrgio/snf-ganeti,grnet/snf-ganeti,grnet/snf-ganeti,mbakke/ganeti,leshchevds/ganeti,apyrgio/snf-ganeti,andir/ganeti,mbakke/ganeti,dimara/ganeti,mbakke/ganeti,bitemyapp/ganeti,andir/ganeti,yiannist/ganeti,onponomarev/ganeti