Monitor now consists of three threads instead of two: 
Monitor now consists of three threads instead of two: 

MainThread is in charge of the standardAllocationLoop.  However, it has had its responsibilities to cleanup old empty directories taken away and assigned to a new thread.  Free of that responsibility, it no longer has to walk the entire directory tree each time it wants to look for new jobs.  It begins each directory walk at the point that it left off the previous time.  This will save tons of time.  

The main thread also has new responsibilities of reassigning jobs from dead processors to live processors.  Prior to this, a dead processor's job were just deleted from the jobs table: the full directory walk would rediscover the jobs and reassign them to live processors.  Instead, a reassignment is done on the fly as necessary at the beginning of each iteration of the standardAllocationLoop.

The priorityLoopingThread keeps its current responsibilities with the priorityJobAllocationLoop, but it has had its walking of the file system cleaned up in the same way as was done for the standardAllocationLoop.  It only walks where it has not been before.  Again this will save a lot of time.

A new thread, directoryCleanupThread, walks the entire file system every four hours (changeable via cleanupLoopDelay in the config file).  Its responsibility is to delete empty old directories.

This new monitor has a couple new configuration settings.  They can be found in .../socorro/monitor/config.py.dist.  They are  'cleanupLoopDelay' and 'fileSystemDateThreshold'.

'cleanupLoopDelay' – the number of seconds that the  directoryCleanupThread is idle between runs through the cleaning up the filesystem.  I've arbitrarily set it to 4 hours.

'fileSystemDateThreshold' – while the  standardAllocationLoop and priorityJobAllocationLoop will only look at the the parts of the file system that they haven't seen before, the first time monitor runs we don't want them walking the entire file system.  This setting is way to jump start the “most recently seen” value.  It is a datetime in a form similar to how the datetime appears in the file system paths: YYYY/MM/DD/HH.  No minutes or seconds are used.  This value should be set to an approximation of where the previous version of monitor may have left off.  Since it is taking about two hours to run through the entire file system, this value should be set to something like a couple hours behind the time when monitor last walked the tree.  For example, if you killed the old monitor on 2008/7/3 at 10AM, set the value to '2008/7/3/8'.  It is not terribly important that this number be accurate, err on the side of allowing monitor to rescan parts of the filesystem that it has seen before.  The second time around the loop, both allocation loops will optimize themselves.

To use this version of monitor, you do not need to stop the processors.  Stop the current monitor, update both .../trunk/socorro/monitor and .../trunk/socorro/lib to the latest from svn and then restart the monitor.




git-svn-id: 0429bb2eb6f71f8cd87fe9410a915008a087abca@401 efcc4bdc-162a-0410-a8af-5b4b46c5376d

Tchanders/socorro,Tayamarn/socorro,Tchanders/socorro,Serg09/socorro,mozilla/socorro,m8ttyB/socorro,bsmedberg/socorro,lonnen/socorro,Tayamarn/socorro,linearregression/socorro,mozilla/socorro,m8ttyB/socorro,Serg09/socorro,linearregression/socorro,yglazko/socorro,cliqz/socorro,KaiRo-at/socorro,AdrianGaudebert/socorro,AdrianGaudebert/socorro,KaiRo-at/socorro,cliqz/socorro,cliqz/socorro,adngdb/socorro,AdrianGaudebert/socorro,linearregression/socorro,yglazko/socorro,yglazko/socorro,cliqz/socorro,rhelmer/socorro,KaiRo-at/socorro,spthaolt/socorro,luser/socorro,luser/socorro,m8ttyB/socorro,spthaolt/socorro,pcabido/socorro,Serg09/socorro,pcabido/socorro,lonnen/socorro,yglazko/socorro,rhelmer/socorro,spthaolt/socorro,KaiRo-at/socorro,twobraids/socorro,adngdb/socorro,AdrianGaudebert/socorro,adngdb/socorro,pcabido/socorro,bsmedberg/socorro,Tayamarn/socorro,adngdb/socorro,pcabido/socorro,Tayamarn/socorro,yglazko/socorro,mozilla/socorro,mozilla/socorro,m8ttyB/socorro,Tchanders/socorro,Tchanders/socorro,lonnen/socorro,mozilla/socorro,m8ttyB/socorro,Tayamarn/socorro,twobraids/socorro,KaiRo-at/socorro,luser/socorro,cliqz/socorro,rhelmer/socorro,AdrianGaudebert/socorro,linearregression/socorro,AdrianGaudebert/socorro,pcabido/socorro,spthaolt/socorro,spthaolt/socorro,rhelmer/socorro,twobraids/socorro,luser/socorro,twobraids/socorro,adngdb/socorro,yglazko/socorro,Tchanders/socorro,Serg09/socorro,bsmedberg/socorro,twobraids/socorro,adngdb/socorro,luser/socorro,cliqz/socorro,linearregression/socorro,KaiRo-at/socorro,mozilla/socorro,Tayamarn/socorro,m8ttyB/socorro,bsmedberg/socorro,Serg09/socorro,luser/socorro,Tchanders/socorro,linearregression/socorro,twobraids/socorro,spthaolt/socorro,lonnen/socorro,Serg09/socorro,pcabido/socorro,rhelmer/socorro,bsmedberg/socorro,rhelmer/socorro