Bug 1162682 - Catch exceptions whilst processing objectstore jobs
Bug 1162682 - Catch exceptions whilst processing objectstore jobs

Now if an exception occurs during _load_ref_and_job_data_structs(), we
mark that job in the objectstore as errored and continue inserting the
other jobs. Previously the exception would have meant all of the other
jobs were not inserted, causing up to 100 rows in the objectstore to be
stuck in the 'loading' processed_state indefinitely.

The exception string passed to mark_object_error() isn't ideal, but it's
the same as the handling above, so will do for now until we remove the
objectstore.

In addition, this change means that we lose visibility in New Relic for
these exceptions - and someone has to manually check the objectstore for
jobs with error = "Y". However short term this seems preferable to
dropping 100 jobs every time we get an exception, particularly since
this is already the case for deserialisation exceptions. In a followup
bug we could always try using the New Relic Python agent's
record_exception() to maintain reporting without having to re-raise the
exception ourselves.

gbrmachado/treeherder,KWierso/treeherder,akhileshpillai/treeherder,rail/treeherder,KWierso/treeherder,gbrmachado/treeherder,avih/treeherder,gbrmachado/treeherder,akhileshpillai/treeherder,KWierso/treeherder,vaishalitekale/treeherder,jgraham/treeherder,akhileshpillai/treeherder,edmorley/treeherder,deathping1994/treeherder,moijes12/treeherder,vaishalitekale/treeherder,glenn124f/treeherder,sylvestre/treeherder,deathping1994/treeherder,jgraham/treeherder,kapy2010/treeherder,tojonmz/treeherder,wlach/treeherder,rail/treeherder,tojon/treeherder,adusca/treeherder,tojonmz/treeherder,moijes12/treeherder,glenn124f/treeherder,jgraham/treeherder,glenn124f/treeherder,deathping1994/treeherder,parkouss/treeherder,vaishalitekale/treeherder,vaishalitekale/treeherder,adusca/treeherder,parkouss/treeherder,tojonmz/treeherder,wlach/treeherder,deathping1994/treeherder,parkouss/treeherder,kapy2010/treeherder,sylvestre/treeherder,wlach/treeherder,moijes12/treeherder,parkouss/treeherder,KWierso/treeherder,gbrmachado/treeherder,deathping1994/treeherder,jgraham/treeherder,tojon/treeherder,adusca/treeherder,tojonmz/treeherder,tojonmz/treeherder,sylvestre/treeherder,avih/treeherder,wlach/treeherder,glenn124f/treeherder,rail/treeherder,tojon/treeherder,akhileshpillai/treeherder,edmorley/treeherder,deathping1994/treeherder,edmorley/treeherder,kapy2010/treeherder,glenn124f/treeherder,rail/treeherder,parkouss/treeherder,adusca/treeherder,akhileshpillai/treeherder,avih/treeherder,avih/treeherder,rail/treeherder,vaishalitekale/treeherder,jgraham/treeherder,kapy2010/treeherder,tojon/treeherder,sylvestre/treeherder,moijes12/treeherder,edmorley/treeherder,sylvestre/treeherder,avih/treeherder,adusca/treeherder,moijes12/treeherder,wlach/treeherder,parkouss/treeherder,moijes12/treeherder,rail/treeherder,tojonmz/treeherder,avih/treeherder,adusca/treeherder,vaishalitekale/treeherder,glenn124f/treeherder,jgraham/treeherder,kapy2010/treeherder,akhileshpillai/treeherder,sylvestre/treeherder,gbrmachado/treeherder,gbrmachado/treeherder,wlach/treeherder