Updated REGRESSIONS and sources of noise in nightly regressions mail summary
Updated REGRESSIONS and sources of noise in nightly regressions mail summary

These updates are based on the past 2.5 weeks or so and are
categorized into apparent new regressions followed by sources of noise
in the nightly regressions emails that could be removed by getting rid
of systemic or spurious failures.  I've added notes in the discussion
of the spurious errors below.

main issues, sorted by approximate order of importance/ease-of-fix
------------------------------------------------------------------
- CMO_array failing (due to rank change interface change?)
- internal error SYM0172 for test_array_type_alias, simple-4 with --no-local
- illegal use of on in serial iterator error on iteratorWithOn for --no-local
- test_recursive_leader3 failing for --baseline --no-local
- new XMT unreachable line errors: fft, slices
- nbody_record_domain_2a: error matching performance keys when not timing out
- franzf/v1/chpl/main timeouts due to recent -O changes muddy opts testing
- fannkuch timing out for several opts testing cases as well
- new intel compilation timeouts: test_param_fold5, select1

systemic errors
---------------
* XT environment failures
* Lee pvm checkout failures

spurious errors
---------------
* Timed out executing program studies/madness/.../test_reconstruct: 12x

  - not clear whether this is truly a timeout (e.g., upping the
    TIMEOUT would fix it) or a race-y deadlock.  Would be nice to
    address this since it's our #1 spurious failure


* Timed out compilation for users/franzf/v0/chpl/main: 5x

  - this program has huge basic blocks which we make into even more
    huge C basic blocks.  We used to compile the generated source -O
    which resulted in spurious compilation timeout issues in the C
    compiler.  We still seem to get them in spite of removing this (or
    else I didn't remove it properly -- see r16492)

* Timed out compilation for release/examples/hpcc/fft: 5x

  - though the basic blocks for this fft are not as large, I suspect
    that this is running into similar bottlenecks in our compiler

* Timed out executing program studies/madness/aniruddha/.../test_compress: 4x

  - See test_reconstruct above

* Timed out compilation for users/franzf/v1/chpl/main: 4x

  - See v0 version above

* Timed out compilation for studies/hpcc/FFT/diten/fft: 3x
* Timed out compilation for studies/hpcc/FFT/fft-testPow4: 3x

  - see release/examples/hpcc/fft notes above

* Timed out executing program npb/cg/bradc/cg-sparse: 3x

  - just a long-running benchmark, I'm guessing.  Could use a smaller
    fake problem size for correctness testing and problem size S for
    performance testing?

* Timed out executing program users/agnieszka/test2: 2x
* Timed out executing program users/agnieszka/multiArrayI10: 1x

  - These are a mystery to me.  The time or two that I poked through
    them (or that Steve looked into them), they looked like they were
    probably reasonable codes.  I'm not sure whether they're merely
    long-running or running into some sort of race-y deadlock that we
    don't see in other programs.




git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@16690 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

sungeunchoi/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,sungeunchoi/chapel,hildeth/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,sungeunchoi/chapel,chizarlicious/chapel,CoryMcCartan/chapel,hildeth/chapel,hildeth/chapel,chizarlicious/chapel,sungeunchoi/chapel,hildeth/chapel,sungeunchoi/chapel,hildeth/chapel,CoryMcCartan/chapel,chizarlicious/chapel,chizarlicious/chapel,hildeth/chapel,chizarlicious/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,CoryMcCartan/chapel,chizarlicious/chapel,CoryMcCartan/chapel,hildeth/chapel,chizarlicious/chapel