Make the runtime startup code more symmetric across locales.
Make the runtime startup code more symmetric across locales.

[Reviewed by Tom.]

It used to be that the control flow for locale 0 and the other locales
occurred directly in the runtime startup code.  On locale 0 we launched
a tasking layer task to execute a wrapper that called the user's main
program.  Among other things the compiler-emitted code for the user's
main program calls the pre-user-code hook, a function in the runtime
that needs to be calledafter module initialization and before the actual
user code runs.  On the other locales we just call this pre-user-code
hook directly from the runtime startup code itself.

The change here is to launch a tasking layer task on all locales, rather
than just on locale 0, and move the control divergence for the 0 and
non-0 locales into the body of that launched task.  We also now call the
post-user-code hook on the non-0 locales, which we didn't do at all
before.  This makes the main control flow more symmetric, and the code a
bit more elegant, but it has a functional effect too: the post-user-code
hook is called on all locales just like the pre-user-code hook is, and
both hooks are called from a tasking layer task on all locales, rather
than (for the pre- hook) from a task on locale 0 and directly on the
other locales.

This also resolves a startup failure which has been occurring in all
numa+gasnet+qthreads tests that run on more than one locale.  The
failure was happening in chpl_setMemFlags(), which is called from the
pre-user-code hook.  A (Chapel) callee of that function ends up needing
to access qthreads task-locale storage (in module code having to do with
sublocales for numa), but that storage doesn't exist unless the code is
running in a qthreads task.  So when the hook was called directly from
the main startup code it failed an internal check in qthreads.  Now that
the pre-use-code hook is called from a task, the reference to task local
storage works.  As a result of this change, in gasnet+numa+qthreads
examples testing I saw the current 22 failures (all of the tests that
actually run on multiple locales) drop to just 2, both of which were
SSCA2 timeouts.


git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@22787 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

CoryMcCartan/chapel,chizarlicious/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,sungeunchoi/chapel,hildeth/chapel,CoryMcCartan/chapel,hildeth/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,chizarlicious/chapel,sungeunchoi/chapel,chizarlicious/chapel,sungeunchoi/chapel,hildeth/chapel,CoryMcCartan/chapel,chizarlicious/chapel,sungeunchoi/chapel,sungeunchoi/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,hildeth/chapel,chizarlicious/chapel,chizarlicious/chapel,hildeth/chapel,chizarlicious/chapel,hildeth/chapel,hildeth/chapel,CoryMcCartan/chapel