Wait for private broadcast completion.
Wait for private broadcast completion.

Currently the GASNet comm layer does non-blocking private broadcasts.
It fires an Active Message request to each remote locale to update the
appropriate chpl_private_broadcast_table[] entry, but does not wait for
an acknowledgement that that update has occurred before it goes on.  We
became concerned that there might be a scenario in which this would
allow a locale to reference a chpl_private_broadcast_table[] entry
before it was initialized, so this commit makes the private broadcast
logic wait for remote acknowledgement before continuing.

We add a vector of done flags to the broadcast function, one flag for
each locale.  Each transmission in the broadcast includes the address of
the appropriate flag in the descriptor that is sent to the AM handler.
The AM handler does a reply to set the flag after it has updated the
remote chpl_private_broadcast_table[] entry.  In order to minimize the
total time spent waiting for acknowledgements, the sender does all the
transmits and only then waits on all the flags.

This was reviewed by Sung.


git-svn-id: 88467cb1fb04b8a755be7e1ee1026be4190196ef@19868 3a8e244f-b0f2-452b-bcba-4c88e055c3ca

hildeth/chapel,CoryMcCartan/chapel,sungeunchoi/chapel,hildeth/chapel,chizarlicious/chapel,sungeunchoi/chapel,hildeth/chapel,hildeth/chapel,CoryMcCartan/chapel,chizarlicious/chapel,chizarlicious/chapel,CoryMcCartan/chapel,CoryMcCartan/chapel,CoryMcCartan/chapel,hildeth/chapel,chizarlicious/chapel,chizarlicious/chapel,sungeunchoi/chapel,chizarlicious/chapel,sungeunchoi/chapel,hildeth/chapel,sungeunchoi/chapel,sungeunchoi/chapel,sungeunchoi/chapel,sungeunchoi/chapel,hildeth/chapel,chizarlicious/chapel,CoryMcCartan/chapel,CoryMcCartan/chapel