[SPARK-18523][PYSPARK] Make SparkContext.stop more reliable
[SPARK-18523][PYSPARK] Make SparkContext.stop more reliable

## What changes were proposed in this pull request?

This PR fixes SparkContext broken state in which it may fall if spark driver get crashed or killed by OOM.

## How was this patch tested?

1. Start SparkContext;
2. Find Spark driver process and `kill -9` it;
3. Call `sc.stop()`;
4. Create new SparkContext after that;

Without this patch you will crash on step 3 and won't be able to do step 4 without manual reset private attibutes or IPython notebook / shell restart.

Author: Alexander Shorin <kxepal@apache.org>

Closes #15961 from kxepal/18523-make-spark-context-stop-more-reliable.

(cherry picked from commit 71352c94ad2a60d1695bd7ac0f4452539270e10c)
Signed-off-by: Reynold Xin <c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com>

big-pegasus/spark,Panos-Bletsos/spark-cost-model-optimizer,huang1900/spark,metamx/spark,kimoonkim/spark,SnappyDataInc/spark,huang1900/spark,huang1900/spark,huang1900/spark,ibm-research-ireland/sparkoscope,ibm-research-ireland/sparkoscope,SnappyDataInc/spark,metamx/spark,SnappyDataInc/spark,big-pegasus/spark,ibm-research-ireland/sparkoscope,kimoonkim/spark,Panos-Bletsos/spark-cost-model-optimizer,metamx/spark,Panos-Bletsos/spark-cost-model-optimizer,SnappyDataInc/spark,huang1900/spark,big-pegasus/spark,big-pegasus/spark,metamx/spark,kimoonkim/spark,ibm-research-ireland/sparkoscope,big-pegasus/spark,Panos-Bletsos/spark-cost-model-optimizer,Panos-Bletsos/spark-cost-model-optimizer,metamx/spark,SnappyDataInc/spark,kimoonkim/spark,ibm-research-ireland/sparkoscope,kimoonkim/spark,metamx/spark,ibm-research-ireland/sparkoscope,huang1900/spark,metamx/spark,big-pegasus/spark,ibm-research-ireland/sparkoscope,Panos-Bletsos/spark-cost-model-optimizer,kimoonkim/spark,big-pegasus/spark,SnappyDataInc/spark,Panos-Bletsos/spark-cost-model-optimizer,huang1900/spark,kimoonkim/spark,SnappyDataInc/spark