[SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
[SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API

### What changes were proposed in this pull request?

Currently ,we sets the environment variable `PYSPARK_PIN_THREAD` at the client side of `InhertiableThread` API for Py4J (`python/pyspark/util.py`). If the Py4J gateway is created somewhere else (e.g., Zeppelin, etc), it could introduce a breakage at:

```python
from pyspark import SparkContext
jvm = SparkContext._jvm
thread_connection = jvm._gateway_client.get_thread_connection()
# `AttributeError: 'GatewayClient' object has no attribute 'get_thread_connection'` (non-pinned thread mode)
# `get_thread_connection` is only in 'ClientServer' (pinned thread mode)
```

This PR proposes to check the given gateway created, and do the pinned thread mode behaviour accordingly so we can avoid any breakage when Py4J server/gateway is created separately from somewhere else without a pinned thread mode.

### Why are the changes needed?

To avoid any potential breakage.

### Does this PR introduce _any_ user-facing change?

No, the change happened only in the master (https://github.com/apache/spark/commit/fdd7ca5f4e35a906090f3c6b160bdba9ac9fd0ca).

### How was this patch tested?

This is actually a partial revert of https://github.com/apache/spark/commit/fdd7ca5f4e35a906090f3c6b160bdba9ac9fd0ca. As long as the existing tests pass, I guess we're all good.

I also manually tested to make doubly sure:

**Before**:

```python
>>> from pyspark import InheritableThread, inheritable_thread_target
>>> InheritableThread(lambda: 1).start()
>>> inheritable_thread_target(lambda: 1)()
Traceback (most recent call last):
  File "/.../python3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/.../python3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/.../spark/python/pyspark/util.py", line 361, in copy_local_properties
    InheritableThread._clean_py4j_conn_for_current_thread()
  File "/.../spark/python/pyspark/util.py", line 381, in _clean_py4j_conn_for_current_thread
    thread_connection = jvm._gateway_client.get_thread_connection()
AttributeError: 'GatewayClient' object has no attribute 'get_thread_connection'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/util.py", line 324, in wrapped
    InheritableThread._clean_py4j_conn_for_current_thread()
  File "/.../spark/python/pyspark/util.py", line 381, in _clean_py4j_conn_for_current_thread
    thread_connection = jvm._gateway_client.get_thread_connection()
AttributeError: 'GatewayClient' object has no attribute 'get_thread_connection'
```

**After**:

```python
>>> from pyspark import InheritableThread, inheritable_thread_target
>>> InheritableThread(lambda: 1).start()
>>> inheritable_thread_target(lambda: 1)()
1
```

Closes #33147 from HyukjinKwon/SPARK-35946.

Authored-by: Hyukjin Kwon <2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org>
Signed-off-by: Dongjoon Hyun <6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org>

mahak/spark,gengliangwang/spark,chuckchen/spark,witgo/spark,holdenk/spark,holdenk/spark,cloud-fan/spark,nchammas/spark,vinodkc/spark,apache/spark,vinodkc/spark,wangmiao1981/spark,shaneknapp/spark,xuanyuanking/spark,HyukjinKwon/spark,wangmiao1981/spark,xuanyuanking/spark,dongjoon-hyun/spark,witgo/spark,witgo/spark,apache/spark,JoshRosen/spark,cloud-fan/spark,gengliangwang/spark,milliman/spark,vinodkc/spark,rednaxelafx/apache-spark,wangyum/spark,witgo/spark,apache/spark,hvanhovell/spark,xuanyuanking/spark,mahak/spark,witgo/spark,hvanhovell/spark,wangyum/spark,WeichenXu123/spark,milliman/spark,zzcclp/spark,srowen/spark,ueshin/apache-spark,HyukjinKwon/spark,apache/spark,wangyum/spark,jiangxb1987/spark,wangmiao1981/spark,zzcclp/spark,HyukjinKwon/spark,milliman/spark,shaneknapp/spark,ueshin/apache-spark,mahak/spark,zero323/spark,zero323/spark,zero323/spark,cloud-fan/spark,rednaxelafx/apache-spark,WeichenXu123/spark,jiangxb1987/spark,dongjoon-hyun/spark,chuckchen/spark,xuanyuanking/spark,wangmiao1981/spark,zzcclp/spark,wangyum/spark,JoshRosen/spark,ueshin/apache-spark,HyukjinKwon/spark,wangyum/spark,dongjoon-hyun/spark,xuanyuanking/spark,rednaxelafx/apache-spark,zero323/spark,milliman/spark,hvanhovell/spark,taroplus/spark,shaneknapp/spark,srowen/spark,cloud-fan/spark,jiangxb1987/spark,chuckchen/spark,zzcclp/spark,dongjoon-hyun/spark,dongjoon-hyun/spark,srowen/spark,shaneknapp/spark,chuckchen/spark,JoshRosen/spark,jiangxb1987/spark,WeichenXu123/spark,gengliangwang/spark,milliman/spark,holdenk/spark,taroplus/spark,chuckchen/spark,HyukjinKwon/spark,WeichenXu123/spark,rednaxelafx/apache-spark,jiangxb1987/spark,jiangxb1987/spark,zero323/spark,JoshRosen/spark,shaneknapp/spark,gengliangwang/spark,dongjoon-hyun/spark,taroplus/spark,zero323/spark,xuanyuanking/spark,apache/spark,zero323/spark,vinodkc/spark,wangmiao1981/spark,hvanhovell/spark,srowen/spark,rednaxelafx/apache-spark,rednaxelafx/apache-spark,mahak/spark,shaneknapp/spark,taroplus/spark,vinodkc/spark,ueshin/apache-spark,HyukjinKwon/spark,cloud-fan/spark,taroplus/spark,JoshRosen/spark,ueshin/apache-spark,hvanhovell/spark,nchammas/spark,WeichenXu123/spark,WeichenXu123/spark,rednaxelafx/apache-spark,mahak/spark,zzcclp/spark,taroplus/spark,srowen/spark,apache/spark,nchammas/spark,chuckchen/spark,shaneknapp/spark,holdenk/spark,hvanhovell/spark,mahak/spark,vinodkc/spark,wangmiao1981/spark,zzcclp/spark,jiangxb1987/spark,JoshRosen/spark,HyukjinKwon/spark,taroplus/spark,hvanhovell/spark,dongjoon-hyun/spark,wangmiao1981/spark,WeichenXu123/spark,JoshRosen/spark,milliman/spark,gengliangwang/spark,chuckchen/spark,cloud-fan/spark,nchammas/spark,nchammas/spark,milliman/spark,ueshin/apache-spark,wangyum/spark,ueshin/apache-spark,holdenk/spark,gengliangwang/spark,apache/spark,vinodkc/spark,wangyum/spark,nchammas/spark,witgo/spark,cloud-fan/spark,xuanyuanking/spark,srowen/spark,witgo/spark,holdenk/spark,zzcclp/spark,srowen/spark,nchammas/spark,mahak/spark,holdenk/spark,gengliangwang/spark