[SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
[SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided

### What changes were proposed in this pull request?

Add `log4j-core` and `log4j-api` to classpath if active hadoop-provided.

### Why are the changes needed?

`log4j-core` is needed:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.Filter
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
```

`log4j-api` is needed:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
	at org.apache.spark.deploy.yarn.SparkRackResolver.<init>(SparkRackResolver.scala:42)
	at org.apache.spark.deploy.yarn.SparkRackResolver$.get(SparkRackResolver.scala:114)
	at org.apache.spark.scheduler.cluster.YarnScheduler.<init>(YarnScheduler.scala:31)
	at org.apache.spark.scheduler.cluster.YarnClusterManager.createTaskScheduler(YarnClusterManager.scala:35)
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2985)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:563)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:327)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:159)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.LogManager
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 26 more
```
`log4j-slf4j-impl` is not needed: https://github.com/apache/spark/pull/35811#issuecomment-1064855439

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual test:
```shell
 ./dev/make-distribution.sh --name SPARK-38516 --tgz  -Phive -Phive-thriftserver  -Pyarn -Phadoop-2 -Phadoop-provided
```

Closes #35811 from wangyum/SPARK-38516.

Lead-authored-by: Yuming Wang <a045e9e381e8d6ed98f3af6ea924f96c8dd7fbad@ebay.com>
Co-authored-by: Yuming Wang <1d300f9117ecd097ba14842082d2c83575f79225@gmail.com>
Signed-off-by: Dongjoon Hyun <6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org>

JoshRosen/spark,wangyum/spark,shaneknapp/spark,srowen/spark,vinodkc/spark,WeichenXu123/spark,zzcclp/spark,WeichenXu123/spark,JoshRosen/spark,cloud-fan/spark,mahak/spark,WeichenXu123/spark,wangyum/spark,apache/spark,hvanhovell/spark,rednaxelafx/apache-spark,zzcclp/spark,HyukjinKwon/spark,srowen/spark,JoshRosen/spark,rednaxelafx/apache-spark,HyukjinKwon/spark,apache/spark,WeichenXu123/spark,shaneknapp/spark,mahak/spark,WeichenXu123/spark,rednaxelafx/apache-spark,WeichenXu123/spark,hvanhovell/spark,nchammas/spark,apache/spark,wangyum/spark,shaneknapp/spark,HyukjinKwon/spark,HyukjinKwon/spark,mahak/spark,hvanhovell/spark,zero323/spark,witgo/spark,zero323/spark,mahak/spark,vinodkc/spark,shaneknapp/spark,nchammas/spark,nchammas/spark,ueshin/apache-spark,holdenk/spark,zzcclp/spark,cloud-fan/spark,apache/spark,apache/spark,gengliangwang/spark,zzcclp/spark,zero323/spark,vinodkc/spark,holdenk/spark,srowen/spark,zzcclp/spark,witgo/spark,srowen/spark,wangyum/spark,gengliangwang/spark,witgo/spark,hvanhovell/spark,rednaxelafx/apache-spark,apache/spark,holdenk/spark,JoshRosen/spark,cloud-fan/spark,gengliangwang/spark,nchammas/spark,hvanhovell/spark,shaneknapp/spark,witgo/spark,HyukjinKwon/spark,shaneknapp/spark,dongjoon-hyun/spark,rednaxelafx/apache-spark,dongjoon-hyun/spark,JoshRosen/spark,ueshin/apache-spark,srowen/spark,nchammas/spark,witgo/spark,zero323/spark,hvanhovell/spark,JoshRosen/spark,gengliangwang/spark,vinodkc/spark,mahak/spark,holdenk/spark,JoshRosen/spark,zero323/spark,apache/spark,wangyum/spark,mahak/spark,HyukjinKwon/spark,ueshin/apache-spark,holdenk/spark,gengliangwang/spark,srowen/spark,nchammas/spark,dongjoon-hyun/spark,gengliangwang/spark,cloud-fan/spark,ueshin/apache-spark,cloud-fan/spark,zero323/spark,holdenk/spark,WeichenXu123/spark,zzcclp/spark,hvanhovell/spark,ueshin/apache-spark,cloud-fan/spark,vinodkc/spark,nchammas/spark,dongjoon-hyun/spark,ueshin/apache-spark,HyukjinKwon/spark,cloud-fan/spark,rednaxelafx/apache-spark,vinodkc/spark,witgo/spark,mahak/spark,witgo/spark,dongjoon-hyun/spark,zzcclp/spark,zero323/spark,ueshin/apache-spark,rednaxelafx/apache-spark,vinodkc/spark,srowen/spark,wangyum/spark,dongjoon-hyun/spark,holdenk/spark,shaneknapp/spark,dongjoon-hyun/spark,gengliangwang/spark,wangyum/spark