Added LD_LIBRARY_PATH to the ExecutorEnv. This is need so that the nodes can find libjvm.so, or if the master has a different LD_LIBRARY_PATH then the nodes. Make sure to export LD_LIBRARY_PATH  that includes the path to libjvm.so in the nodes.
Added LD_LIBRARY_PATH to the ExecutorEnv. This is need so that the nodes can find libjvm.so, or if the master has a different LD_LIBRARY_PATH then the nodes. Make sure to export LD_LIBRARY_PATH  that includes the path to libjvm.so in the nodes.

To Run:

USE_YARN=1 SPARK_YARN_VERSION=2.4.0 SPARK_HADOOP_VERSION=2.4.0 ./install-dev.sh
export YARN_CONF_DIR=<yarn_conf_dir>
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/ to/ libjvm.so/ on/ nodes/"
MASTER="yarn-client" ./sparkR


If LD_LIBRARY_PATH not properly set you will find the following error in the yarn logs

Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/usr/lib/Rlibrary/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file: No such file or directory
Error: package ‘rJava’ could not be loaded
Execution halted
Draki/SparkR-pkg,RevolutionAnalytics/SparkR-pkg,amplab-extras/SparkR-pkg,rickyHong/sparkR-dist,EugenePig/SparkR-pkg,LeoYao/SparkR-pkg,7c00/SparkR-pkg,LeoYao/SparkR-pkg,EugenePig/SparkR-pkg,Liziyao/SparkR-pkg,xzw0005/SparkR-pkg,RevolutionAnalytics/SparkR-pkg,7c00/SparkR-pkg,amplab-extras/SparkR-pkg,rickyHong/sparkR-dist,rickyHong/sparkR-dist,kimchitsigai/SparkR-pkg,kcompher/SparkR-pkg,xzw0005/SparkR-pkg,nvoron23/SparkR-pkg,kimchitsigai/SparkR-pkg,Liziyao/SparkR-pkg,Liziyao/SparkR-pkg,kimchitsigai/SparkR-pkg,kuch4r/SparkR-pkg,liyuqi/SparkR-pkg,EDMunplugged/SparkR-pkg,RevolutionAnalytics/SparkR-pkg,nvoron23/SparkR-pkg,amplab-extras/SparkR-pkg,liyuqi/SparkR-pkg,RevolutionAnalytics/SparkR-pkg,rickyHong/sparkR-dist,Draki/SparkR-pkg,kcompher/SparkR-pkg,kcompher/SparkR-pkg,kuch4r/SparkR-pkg,7c00/SparkR-pkg,xzw0005/SparkR-pkg,Liziyao/SparkR-pkg,EDMunplugged/SparkR-pkg,LeoYao/SparkR-pkg,liyuqi/SparkR-pkg,amplab-extras/SparkR-pkg,EDMunplugged/SparkR-pkg,Draki/SparkR-pkg,xzw0005/SparkR-pkg,7c00/SparkR-pkg,kcompher/SparkR-pkg,nvoron23/SparkR-pkg,kuch4r/SparkR-pkg,nvoron23/SparkR-pkg,EugenePig/SparkR-pkg,liyuqi/SparkR-pkg,LeoYao/SparkR-pkg,EDMunplugged/SparkR-pkg,kuch4r/SparkR-pkg