Could you please let me know how to change to latest version of the spark. when I give spark-shell, it it taking us to 2.11 version.
We usually install the spark in /usr folder. You can find it there.
ls /usr
bin etc games hdp include jdk64 lib lib64 libexec local sbin share spark2.3 spark2.4.3 src tmp
You can see there are two spark installations: /usr/spark2.3 and /usr/spark2.4.3
To use spark 2.3, you can run the following commands in console. The names of commands are self explainatory.
/usr/spark2.3/bin/sparkR
/usr/spark2.3/bin/spark-submit
/usr/spark2.3/bin/pyspark
/usr/spark2.3/bin/spark-sql
/usr/spark2.3/bin/spark-shell
Similarly, you can use the spark2.4.3.