Could you please let me know how to change to latest version of the spark. when I give spark-shell, it it taking us to 2.11 version.
We usually install the spark in /usr folder. You can find it there.
bin etc games hdp include jdk64 lib lib64 libexec local sbin share spark2.3 spark2.4.3 src tmp
You can see there are two spark installations: /usr/spark2.3 and /usr/spark2.4.3
To use spark 2.3, you can run the following commands in console. The names of commands are self explainatory.
Similarly, you can use the spark2.4.3.