Latest Spark-Version

Could you please let me know how to change to latest version of the spark. when I give spark-shell, it it taking us to 2.11 version.

We usually install the spark in /usr folder. You can find it there.

ls /usr

bin etc games hdp include jdk64 lib lib64 libexec local sbin share spark2.3 spark2.4.3 src tmp

You can see there are two spark installations: /usr/spark2.3 and /usr/spark2.4.3

To use spark 2.3, you can run the following commands in console. The names of commands are self explainatory.


Similarly, you can use the spark2.4.3.