Getting below error , As I understand this is due to running with spark versions below 2.3.1 , Is there any way to use spark-submit with higher version as the default version is 2.1.1
21/06/27 20:08:09 INFO EventLoggingListener: Logging events to hdfs:///spark2-history/local-1624824487845
21/06/27 20:08:09 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hi
ve.metastore.warehouse.dir (’/apps/hive/warehouse’).
21/06/27 20:08:09 INFO SharedState: Warehouse path is ‘/apps/hive/warehouse’.
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport
-
at java.lang.ClassLoader.defineClass1(Native Method)* -
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)* -
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)*