Pyspark and SparkSQl features are not available in Jupyter notebook

Sundeep_Naidu · April 13, 2020, 5:08pm

Hi,

Pyspark and SparkSQl features are not available in Jupyter notebook.

How to integrate current python 3 with py4j ? to ge the pyspark in my Jupyter notebook?

Thanks,
Sundeep

sgiri · April 13, 2020, 5:25pm

These notebook should be helpful to you:

github.com

cloudxlab/bigdata/blob/master/spark/pyspark.md

If you are learning pyspark i.e. Spark using Python, please note that the API is almost same as Scala.
Here are the links to the code:

## Spark with Python
https://github.com/cloudxlab/bigdata/tree/master/spark/python
https://github.com/cloudxlab/bigdata/tree/master/spark/examples/python

## GraphX
https://github.com/cloudxlab/bigdata/blob/master/spark/examples/graphx/pagerank.py.md

## Mllib
https://github.com/cloudxlab/bigdata/blob/master/spark/examples/mllib/mllib_random_forrest.ipynb
https://github.com/cloudxlab/bigdata/blob/master/spark/examples/mllib/movie-recommendations.py

## Streaming with Pyspark
https://github.com/cloudxlab/bigdata/blob/master/spark/examples/streaming/word_count/word_count.py

## Streaming with Pyspark and Kafka
https://github.com/cloudxlab/bigdata/blob/master/spark/examples/streaming/word_count_kafka/word_count_kafka_spark_streaming.md

This file has been truncated. show original

Sundeep_Naidu · April 13, 2020, 7:58pm

Hello Giri,

Thanks for sharing the information.