Kafka to spark integration

Technical Discussions Apache Spark
#1

Hi Guys,

Does anyone have any docs or info about reading Kafka consumer messages from a topic using pyspark in cloudxlab.

If you can share that will be great
thanks

#2

Hi @Arun_Nattamai,

Just curious if you have found any official doc on the same?

#3

No not yet… once i find and try it out… i wil share with the rest

#4

@Arun_Nattamai,

Please refer below GitHub Gist link for Spark Streaming with Kafka (Spark 1.6.3):
https://gist.github.com/ravi-tejarockon/b68e69b868a1271e112982f603eac2a9

Spark Structured Streaming with Kafka (Spark 2.x):
https://gist.github.com/ravi-tejarockon/0d33e1e317180cd1c3e1068d85a7e5d5

Create a topic first & write data to Topic using any console or any type producer, then use above scripts as consumer.

1 Like
#5

thanks Raviteja for your inputs. I will check the links. thanks again.

1 Like