Apache Kafka, ozzie and Zookeeper

Hi,

I need to work on Apache kafka. And also need to get hands-on on scheduling jobs using oozie, and zookeeper. Could you please help me in getting links for the same.

Appreciate your help well in advance.

Thanks,
Mahesh .H

Hi @maheshharuvaiah8134,

There are tons of articles available on the same on the internet. You can also go through CloudxLab playlist on Youtube.

Hi Abhinav,

I went through one of the cloudxlab youtube video and did the below step: but not successfully. Please check and reply me ASAP…

export Path=$PATH:/usr/hdp/current/kafka-broker/bin

[maheshharuvaiah8134@ip-172-31-38-146 ~]$ export Path=$PATH:/usr/hdp/current/kafka-broker/bin
[maheshharuvaiah8134@ip-172-31-38-146 ~]$ kafka
Usage: /bin/kafka {start|stop|status|clean}
[maheshharuvaiah8134@ip-172-31-38-146 ~]$ bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181
-bash: bin/kafka-topic.sh: No such file or directory

Now, i want to start 3 server brokers, producers, consumers and zookeeper. Please guide me.

These are the steps i need to create ==>

  1. Starting zookeeper
  2. Starting the broker servers(i am looking for 3 brokers)
  3. Creating a topic to pull from publisher
  4. Creating consumer program to take data from that topic periodically.

EX:
a. STARTING ZOOKEEPER SERVICE-

/usr/local/kafka_2.10-0.10.2.1]$ bin/zookeeper-server-start.sh config/zk1.properties

b. STARTING KAFKA BROKER SERVICES

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server1.properties

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server2.properties

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server3.properties

c. CREATING A TOPIC

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

Thanks,
Mahesh .H

Hi Abhinav,

Could you please check and update my queries. I have waiting for your reply since long time or route me to some one who can help me. The video on youtube did not help me much. Basically i need to start zookeeper, kafka brokers, producers and consumers in the console and capture the data from topic to consumer. Also i need to write a program using SBT and do the same. Please help.

Thanks,
Mahesh .H

Any update on the below query.

Since you have set the home directory clearly, you don’t need to use bin/kafka-topics.sh.

So instead of :
bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

You should use:
kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

I would strongly suggest going thru linux basics: https://cloudxlab.com/assessment/playlist-intro/2/linux-basics