Apache Kafka, ozzie and Zookeeper

Technical Discussions Apache Kafka
#1

Hi,

I need to work on Apache kafka. And also need to get hands-on on scheduling jobs using oozie, and zookeeper. Could you please help me in getting links for the same.

Appreciate your help well in advance.

Thanks,
Mahesh .H

#2

Hi @maheshharuvaiah8134,

There are tons of articles available on the same on the internet. You can also go through CloudxLab playlist on Youtube.

#3

Hi Abhinav,

I went through one of the cloudxlab youtube video and did the below step: but not successfully. Please check and reply me ASAP…

export Path=$PATH:/usr/hdp/current/kafka-broker/bin

[maheshharuvaiah8134@ip-172-31-38-146 ~]$ export Path=$PATH:/usr/hdp/current/kafka-broker/bin
[maheshharuvaiah8134@ip-172-31-38-146 ~]$ kafka
Usage: /bin/kafka {start|stop|status|clean}
[maheshharuvaiah8134@ip-172-31-38-146 ~]$ bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181
-bash: bin/kafka-topic.sh: No such file or directory

Now, i want to start 3 server brokers, producers, consumers and zookeeper. Please guide me.

These are the steps i need to create ==>

  1. Starting zookeeper
  2. Starting the broker servers(i am looking for 3 brokers)
  3. Creating a topic to pull from publisher
  4. Creating consumer program to take data from that topic periodically.

EX:
a. STARTING ZOOKEEPER SERVICE-

/usr/local/kafka_2.10-0.10.2.1]$ bin/zookeeper-server-start.sh config/zk1.properties

b. STARTING KAFKA BROKER SERVICES

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server1.properties

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server2.properties

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-server-start.sh config/server3.properties

c. CREATING A TOPIC

/usr/local/kafka_2.10-0.10.2.1]$ bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

Thanks,
Mahesh .H

#4

Hi Abhinav,

Could you please check and update my queries. I have waiting for your reply since long time or route me to some one who can help me. The video on youtube did not help me much. Basically i need to start zookeeper, kafka brokers, producers and consumers in the console and capture the data from topic to consumer. Also i need to write a program using SBT and do the same. Please help.

Thanks,
Mahesh .H

#5

Any update on the below query.

#6

Since you have set the home directory clearly, you don’t need to use bin/kafka-topics.sh.

So instead of :
bin/kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

You should use:
kafka-topic.sh --create --topic testtopic1 --partitions 3 --replication-factor 3 --zookeeper localhost:2181

I would strongly suggest going thru linux basics: https://cloudxlab.com/assessment/playlist-intro/2/linux-basics