Common Commands to interact with various tools

Technical Discussions
#1

Please post the common commands to interact with various tools such as Sqoop, Flume, HDFS, Hadoop Streaming, Python3, Kafka etc.

2 Likes
#2

Here are some example commands:

Python 3

export PATH=/usr/local/anaconda/bin:$PATH
python

This is how it would look:

Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

HDFS

list files

hadoop fs -ls

documentation

hadoop fs -help

Sqoop

Example:

sqoop import --connect jdbc:mysql://cxln2.c.thelab-240901.internal/sqoopex --table widgets -m 2 --hive-import --username sqoopuser -P --hive-database sqoop_testing --split-by id

Hive:

hive

[sandeepgiri9034@cxln4 ~]$ hive
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.2.0-205/0/hive-log4j.properties
hive> show databases

Kafka

export PATH=$PATH:/usr/hdp/current/kafka-broker/bin/

kafka-topics.sh --zookeeper localhost:2181 --list

kafka-topics.sh --zookeeper localhost:2181 --create --topic sandeepgiri9034_test2

kafka-console-producer.sh  --broker-list localhost:6667,c.cloudxlab.com --topic sandeepgiri9034_test

kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic sandeepgiri9034_test