I am a newbie into hadoop world learned pig and HIVE and Spark. Can any of you provide me the link that better explains the practical steps to automate using Oozie.
oozie manual itself narrates the steps to create a workflow.
https://oozie.apache.org/docs/4.1.0/
What you need to understand is, Oozie works in DAG Directed Acyclic Graph. You can compare it to a Flowchat. Oozie has nodes,
- Start node
- Action nodes
- Kill nodes
- end node
An oozie workflow is generated in XML format with one element for each node,
all oozie workflow has start, kill and end node. Your action node is where you have to place your MR, Hive or other module.
There is another important thing to remember that Oozie workflow XML file must be placed in HDFS system.
Thus editing the file for paramater change etc is difficult thus we have another file called job.properties which must be in local file system(easy to edit). This job.properties files is used for passing parameters to workflow. Parameters like : Output directory of hive, MR input format, MR output format, MR output key class, MR output value class etc.
This blogs explains -> http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/