I have configured a flume agent to use spool directory as source and hdfs as sink. The configuration is as follows.
Naming the components
retail.sources = e1
retail.channels = c1
retail.sinks = k1
Configuring the sources
retail.sources.e1.type = spooldir
retail.sources.e1.spoolDir = /home/shanthancheruku2610/GiHubDocs
Configuring the sinks
retail.sinks.k1.type = hdfs
retail.sinks.k1.hdfs.path = /user/shanthancheruku2610/retaildb/
retail.sinks.k1.hdfs.writeFormat = Text
retail.sinks.k1.hdfs.batchSize = 1000
retail.sinks.k1.hdfs.fileType = DataStream
Configuring Channels
retail.channels.c1.type = file
retail.channels.c1.capacity = 20000
retail.channels.c1.transactionCapacity = 1000
Linking Sources and Sinks to Channels
retail.sources.e1.channels = c1
retail.sinks.k1.channel = c1
When i run the flume agent, an error is poping up as follows.
18/02/08 11:51:47 ERROR file.FileChannel: Failed to start the file channel [channel=c1]
java.io.IOException: File /home/shanthancheruku2610/.flume/file-channel/data/log-8 has bad version 7f000000
at org.apache.flume.channel.file.LogFileFactory.getRandomReader(LogFileFactory.java:98)
at org.apache.flume.channel.file.Log.replay(Log.java:405)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:289)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
18/02/08 11:51:48 INFO node.Application: Starting Sink k1
18/02/08 11:51:48 INFO node.Application: Starting Source e1
18/02/08 11:51:48 INFO source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /home/shanthancheruku2610/GiHubDocs
18/02/08 11:51:48 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/02/08 11:51:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
18/02/08 11:51:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: e1: Successfully registered new MBean.
18/02/08 11:51:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
18/02/08 11:51:48 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: e1 started
18/02/08 11:51:48 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.io.IOException: File /home/shanthancheruku2610/.flume/file-channel/data/log-8 has bad ver
sion 7f000000
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:339)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: File /home/shanthancheruku2610/.flume/file-channel/data/log-8 has bad version 7f000000
at org.apache.flume.channel.file.LogFileFactory.getRandomReader(LogFileFactory.java:98)
at org.apache.flume.channel.file.Log.replay(Log.java:405)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:289)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
Is there any issue with my flume agent configuration or is it something else. Somebody please help me with this issue. Thanks & Regards…