山东省建设厅执业资格注册中心网站,装饰工程有限公司,学ui设计难不难,西峡网站优化flume的作用是从接受外界的日志信息#xff0c;然后输出到本地的一个框架。agent是Flume很重要的组成#xff0c;包括有source#xff0c;channel#xff0c;sink。source是从外部接受日志。channel跟内存相似#xff0c;读满了之后再写到sink中。sink是将数据写到本地然后输出到本地的一个框架。agent是Flume很重要的组成包括有sourcechannelsink。source是从外部接受日志。channel跟内存相似读满了之后再写到sink中。sink是将数据写到本地可以写在HDFS上也能先写在Fafka等等。配置1.首先下载包解压并将bin路径配置到~/.bash_profile复制flume-env.sh.template配置文件并修改指定jdk的路径。2.在conf文件夹下新建conf文件# example.conf: A single-node Flume configuration# Name the components on this agenta1.sourcesr1a1.sinksk1a1.channelsc1#Describe/configure the sourcea1.sources.r1.typenetcata1.sources.r1.bindlocalhosta1.sources.r1.port44444# Describe the sinka1.sinks.k1.typelogger# Use a channel which buffers events in memorya1.channels.c1.typememorya1.channels.c1.capacity1000a1.channels.c1.transactionCapacity100# Bind the source and sink to the channela1.sources.r1.channelsc1a1.sinks.k1.channelc1启动agentflume-ng agent \--name a1 \--conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/example.conf \-Dflume.root.loggerINFO.console在另一个控制台里输入telnet hadoop000 44444 输入些文字测试。。。以上是Flume监听端口信息接下来实时监听本地文件信息只需将#Describe/configure the source下的三行改为a1.sources.r1.type execa1.sources.r1.command tail -F /home/hadoop/data/data.loga1.sources.r1.shell /bin/sh -C在conf目录下新建 exec-memory-logger.conf文件写入以上配置信息然后启动agent。flume-ng agent \--name a1 \--conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/exec-memory-logger.conf \-Dflume.root.loggerINFO.console在另一个控制台里往data.log里写数据echo hello data.logecho world data.log这样agent就会输出日志信息了。接下来将A服务器上的日志实时采集到B服务器上、图1机器A的conf配置# exec-memory-avro.conf:# Name the components on this agentexec-memory-avro.sourcesexec-sourceexec-memory-avro.sinksavro-sinkexec-memory-avro.channels memory-channel#Describe/configure the sourceexec-memory-avro.sources.exec-source.typeexecexec-memory-avro.sources.exec-source.commandtail -F /home/hadoop/data/data.logexec-memory-avro.sources.exec-source.shell /bin/sh -c# Describe the sinkexec-memory-avro.sinks.avro-sink.typeavroexec-memory-avro.sinks.avro-sink.hostname hadoop000exec-memory-avro.sinks.avro-sink.port 44444# Use a channel which buffers events in memoryexec-memory-avro.channels.memory-channel.typememory# Bind the source and sink to the channelexec-memory-avro.sources.exec-source.channelsmemory-channelexec-memory-avro.sinks.avro-sink.channelmemory-channel机器B的conf文件avro-memory-logger.sources avro-sourceavro-memory-logger.sinks logger-sinkavro-memory-logger.channels memory-channelavro-memory-logger.sources.avro-source.type avroavro-memory-logger.sources.avro-source.bind hadoop000avro-memory-logger.sources.avro-source.port 44444avro-memory-logger.sinks.logger-sink.type loggeravro-memory-logger.channels.memory-channel.type memoryavro-memory-logger.sources.avro-source.channels memory-channelavro-memory-logger.sinks.logger-sink.channel memory-channel配置完成。先启动机器B(第一个控制台)flume-ng agent \--name avro-memory-logger \--conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/avro-memory-logger.conf \-Dflume.root.loggerINFO,console再启动机器A(第二个控制台)flume-ng agent \--name exec-memory-avro \--conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \-Dflume.root.loggerINFO,console在第三个控制台的data目录输入echo welcome data.log第一个控制台就会有日志信息显示了。。。
