您的位置: 首页  >  文章  >  Apache Storm

Apache Storm

分类: 文章 • 2023-11-09 21:56:22

Apache Storm体系结构

三、安装和配置Apache Storm

（一）Storm的伪分布环境: tanzhou111

1、搭建一个ZooKeeper（模式：standalone模式下）

（*）解压tar -zxvf zookeeper-3.4.10.tar.gz -C ~/training/

（*）设置环境变量

ZOOKEEPER_HOME=/root/training/zookeeper-3.4.10

export ZOOKEEPER_HOME

export PATH=$ZOOKEEPER_HOME/bin:$PATH

./zkServer.sh start (如果启动不了直接删掉重新解压zk,检查firewalld)

同步时间的命令 date -s 2017-00-00

（*）创建ZK保存数据的目录 mkdir /root/training/zookeeper-3.4.10/tmp

（*）修改配置文件 zoo.cfg

dataDir=/root/training/zookeeper-3.4.10/tmp

server.1=tanzhou111:2888:3888

2888: 通信的端口

3888：选举的端口

（*）在/root/training/zookeeper-3.4.10/tmp下创建 myid 的文件

输入1

2、Storm的伪分布环境

（*）tar -zxvf apache-storm-1.0.3.tar.gz -C ~/training/

（*）设置环境变量

STORM_HOME=/root/training/apache-storm-1.0.3

export STORM_HOME

export PATH=$STORM_HOME/bin:$PATH

（*）修改配置文件 storm.yaml

一定注意：后面有个空格 （指定zookeeper服务地址）

Apache Storm

storm.zookeeper.servers:

- "192.168.157.111"

指定主节点的nimbus

nimbus.seeds: ["192.168.157.111"]

指定任务提交到nimbus后，保存的目录

storm.local.dir: "/root/training/apache-storm-1.0.3/tmp"

指定每个从节点的supervisor上运行几个worker

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

（*）启动：

（1）启动ZK

（2）启动主节点: storm nimbus &

启动从节点：storm supervisor &

启动UI：storm ui &

端口: http://ip:8080

（二）Storm的全分布环境: tanzhou112 tanzhou113 tanzhou114

（*）已经有ZK

（*）在主节点上搭建：tanzhou112

（*）修改配置文件 storm.yaml

一定注意：后面有个空格

storm.zookeeper.servers:

- "192.168.157.112"

- "192.168.157.113"

- "192.168.157.114"

指定主节点的nimbus

nimbus.seeds: ["192.168.157.112"]

 

storm.zookeeper.port:2181

 

指定任务提交到nimbus后，保存的目录

storm.local.dir: "/root/training/apache-storm-1.0.3/tmp"

指定每个从节点的supervisor上运行几个worker

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

Apache Storm

（*）复制到其他节点

scp -r apache-storm-1.0.3/ [email protected]:/root/training

scp -r apache-storm-1.0.3/ [email protected]:/root/training

（*）启动

tanzhou112： storm nimbus &

storm ui &

tanzhou113和tanzhou114: storm supervisor &

（三）Storm的HA环境

nimbus.seeds: ["192.168.157.112","192.168.157.113"]

启动

tanzhou112： storm nimbus &

storm ui &

storm logviewer & 启动日志查看器

tanzhou113： storm nimbus &

storm supervisor &

storm logviewer & 启动日志查看器

tanzhou114: storm supervisor &

storm logviewer & 启动日志查看器

 

Apache Storm

四、Demo演示: 例子：WordCount 单词计数

/root/training/apache-storm-1.0.3/examples/storm-starter/storm-starter-topologies-1.0.3.jar

查看README文件

[WordCountTopology](src/jvm/org/apache/storm/starter/WordCountTopology.java)

格式：

storm jar 任务的jar包 任务的Class名字 别名

在tanzhou112上提交任务

storm jar storm-starter-topologies-1.0.3.jar org.apache.storm.starter.WordCountTopology MyDemoWordCount

 

是带 s 的

启动Debug开关： "topology.eventlogger.executors": 1

Apache Storm

前面需要有空格一个

 

五、Storm任务调度的过程和WordCount数据流动的过程

提交任务完毕 Finish

Apache Storm

 

Apache Storm

[[email protected] storm-starter]# storm jar storm-starter-topologies-1.0.3.jar org.apache.storm.starter.WordCountTopology MyDemoWordCount

Running: /root/training/jdk1.8.0_144/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/root/training/apache-storm-1.0.3 -Dstorm.log.dir=/root/training/apache-storm-1.0.3/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /root/training/apache-storm-1.0.3/lib/storm-core-1.0.3.jar:/root/training/apache-storm-1.0.3/lib/kryo-3.0.3.jar:/root/training/apache-storm-1.0.3/lib/reflectasm-1.10.1.jar:/root/training/apache-storm-1.0.3/lib/asm-5.0.3.jar:/root/training/apache-storm-1.0.3/lib/minlog-1.3.0.jar:/root/training/apache-storm-1.0.3/lib/objenesis-2.1.jar:/root/training/apache-storm-1.0.3/lib/clojure-1.7.0.jar:/root/training/apache-storm-1.0.3/lib/disruptor-3.3.2.jar:/root/training/apache-storm-1.0.3/lib/log4j-api-2.1.jar:/root/training/apache-storm-1.0.3/lib/log4j-core-2.1.jar:/root/training/apache-storm-1.0.3/lib/log4j-slf4j-impl-2.1.jar:/root/training/apache-storm-1.0.3/lib/slf4j-api-1.7.7.jar:/root/training/apache-storm-1.0.3/lib/log4j-over-slf4j-1.6.6.jar:/root/training/apache-storm-1.0.3/lib/servlet-api-2.5.jar:/root/training/apache-storm-1.0.3/lib/storm-rename-hack-1.0.3.jar:storm-starter-topologies-1.0.3.jar:/root/training/apache-storm-1.0.3/conf:/root/training/apache-storm-1.0.3/bin -Dstorm.jar=storm-starter-topologies-1.0.3.jar org.apache.storm.starter.WordCountTopology MyDemoWordCount

1628 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5974762871224462582:-5889441621206356724

1824 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []

1960 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : tanzhou112:6627

1984 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : tanzhou112:6627

2015 [main] INFO o.a.s.StormSubmitter - Uploading topology jar storm-starter-topologies-1.0.3.jar to assigned location: /root/training/apache-storm-1.0.3/tmp/nimbus/inbox/stormjar-4c165f6f-b736-4638-8686-1bbbf1d240ee.jar

Start uploading file 'storm-starter-topologies-1.0.3.jar' to '/root/training/apache-storm-1.0.3/tmp/nimbus/inbox/stormjar-4c165f6f-b736-4638-8686-1bbbf1d240ee.jar' (73522925 bytes)

[==================================================] 73522925 / 73522925

File 'storm-starter-topologies-1.0.3.jar' uploaded to '/root/training/apache-storm-1.0.3/tmp/nimbus/inbox/stormjar-4c165f6f-b736-4638-8686-1bbbf1d240ee.jar' (73522925 bytes)

4597 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /root/training/apache-storm-1.0.3/tmp/nimbus/inbox/stormjar-4c165f6f-b736-4638-8686-1bbbf1d240ee.jar

4612 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : tanzhou112:6627

4614 [main] INFO o.a.s.StormSubmitter - Submitting topology MyDemoWordCount in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-5974762871224462582:-5889441621206356724","topology.workers":3,"topology.debug":true}

6100 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: MyDemoWordCount

 

 

Apache Storm

8080

Apache Storm

一个storm任务叫topology 并且可以级联

Apache Storm

 

Apache Storm

 

 

linux查看主机端口进程命令

-- jps 也能看到端口

1、查看主机信息

# more /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 xyn-redhat localhost.localdomain localhost

2、查看进程

# ps -aux|grep java //显示所有进程名中包含java字符串的进程

说明：

grep是查找字符串，把包含有匹配字符串的那行显示出来。

-aux 显示所有包含其他使用者的进程

au(x) 输出格式 :

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

USER: 进程拥有者

PID: pid

%CPU: 占用的 CPU 使用率

%MEM: 占用的内存使用率

VSZ: 占用的虚拟内存大小

RSS: 占用的内存大小

TTY: 终端的次要装置号码 (minor device number of tty)

STAT: 该进程的状态

START: 进程开始时间

TIME: 执行的时间

COMMAND:所执行的指令

3、杀进程

# kill pid //根据进程id杀进程

# killall process_name //kill掉所有进程名为指定参数值的进程

如果在前面加上-9表示强制kill，如：kill -9 2992

4、查看网络状态

netstat -tln 命令是用来查看linux的端口使用情况

netstat -a 查看所有的服务端口

netstat -an | grep 端口号 来检验下是不是已经打开了某某端口

#netstat -an //显示所有跟本机连接的IP以及连接的端口

查看8080端口占用情况

# netstat -aln|grep 8080

tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN

tcp 0 0 192.168.80.196:8080 192.168.80.180:1379 SYN_RECV

相关推荐