Storm-0.10.0集群安装
1. 下载地址:
2. 依赖环境
Zookeeper,Python,Java(JDK 6+)
确定系统自带的Python版本(python -V ),如果是2.6.6+可以不用再安装。
搭建ZooKeeper集群
- [email protected]:/usr/local# wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.7/zookeeper-3.4.7.tar.gz
- [email protected]:/usr/local# tar -zxvf zookeeper-3.4.7.tar.gz
- [email protected]:/usr/local# cd zookeeper-3.4.7/conf
- [email protected]:/usr/local/zookeeper-3.4.7/conf# cp -p zoo_sample.cfg zoo.cfg
- [email protected]:/usr/local/zookeeper-3.4.7/conf# vim zoo.cfg
- # The number of milliseconds of each tick
- tickTime=2000
- # The number of ticks that the initial
- # synchronization phase can take
- initLimit=10
- # The number of ticks that can pass between
- # sending a request and getting an acknowledgement
- syncLimit=5
- # the directory where the snapshot is stored.
- # do not use /tmp for storage, /tmp here is just
- # example sakes.
- dataDir=/opt/zookeeper/data
- # the port at which the clients will connect
- clientPort=2181
- server.1=storm1:7000:7001
- server.2=storm2:7000:7001
- # the maximum number of client connections.
- # increase this if you need to handle more clients
- #maxClientCnxns=60
- #
- # Be sure to read the maintenance section of the
- # administrator guide before turning on autopurge.
- #
- # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
- #
- # The number of snapshots to retain in dataDir
- #autopurge.snapRetainCount=3
- # Purge task interval in hours
- # Set to "0" to disable auto purge feature
- #autopurge.purgeInterval=1
- # 手动创建dataDir目录
- [email protected]:/opt# mkdir zookeeper
- [email protected]:/opt# cd zookeeper
- [email protected]:/opt/zookeeper# mkdir data
- # 在dataDir目录下创建myid文件,写id号,用来标识当前主机
- [email protected]:/opt/zookeeper/data# echo "1" > myid
- # 启动ZooKeeper
- [email protected]:/usr/local/zookeeper-3.4.7# bin/zkServer.sh start
- [email protected]:/usr/local/zookeeper-3.4.7# bin/zkServer.sh start
- # 查看ZooKeeper状态
- [email protected]:/usr/local/zookeeper-3.4.7# bin/zkServer.sh status
- ZooKeeper JMX enabled by default
- Using config: /usr/local/zookeeper-3.4.7/bin/../conf/zoo.cfg
- Mode: follower
- [email protected]:/usr/local/zookeeper-3.4.7# bin/zkServer.sh status
- ZooKeeper JMX enabled by default
- Using config: /usr/local/zookeeper-3.4.7/bin/../conf/zoo.cfg
- Mode: leader
- # 下载安装Storm
- [email protected]:/usr/local# wget wget http://mirrors.cnnic.cn/apache/storm/apache-storm-0.10.0/apache-storm-0.10.0.tar.gz
- [email protected]:/usr/local# tar -zxvf apache-storm-0.10.0.tar.gz
- [email protected]:/usr/local# mv apache-storm-0.10.0 storm-0.10.0
-
[email protected]:/usr/local# vimstorm-0.10.0/conf/storm.yaml
- # 省略...
- ########### These MUST be filled in for a storm configuration
-
#配置zookeeper内的所有节点
- storm.zookeeper.servers:
- - "storm1"
- - "storm2"
-
#配置nimbus所在节点,在集群内选一个作为nimbus,BUT有单点问题。
- nimbus.host: "storm1"
-
- #storm的工作目录,据说上传的JAR包什么的,在里面
- storm.local.dir: "/opt/storm/data"
-
-
#supervisor节点上,每个端口对应一个slot,
- supervisor.slots.port:
- - 6700
- - 6701
- - 6702
- - 6703
- nimbus.childopts: "-Xmx1024m"
- ui.childopts: "-Xmx768m"
-
- #UI的端口号,老实改掉,8080太容易冲突了
-
ui.port:58080
-
- #
- # ##### These may optionally be filled in:
- #
- ## List of custom serializations
- # topology.kryo.register:
- # - org.mycompany.MyType
- # - org.mycompany.MyType2: org.mycompany.MyType2Serializer
- #
- ## List of custom kryo decorators
- # topology.kryo.decorators:
- # - org.mycompany.MyDecorator
- #
- ## Locations of the drpc servers
-
#drpc服务起的节点和端口号
- drpc.servers:
- - "server1"
- - "server2"
- #端口可以不用配置
- # drpc.port: 1234
- ## Metrics Consumers
- # topology.metrics.consumer.register:
- # - class: "backtype.storm.metric.LoggingMetricsConsumer"
- # parallelism.hint: 1
- # - class: "org.mycompany.MyMetricsConsumer"
- # parallelism.hint: 1
- # argument:
- # - endpoint: "metrics-collector.mycompany.org"
- # 创建storm.local.dir目录
- [email protected]:/opt# mkdir storm
- [email protected]:/opt/storm# mkdir data
- [email protected]:/opt/storm/data# pwd
- /opt/storm/data
- # 启动Storm(确保ZooKeeper已正常启动)
- [email protected]:/usr/local/storm-0.10.0# bin/storm nimbus >/dev/null 2>&1 &
- [email protected]:/usr/local/storm-0.10.0# bin/storm supervisor >/dev/null 2>&1 &
- [email protected]:/usr/local/storm-0.10.0# bin/storm ui >/dev/null 2>&1 &
- # 查看启动进程
- [email protected]:~# jps
- 2658 QuorumPeerMain
- 2696 nimbus
- 2813 core
- 3334 Jps
- [email protected]:~# jps
- 2673 supervisor
- 3287 Jps
- 2632 QuorumPeerMain
nimbus: nohup ./bin/storm nimbus &
supervisor:nohup ./bin/storm supervisor&
ui: nohup ./bin/storm ui &
drpc: nohup ./bin/storm drpc &
# 所有节点打开logviewer
nohup ./bin/storm logviewer &
七、遇到的问题
如果出现IPv6引起的连接问题可以修改Storm启动JVM参数,如下:
nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
问题排查:
1:如果发现无法在webui界面查看日志信息
只可能是两个问题
一:可能是logviewer进程没启动
二:主机名和ip的映射关系没有配置
2:服务器的8080端口被占用导致ui进程启动失败
修改storm.yaml文件
ui.port: 8081
最完整的属性配置在storm-core.jar中的defaults.yaml文件中