The following steps are for Ubuntu 12.04 LT or 14.05 LT. Storm has lot of moving parts as of now and the easiest configuration happens on Ubuntu. I tried configuring on CentOS and found quite challenging. Before trying storm configuration on CentOS I suggest you first try on Ubuntu. ——————————- PRE Requisites—————————-
- Make sure your Ubuntu is updated. You can update it using $ sudo apt-get update
- Install your favorite JDK. For example : sudo apt-get install openjdk-6-jdk
$ sudo apt-get install git -y
$ sudo apt-get install libtool -y
$ sudo apt-get install automake -y
$ sudo apt-get install uuid-dev
$ sudo apt-get install g++ -y
$ sudo apt-get install gcc-multilib -y
ZooKeeper provides a service for maintaining centralized information in a distributed environment using a small set of primitives and group services. Storm uses ZooKeeper primarily to coordinate state information such as task assignments, worker status, and topology metrics between nimbus and supervisors in a cluster. 1) Get the Zookeeper Download the zookeeper setup ( latest at the time of writing is : 3.4.6 ). You can download from browser or with wget. wget http://www.eng.lsu.edu/mirrors/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz 2) Extract the tarball. 3) Rename the Zookeeper extracted directory : $ mv zookeeper-3.4.6 zookeeper 4) Optionally : a) Add ZOOKEEPER_HOME under .bashrc b) Add ZOOKEEPER_HOME/bin to the PATH variable 5) Create a data directory on your favorite place. $ mkdir zookeeper-data/ 6) Create a configuration file under ZOOKEEPER_HOME/conf/ directory say zoo.cfg 7) Add ticktime, dataDir and clientPort properties in the zoo.cfg file. 8) Verify that you are able to start the zookeeper server : after starting using : $ zkServer.sh start. ———————- Zero MQ ———————————- Storm internally uses ZerMQ, in the current version it is to be installed explicitly, in the future releases they are planning to include this dependency as a part of the storm distribution. 1) Get the ZeroMQ $ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz 2) Untar the tarball. $ tar –xvf zeromq-2.1.7.tar.gz ————- Configuring ZeroMQ——————- 1) $ cd zeromq-2.1.7 2) $ ./configure 3) $ make 4) $ sudo make install ————————– Java Bindings for ZeroMQ —————————– 1) Get the java binding for ZeroMQ $ git clone https://github.com/nathanmarz/jzmq.git This will create a folder with name jzmq Configurating jzmq: 1) $ cd jzmq 2) $ sed -i ‘s/classdist_noinst.stamp/classnoinst.stamp/g’ src/Makefile.am 3) $ ./autogen.sh 4) $ ./configure 5) $ make 6) $ sudo make install ————————————– Configuring Storm ———————————— 1) Download storm binaries $ wget http://mirror.tcpdiag.net/apache/incubator/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz 2) Untar the tarball $ tar -xvf apache-storm-0.9.1-incubating.tar.gz 2) Rename the extracted directory to Storm $ mv apache-storm-0.9.1-incubating storm 3) Optionally : Add STORM_HOME in .bashrc file. Add STORM_HOME/bin to the PATH. 4) Add a data directory for storm to store the temporary data and topology jars. I am creating under $STORM_HOME $ cd $STORM_HOME $ mkdir data 5) Edit $STORM_HOME/conf/storm.yaml file. 6) Edit the values of various parameters like following : This is very important. And this is the major part of Storm configuration. ————- Start the demeans and verify that installation is successful.——————— Note : If you have not added Storm_Home/bin to the path then you will require to go to STORM_HOME/bin directory and issue the commands on terminal…
1) Start Nimbus :
$ storm nimbus
2) Start Supervisor ( Do not close previous terminal Open another terminal window and type following )
3) Start UI ( Open a new terminal, change the directory to storm and start UI . Don’t close the previous terminal)
4) Check the UI ( Hit the URL with IP Address of the UI Port defined in storm.yaml file)
Troubleshooting Notes :
Most commonly faced issues are two :
1) Exception in thread “main” java.lang.RuntimeException: org.apache.thrift7.transport.TransportException:
java.net.ConnectException: Connection refused
2) Nimbus starts , but stops after few seconds.
The problem could be one of the following
- 1. Nimbus is not started correctly. Or Nimbus stopped some time after starting. Check the nimbus logs for errors.
- Nimbus or zookeeper is not correctly configured. Please check the storm.yaml file.
- dataDir defined in the storm.yaml does not exist or has permission issues.
- storm.local.dir defined in the storm.yaml, does not exist or it has permission issues.
- There could be connectivity issues between the machines. Please check your network settings.
Tagged: Storm configuration single-node