multi threading storm Bolt parallelism - multithreading

Question

Components for Testing are
Kafka Procucer that reads a file from machine, file is composed of 1000 lines.
String sCurrentLine;
br = new BufferedReader(new FileReader("D:\\jsonLogTest.txt"));
while ((sCurrentLine = br.readLine()) != null) {
//System.out.println(sCurrentLine);
KeyedMessage<String, String> message =new KeyedMessage<String, String>(TOPIC,sCurrentLine);
producer.send(message);
}
Storm Consumer with Three Bolts, BoltOne is supposed to receive the stream and divide it on two different Streams (Stream1 & Stream2). BoltTwo and BoltThree should subscribe to these Streams.
(In simple words I am looking to process the tuple in BoltOne parley like Bolt2 processes first 500 lines and BolltThree last 500 lines.
Topology
builder.setSpout("line-reader-spout",kafkaSpout,1);
builder.setBolt("bolt-one", new BoltOne(),1).shuffleGrouping("line-reader-spout");
builder.setBolt("bolt-two", new BoltTwo(),1).shuffleGrouping("bolt-one","stream1");
builder.setBolt("bolt-three", new BoltThree(),1).shuffleGrouping("bolt-one","stream2");
BoltOne
collector.emit("stream1", new Values(input.getString(0)));
collector.emit("stream2", new Values(input.getString(0)));
x++;System.out.println("" + x);
collector.ack(input);
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
// TODO Auto-generated method stub
outputFieldsDeclarer.declareStream("stream1", new Fields("field1"));
outputFieldsDeclarer.declareStream("stream2", new Fields("field2"));
}
BoltTwo & BoltThree
public void execute(Tuple input) {
String sentence = input.getString(0);
System.out.println("*********B2*************");
}
StackTrace
*********B2*************
1
*********B3*************
2
*********B2*************
*********B3*************
3
*********B3*************
*********B2*************
4
*********B3*************
*********B2*************
5
*********B2*************
*********B3*************
6
*********B2*************
*********B3*************
7
*********B3*************
*********B2*************
Totally confused with splitting streams and parallelism. Example would be helpful.
Updated Solution I came up with for now :
public void execute(Tuple input) {
#SuppressWarnings("unused")
String sentence = input.getString(0);
if (x%2==0) {
collector.emit("stream1", new Values(input.getString(0)));
}
else{
collector.emit("stream2", new Values(input.getString(0)));
}
x++;
collector.ack(input);
}
I just divided the stream on even-Odd basis, and the time to process becomes half, While the BoltTwo processes a tuple other is processed by BoltThree.

Answer 1

I guess you run everything using LocalCluster. As there are multiple threads running, the output via println(...) is not synchronized and internal buffering can mess up the order of the output... Thus, the stuff you see in not reliable -- the order is only preserved within a single spout/bolt.
Furthermore, what is the behavior you want to get?
Right now, you have
Spout => Bolt1 =+=> Bolt2
+=> Bolt3
Ie, the output of Bolt1 is duplicated and Bolt2 and Bolt3 both receive all output tuple from Bolt1. Thus, Bolt1 counts from 1 to 7, and each output tuple of Bolt1 triggers execute() of Bolt2 and Bolt3.
As Bolt2 and Bolt3 do the same thing, I guess you want to have two copies of the same Bolt and partition the input to both. For this, you only add a single bolt and set the parallelism to 2:
builder.setSpout("line-reader-spout",kafkaSpout,1);
builder.setBolt("bolt-one", new BoltOne(),1).shuffleGrouping("line-reader-spout");
builder.setBolt("bolt-two", new BoltTwo(),2).shuffleGrouping("bolt-one","stream1");
Furthermore, Bolt1 only need to declare a single output stream (and not two). If you declare multiple output streams and write to both, you replicate the data...