Multiple thread completion time measurement - multithreading

Question

Use case is: I have a huge log file, which I'm reading on main thread chunk by chunk (equal size, IO read). Every chunk read approximately takes 1s in my test machine. After reading each chunk I'm using a threadpool to create a thread for each chunk to put it in 2 DB instances. Now I have 2 challenges:
I have to alternatively insert chunks into 2 DBS. i.e. odd chunks go to 1st DB and even chunks go to 2nd DB. I don't have anything in the chunk model to denote me the number of chunk on which I can depend. I tried to create a wrapper on that chunk model to have a "chunkCount" but where do I increment the chunkCount?
How do I measure the time for each insert which would be running on different threads from the threadpool?
Following code I tried on experiment basis, but it's not yielding any result:
logEventsChunk = logFetcher.GetNextLogEventsChunk();
chunkModel = new LogEventChunkModel();
stw = new Stopwatch();
chunkModel.ChunkCount = chunkCount;
chunkModel.LogeventChunk = logEventsChunk;
//chunkCount++;
ThreadPool.QueueUserWorkItem(new WaitCallback(delegate(object state)
{ InsertChunk(chunkModel, collection, secondCollection, stw); }), null);
The InsertChunk method is here:
private void InsertChunk(LogEventChunkModel logEventsChunk, MongoCollection<LogEvent> collection, MongoCollection<LogEvent> secondCollection,Stopwatch stw)
{
chunkCount++;
stw.Start();
MongoInsertOptions options = new MongoInsertOptions();
options.WriteConcern = WriteConcern.Unacknowledged;
options.CheckElementNames = true;
string db = string.Empty;
{
//DateTime dtWrite = DateTime.Now;
if (logEventsChunk.ChunkCount % 2 == 0)
{
DateTime dtWrite1 = DateTime.Now;
collection.InsertBatch(logEventsChunk.LogeventChunk.LogEvents, options);
db = "FirstDB";
//Console.WriteLine("Time taken to write the chunk: " + DateTime.Now.Subtract(dtWrite1).TotalSeconds.ToString() + " s. " + db);
}
else
{
DateTime dtWrite2 = DateTime.Now;
secondCollection.InsertBatch(logEventsChunk.LogeventChunk.LogEvents, options);
db = "SecondDB";
//Console.WriteLine("Time taken to write the chunk: " + DateTime.Now.Subtract(dtWrite2).TotalSeconds.ToString() + " s. " + db);
}
Console.WriteLine("Thread Completed: {0} **********", Thread.CurrentThread.GetHashCode() );
stw.Stop();
Console.WriteLine("Time taken to write the chunk: " + stw.ElapsedMilliseconds + " ms. " + db + " Chunk Count: " + logEventsChunk.ChunkCount);
stw.Reset();
//+ "Chunk Count: " + chunkCount.ToString()
//Console.WriteLine("Time taken to write the chunk: " + DateTime.Now.Subtract(dtWrite).TotalSeconds.ToString() + " s. "+db);
//mongoDBInsertionTotalTime += DateTime.Now.Subtract(dtWrite).TotalSeconds;
}
}
Please ignore those commented lines since they are part of some experiments only.

Answer 1

Rather than starting a new thread for each insertion, and trying to make the thread figure out which database to write to, start two persistent threads, each of which writes to a single database. Those threads get their data from queues. This is a pretty standard producer/consumer setup using BlockingCollection<T>.
So, you have:
// Maximum number of items in queue (to avoid out of memory errors)
const int MaxQueueSize = 10000;
BlockingCollection<LogEventChunkModel> Db1Queue = new BlockingCollection<LogEventChunkModel>(MaxQueueSize);
BlockingCollection<LogEventChunkModel> Db2Queue = new BlockingCollection<LogEventChunkModel>(MaxQueueSize);
In your main thread, start the database update threads:
var t1 = new Thread(DbWriteThreadProc);
t1.Start(new Tuple<string, BlockingCollection<LogEventChunkModel>>("FirstDB", Db1Queue));
var t2 = new Thread(DbWriteThreadProc);
t2.Start(new Tuple<string, BlockingCollection<LogEventChunkModel>>("SecondDb", Db2Queue));
Then, begin reading the log file and placing alternate chunks into the queues:
int chunk = 0;
while (!EndOfLogFile)
{
var chunk = GetNextChunk();
if ((chunk % 0) == 0)
Db1Queue.Add(chunk);
else
Db2Queue.Add(chunk);
++chunk;
}
// end of data, so mark the queues as complete
Db1Queue.CompleteAdding();
Db2Queue.CompleteAdding();
// and wait for threads to complete processing the queues
t1.Join();
t2.Join();
Your write thread proc is pretty simple. All it does is service the queue and write to the database:
void DbWriteThreadProc(object state)
{
// passed object is a Tuple<string, BlockingCollection>
// Get the items from it
var threadData = (Tuple<string, BlockingCollection>)state;
string dbName = threadData.Item1;
BlockingCollection<LogEventChunk> queue = threadData.Item2;
// now read the queue and write to the database
foreach (var chunk in queue.GetConsumingEnumerable())
{
var sw = Stopwatch.StartNew();
// write chunk to the database.
sw.Stop();
Console.WriteLine("Time to write = {0:N0} ms", sw.ElapsedMilliseconds);
}
}
GetConsumingEnumerable does a non-busy wait on the queue, so it's not continually polling. The loop will complete when the queue is empty and the queue is marked as complete for adding (which is why the main thread calls CompleteAdding).
This approach has several advantages over what you had. In particular, it simplifies determining which database chunks get written to. In addition, it uses at most three threads and guarantees that chunks are added to the database in the same order in which they were read from the log file. Your approach using QueueUserWorkItem does not guarantee insertion order. It also creates a new thread for each insertion, and could end up with a huge number of concurrent threads.