I am logging market data with Poco asynchronously using FileChannel and AsyncChannel, which is creating huge amounts of log entries per second.
It appears that Poco writes every single message separately to file and does not buffer.
I believe this is putting considerable strain on my HDD/filesystem, and I had an application crash which I believe is related.
Is there anyway to have Poco only save the log to disk in say 1Mb increments, and also write anything remaining in the buffer to file on close of the logger?
And separately, is there any chance this is going to create huge numbers of threads? From what i've read the AsyncChannel just puts messages into a queue, so I guess only 1 additional thread is created?
Following is basically the code I am using:
#include "Poco/Message.h"
#include "Poco/FormattingChannel.h"
#include "Poco/PatternFormatter.h"
#include "Poco/Logger.h"
#include "Poco/FileChannel.h"
#include "Poco/AutoPtr.h"
#include "Poco/AsyncChannel.h"
class APocoClass
{
private:
Poco::AutoPtr<Poco::FileChannel> pFileChannel;
Poco::AutoPtr<Poco::PatternFormatter> pPF;
Poco::AutoPtr<Poco::FormattingChannel> pFormattingChannel;
Poco::AutoPtr<Poco::AsyncChannel> pFileChannelAsync;
Poco::Logger & _pocoLogger;
public:
APocoClass() :
pFileChannel(new Poco::FileChannel()),
pPF(new Poco::PatternFormatter("%Y%m%d %H:%M:%S.%F: %t")),
pFormattingChannel(new Poco::FormattingChannel(pPF, pFileChannel)),
pFileChannelAsync(new Poco::AsyncChannel(pFormattingChannel)),
_pocoLogger(Poco::Logger::create("PocoLogger", pFileChannelAsync, Poco::Message::PRIO_INFORMATION))
{
pFileChannelAsync->setProperty("priority", "lowest");
pFileChannel->setProperty("path", "MsgStorage/poco.log");
pFileChannel->setProperty("rotation", "daily");
pFileChannel->setProperty("times", "utc");
pFileChannel->setProperty("archive", "timestamp");
}
~APocoClass() {
_pocoLogger.shutdown();
_pocoLogger.close();
pFileChannelAsync = nullptr;
pFileChannel = nullptr;
}
//following is called every time we have a new market data message to log
void MessageReceived(const string & message) {
Poco::Message m("PocoLogger", message, Poco::Message::Priority::PRIO_INFORMATION);
_pocoLogger.log(m);
}
}
Is there anyway to have Poco only save the log to disk in say 1Mb increments, and also write anything remaining in the buffer to file on close of the logger?
You don't have such precise level of control through FileChannel, but you have available flush property (default true) which determines whether buffers are flushed on every log entry. Set it to false and see if things improve.
If that does not satisfy your performance requirements, then you have an option of writing your own wrapper, see LogStream for an example - obviously, you'll want to implement your own logic for LogStreamBuf::writeToDevice(). (It would be much simpler if library allowed you to simply pass in your own streambuf, but unfortunately it does not.)
And separately, is there any chance this is going to create huge numbers of threads?
No, AsyncChannel will launch itself in a thread and process all notifications (ie. log messages) in that single thread.
Related
The following functions and fields are part of the same class in a Visual Studio DLL. Data is continuously being read and processed using the run function on a thread. However, getPoints is being accessed in a Qt app on a QTimer. I don't wan't to miss a single processed vector, because it seems it could be skipping leading to jumpy data. What's the safest way to get the points to the updated version?
If possible I'd like an answer that uses the C++ standard library as I've been exploring mutex-es, but it still seems to lead to jumpy data.
vector<float> points;
// std::mutex ioMutex;
// function running on a thread
void run(){
while(running){
//ioMutex.lock()
vector<byte> data = ReadData()
points = processData(data);
//ioMutex.unlock()
}
}
vector<float> getPoints(){
return points;
}
I believe there is a mistake in your code. The while loop will consume all the process activity and will not allow proper functionality of other functions. In Qt, in such continuous loops, usually it is a good habit to use the following because it actually gives other process time to access the event buffer properly. If this dll is written in Qt, please add the following within the while loop
QCoreApplication::processEvents();
The safest (and probably easiest) way to deliver your points-data to the main thread is by calling qApp->postEvent() with an object of a custom QEvent-subclass that contains your vector<float> as a member-variable.
That will cause the event(QEvent *) method of (whatever Qt object you specified as the first argument to postEvent()) to be called from inside the main/GUI thread, and so you can override that method to read the vector<float> out of the QEvent-subclassed object and update the GUI with that data.
First of all sorry for my poor English.
I made some iTextSharp review and profiling and found a huge amount of perfomance-related problems especially in multi-thread scenario. I made some massive refactoring which improve perfomance up to 300% (from ~100 seconds to ~25 seconds). There was too much unnecessary locks at one side and some stupid threading errors at other side. For example "Image.cs line 1352"
/// <summary>
/// generates new serial id
/// </summary>
static protected long GetSerialId() {
lock (serialId) {
serialId = (long)serialId + 1L;
return (long)serialId;
}
}
this is NOT a thread safe code since you change lock object. let's say 2 threads (A and B) will wait for lock while another thread (C) is in lock section. So those A nd B threads waits for value "1" but thread C changed them to "2". Once C exit lock race betwen A and B begins (they both can enter a section in some time because "1" is no more locked).
There are much better Interlocked.Increment function
So my changes do
review every lock() into code, replace 90% with more effective Interocked and ReaderWriterLockSlim
since PdfName used extreme often as dictionary key -> review PdfName.GetHashCode (replace with FNV1a hashe), review PdfName.Equals (replace with fast byte array comparition)
review and remove unnececcary lock in PdfObject constructor
replace uneffective walk-over-PdfDictionary pattern:
foreach(var name in dic.Keys){
var obj = dic.Get(name);// unnececarry hashtable lookup here
}
foreach(var kv in dic){ // PdfDictionary now implements IEnumerable{KeyValue}
var name = kv.Key;
var obj = kv.Value;
}
some improvements in IRandomAccessSource. File-based random source are now thread safe and very fast since it used MemoryMapped file instead of file stream
ability to provide my own ZLib compression via global static delegate (replacing managed ZLib to unmananaged got perfomance boost)
Unfortunally I have no time to deal with Github to create pull request etc. Is community interested in my work? I can upload zipped sources somewhere so anybody can help me to create pull request and commit those changes into main repository...
Consider an application with two threads, Producer and Consumer.
Both threads are running approximately equally frequent, multiple times in a second.
Both threads access the same memory region, where Producer writes to the memory, and Consumer reads the current chunk of data and does something with it, without invalidating the data.
A classical approach is this one:
int[] sharedData;
//Called frequently by thread Producer
void WriteValues(int[] data)
{
lock(sharedData)
{
Array.Copy(data, sharedData, LENGTH);
}
}
//Called frequently by thread Consumer
void WriteValues()
{
int[] data;
lock(sharedData)
{
Array.Copy(sharedData, data, LENGTH);
}
DoSomething(data);
}
If we assume that the Array.Copy takes time, this code would run slow, since Producer always has to wait for Consumer during copying and vice versa.
An approach to this problem would be to create two buffers, one which is accessed by the Consumer, and one which is written to by the Producer, and swap the buffers, as soon as writing has finished.
int[] frontBuffer;
int[] backBuffer;
//Called frequently by thread Producer
void WriteValues(int[] data)
{
lock(backBuffer)
{
Array.Copy(data, backBuffer, LENGTH);
int[] temp = frontBuffer;
frontBuffer = backBuffer;
backBuffer = temp;
}
}
//Called frequently by thread Consumer
void WriteValues()
{
int[] data;
int[] currentFrontBuffer = frontBuffer;
lock(currentForntBuffer)
{
Array.Copy(currentFrontBuffer , data, LENGTH);
}
DoSomething(currentForntBuffer );
}
Now, my questions:
Is locking, as shown in the 2nd example, safe? Or does the change of references introduce problems?
Will the code in the 2nd example execute faster than the code in the 1st example?
Are there any better methods to efficiently solve the problem described above?
Could there be a way to solve this problem without locks? (Even if I think it is impossible)
Note: this is no classical producer/consumer problem: It is possible for Consumer to read the values multiple times before Producer writes it again - the old data stays valid until Producer writes new data.
Is locking, as shown in the 2nd example, safe? Or does the change of references introduce problems?
As far as I can tell, because reference assignment is atomic, this may be safe but not ideal. Because the WriteValues() method reads from frontBuffer without a lock or memory barrier forcing a cache refresh, there no guarantee that the variable will ever be updated with new values from main memory. There is then a potential to continuously read the stale, cached values of that instance from the local register or CPU cache. I'm unsure of whether the compiler/JIT might infer a cache refresh anyway based on the local variable, maybe somebody with more specific knowledge can speak to this area.
Even if the values aren't stale, you may also run into more contention than you would like. For example...
Thread A calls WriteValues()
Thread A takes a lock on the instance in frontBuffer and starts copying.
Thread B calls WriteValues(int[])
Thread B writes its data, moves the currently locked frontBuffer instance into backBuffer.
Thread B calls WriteValues(int[])
Thread B waits on the lock for backBuffer because Thread A still has it.
Will the code in the 2nd example execute faster than the code in the 1st example?
I suggest that you profile it and find out. X being faster than Y only matters if Y is too slow for your particular needs, and you are the only one who knows what those are.
Are there any better methods to efficiently solve the problem described above?
Yes. If you are using .Net 4 and above, there is a BlockingCollection type in System.Collections.Concurrent that models the Producer/Consumer pattern well. If you consistently read more than you write, or have multiple readers to very few writers, you may also want to consider the ReaderWriterLockSlim class. As a general rule of thumb, you should do as little within a lock as you can, which will also help to alleviate your time issue.
Could there be a way to solve this problem without locks? (Even if I think it is impossible)
You might be able to, but I wouldn't suggest trying that unless you are extremely familiar with multi-threading, cache coherency, and potential compiler/JIT optimizations. Locking will most likely be fine for your situation and it will be much easier for you (and others reading your code) to reason about and maintain.
For example, I have multi-threaded application which can be presented as:
Data bigData;
void thread1()
{
workOn(bigData);
}
void thread2()
{
workOn(bigData);
}
void thread3()
{
workOn(bigData);
}
There are few threads that are working on data. I could leave it as it is, but the problem is that sometimes (very seldom) data are modified by thread4.
void thread4()
{
sometimesModifyData(bigData);
}
Critical sections could be added there, but it would make no sense to multi-threading, because only one thread could work on data at the same time.
What is the best method to make it sense multi-threading while making it thread safe?
I am thinking about kind of state (sempahore?), that would prevent reading and writing at the same time but would allow parallel reading.
This is called a readers–writer lock. You could implement what is called a mutex to make sure no one reads when write is going on and no one writes when reads are going on. One way to solve the problem would be to have flags. If the writer is got something to modify, then switch on a lock. Upon which NO MORE readers will get to read and after all the current readers have finished, the writer will get to do its job and then again the readers read.
Is there any tool available for tracing communication among threads;
1. running in a single process
2. running in different processes (IPC)
I am presuming you need to trace this for debugging. Under normal circumstances it's hard to do this, without custom written code. For a similar problem that I faced, I had a per-processor tracing buffer, which used to record briefly the time and interesting operation that was performed by the running thread. The log was a circular trace which used to store data like this:
struct trace_data {
int op;
void *data;
struct time t;
union {
struct {
int op1_field1;
int op1_field2;
} d1;
struct {
int op2_field1;
int op2_field2;
} d2
} u;
}
The trace log was an array of these structures of length 1024, one for each processor. Each thread used to trace operations, as well as time to determine causality of events. The fields which were used to store data in the "union" depended upon the operation being done. The "data" pointer's meaning depended upon the "op" as well. When the program used to crash, I'd open the core in gdb and I had a gdb script which would go through the logs in each processor and print out the ops and their corresponding data, to find out the history of events.
For different processes you could do such logging to a file instead - one per process. This example is in C, but you can do this in whatever language you want to use, as long as you can figure out the CPU id on which the thread is running currently.
You might be looking for something like the Intel Thread Checker as long as you're using pthreads in (1).
For communication between different processes (2), you can use Aspect-Oriented Programming (AOP) if you have the source code, or write your own wrapper for the IPC functions and LD_PRELOAD it.
Edit: Whoops, you said tracing, not checking.
It will depend so much on the operating system and development environment that you are using. If you're with Visual Studio, look at the tools in Visual Studio 2010.