I have an old application which uses CString through out the code.
Maximum size of the string which is written to CString is 8,9 characters, but I noticed that it allocates more. (at least 128 bytes per CString)
Is there a way to limit the size of CString buffer. Fox example to 64bytes?
Thanks in advance,
No.
In detail:
The CString implementation is internal. You find the code in CSimpleStringT::PrepareWrite2 and in the Reallocate function of the string manager.
PrepareWrite2 allocates the buffer. If there was no buffer before, it requests the exact size. If the buffer is changes. The buffer is newLength*1.5.
Finally the request is passed to the Reallocate function of the string manager. Finally this size is passed to the CRT function realloc.
Keep in mind that the memory manager itself decides again what blocksize is "effective" and might change the size again.
So as I see (in VS-2013/VS-2010) you have no chance to change the blocksize. The job is finally done by realloc. And even this function passes its request to HeapAlloc...
Related
When I read a 16MB file in pieces of 64Kb, and do Buffer.concat on each piece, the latter proves to be incredibly slow, takes a whole 4s to go through the lot.
Is there a better way to concatenate a buffer in Node.js?
Node.js version used: 7.10.0, under Windows 10 (both are 64-bit).
This question is asked while researching the following issue: https://github.com/brianc/node-postgres/issues/1286, which affects a large audience.
The PostgreSQL driver reads large bytea columns in chunks of 64Kb, and then concatenates them. We found out that calling Buffer.concat is the culprit behind a huge loss of performance in such examples.
Rather than concatenating every time (which creates a new buffer each time), just keep an array of all of your buffers and concat at the end.
Buffer.concat() can take a whole list of buffers. Then it's done in one operation. https://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength
If you read from a file and know the size of that file, then you can pre-allocate the final buffer. Then each time you get a chunk of data, you can simply write it to that large 16Mb buffer.
// use the "unsafe" version to avoid clearing 16Mb for nothing
let buf = Buffer.allocUnsafe(file_size)
let pos = 0
file.on('data', (chunk) => {
buf.fill(chunk, pos, pos + chunk.length)
pos += chunk.length
})
if(pos != file_size) throw new Error('Ooops! something went wrong.')
The main difference with #Brad's code sample is that you're going to use 16Mb + size of one chunk (roughly) instead of 32Mb + size of one chunk.
Also, each chunk has a header, various pointers, etc. so you are not unlikely to use 33Mb or even 34Mb... that's a lot more RAM. The amount of RAM copied is otherwise the same. That being said, it could be that Node starts reading the next chunk while you copy so it could make it transparent. When done in one large chunk in the 'end' event, you're going to have to wait for the contact() to complete while doing nothing else in parallel.
In case you are receiving an HTTP POST and are reading it. Remember that you get a Content-Length parameter so you also have the length in that case and can pre-allocate the entire buffer before reading the data.
I have a pipeline, which takes data from webcam and process it.
For the processing i need to pull that buffer to appsink and push it into pipeline by using appsrc element.
While pushing i had used gst_buffer_new_wrapped function.
Then a new buffer is allocated every time i am pushing the data. But how to free that memory is the problem.
I had tried gst_buffer_unref(buffer);
Then got below error.
Error in `./uuHiesSoaServer': free(): invalid pointer: 0x00007fddf52f6000
I had take the data into an unsigned char pointer and then wrapped into a gstbuffer based on the size.
Now how to free the allocated memory?
g_signal_emit_by_name (Source, "push-buffer", Buffer, &ret);
I had used above function for pushing data into Source(appsrc).
That function will continuously call on a separate thread.
When data available to it, then the thread function will create a buffer using
gst_buffer_new_wrapped((void *)data, Size);
When checking in valgrind, for memory leaks, above line was shown as a leak.
How to solve this?
How do you push the buffer into appsrc?
If you use gst_app_src_push_buffer function I guess you do not have to free resources because gst_app_src_push_buffer will own the buffer (which means it also frees it)
Check this example
If you use need-data callback you may need to free data - check this example
HTH
Is it good to construct CString from char[] ?
char R[5000];
CString s = R;
In these lines sometimes I have exception:
Windows has triggered a breakpoint in tst.exe.
This may be due to a corruption of the heap, which indicates a bug in tst.exe or any of the DLLs it has loaded.
This may also be due to the user pressing F12 while tst.exe has focus.
The output window may have more diagnostic information
The CString constructor attempts to copy a null-terminated string from the input argument (more precisely, from the memory address pointed by the input argument).
So it copies characters until reaching a 0 (null) character, and since the contents of R[5000] are not initialized, there's a good chance that none of the characters in it is equal to 0.
If there is no such character within the legal memory region pointed by the input argument, then the CString constructor exceeds that memory region, and most likely causes an illegal memory access.
so basically the method of adding and handling string in general is being replaced with other ways because it causes confusion and waste of resources.
i agree but i want to know what exactly causes this waste of resources...as said here.
'..and you’re running on an implementation that doesn’t have
sophisticated code for handling strings you can end up doing a lot of
wasted allocations...'
Link
how string buffer method avoids this wasteful...
My explanation of this is coming from a Java / .NET background, however the same logic applies.
1. You must learn the concept of mutable and immutable objects...
Objects like Int32 / Integer are mutable objects, meaning they can be changed in their current memory location after instantiation. This is because regardless of the value of the object, it's size in memory does not need to change.
Strings are immutable objects which means that once they are allocated they cannot be changed in their current memory location. This is because by nature a string can be of arbitrary length, and therefore, every time the string changes length, the system/runtime must find a new location in memory to store the string.
2. Concatenation vs. StringBuilder / StringBuffer
Since strings are immutable, every concatenation forces reallocation of memory. Lets assume the following example uses ASCII encoding (1 byte per char)
var message = "Hello World";
At this point, the system has allocated 11 bytes of memory to store your string.
message += "Hello Universe";
At this point, the system must allocate another 14 bytes to your original string. Your existing 11 bytes of memory can no longer store your new string!
Why "sophisticated code for handling strings" (StringBuffer / StringBuilder) helps you!
every time you append a string to the buffer/builder, it allocates the memory once, and keeps a pointer to that string in memory. The next time you allocate a string, it does it in a new location, without affecting the last one. Once you have finished building your string, the buffer/builder, concatenates everything in once pass into a single string, therefore your string allocation is vastly reduced as you are not doing it every time you append something to your buffer/builder!
Example:
StringBuilder builder = new StringBuilder();
builder.Append("Hello World");
At this point the builder has allocated 11 bytes, and leaves that allocation as-is!
builder.Append("Hello Universe");
At this point, the builder allocated another 14 bytes, leaving the last string in tact.
builder.ToString();
At this point the builder concatenates all the strings in memory into one single string!
Summary:
Concatenation is a waste of resources because:
The system/runtime must clean out old, de-referenced memory locations, this takes some CPU time. In Java/.NET its called garbage collection.
Every re-allocation of memory is a waste, until the garbage collector can go and clean it out!
Therefore, concatenation reduces performance of CPU and memory usage!
Is there any simple functions to check how much data is buffered but unread? FD_ISSET only indicates the presence of data in the buffer. Is possible not to create a second buffer in the program for greater control of buffer?
You could use recv() with the MSG_PEEK and MSG_DONTWAIT flags, but there's no firm guarantee that there aren't more bytes available than recv() returned in that case.
Using a buffer within your program is the normal and accepted way to solve the problem.