PHP have an internal data-structure called smart string (smart_str?), where they store both length and buffer size. That is, more memory than the length of the string is allocated to improve concatenation performance. Why isn't this data-structure used for the actual PHP strings? Wouldn't that lead to fewer memory allocations and better performance?
Normal PHP strings (as of PHP 7) are represented by the zend_string type, which includes both the length of the string and its character data array. zend_strings are usually allocated to fit the character data precisely (alignment notwithstanding): They will not leave place to append additional characters.
The smart_str structure includes a pointer to a zend_string and an allocation size. This time, the zend_string will not be precisely allocated. Instead the allocation will be made too large, so that additional characters can be appended without expensive reallocations.
The reallocation policy for smart_str is as follows: First, it will be allocated to have a total size of 256 bytes (minus the zend_string header, minus allocator overhead). If this size is exceeded it will be reallocated to 4096 bytes (minus overhead). After that, the size will increase in increments of 4096 bytes.
Now, imagine that we replace all strings with smart_strings. This would mean that even a single character string would have a minimum allocation size of 256 bytes. Given that most strings in use are small, this is an unacceptable overhead.
So essentially, this is a classic performance/memory tradeoff. We use a memory-compact representation by default and switch to a faster, but less memory-effective representation in the cases that benefit most from it, i.e. cases where large strings are constructed from small parts.
Related
I want to create an empty list of lists:
shape = (70000,70000)
corr = np.empty(shape).tolist()
How can I know how much RAM I need to hold this list using windows operating system (64 bit)?
This will create a list-of-lists-of-floats. About half of the RAM used is for the floats themselves and half is for the references to them. The size of each reference is 8 bytes and the size of each float is also 8 bytes. That makes 70000 * 70000 * 8 * 2 bytes (approx 80G).
Lists look like this in memory:
image source: here
The 70001 lists objects themselves also have overhead (they maintain pointer into storage array, and their own length), but this will be negligible in comparison (probably ~4 MB).
Also note that Python lists overallocate space by an implementation-dependent factor, so consider these numbers a lower bound. Memory is over-allocated so that there are always some free slots available, which makes appends and inserts faster. The space allocated increased by about 12.5% when full.
In node.js, what is the difference between memory allocated in the node.js heap vs. memory reported as "external" as seen in process.memoryUsage()? As there advantages or disadvantages to one over the other? If you were going to have a particularly large object (gigabytes), does it matter which one is used? That doc only says this:
external refers to the memory usage of C++ objects bound to JavaScript
objects managed by V8.
It doesn't tell you what any consequences or tradeoffs might be of doing allocations that use this "external" memory instead of the heap memory.
I'm examining tradeoffs between using a large Array (regular Javascript Array object) vs. a large Uint32Array object. Apparently, an Array is always allocated in the heap, but a Uint32Array is always allocated from "external" memory (not in the heap).
Here the background. I have an object that needs a fairly large array of 32 bit integers as part of what it does. This array can be as much as a billion units long. I realized that I could cut the storage occupied by the array in half if I switched to a Uint32Array instead of a regular Array because a regular Array stores things as double precision floats (64 bits long) vs. a Uint32Array stores things as 32 bit values.
In a test jig, I measured that the Uint32Array does indeed use 1/2 the total storage for the same number of units long (32-bits per unit instead of 64-bits per unit). But, when looking at the results of process.memoryUsage(), I noticed that the Uint32Array storage is in the "external" bucket of storage whereas the array is in the "heapUsed" bucket of storage.
To add to the context a bit more, because a Uint32Array is not resizable, when I do need to change its size, I have to allocate a new Uint32Array and then copy data over from the original (which I suppose could lead to more memory fragmentation opportunites). With an Array, you can just change the size and the JS engine takes care of whatever has to happen internally to adjust to the new size.
What are the advantages/disadvantages of making a large storage allocation that's "external" vs. in the "heap"? Any reason to care?
According to the Rust Reference:
The isize type is a signed integer type with the same number of bits as the platform's pointer type. The theoretical upper bound on object and array size is the maximum isize value. This ensures that isize can be used to calculate differences between pointers into an object or array and can address every byte within an object along with one byte past the end.
This obviously constrain an array to at most 2G elements on 32 bits system, however what is not clear is whether an array is also constrained to at most 2GB of memory.
In C or C++, you would be able to cast the pointers to the first and one past last elements to char* and obtain the difference of pointers from those two; effectively limiting the array to 2GB (lest it overflow intptr_t).
Is an array in 32 bits also limited to 2GB in Rust? Or not?
The internals of Vec do cap the value to 4GB, both in with_capacity and grow_capacity, using
let size = capacity.checked_mul(mem::size_of::<T>())
.expect("capacity overflow");
which will panic if the pointer overflows.
As such, Vec-allocated slices are also capped in this way in Rust. Given that this is because of an underlying restriction in the allocation API, I would be surprised if any typical type could circumvent this. And if they did, Index on slices would be unsafe due to pointer overflow. So I hope not.
It might still not be possible to allocate all 4GB for other reasons, though. In particular, allocate won't let you allocate more than 2GB (isize::MAX bytes), so Vec is restricted to that.
Rust uses LLVM as compiler backend. The LLVM instruction for pointer arithmetic (GetElementPtr) takes signed integer offsets and has undefined behavior on overflow, so it is impossible to index into arrays larger than 2GB when targeting a 32-bit platform.
To avoid undefined behavior, Rust will refuse to allocate more than 2 GB in a single allocation. See Rust issue #18726 for details.
How to increase the size of the CString, if CString Object get maximum size. or tell me the function which can hold maximum data more than the CString
CString uses heap allocation for the string buffer so actual limit for the string length depends on a number of conditions and is some hundreds megabytes.
In general, each time the string needs to grow its buffer it allocates a new buffer greater then the previous one - there's a strategy for how to determine the new size of the buffer. Depending on actual amount of available memory in the system this reallocation may either fail or succeed. If it fails you have very little options of what you can do - the best choice is usually to restart the program.
For the task you solve - working with a COM port - you can use an MFC::CArray which is very convenient to use as a variable size array. You could also use std::vector for the same.
In CString, the string actual size and allocated buffer are held by signed ints (check out CStringData). The string buffer itself is dynamically allocated. This means the theoretical limit is 2^31 characters. Practically, on a 32 bit environment you'll be able to get much less due to memory fragmantation. Also, if you're using Unicode CString, each character is two bytes, which means the CString buffer will hold less text. On a 64 bit environment you might be able to get as much as 2^31 characters.
Having that said, are you really trying to work with strings that long? There's probably a lot to do before you hit the CString length limit.
Why is there a hardcoded chunk limit (.5 meg after compression) in memcached? Has anyone recompiled theirs to up it? I know I should not be sending big chunks like that around, but these extra heavy chunks happen for me from time to time and wreak havoc.
This question used to be in the official FAQ
What are some limits in memcached I might hit? (Wayback Machine)
To quote:
The simple limits you will probably see with memcache are the key and
item size limits. Keys are restricted to 250 characters. Stored data
cannot exceed 1 megabyte in size, since that is the largest typical
slab size."
The FAQ has now been revised and there are now two separate questions covering this:
What is the maxiumum key length? (250 bytes)
The maximum size of a key is 250 characters. Note this value will be
less if you are using client "prefixes" or similar features, since the
prefix is tacked onto the front of the original key. Shorter keys are
generally better since they save memory and use less bandwidth.
Why are items limited to 1 megabyte in size?
Ahh, this is a popular question!
Short answer: Because of how the memory allocator's algorithm works.
Long answer: Memcached's memory storage engine (which will be
pluggable/adjusted in the future...), uses a slabs approach to memory
management. Memory is broken up into slabs chunks of varying sizes,
starting at a minimum number and ascending by a factorial up to the
largest possible value.
Say the minimum value is 400 bytes, and the maximum value is 1
megabyte, and the factorial is 1.20:
slab 1 - 400 bytes slab 2 - 480 bytes slab 3 - 576 bytes ... etc.
The larger the slab, the more of a gap there is between it and the
previous slab. So the larger the maximum value the less efficient the
memory storage is. Memcached also has to pre-allocate some memory for
every slab that exists, so setting a smaller factorial with a larger
max value will require even more overhead.
There're other reason why you wouldn't want to do that... If we're
talking about a web page and you're attempting to store/load values
that large, you're probably doing something wrong. At that size it'll
take a noticeable amount of time to load and unpack the data structure
into memory, and your site will likely not perform very well.
If you really do want to store items larger than 1MB, you can
recompile memcached with an edited slabs.c:POWER_BLOCK value, or use
the inefficient malloc/free backend. Other suggestions include a
database, MogileFS, etc.