According to the Rust Reference:
The isize type is a signed integer type with the same number of bits as the platform's pointer type. The theoretical upper bound on object and array size is the maximum isize value. This ensures that isize can be used to calculate differences between pointers into an object or array and can address every byte within an object along with one byte past the end.
This obviously constrain an array to at most 2G elements on 32 bits system, however what is not clear is whether an array is also constrained to at most 2GB of memory.
In C or C++, you would be able to cast the pointers to the first and one past last elements to char* and obtain the difference of pointers from those two; effectively limiting the array to 2GB (lest it overflow intptr_t).
Is an array in 32 bits also limited to 2GB in Rust? Or not?
The internals of Vec do cap the value to 4GB, both in with_capacity and grow_capacity, using
let size = capacity.checked_mul(mem::size_of::<T>())
.expect("capacity overflow");
which will panic if the pointer overflows.
As such, Vec-allocated slices are also capped in this way in Rust. Given that this is because of an underlying restriction in the allocation API, I would be surprised if any typical type could circumvent this. And if they did, Index on slices would be unsafe due to pointer overflow. So I hope not.
It might still not be possible to allocate all 4GB for other reasons, though. In particular, allocate won't let you allocate more than 2GB (isize::MAX bytes), so Vec is restricted to that.
Rust uses LLVM as compiler backend. The LLVM instruction for pointer arithmetic (GetElementPtr) takes signed integer offsets and has undefined behavior on overflow, so it is impossible to index into arrays larger than 2GB when targeting a 32-bit platform.
To avoid undefined behavior, Rust will refuse to allocate more than 2 GB in a single allocation. See Rust issue #18726 for details.
Related
HVM is a functional runtime which represents pointers as 32-bit values. Its allocator reserves a huge (4 GB) buffer preemptively, which it uses to create internal objects. This is not ideal. Instead, I'd like to use the system allocator, but that's not possible, since it returns 64-bit pointers, which may be larger than the space available to store them. Is there any cross-platform way in Rust to allocate a buffer, such that the pointer to the buffer is guaranteed to fit in an u32? In other words, I'm looking for something akin to:
let ptr = Box::new_with_small_ptr(size);
assert!(ptr as u64 + size < u32::MAX);
There isn't, because it's an extremely niche need and requires a lot of care.
It's not as simple as "just returning a low pointer" - you need to actually allocate that space from the OS. Your entry point into that would be mmap. Be prepared to do some low-level work with MAP_FIXED and reading /proc/self/maps, and also implementing an allocator on top of the memory region you get from mmap.
If your concern is just excess memory usage, note that Linux overcommits memory by default - allocating 4GB of memory won't reserve physical memory unless you actually try to use it all.
The following is an excerpt from The Rust Programming Language chapter on ownership:
Now consider the following code fragment:
let v = vec![1, 2, 3];
let mut v2 = v;
The first line allocates memory for the vector object v on the stack
like it does for x above. But in addition to that it also allocates
some memory on the heap for the actual data ([1, 2, 3]). Rust copies
the address of this heap allocation to an internal pointer, which is
part of the vector object placed on the stack (let's call it the data
pointer).
It is worth pointing out (even at the risk of stating the obvious)
that the vector object and its data live in separate memory regions
instead of being a single contiguous memory allocation (due to reasons
we will not go into at this point of time). These two parts of the
vector (the one on the stack and one on the heap) must agree with each
other at all times with regards to things like the length, capacity,
etc.
When we move v to v2, Rust actually does a bitwise copy of the vector
object v into the stack allocation represented by v2. This shallow
copy does not create a copy of the heap allocation containing the
actual data. Which means that there would be two pointers to the
contents of the vector both pointing to the same memory allocation on
the heap. It would violate Rust’s safety guarantees by introducing a
data race if one could access both v and v2 at the same time.
For example if we truncated the vector to just two elements through
v2:
v2.truncate(2);
and v were still accessible we'd end up with an invalid vector since v
would not know that the heap data has been truncated. Now, the part of
the vector v on the stack does not agree with the corresponding part
on the heap. v still thinks there are three elements in the vector and
will happily let us access the non existent element v[2] but as you
might already know this is a recipe for disaster. Especially because
it might lead to a segmentation fault or worse allow an unauthorized
user to read from memory to which they don't have access.
After truncating the vector using v2, the truncated values are updated on the heap memory. v1 still sees the heap memory and after truncation, it sees the new values. So why does the book say
and v were still accessible we'd end up with an invalid vector since v would not know that the heap data has been truncated
What you're missing is that a Vec contains both a pointer to the heap and a len, or length of the heap's data. If v2 truncates the Vec, it's possible that the truncated data has been freed, and v1 still believes that the truncated memory is still part of the vector.
I guess I understand it. The key is to know that Rust has this internal pointer which is again a space on the stack that Rust uses to locate the address of the data on the heap!
If v2 changes this data on the heap, v2's internal pointer gets updated with the new data allocation address, while v1's internal pointer would still refer to the old data allocation address!
When I was looking at the way std::string is implemented in gcc I noticed that sizeof(std::string) is exactly equal to the size of pointer (4 bytes in x32 build, and 8 bytes for x64). As string should hold a pointer to string buffer and its length as a bare minimum, this made me think that std::string object in GCC is actually a pointer to some internal structure that holds this data.
As a consequence when new string is created one dynamic memory allocation should occur (even if the string is empty).
In addition to performance overhead this also cause memory overhead (that happens when we are allocating very small chunk of memory).
So I see only downsides of such design. What am I missing? What are the upsides and what is the reason for such implementation in the first place?
Read the long comment at the top of <bits/basic_string.h>, it explains what the pointer points to and where the string length (and reference count) are stored and why it's done that way.
However, C++11 doesn't allow a reference-counted Copy-On-Write std::string so the GCC implementation will have to change, but doing so would break the ABI so is being delayed until an ABI change is inevitable. We don't want to change the ABI, then have to change it again a few months later, then again. When it changes it should only change once to minimise the hassles for users.
I also want to know whether glibc malloc() does this.
Suppose that you have the structure.
struct S {
short a;
int b;
char c, d;
};
Without alignment, it would be laid out in memory like this (assuming a 32-bit architecture):
0 1 2 3 4 5 6 7
|a|a|b|b|b|b|c|d| bytes
| | | words
The problem is that on some CPU architectures, the instruction to load a 4-byte integer from memory only works on word boundaries. So your program would have to fetch each half of b with separate instructions.
But if the memory was laid out as:
0 1 2 3 4 5 6 7 8 9 A B
|a|a| | |b|b|b|b|c|d| | |
| | | |
Then access to b becomes straightforward. (The disadvantage is that more memory is required, because of the padding bytes.)
Different data types have different alignment requirements. It's common for char to be 1-byte aligned, short to be 2-byte aligned, and 4-byte types (int, float, and pointers on 32-bit systems) to be 4-byte aligned.
malloc is required by the C standard to return a pointer that's properly aligned for any data type.
glibc malloc on x86-64 returns 16-byte-aligned pointers.
Alignment requirements specify what address offsets can be assigned to what types. This is completely implementation-dependent, but is generally based on word size. For instance, some 32-bit architectures require all int variables start on a multiple of four. On some architectures, alignment requirements are absolute. On others (e.g. x86) flouting them only comes with a performance penalty.
malloc is required to return an address suitable for any alignment requirement. In other words, the returned address can be assigned to a pointer of any type. From C99 §7.20.3 (Memory management functions):
The pointer returned if the allocation
succeeds is suitably aligned so that
it may be assigned to a pointer to any
type of object and then used to access
such an object or an array of such
objects in the space allocated (until
the space is explicitly deallocated).
The malloc() documentation says:
[...] the allocated memory that is suitably aligned for any kind of variable.
Which is true for most everything you do in C/C++. However, as pointed out by others, many special cases exist and require a specific alignment. For example, Intel processors support a 256 bit type: __m256, which is most certainly not taken in account by malloc().
Similarly, if you want to allocate a memory buffer for data that is to be paged (similar to addresses returned by mmap(), etc.) then you need a possibly very large alignment which would waste a lot of memory if malloc() was to return buffers always aligned to such boundaries.
Under Linux or other Unix systems, I suggest you use the posix_memalign() function:
int posix_memalign(void **memptr, size_t alignment, size_t size);
This is the most current function that one wants to use for such needs.
As a side note, you could still use malloc(), only in that case you need to allocate size + alignment - 1 bytes and do your own alignment on the returned pointer: (ptr + alignment - 1) & -alignment (not tested, all casts missing). Also the aligned pointer is not the one you'll use to call free(). In other words, you have to store the pointer that malloc() returned to be able to call free() properly. As mentioned above, this means you lose up to alignment - 1 byte per such malloc(). In contrast, the posix_memalign() function should not lose more than sizeof(void*) * 4 - 1 bytes, although since your size is likely a multiple of alignment, you would only lose sizeof(void*) * 2... unless you only allocate such buffers, then you lose a full alignment bytes each time.
If you have particular memory alignemnt needs (for particular hardware or libraries), you can check out non-portable memory allocators such as _aligned_malloc() and memalign(). These can easily be abstracted behind a "portable" interface, but are unfortunately non-standard.
How to increase the size of the CString, if CString Object get maximum size. or tell me the function which can hold maximum data more than the CString
CString uses heap allocation for the string buffer so actual limit for the string length depends on a number of conditions and is some hundreds megabytes.
In general, each time the string needs to grow its buffer it allocates a new buffer greater then the previous one - there's a strategy for how to determine the new size of the buffer. Depending on actual amount of available memory in the system this reallocation may either fail or succeed. If it fails you have very little options of what you can do - the best choice is usually to restart the program.
For the task you solve - working with a COM port - you can use an MFC::CArray which is very convenient to use as a variable size array. You could also use std::vector for the same.
In CString, the string actual size and allocated buffer are held by signed ints (check out CStringData). The string buffer itself is dynamically allocated. This means the theoretical limit is 2^31 characters. Practically, on a 32 bit environment you'll be able to get much less due to memory fragmantation. Also, if you're using Unicode CString, each character is two bytes, which means the CString buffer will hold less text. On a 64 bit environment you might be able to get as much as 2^31 characters.
Having that said, are you really trying to work with strings that long? There's probably a lot to do before you hit the CString length limit.