I am working on a project where data is read from memory. Some of this data are integers, and there was a problem accessing them at unaligned addresses. My idea would be to use memcpy for that, i.e.
uint32_t readU32(const void* ptr)
{
uint32_t n;
memcpy(&n, ptr, sizeof(n));
return n;
}
The solution from the project source I found is similar to this code:
uint32_t readU32(const uint32_t* ptr)
{
union {
uint32_t n;
char data[4];
} tmp;
const char* cp=(const char*)ptr;
tmp.data[0] = *cp++;
tmp.data[1] = *cp++;
tmp.data[2] = *cp++;
tmp.data[3] = *cp;
return tmp.n;
}
So my questions:
Isn't the second version undefined behaviour? The C standard says in 6.2.3.2 Pointers, at 7:
A pointer to an object or incomplete type may be converted to a pointer to a different
object or incomplete type. If the resulting pointer is not correctly aligned 57) for the
pointed-to type, the behavior is undefined.
As the calling code has, at some point, used a char* to handle the memory, there must be some conversion from char* to uint32_t*. Isn't the result of that undefined behaviour, then, if the uint32_t* is not corrently aligned? And if it is, there is no point for the function as you could write *(uint32_t*) to fetch the memory. Additionally, I think I read somewhere that the compiler may expect an int* to be aligned correctly and any unaligned int* would mean undefined behaviour as well, so the generated code for this function might make some shortcuts because it may expect the function argument to be aligned properly.
The original code has volatile on the argument and all variables because the memory contents could change (it's a data buffer (no registers) inside a driver). Maybe that's why it does not use memcpy since it won't work on volatile data. But, in which world would that make sense? If the underlying data can change at any time, all bets are off. The data could even change between those byte copy operations. So you would have to have some kind of mutex to synchronize access to this data. But if you have such a synchronization, why would you need volatile?
Is there a canonical/accepted/better solution to this memory access problem? After some searching I come to the conclusion that you need a mutex and do not need volatile and can use memcpy.
P.S.:
# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 1581.05
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10
This code
uint32_t readU32(const uint32_t* ptr)
{
union {
uint32_t n;
char data[4];
} tmp;
const char* cp=(const char*)ptr;
tmp.data[0] = *cp++;
tmp.data[1] = *cp++;
tmp.data[2] = *cp++;
tmp.data[3] = *cp;
return tmp.n;
}
passes the pointer as a uint32_t *. If it's not actually a uint32_t, that's UB. The argument should probably be a const void *.
The use of a const char * in the conversion itself is not undefined behavior. Per 6.3.2.3 Pointers, paragraph 7 of the C Standard (emphasis mine):
A pointer to an object type may be converted to a pointer to a
different object type. If the resulting pointer is not correctly
aligned for the referenced type, the behavior is undefined.
Otherwise, when converted back again, the result shall compare
equal to the original pointer. When a pointer to an object is
converted to a pointer to a character type, the result points to the
lowest addressed byte of the object. Successive increments of the
result, up to the size of the object, yield pointers to the remaining
bytes of the object.
The use of volatile with respect to the correct way to access memory/registers directly on your particular hardware would have no canonical/accepted/best solution. Any solution for that would be specific to your system and beyond the scope of standard C.
Implementations are allowed to define behaviors in cases where the Standard does not, and some implementations may specify that all pointer types have the same representation and may be freely cast among each other regardless of alignment, provided that pointers which are actually used to access things are suitably aligned.
Unfortunately, because some obtuse compilers compel the use of "memcpy" as an
escape valve for aliasing issues even when pointers are known to be aligned,
the only way compilers can efficiently process code which needs to make
type-agnostic accesses to aligned storage is to assume that any pointer of a type requiring alignment will always be aligned suitably for such type. As a result, your instinct that approach using uint32_t* is dangerous is spot on. It may be desirable to have compile-time checking to ensure that a function is either passed a void* or a uint32_t*, and not something like a uint16_t* or a double*, but there's no way to declare a function that way without allowing a compiler to "optimize" the function by consolidating the byte accesses into a 32-bit load that will fail if the pointer isn't aligned.
Related
I'm relatively new to C++ and I searched for an answer to my question, however I got more confused. As I understand, string literals must be pointed by "const" pointers, since are considered to be readable only. I also understand the pointer itself is not constant (and could be changed), but actually it is pointing to a string constant.I also understand that the string itself cannot be modified. So in this example:
const char* cstr="string";
*cstr = 'a';
I get an error: "assignment of read-only location."
Now, if I define my C-string as following, and define a pointer to it, I'll be able to change the string:
char str[7]="string";
char* cstr = str;
*cstr = 'a';
cout << cstr <<endl;
the string will be modified (output --> a), means the first element of the string is changes. My two questions are:
1- why in the second example I am able to modify the C-string but in the first case I cannot make any changes to the string? 2- In both cases I am using pointers, but in the first case I should Use constant char pointer?
When you use the syntax
const char* cstr="string";
C++ defines:
An array of 7 character in the read-only section of memory, with the contents string\0 in it.
pointer on the stack (or in the writable global section of memory), with the address of that array.
However, when you use the syntax:
char str[7]="string";
C++ defines:
An array of 7 character on the stack (or in the writable global section of memory), with the contents "string\0" in it.
In the first case, the actual values are in read-only memory, so you can't change them. In the second case, they are in writable memory (stack or global).
C++ tries to enforce this semantic, so if the definition is read-only memory, you should use a const pointer.
Note that not all architectures have read-only memory, but because most of them do, and C++ might want to use the read-only memory feature (for better correctness), then C++ programmers should assume (for the purpose of pointer types) that constants are going to be placed in read-only memory.
I read the book "Linux Kernel Development", and find some functions that make me confused, listed as bellow:
struct page *alloc_pages(gfp_t gfp_mask, unsigned int order)
void __free_pages(struct page *page, unsigned int order)
unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)
void free_pages(unsigned long addr, unsigned int order)
The problem is the use of the two underline in the function name, and how the function pairs.
1. when will the linux kernel uses two underline in its function name?
2. why alloc_pages is paired with __free_pages, but not free_pages?
As you can notice:
alloc_pages() / __free_pages() takes "page *" (page descriptor) as argument.
They are ususally used internally by some infrastrcture kernel code, like page fault handler, which wish to manipulate page descriptor instead of memory block content.
__get_free_pages() / free_pages() takes "unsigned long" (virtual address of memory block) as argument
They could be used by code which wish to use the memory block itself, after allocation, you can read / write to this memory block.
As for their name and double underscore "__", you don't need to bother too much. Sometimes kernel functions were named casually without too much consideration when they were first written. And when people think of that the names are not proper, but later those functions are already used wildly in kernel, and kernel guys are simply lazy to change them.
I have seen that __iomem is used to store the return type of ioremap(), but I have used u32 in ARM architecture for it and it works well.
So what difference does __iomem make here? And in which circumstances should I use it exactly?
Lots of type casts are going to just "work well". However, this is not very strict. Nothing stops you from casting a u32 to a u32 * and dereference it, but this is not following the kernel API and is prone to errors.
__iomem is a cookie used by Sparse, a tool used to find possible coding faults in the kernel. If you don't compile your kernel code with Sparse enabled, __iomem will be ignored anyway.
Use Sparse by first installing it, and then adding C=1 to your make call. For example, when building a module, use:
make -C $KPATH M=$PWD C=1 modules
__iomem is defined like this:
# define __iomem __attribute__((noderef, address_space(2)))
Adding (and requiring) a cookie like __iomem for all I/O accesses is a way to be stricter and avoid programming errors. You don't want to read/write from/to I/O memory regions with absolute addresses because you're usually using virtual memory. Thus,
void __iomem *ioremap(phys_addr_t offset, unsigned long size);
is usually called to get the virtual address of an I/O physical address offset, for a specified length size in bytes. ioremap() returns a pointer with an __iomem cookie, so this may now be used with inline functions like readl()/writel() (although it's now preferable to use the more explicit macros ioread32()/iowrite32(), for example), which accept __iomem addresses.
Also, the noderef attribute is used by Sparse to make sure you don't dereference an __iomem pointer. Dereferencing should work on some architecture where the I/O is really memory-mapped, but other architectures use special instructions for accessing I/Os and in this case, dereferencing won't work.
Let's look at an example:
void *io = ioremap(42, 4);
Sparse is not happy:
warning: incorrect type in initializer (different address spaces)
expected void *io
got void [noderef] <asn:2>*
Or:
u32 __iomem* io = ioremap(42, 4);
pr_info("%x\n", *io);
Sparse is not happy either:
warning: dereference of noderef expression
In the last example, the first line is correct, because ioremap() returns its value to an __iomem variable. But then, we deference it, and we're not supposed to.
This makes Sparse happy:
void __iomem* io = ioremap(42, 4);
pr_info("%x\n", ioread32(io));
Bottom line: always use __iomem where it's required (as a return type or as a parameter type), and use Sparse to make sure you did so. Also: do not dereference an __iomem pointer.
Edit: Here's a great LWN article about the inception of __iomem and functions using it.
Simple, Straight and Short (S3) Explanation.
There is an article https://lwn.net/Articles/653585/ for more details.
I have recently been reading about a family of automatic memory management techniques that rely on storing information in the pointer returned by the allocator, i.e. few bits of header e.g. to differentiate between pointers or to store thread-related information (note that I'm not talking about limited-field reference counting here, only immutable information).
I'd like to toy with these techniques. Now, to implement them, I need to be able to return pointers with a specific shape from my allocator. I suppose I could play with the least weight bits but this would require padding that looks extremely memory consuming, so I believe that I should play with the heaviest bits. However, I have no good idea on how to do this. Is there a way for me to, call malloc or malloc_create_zone or some related function and request a pointer that always starts with the given bits?
Thanks everyone!
The amount of information you can actually store in a pointer is pretty limited (typically one or two bits per pointer). And every attempt to dereference the pointer has to first mask out the magic information. The technique is often called tagging, BTW.
#define TAG_MASK 0x3
#define CONS_TAG 0x1
#define STRING_TAG 0x2
#define NUMBER_TAG 0x3
typedef uintptr_t value_t;
typedef struct cons {
value_t car;
value_t cdr;
} cons_t;
value_t
create_cons(value_t t1, value_t t2)
{
cons_t* pair = malloc(sizeof(cons_t));
value_t addr = (value_t)pair;
pair->car = t1;
pair->cdr = t2;
return addr | CONS_TAG;
}
value_t
car_of_cons(value_t v)
{
if ((v % TAG_MASK) != CONS_TAG) error("wrong type of argument");
return ((cons_t*) (v & ~TAG_MASK))->car;
}
One advantage of this technique is, that you can directly infer the type of the object from the pointer itself. You don't need to dereference it (say, in order to read a special type field or similar). Many language implementations using this scheme also have a special tag combination for "immediate" numbers and other small values, which can be represented direcly using the "pointer".
The disadvatage is, that the amount of information, which can be stored, is pretty limited. Also, as the example code shows, you have to be aware of the tagging in every access to the object, and need to "untag" the pointer before actually using it.
The use of the least significant bits for tagging stemms from the observation, that on most platforms, all pointer to malloced memory is actually aligned on a non-byte boundary (usually 8 bytes), so the least significant bits are always zero.
This question is not about any error I'm currently seeing, it's more about theory and getting educated on the variations in HW architecture design and implementation.
Scenario 1: Assuming a 16-bit processor with 16-bit registers, 16-bit addressing, and sizeof(int) = 16 bits:
unsigned int a, b, c, d;
a=0xFFFF;
b=0xFFFF;
c=a+b;
Is it possible for the memory location next to c to be overwritten? (In this case I would expect that an overflow flag would be raised during the add operation, and c either remain unchanged or be filled with undefined data.)
Scenario 2: Assuming a 32-bit processor with 32-bit registers, 32-bit addressing, sizeof(int) = 32 bits, and sizeof(short int)=16 bits:
unsigned int a, b;
unsigned short int c, d;
a=0xFFFF;
b=0xFFFF;
c=a+b;
Is it possible for the memory location next to c to be overwritten? (I would expect that no overflow flag be raised during the add operation, but whether or not a memory access or overflow flag be raised during the assignment operation would depend on the actual design and implementation of the HW. If d was located in the upper 16 bits of the same 32-bit address location (probably not even possible with 32-bit addressing), it might be overwritten.)
Scenario 3: Assuming a 32-bit processor with 32-bit registers, 16-bit addressing, sizeof(int) = 32 bits, and sizeof(short int)=16 bits:
unsigned int a, b;
unsigned short int c, d;
a=0xFFFF;
b=0xFFFF;
c=a+b;
Is it possible for the memory location next to c to be overwritten? (I would expect some overflow flag or memory violation flag to be raised during the type conversion and assignment operation.)
Scenario 4: Assuming a 32-bit processor with 32-bit registers, 32-bit addressing, and sizeof(int) = 32 bits:
unsigned int a, b;
struct {
unsigned int c:16;
unsigned int d:16;
} e;
a=0xFFFF;
b=0xFFFF;
e.c=a+b;
Is it possible for the memory location next to c, namely d, to be overwritten? (In this case, since c and d are expected to reside in the same 32-bit address and both are technically 32-bit integers, it's conceivable that no overflow flags be raised during addition or assignment, and d could be affected.)
I have not tried to test this on actual hardware because my question is more about theory and possible variations in design and implementation. Any insight would be appreciated.
Is it possible for a binary integer operation that results in an overflow to overwrite adjacent memory?
Are there currently any HW implementations that suffer from similar memory overwrite problems, or have there been systems in the past that had this problem?
What devices do typical processors use to guard against neighboring memory from being overwritten by arithmetic and assignment operations?
None of your 1,2,3,4 will result in any memory corruption, or writing "past" the storage for the integer location you are performing arithmetic on. C specifies what should happen on an overflow of unsigned integers. This is assuming the compiler produces the code it's supposed to. Nothing will guard you acainst a buggy compiler that generates code which copies 4 bytes into a 2 byte variable.
Here's what C99 (6.2.5) says:
"A computation involving unsigned operands can never overflow,
because a result that cannot be represented by the resulting unsigned integer type is
reduced modulo the number that is one greater than the largest value that can be
represented by the resulting type."
So, it's well defined what will happen when you try to "overflow" an unsigned integer.
Now, if your integers had been signed integers, it is an other story. According to C, overflowing a signed integer results in undefined behavior. And undefined behavior means anything, including memory corruption, could occur. I have not yet seen a C compiler that would corrupt anything regarding overflowing an integer though.
What devices do typical processors use to guard against neighboring memory from being overwritten by arithmetic and assignment operations?
There's no guards against neighboring memory regarding assignment and arithmetic operations. The processor simply executes the machine code instructions given to it (And if those instructions overwrites memory it was not suppoed to do as expressed in a higher level language, the processor does not care).
At a slightly different level, the CPU might issue traps if it cannot carry out the operation(e.g. the operation specified memory location that does not exist), or tries to do some illegal operation (e.g. division by zero, or encounters an op code the processor does not understand, or tries to do unaligned access to data).
The addition operation is processed inside the processor afaik, so whatever you do, the add operation will be done inside the CPU (in the ALU more precisely)
The overflow register would be set in case of overflow, and the result will still be in the register, and then copied back to your memory location without risk of corrupting adjacent memory.
This is how the code would (sort of) be translated in asm:
mov ax, ptr [memory location of a]
mov bx, ptr [memory location of b]
add ax,bx
mov ptr [memory location of c], ax
so as you can see, c would only hold what was in ax (which is of a known and fixed size) no matter if an overflow occurred or not
In C, the behavior of overflows with unsigned types is well-defined. Any implementation in which overflow causes any result other than what the standard predicts is non-compliant.
The behavior of overflows with signed types is undefined. While the most common effects of an overflow would either be the assignment of some erroneous value or an outright crash, nothing in the C standard guarantees that the processor won't trigger an overflow fault in a way that the compiled code tries to recover from, but which for some reason trashes the contents of a stack variable or a register. It's also possible that an overflow could cause a flag to get set in a way that code doesn't expect, and that such a flag might cause erroneous behavior on future computations.
Other languages may have different semantics for what happens when an overflow occurs.
note: I've seen processors which trap on overflow, processors where unexpected traps that happen at the same time as an external interrupt may cause data corruption, and processors where an unexpected unsigned overflow could cause a succeeding computation to be off by one. I don't know of any where a signed overflow would latch a flag that would interfere with subsequent calculations, but some DSP's have interesting overflow behaviors, so I wouldn't be surprised if one exists.