Stack canaries can be disabled by compiler? - security

Question

Who is responsible for inserting the stack canaries in the stack? Is it the OS?
If yes, how can the gcc compiler disable them by using the -fno-stack-protector option? Or it is only a flag created using that option and added to the binary to tell the OS to not insert canaries in the stack where the binary is loaded at runtime?
EDIT: one more question
Who checks the value of the canaries if they were changed over the execution?
Again if inserted by the compiler, how can be checked by the OS? If inserted by the OS how can it be disabled by the compiler (main question)?

Answer 1

Who is responsible for inserting the stack canaries in the stack?
The compiler. The code for creating and checking stack canaries is a subset of the code generated by the compiler from the program source code.
For GCC:
-fstack-protector
Emit extra code to check for buffer overflows, such as stack smashing attacks. This is done by adding a guard variable to functions with vulnerable objects. This includes functions that call alloca, and functions with buffers larger than 8 bytes. The guards are initialized when a function is entered and then checked when the function exits. If a guard check fails, an error message is printed and the program exits.
The aforementioned "guard variable" is commonly referred to as a canary:
The basic idea behind stack protection is to push a "canary" (a randomly chosen integer) on the stack just after the function return pointer has been pushed. The canary value is then checked before the function returns; if it has changed, the program will abort. Generally, stack buffer overflow (aka "stack smashing") attacks will have to change the value of the canary as they write beyond the end of the buffer before they can get to the return pointer. Since the value of the canary is unknown to the attacker, it cannot be replaced by the attack. Thus, the stack protection allows the program to abort when that happens rather than return to wherever the attacker wanted it to go.1
Example program:
Source code:
int test(int i) {
return i;
}
int main(void) {
int x;
int i = 10;
x = test(i);
return x;
}
Function from binary compiled without -fstack-protector-all:
$ objdump -dj .text test | grep -A7 "<test>:"
00000000004004ed <test>:
4004ed: 55 push %rbp
4004ee: 48 89 e5 mov %rsp,%rbp
4004f1: 89 7d fc mov %edi,-0x4(%rbp)
4004f4: 8b 45 fc mov -0x4(%rbp),%eax
4004f7: 5d pop %rbp
4004f8: c3 retq
Function from binary compiled with -fstack-protector-all:
$ objdump -dj .text protected_test | grep -A20 "<test>:"
000000000040055d <test>:
40055d: 55 push %rbp
40055e: 48 89 e5 mov %rsp,%rbp
400561: 48 83 ec 20 sub $0x20,%rsp
400565: 89 7d ec mov %edi,-0x14(%rbp)
400568: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax <- get guard variable value
40056f: 00 00
400571: 48 89 45 f8 mov %rax,-0x8(%rbp) <- save guard variable on stack
400575: 31 c0 xor %eax,%eax
400577: 8b 45 ec mov -0x14(%rbp),%eax
40057a: 48 8b 55 f8 mov -0x8(%rbp),%rdx <- move it to register
40057e: 64 48 33 14 25 28 00 xor %fs:0x28,%rdx <- check it against original
400585: 00 00
400587: 74 05 je 40058e <test+0x31>
400589: e8 b2 fe ff ff callq 400440 <__stack_chk_fail#plt>
40058e: c9 leaveq
40058f: c3 retq
1. "Strong" stack protection for GCC