system delay functions are not producing the correct timings - delay

If anyone can help me with this problem I would be very grateful.
Here's what i know.
I have a custom PCB based around an atmega328p processor.
I am using atmelstudio 7.0.1645 IDE.
I have used the ASF wizard to set up my project.
I am programming using a USBTinyISP.
In my main loop I am toggling an LED and delaying 1 second.
Yet when i check on a scope the LED flashes at a rate of 1.6s
I have checked the fuses and all are set correctly as default (E=0xff, H=0xD9, L=0x62)
this should give a 1MHz clock. (8MHz internal RC plus divide by 8 prescaler)
I am using the ASF's System Clock Control Service and the Delay Routines Service.
calling sysclk_get_main_hz(); returns 16000000.
calling sysclk_get_source_clock_hz(); returns 1000000.
calling sysclk_get_cpu_hz(); returns 1000000.
Which all seems correct to me.
Also if i compile and program a similar project from arduino ide the delay functions are spot on so I feel i can rule out any obvious hardware issues.
I am just using the internal RC anyway.
I have trauled goodle for similar problems but just get the usual Clock setup problems that i feel I have exhausted.
Any extra help would be very much appreciated.
Thanks.
main.c
/**
* \file
*
* \brief Empty user application template
*
*/
/**
* \mainpage User Application template doxygen documentation
*
* \par Empty user application template
*
* Bare minimum empty user application template
*
* \par Content
*
* -# Include the ASF header files (through asf.h)
* -# "Insert system clock initialization code here" comment
* -# Minimal main function that starts with a call to board_init()
* -# "Insert application code here" comment
*
*/
/*
* Include header files for all drivers that have been imported from
* Atmel Software Framework (ASF).
*/
/*
* Support and FAQ: visit Atmel Support
*/
#include <asf.h>
uint32_t main_hz = 0;
uint32_t source_clock_hz = 0;
uint32_t cpu_hz = 0;
int main (void)
{
sysclk_init();
board_init();
/* Insert application code here, after the board has been initialized. */
main_hz = sysclk_get_main_hz();
source_clock_hz = sysclk_get_source_clock_hz();
cpu_hz = sysclk_get_cpu_hz();
while(1){
ioport_toggle_pin(LED_STRIP);
delay_ms(1000);
}
}
conf_clock.h
/**
* \file
*
* \brief Chip-specific system clock manager configuration
*
* Copyright (c) 2014-2015 Atmel Corporation. All rights reserved.
*
* \asf_license_start
*
* \page License
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* 3. The name of Atmel may not be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* 4. This software may only be redistributed and used in connection with an
* Atmel microcontroller product.
*
* THIS SOFTWARE IS PROVIDED BY ATMEL "AS IS" AND ANY EXPRESS OR IMPLIED
* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE
* EXPRESSLY AND SPECIFICALLY DISCLAIMED. IN NO EVENT SHALL ATMEL BE LIABLE FOR
* ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* \asf_license_stop
*
*/
/*
* Support and FAQ: visit Atmel Support
*/
#ifndef CONF_CLOCK_H_INCLUDED
#define CONF_CLOCK_H_INCLUDED
/* ===== System Clock Source Options */
#define SYSCLK_SRC_RC16MHZ 0
#define SYSCLK_SRC_RC128KHZ 1
#define SYSCLK_SRC_TRS16MHZ 2
#define SYSCLK_SRC_RC32KHZ 3
#define SYSCLK_SRC_XOC16MHZ 4
#define SYSCLK_SRC_EXTERNAL 5
/* ===== Select connected clock source */
#define SYSCLK_SOURCE SYSCLK_SRC_RC16MHZ
/* #define SYSCLK_SOURCE SYSCLK_SRC_RC128KHZ */
/* #define SYSCLK_SOURCE SYSCLK_SRC_TRS16MHZ */
/* #define SYSCLK_SOURCE SYSCLK_SRC_XOC16MHZ */
/* ===== System Clock Bus Division Options */
#define CONFIG_SYSCLK_PSDIV SYSCLK_PSDIV_8
#endif /* CONF_CLOCK_H_INCLUDED */
AVRDude Output
avrdude.exe: Version 6.3, compiled on Feb 17 2016 at 09:25:53
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
System wide configuration file is "C:\Program Files (x86)\AVRDude\avrdude.conf"
Using Port : usb
Using Programmer : usbtiny
avrdude.exe: usbdev_open(): Found USBtinyISP, bus:device: bus-0:\\.\libusb0-0001--0x1781-0x0c9f
AVR Part : ATmega328P
Chip Erase delay : 9000 us
PAGEL : PD7
BS2 : PC2
RESET disposition : dedicated
RETRY pulse : SCK
serial program mode : yes
parallel program mode : yes
Timeout : 200
StabDelay : 100
CmdexeDelay : 25
SyncLoops : 32
ByteDelay : 0
PollIndex : 3
PollValue : 0x53
Memory Detail :
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom 65 20 4 0 no 1024 4 0 3600 3600 0xff 0xff
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
flash 65 6 128 0 yes 32768 128 256 4500 4500 0xff 0xff
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
lfuse 0 0 0 0 no 1 0 0 4500 4500 0x00 0x00
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
hfuse 0 0 0 0 no 1 0 0 4500 4500 0x00 0x00
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
efuse 0 0 0 0 no 1 0 0 4500 4500 0x00 0x00
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
lock 0 0 0 0 no 1 0 0 4500 4500 0x00 0x00
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
Programmer Type : USBtiny
Description : USBtiny simple USB programmer, http://www.ladyada.net/make/usbtinyisp/
avrdude.exe: programmer operation not supported
avrdude.exe: Using SCK period of 10 usec
CMD: [ac 53 00 00] [ff fe 53 00]
avrdude.exe: AVR device initialized and ready to accept instructions
Reading | CMD: [30 00 00 00] [00 30 00 1e]
CMD: [30 00 01 00] [00 30 00 95]
################CMD: [30 00 02 00] [00 30 00 0f]
################################## | 100% 0.00s
avrdude.exe: Device signature = 0x1e950f (probably m328p)
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 1, lfuse value: 62
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 2, lfuse value: 62
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 3, lfuse value: 62
avrdude.exe: safemode: lfuse reads as 62
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 1, hfuse value: d9
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 2, hfuse value: d9
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 3, hfuse value: d9
avrdude.exe: safemode: hfuse reads as D9
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 1, efuse value: ff
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 2, efuse value: ff
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 3, efuse value: ff
avrdude.exe: safemode: efuse reads as FF
avrdude.exe: NOTE: "flash" memory has been specified, an erase cycle will be performed
To disable this feature, specify the -D option.
avrdude.exe: erasing chip
CMD: [ac 80 00 00] [00 ac 80 00]
avrdude.exe: Using SCK period of 10 usec
CMD: [ac 53 00 00] [00 ac 53 00]
avrdude.exe: reading input file "\Atmel Studio\7.0\GccBoardProject2\GccBoardProject2\Debug\GccBoardProject2.hex"
avrdude.exe: writing flash (228 bytes):
Writing | CMD: [4c 00 00 00] [92 4c 00 00]
#########################CMD: [4c 00 40 00] [ff 4c 00 40]
######################### | 100% 0.32s
avrdude.exe: 228 bytes of flash written
avrdude.exe: verifying flash memory against \Atmel Studio\7.0\GccBoardProject2\GccBoardProject2\Debug\GccBoardProject2.hex:
avrdude.exe: load data flash data from input file \Atmel Studio\7.0\GccBoardProject2\GccBoardProject2\Debug\GccBoardProject2.hex:
avrdude.exe: input file \Atmel Studio\7.0\GccBoardProject2\GccBoardProject2\Debug\GccBoardProject2.hex contains 228 bytes
avrdude.exe: reading on-chip flash data:
Reading | ################################################## | 100% 0.25s
avrdude.exe: avr_read(): skipping page 2: no interesting data
avrdude.exe: avr_read(): skipping page 3: no interesting data
avrdude.exe: avr_read(): skipping page 4: no interesting data
...
...
avrdude.exe: avr_read(): skipping page 254: no interesting data
avrdude.exe: avr_read(): skipping page 255: no interesting data
avrdude.exe: verifying ...
avrdude.exe: 228 bytes of flash verified
avrdude.exe: reading input file "0xff"
avrdude.exe: writing efuse (1 bytes):
Writing | CMD: [50 08 00 00] [ff 50 08 ff]
################################################## | 100% 0.00s
avrdude.exe: 1 bytes of efuse written
avrdude.exe: verifying efuse memory against 0xff:
avrdude.exe: load data efuse data from input file 0xff:
avrdude.exe: input file 0xff contains 1 bytes
avrdude.exe: reading on-chip efuse data:
Reading | CMD: [50 08 00 00] [00 50 08 ff]
################################################## | 100% 0.00s
avrdude.exe: verifying ...
avrdude.exe: 1 bytes of efuse verified
avrdude.exe: reading input file "0xd9"
avrdude.exe: writing hfuse (1 bytes):
Writing | CMD: [58 08 00 00] [00 58 08 d9]
################################################## | 100% 0.00s
avrdude.exe: 1 bytes of hfuse written
avrdude.exe: verifying hfuse memory against 0xd9:
avrdude.exe: load data hfuse data from input file 0xd9:
avrdude.exe: input file 0xd9 contains 1 bytes
avrdude.exe: reading on-chip hfuse data:
Reading | CMD: [58 08 00 00] [00 58 08 d9]
################################################## | 100% 0.00s
avrdude.exe: verifying ...
avrdude.exe: 1 bytes of hfuse verified
avrdude.exe: reading input file "0x62"
avrdude.exe: writing lfuse (1 bytes):
Writing | CMD: [50 00 00 00] [00 50 00 62]
################################################## | 100% 0.00s
avrdude.exe: 1 bytes of lfuse written
avrdude.exe: verifying lfuse memory against 0x62:
avrdude.exe: load data lfuse data from input file 0x62:
avrdude.exe: input file 0x62 contains 1 bytes
avrdude.exe: reading on-chip lfuse data:
Reading | CMD: [50 00 00 00] [00 50 00 62]
################################################## | 100% 0.00s
avrdude.exe: verifying ...
avrdude.exe: 1 bytes of lfuse verified
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 1, lfuse value: 62
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 2, lfuse value: 62
CMD: [50 00 00 00] [00 50 00 62]
avrdude.exe: safemode read 3, lfuse value: 62
avrdude.exe: safemode: lfuse reads as 62
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 1, hfuse value: d9
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 2, hfuse value: d9
CMD: [58 08 00 00] [00 58 08 d9]
avrdude.exe: safemode read 3, hfuse value: d9
avrdude.exe: safemode: hfuse reads as D9
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 1, efuse value: ff
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 2, efuse value: ff
CMD: [50 08 00 00] [00 50 08 ff]
avrdude.exe: safemode read 3, efuse value: ff
avrdude.exe: safemode: efuse reads as FF
avrdude.exe: safemode: Fuses OK (E:FF, H:D9, L:62)

Related

Why RDATA in EDNS(0) resource record has 1 octet?

I want to understand why EDNS(0) resource records contains an extra octet? I read RFC 6891 and RFC 1035. It says nothing about case when RDLENGHT == 0 but RDATA == "\0".
To test this here python code
import binascii
import socket
def send_udp_message(message, address, port):
"""send_udp_message sends a message to UDP server
message should be a hexadecimal encoded string
"""
message = message.replace(" ", "").replace("\n", "")
server_address = (address, port)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
sock.sendto(binascii.unhexlify(message), server_address)
data, _ = sock.recvfrom(4096)
finally:
sock.close()
return binascii.hexlify(data).decode("utf-8")
def format_hex(hex):
"""format_hex returns a pretty version of a hex string"""
octets = [hex[i:i+2] for i in range(0, len(hex), 2)]
pairs = [" ".join(octets[i:i+2]) for i in range(0, len(octets), 2)]
return "\n".join(pairs)
message = "AA AA 01 00 00 01 00 00 00 00 00 01 " \
"07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01 "
# EDNS(0) resource record
message += "00 00 " # NAME
message += "29 00 " # TYPE
message += "FF 00 00 80 " # TTL
message += "00 00 " # RDLENGTH
# message += "00" # RDATA
response = send_udp_message(message, "8.8.8.8", 53)
print(format_hex(response))
Dns query returns here error. But if uncomment RDATA line it returns success.
You misread section ยง6.1.2 of RFC 6891, so the RFCs are not "lies".
It says:
+------------+--------------+------------------------------+
| Field Name | Field Type | Description |
+------------+--------------+------------------------------+
| NAME | domain name | MUST be 0 (root domain) |
| TYPE | u_int16_t | OPT (41) |
| CLASS | u_int16_t | requestor's UDP payload size |
| TTL | u_int32_t | extended RCODE and flags |
| RDLEN | u_int16_t | length of all RDATA |
| RDATA | octet stream | {attribute,value} pairs |
+------------+--------------+------------------------------+
so 6 pieces of information while your code has only 5 pieces:
# EDNS(0) resource record
message += "00 00 " # NAME
message += "29 00 " # TYPE
message += "FF 00 00 80 " # TTL
message += "00 00 " # RDLENGTH
# message += "00" # RDATA
You are missing the CLASS between TYPE and TTL, hence things are not interpreted the way you think they are.
You are also misreading how the bytes work for the name.
It is not:
00 00 NAME
29 00 TYPE
FF 00 00 80 TTL
00 00 RDLENGTH
but really:
00 NAME (root domain per EDNS(0) specification, which is a sole zero)
00 29 TYPE (41, per specification)
00 FF CLASS (255, considered as payload)
00 00 80 00 TTL, read as 00 = EXTENDED-CODE, 00 = VERSION (mandatory), 80 = DO set plus everything else 0 as the final 00 byte, per specification
00 RDLENGTH
and the final item RDLENGTH is then not properly formatted per the RFC 1035 specification as it is 2 bytes (16 bits).
Once you comment your last line, RDLENGTH becomes 00 00 and then is valid.
And there is indeed no RDATA part in your message.
What you believe being a 00 value in RDATA is in fact the last byte of RDLENGTH but you did not parse the stream correctly.
Had you used dnspython as advised to you, you would have seen the problem immediately with the proper mapping of fields:
In [1]: import dns
In [2]: import dns.message
In [10]: stream = 'AA AA 01 00 00 01 00 00 00 00 00 01 07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01 00 00 29 00 FF 00 00 80 00 00 00'
In [11]: data=''.join(chr(int(x, base=16)) for x in stream.split(' '))
In [12]: m = dns.message.from_wire(data)
In [13]: print m
id 43690
opcode QUERY
rcode NOERROR
flags RD
edns 0
eflags DO
payload 255
;QUESTION
example.com. IN A
;ANSWER
;AUTHORITY
;ADDITIONAL
Had you wanted to do that you could have simulated the expected bytes stream that way:
(starting from your message)
In [31]: m = dns.message.from_wire(data)
In [32]: print m
id 43690
opcode QUERY
rcode NOERROR
flags RD
edns 0
eflags DO
payload 255
;QUESTION
example.com. IN A
;ANSWER
;AUTHORITY
;ADDITIONAL
(creating a new one to look like yours)
In [39]: mm = dns.message.make_query('example.com.', 'A', use_edns=0, payload=255, want_dnssec=True)
In [40]: mm.id=43690
In [41]: print mm
id 43690
opcode QUERY
rcode NOERROR
flags RD
edns 0
eflags DO
payload 255
;QUESTION
example.com. IN A
;ANSWER
;AUTHORITY
;ADDITIONAL
(now looking at its wire representation)
In [46]: print ' '.join(hex(ord(d)) for d in mm.to_wire())
0xaa 0xaa 0x1 0x0 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x1 0x7 0x65 0x78 0x61 0x6d 0x70 0x6c 0x65 0x3 0x63 0x6f 0x6d 0x0 0x0 0x1 0x0 0x1 0x0 0x0 0x29 0x0 0xff 0x0 0x0 0x80 0x0 0x0 0x0
Comparing with your bytestream:
dnspython: AA AA 01 00 00 01 00 00 00 00 00 01 07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01 00 00 29 00 FF 00 00 80 00 00 00
you: AA AA 01 00 00 01 00 00 00 00 00 01 07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01 00 00 29 00 FF 00 00 80 00 00
Note the missing final 00 being in your commented line.
This shows that existing good libraries, as dnspython is, really help sometimes better understanding RFCs or other specifications. And in the world of DNS there are many RFCs, sometimes conflicting between each other, with ambiguous parts, etc. So using existing libraries for tests and/or studying their source code really helps, if you want this advice.

How to make an executable ELF file in Linux using a hex editor?

Just curious. This obviously isn't a very good solution for actual programming, but say I wanted to make an executable in Bless (a hex editor).
My architecture is x86. What's a very simple program I can make? A hello world? An infinite loop? Similar to this question, but in Linux.
Decompile a NASM hello world and understand every byte in it
Version of this answer with a nice TOC and more content: http://www.cirosantilli.com/elf-hello-world (hitting the 30k char limit here)
Standards
ELF is specified by the LSB:
core generic: http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/elf-generic.html
core AMD64: http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-AMD64/LSB-Core-AMD64/book1.html
The LSB basically links to other standards with minor extensions, in particular:
generic (both by SCO):
System V ABI 4.1 (1997) http://www.sco.com/developers/devspecs/gabi41.pdf, no 64 bit, although a magic number is reserved for it. Same for core files.
System V ABI Update DRAFT 17 (2003) http://www.sco.com/developers/gabi/2003-12-17/contents.html, adds 64 bit. Only updates chapters 4 and 5 of the previous document: the others remain valid and are still referenced.
architecture specific:
IA-32: http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-IA32/LSB-Core-IA32/elf-ia32.html, points mostly to http://www.sco.com/developers/devspecs/abi386-4.pdf
AMD64: http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-AMD64/LSB-Core-AMD64/elf-amd64.html, points mostly to http://www.x86-64.org/documentation/abi.pdf
A handy summary can be found at:
man elf
Its structure can be examined in a human readable way via utilities like readelf and objdump.
Generate the example
Let's break down a minimal runnable Linux x86-64 example:
section .data
hello_world db "Hello world!", 10
hello_world_len equ $ - hello_world
section .text
global _start
_start:
mov rax, 1
mov rdi, 1
mov rsi, hello_world
mov rdx, hello_world_len
syscall
mov rax, 60
mov rdi, 0
syscall
Compiled with:
nasm -w+all -f elf64 -o 'hello_world.o' 'hello_world.asm'
ld -o 'hello_world.out' 'hello_world.o'
Versions:
NASM 2.10.09
Binutils version 2.24 (contains ld)
Ubuntu 14.04
We don't use a C program as that would complicate the analysis, that will be level 2 :-)
Hexdumps
hd hello_world.o
hd hello_world.out
Output at: https://gist.github.com/cirosantilli/7b03f6df2d404c0862c6
Global file structure
An ELF file contains the following parts:
ELF header. Points to the position of the section header table and the program header table.
Section header table (optional on executable). Each has e_shnum section headers, each pointing to the position of a section.
N sections, with N <= e_shnum (optional on executable)
Program header table (only on executable). Each has e_phnum program headers, each pointing to the position of a segment.
N segments, with N <= e_phnum (optional on executable)
The order of those parts is not fixed: the only fixed thing is the ELF header that must be the first thing on the file: Generic docs say:
ELF header
The easiest way to observe the header is:
readelf -h hello_world.o
readelf -h hello_world.out
Output at: https://gist.github.com/cirosantilli/7b03f6df2d404c0862c6
Bytes in the object file:
00000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............|
00000010 01 00 3e 00 01 00 00 00 00 00 00 00 00 00 00 00 |..>.............|
00000020 00 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 |........#.......|
00000030 00 00 00 00 40 00 00 00 00 00 40 00 07 00 03 00 |....#.....#.....|
Executable:
00000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............|
00000010 02 00 3e 00 01 00 00 00 b0 00 40 00 00 00 00 00 |..>.......#.....|
00000020 40 00 00 00 00 00 00 00 10 01 00 00 00 00 00 00 |#...............|
00000030 00 00 00 00 40 00 38 00 02 00 40 00 06 00 03 00 |....#.8...#.....|
Structure represented:
typedef struct {
unsigned char e_ident[EI_NIDENT];
Elf64_Half e_type;
Elf64_Half e_machine;
Elf64_Word e_version;
Elf64_Addr e_entry;
Elf64_Off e_phoff;
Elf64_Off e_shoff;
Elf64_Word e_flags;
Elf64_Half e_ehsize;
Elf64_Half e_phentsize;
Elf64_Half e_phnum;
Elf64_Half e_shentsize;
Elf64_Half e_shnum;
Elf64_Half e_shstrndx;
} Elf64_Ehdr;
Manual breakdown:
0 0: EI_MAG = 7f 45 4c 46 = 0x7f 'E', 'L', 'F': ELF magic number
0 4: EI_CLASS = 02 = ELFCLASS64: 64 bit elf
0 5: EI_DATA = 01 = ELFDATA2LSB: big endian data
0 6: EI_VERSION = 01: format version
0 7: EI_OSABI (only in 2003 Update) = 00 = ELFOSABI_NONE: no extensions.
0 8: EI_PAD = 8x 00: reserved bytes. Must be set to 0.
1 0: e_type = 01 00 = 1 (big endian) = ET_REl: relocatable format
On the executable it is 02 00 for ET_EXEC.
1 2: e_machine = 3e 00 = 62 = EM_X86_64: AMD64 architecture
1 4: e_version = 01 00 00 00: must be 1
1 8: e_entry = 8x 00: execution address entry point, or 0 if not applicable like for the object file since there is no entry point.
On the executable, it is b0 00 40 00 00 00 00 00. TODO: what else can we set this to? The kernel seems to put the IP directly on that value, it is not hardcoded.
2 0: e_phoff = 8x 00: program header table offset, 0 if not present.
40 00 00 00 on the executable, i.e. it starts immediately after the ELF header.
2 8: e_shoff = 40 7x 00 = 0x40: section header table file offset, 0 if not present.
3 0: e_flags = 00 00 00 00 TODO. Arch specific.
3 4: e_ehsize = 40 00: size of this elf header. TODO why this field? How can it vary?
3 6: e_phentsize = 00 00: size of each program header, 0 if not present.
38 00 on executable: it is 56 bytes long
3 8: e_phnum = 00 00: number of program header entries, 0 if not present.
02 00 on executable: there are 2 entries.
3 A: e_shentsize and e_shnum = 40 00 07 00: section header size and number of entries
3 E: e_shstrndx (Section Header STRing iNDeX) = 03 00: index of the .shstrtab section.
Section header table
Array of Elf64_Shdr structs.
Each entry contains metadata about a given section.
e_shoff of the ELF header gives the starting position, 0x40 here.
e_shentsize and e_shnum from the ELF header say that we have 7 entries, each 0x40 bytes long.
So the table takes bytes from 0x40 to 0x40 + 7 + 0x40 - 1 = 0x1FF.
Some section names are reserved for certain section types: http://www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#special_sections e.g. .text requires a SHT_PROGBITS type and SHF_ALLOC + SHF_EXECINSTR
readelf -S hello_world.o:
There are 7 section headers, starting at offset 0x40:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .data PROGBITS 0000000000000000 00000200
000000000000000d 0000000000000000 WA 0 0 4
[ 2] .text PROGBITS 0000000000000000 00000210
0000000000000027 0000000000000000 AX 0 0 16
[ 3] .shstrtab STRTAB 0000000000000000 00000240
0000000000000032 0000000000000000 0 0 1
[ 4] .symtab SYMTAB 0000000000000000 00000280
00000000000000a8 0000000000000018 5 6 4
[ 5] .strtab STRTAB 0000000000000000 00000330
0000000000000034 0000000000000000 0 0 1
[ 6] .rela.text RELA 0000000000000000 00000370
0000000000000018 0000000000000018 4 2 4
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)
struct represented by each entry:
typedef struct {
Elf64_Word sh_name;
Elf64_Word sh_type;
Elf64_Xword sh_flags;
Elf64_Addr sh_addr;
Elf64_Off sh_offset;
Elf64_Xword sh_size;
Elf64_Word sh_link;
Elf64_Word sh_info;
Elf64_Xword sh_addralign;
Elf64_Xword sh_entsize;
} Elf64_Shdr;
Sections
Index 0 section
Contained in bytes 0x40 to 0x7F.
The first section is always magic: http://www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html says:
If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), e_shnum has the value SHN_UNDEF (0) and the actual number of section header table entries is contained in the sh_size field of the section header at index 0 (otherwise, the sh_size member of the initial entry contains 0).
There are also other magic sections detailed in Figure 4-7: Special Section Indexes.
SHT_NULL
In index 0, SHT_NULL is mandatory. Are there any other uses for it: What is the use of the SHT_NULL section in ELF? ?
.data section
.data is section 1:
00000080 01 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 |................|
000000a0 0d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
80 0: sh_name = 01 00 00 00: index 1 in the .shstrtab string table
Here, 1 says the name of this section starts at the first character of that section, and ends at the first NUL character, making up the string .data.
.data is one of the section names which has a predefined meaning http://www.sco.com/developers/gabi/2003-12-17/ch4.strtab.html
These sections hold initialized data that contribute to the program's memory image.
80 4: sh_type = 01 00 00 00: SHT_PROGBITS: the section content is not specified by ELF, only by how the program interprets it. Normal since a .data section.
80 8: sh_flags = 03 7x 00: SHF_ALLOC and SHF_EXECINSTR: http://www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#sh_flags, as required from a .data section
90 0: sh_addr = 8x 00: in what virtual address the section will be placed during execution, 0 if not placed
90 8: sh_offset = 00 02 00 00 00 00 00 00 = 0x200: number of bytes from the start of the program to the first byte in this section
a0 0: sh_size = 0d 00 00 00 00 00 00 00
If we take 0xD bytes starting at sh_offset 200, we see:
00000200 48 65 6c 6c 6f 20 77 6f 72 6c 64 21 0a 00 |Hello world!.. |
AHA! So our "Hello world!" string is in the data section like we told it to be on the NASM.
Once we graduate from hd, we will look this up like:
readelf -x .data hello_world.o
which outputs:
Hex dump of section '.data':
0x00000000 48656c6c 6f20776f 726c6421 0a Hello world!.
NASM sets decent properties for that section because it treats .data magically: http://www.nasm.us/doc/nasmdoc7.html#section-7.9.2
Also note that this was a bad section choice: a good C compiler would put the string in .rodata instead, because it is read-only and it would allow for further OS optimizations.
a0 8: sh_link and sh_info = 8x 0: do not apply to this section type. http://www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#special_sections
b0 0: sh_addralign = 04 = TODO: why is this alignment necessary? Is it only for sh_addr, or also for symbols inside sh_addr?
b0 8: sh_entsize = 00 = the section does not contain a table. If != 0, it means that the section contains a table of fixed size entries. In this file, we see from the readelf output that this is the case for the .symtab and .rela.text sections.
.text section
Now that we've done one section manually, let's graduate and use the readelf -S of the other sections.
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 2] .text PROGBITS 0000000000000000 00000210
0000000000000027 0000000000000000 AX 0 0 16
.text is executable but not writable: if we try to write to it Linux segfaults. Let's see if we really have some code there:
objdump -d hello_world.o
gives:
hello_world.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <_start>:
0: b8 01 00 00 00 mov $0x1,%eax
5: bf 01 00 00 00 mov $0x1,%edi
a: 48 be 00 00 00 00 00 movabs $0x0,%rsi
11: 00 00 00
14: ba 0d 00 00 00 mov $0xd,%edx
19: 0f 05 syscall
1b: b8 3c 00 00 00 mov $0x3c,%eax
20: bf 00 00 00 00 mov $0x0,%edi
25: 0f 05 syscall
If we grep b8 01 00 00 on the hd, we see that this only occurs at 00000210, which is what the section says. And the Size is 27, which matches as well. So we must be talking about the right section.
This looks like the right code: a write followed by an exit.
The most interesting part is line a which does:
movabs $0x0,%rsi
to pass the address of the string to the system call. Currently, the 0x0 is just a placeholder. After linking happens, it will be modified to contain:
4000ba: 48 be d8 00 60 00 00 movabs $0x6000d8,%rsi
This modification is possible because of the data of the .rela.text section.
SHT_STRTAB
Sections with sh_type == SHT_STRTAB are called string tables.
They hold a null separated array of strings.
Such sections are used by other sections when string names are to be used. The using section says:
which string table they are using
what is the index on the target string table where the string starts
So for example, we could have a string table containing: TODO: does it have to start with \0?
Data: \0 a b c \0 d e f \0
Index: 0 1 2 3 4 5 6 7 8
And if another section wants to use the string d e f, they have to point to index 5 of this section (letter d).
Notable string table sections:
.shstrtab
.strtab
.shstrtab
Section type: sh_type == SHT_STRTAB.
Common name: section header string table.
The section name .shstrtab is reserved. The standard says:
This section holds section names.
This section gets pointed to by the e_shstrnd field of the ELF header itself.
String indexes of this section are are pointed to by the sh_name field of section headers, which denote strings.
This section does not have SHF_ALLOC marked, so it will not appear on the executing program.
readelf -x .shstrtab hello_world.o
Gives:
Hex dump of section '.shstrtab':
0x00000000 002e6461 7461002e 74657874 002e7368 ..data..text..sh
0x00000010 73747274 6162002e 73796d74 6162002e strtab..symtab..
0x00000020 73747274 6162002e 72656c61 2e746578 strtab..rela.tex
0x00000030 7400 t.
The data in this section has a fixed format: http://www.sco.com/developers/gabi/2003-12-17/ch4.strtab.html
If we look at the names of other sections, we see that they all contain numbers, e.g. the .text section is number 7.
Then each string ends when the first NUL character is found, e.g. character 12 is \0 just after .text\0.
.symtab
Section type: sh_type == SHT_SYMTAB.
Common name: symbol table.
First the we note that:
sh_link = 5
sh_info = 6
For SHT_SYMTAB sections, those numbers mean that:
strings that give symbol names are in section 5, .strtab
the relocation data is in section 6, .rela.text
A good high level tool to disassemble that section is:
nm hello_world.o
which gives:
0000000000000000 T _start
0000000000000000 d hello_world
000000000000000d a hello_world_len
This is however a high level view that omits some types of symbols and in which the symbol types . A more detailed disassembly can be obtained with:
readelf -s hello_world.o
which gives:
Symbol table '.symtab' contains 7 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS hello_world.asm
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 2
4: 0000000000000000 0 NOTYPE LOCAL DEFAULT 1 hello_world
5: 000000000000000d 0 NOTYPE LOCAL DEFAULT ABS hello_world_len
6: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 2 _start
The binary format of the table is documented at http://www.sco.com/developers/gabi/2003-12-17/ch4.symtab.html
The data is:
readelf -x .symtab hello_world.o
Which gives:
Hex dump of section '.symtab':
0x00000000 00000000 00000000 00000000 00000000 ................
0x00000010 00000000 00000000 01000000 0400f1ff ................
0x00000020 00000000 00000000 00000000 00000000 ................
0x00000030 00000000 03000100 00000000 00000000 ................
0x00000040 00000000 00000000 00000000 03000200 ................
0x00000050 00000000 00000000 00000000 00000000 ................
0x00000060 11000000 00000100 00000000 00000000 ................
0x00000070 00000000 00000000 1d000000 0000f1ff ................
0x00000080 0d000000 00000000 00000000 00000000 ................
0x00000090 2d000000 10000200 00000000 00000000 -...............
0x000000a0 00000000 00000000 ........
The entries are of type:
typedef struct {
Elf64_Word st_name;
unsigned char st_info;
unsigned char st_other;
Elf64_Half st_shndx;
Elf64_Addr st_value;
Elf64_Xword st_size;
} Elf64_Sym;
Like in the section table, the first entry is magical and set to a fixed meaningless values.
STT_FILE
Entry 1 has ELF64_R_TYPE == STT_FILE. ELF64_R_TYPE is continued inside of st_info.
Byte analysis:
10 8: st_name = 01000000 = character 1 in the .strtab, which until the following \0 makes hello_world.asm
This piece of information file may be used by the linker to decide on which segment sections go.
10 12: st_info = 04
Bits 0-3 = ELF64_R_TYPE = Type = 4 = STT_FILE: the main purpose of this entry is to use st_name to indicate the name of the file which generated this object file.
Bits 4-7 = ELF64_ST_BIND = Binding = 0 = STB_LOCAL. Required value for STT_FILE.
10 13: st_shndx = Symbol Table Section header Index = f1ff = SHN_ABS. Required for STT_FILE.
20 0: st_value = 8x 00: required for value for STT_FILE
20 8: st_size = 8x 00: no allocated size
Now from the readelf, we interpret the others quickly.
STT_SECTION
There are two such entries, one pointing to .data and the other to .text (section indexes 1 and 2).
Num: Value Size Type Bind Vis Ndx Name
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 2
TODO what is their purpose?
STT_NOTYPE
Then come the most important symbols:
Num: Value Size Type Bind Vis Ndx Name
4: 0000000000000000 0 NOTYPE LOCAL DEFAULT 1 hello_world
5: 000000000000000d 0 NOTYPE LOCAL DEFAULT ABS hello_world_len
6: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 2 _start
hello_world string is in the .data section (index 1). It's value is 0: it points to the first byte of that section.
_start is marked with GLOBAL visibility since we wrote:
global _start
in NASM. This is necessary since it must be seen as the entry point. Unlike in C, by default NASM labels are local.
SHN_ABS
hello_world_len points to the special st_shndx == SHN_ABS == 0xF1FF.
0xF1FF is chosen so as to not conflict with other sections.
st_value == 0xD == 13 which is the value we have stored there on the assembly: the length of the string Hello World!.
This means that relocation will not affect this value: it is a constant.
This is small optimization that our assembler does for us and which has ELF support.
If we had used the address of hello_world_len anywhere, the assembler would not have been able to mark it as SHN_ABS, and the linker would have extra relocation work on it later.
SHT_SYMTAB on the executable
By default, NASM places a .symtab on the executable as well.
This is only used for debugging. Without the symbols, we are completely blind, and must reverse engineer everything.
You can strip it with objcopy, and the executable will still run. Such executables are called stripped executables.
.strtab
Holds strings for the symbol table.
This section has sh_type == SHT_STRTAB.
It is pointed to by sh_link == 5 of the .symtab section.
readelf -x .strtab hello_world.o
Gives:
Hex dump of section '.strtab':
0x00000000 0068656c 6c6f5f77 6f726c64 2e61736d .hello_world.asm
0x00000010 0068656c 6c6f5f77 6f726c64 0068656c .hello_world.hel
0x00000020 6c6f5f77 6f726c64 5f6c656e 005f7374 lo_world_len._st
0x00000030 61727400 art.
This implies that it is an ELF level limitation that global variables cannot contain NUL characters.
.rela.text
Section type: sh_type == SHT_RELA.
Common name: relocation section.
.rela.text holds relocation data which says how the address should be modified when the final executable is linked. This points to bytes of the text area that must be modified when linking happens to point to the correct memory locations.
Basically, it translates the object text containing the placeholder 0x0 address:
a: 48 be 00 00 00 00 00 movabs $0x0,%rsi
11: 00 00 00
to the actual executable code containing the final 0x6000d8:
4000ba: 48 be d8 00 60 00 00 movabs $0x6000d8,%rsi
4000c1: 00 00 00
It was pointed to by sh_info = 6 of the .symtab section.
readelf -r hello_world.o gives:
Relocation section '.rela.text' at offset 0x3b0 contains 1 entries:
Offset Info Type Sym. Value Sym. Name + Addend
00000000000c 000200000001 R_X86_64_64 0000000000000000 .data + 0
The section does not exist in the executable.
The actual bytes are:
00000370 0c 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 |................|
00000380 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
The struct represented is:
typedef struct {
Elf64_Addr r_offset;
Elf64_Xword r_info;
Elf64_Sxword r_addend;
} Elf64_Rela;
So:
370 0: r_offset = 0xC: address into the .text whose address this relocation will modify
370 8: r_info = 0x200000001. Contains 2 fields:
ELF64_R_TYPE = 0x1: meaning depends on the exact architecture.
ELF64_R_SYM = 0x2: index of the section to which the address points, so .data which is at index 2.
The AMD64 ABI says that type 1 is called R_X86_64_64 and that it represents the operation S + A where:
S: the value of the symbol on the object file, here 0 because we point to the 00 00 00 00 00 00 00 00 of movabs $0x0,%rsi
A: the addend, present in field r_added
This address is added to the section on which the relocation operates.
This relocation operation acts on a total 8 bytes.
380 0: r_addend = 0
So in our example we conclude that the new address will be: S + A = .data + 0, and thus the first thing in the data section.
Program header table
Only appears in the executable.
Contains information of how the executable should be put into the process virtual memory.
The executable is generated from object files by the linker. The main jobs that the linker does are:
determine which sections of the object files will go into which segments of the executable.
In Binutils, this comes down to parsing a linker script, and dealing with a bunch of defaults.
You can get the linker script used with ld --verbose, and set a custom one with ld -T.
do relocation on text sections. This depends on how the multiple sections are put into memory.
readelf -l hello_world.out gives:
Elf file type is EXEC (Executable file)
Entry point 0x4000b0
There are 2 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000400000 0x0000000000400000
0x00000000000000d7 0x00000000000000d7 R E 200000
LOAD 0x00000000000000d8 0x00000000006000d8 0x00000000006000d8
0x000000000000000d 0x000000000000000d RW 200000
Section to Segment mapping:
Segment Sections...
00 .text
01 .data
On the ELF header, e_phoff, e_phnum and e_phentsize told us that there are 2 program headers, which start at 0x40 and are 0x38 bytes long each, so they are:
00000040 01 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 40 00 00 00 00 00 00 00 40 00 00 00 00 00 |..#.......#.....|
00000060 d7 00 00 00 00 00 00 00 d7 00 00 00 00 00 00 00 |................|
00000070 00 00 20 00 00 00 00 00 |.. ..... |
and:
00000070 01 00 00 00 06 00 00 00 | ........|
00000080 d8 00 00 00 00 00 00 00 d8 00 60 00 00 00 00 00 |..........`.....|
00000090 d8 00 60 00 00 00 00 00 0d 00 00 00 00 00 00 00 |..`.............|
000000a0 0d 00 00 00 00 00 00 00 00 00 20 00 00 00 00 00 |.......... .....|
Structure represented http://www.sco.com/developers/gabi/2003-12-17/ch5.pheader.html:
typedef struct {
Elf64_Word p_type;
Elf64_Word p_flags;
Elf64_Off p_offset;
Elf64_Addr p_vaddr;
Elf64_Addr p_paddr;
Elf64_Xword p_filesz;
Elf64_Xword p_memsz;
Elf64_Xword p_align;
} Elf64_Phdr;
Breakdown of the first one:
40 0: p_type = 01 00 00 00 = PT_LOAD: TODO. I think it means it will be actually loaded into memory. Other types may not necessarily be.
40 4: p_flags = 05 00 00 00 = execute and read permissions, no write TODO
40 8: p_offset = 8x 00 TODO: what is this? Looks like offsets from the beginning of segments. But this would mean that some segments are intertwined? It is possible to play with it a bit with: gcc -Wl,-Ttext-segment=0x400030 hello_world.c
50 0: p_vaddr = 00 00 40 00 00 00 00 00: initial virtual memory address to load this segment to
50 8: p_paddr = 00 00 40 00 00 00 00 00: initial physical address to load in memory. Only matters for systems in which the program can set it's physical address. Otherwise, as in System V like systems, can be anything. NASM seems to just copy p_vaddrr
60 0: p_filesz = d7 00 00 00 00 00 00 00: TODO vs p_memsz
60 8: p_memsz = d7 00 00 00 00 00 00 00: TODO
70 0: p_align = 00 00 20 00 00 00 00 00: 0 or 1 mean no alignment required TODO what does that mean? otherwise redundant with other fields
The second is analogous.
Then the:
Section to Segment mapping:
section of the readelf tells us that:
0 is the .text segment. Aha, so this is why it is executable, and not writable
1 is the .data segment.
As mentioned in my comment, you will essentially be writing your own elf-header for the executable eliminating the unneeded sections. There are still several required sections. The documentation at Muppetlabs-TinyPrograms does a fair job explaining this process. For fun, here are a couple of examples:
The equivalent of /bin/true (45 bytes):
00000000 7F 45 4C 46 01 00 00 00 00 00 00 00 00 00 49 25 |.ELF..........I%|
00000010 02 00 03 00 1A 00 49 25 1A 00 49 25 04 00 00 00 |......I%..I%....|
00000020 5B 5F F2 AE 40 22 5F FB CD 80 20 00 01 |[_..#"_... ..|
0000002d
Your classic 'Hello World!' (160 bytes):
00000000 7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00 |.ELF............|
00000010 02 00 03 00 01 00 00 00 74 80 04 08 34 00 00 00 |........t...4...|
00000020 00 00 00 00 00 00 00 00 34 00 20 00 02 00 28 00 |........4. ...(.|
00000030 00 00 00 00 01 00 00 00 74 00 00 00 74 80 04 08 |........t...t...|
00000040 74 80 04 08 1f 00 00 00 1f 00 00 00 05 00 00 00 |t...............|
00000050 00 10 00 00 01 00 00 00 93 00 00 00 93 90 04 08 |................|
00000060 93 90 04 08 0d 00 00 00 0d 00 00 00 06 00 00 00 |................|
00000070 00 10 00 00 b8 04 00 00 00 bb 01 00 00 00 b9 93 |................|
00000080 90 04 08 ba 0d 00 00 00 cd 80 b8 01 00 00 00 31 |...............1|
00000090 db cd 80 48 65 6c 6c 6f 20 77 6f 72 6c 64 21 0a |...Hello world!.|
000000a0
Don't forget to make them executable...

sock_sendmsg(): implicit function calls by Kernel

I tried to dump all data sent by a specific process on Linux by hooking a handler to the Kernel's function sock_sendmsg() defined in linux/socket.c.
I could do that by writing a systemtap probe handler for probe kernel.function("sock_sendmsg#net/socket.c") that dumps all data blocks passed with the 2nd argument struct msghdr *msg.
Here's the excerpt from net/socket.c:
int sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
{
struct kiocb iocb;
struct sock_iocb siocb;
int ret;
init_sync_kiocb(&iocb, NULL);
iocb.private = &siocb;
ret = __sock_sendmsg(&iocb, sock, msg, size);
if (-EIOCBQUEUED == ret)
ret = wait_on_sync_kiocb(&iocb);
return ret;
}
I tested my systemtap script hook_sendmsg.stp. First I ran hook_sendmsg.stp in one terminal. Then I opened another terminal and invoked telnet command to connect to stackoverflow.com and typed HEAD / HTTP/1.0<Enter twice> in the terminal. I got the following output from hook_sendmsg.stp:
root#debian:~# stap -g hook_sendmsg.stp
message block [0]; type=3(raw); state=1(unconnected)
14 00 00 00 16 00 01 03 ec 95 f4 52 00 00 00 00 |...........R....|
00 00 00 00 |................|
message block [0]; type=3(raw); state=1(unconnected)
14 00 00 00 16 00 01 03 ec 95 f4 52 00 00 00 00 |...........R....|
00 00 00 00 |................|
message block [0]; type=2(udp); state=1(unconnected)
4d 0d 01 00 00 01 00 00 00 00 00 00 0d 73 74 61 |M............sta|
63 6b 6f 76 65 72 66 6c 6f 77 03 63 6f 6d 00 00 |ckoverflow.com..|
01 00 01 |................|
message block [0]; type=2(udp); state=1(unconnected)
0f 1e 01 00 00 01 00 00 00 00 00 00 0d 73 74 61 |.............sta|
63 6b 6f 76 65 72 66 6c 6f 77 03 63 6f 6d 00 00 |ckoverflow.com..|
1c 00 01 |................|
message block [0]; type=1(tcp); state=3(connected)
48 45 41 44 20 2f 20 48 54 54 50 2f 31 2e 30 0d |HEAD / HTTP/1.0.|
0a |................|
message block [0]; type=1(tcp); state=3(connected)
0d 0a |................|
This shows that totally sock_sendmsg() were called 6 times in the context of telnet. Obviously the 3rd and 4th are DNS queries to Google's public DNS servers 8.8.8.8 and 8.8.4.4. The 5th and 6th are the two lines of HTTP request sent from telnet. But what were the 1st and 2nd called for? Did they called internally by the Kernel?
Thanks in advance.
They are NETLINK messages, as you can see if you monitor the telnet command from userspace with strace (check the socket family):
sendto(3, "\24\0\0\0\26\0\1\3?\254\364R\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
sendto(3, "\24\0\0\0\26\0\1\3?\254\364R\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
send(3, "\212\355\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\0\0"..., 35, MSG_NOSIGNAL) = 35
send(3, "\241\353\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\10i"..., 50, MSG_NOSIGNAL) = 50
send(3, "|\341\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\0\0"..., 35, MSG_NOSIGNAL) = 35
send(3, "HEAD / HTTP/1.0\r\n", 17, 0) = 17
send(3, "\r\n", 2, 0) = 2
Netlink is an interface used by the userspace processes to communicate with kernel (so, they are send by the telnet), usually for routing, firewalling... pourposes [1]. Some usual network C functions (i.e. getsockname) works internally with this kind of NETLINK messages.
If you want to investigate a bit more you should translate the payload of the send_message to the netlink header structure and check what type of messages are.:
struct nlmsghdr {
__u32 nlmsg_len; /* Length of message including header. */
__u16 nlmsg_type; /* Type of message content. */
__u16 nlmsg_flags; /* Additional flags. */
__u32 nlmsg_seq; /* Sequence number. */
__u32 nlmsg_pid; /* Sender port ID. */
};
[1]http://man7.org/linux/man-pages/man7/netlink.7.html

nodejs write binary data into WritableStream with Buffer

The following code tries to write the header of a WAV file to a stream
using a Buffer and then writing it to a WritableStream.
var fs = require("fs")
var samplesLength = 1000;
var sampleRate = 44100;
var outStream = fs.createWriteStream("zaz.wav")
var b = new Buffer(1024)
b.write('RIFF', 0);
/* file length */
b.writeUInt32LE(32 + samplesLength * 2, 4);
//b.writeUint32LE(0, 4);
b.write('WAVE', 8);
/* format chunk identifier */
b.write('fmt ', 12);
/* format chunk length */
b.writeUInt32LE(16, 16);
/* sample format (raw) */
b.writeUInt16LE(1, 20);
/* channel count */
b.writeUInt16LE(2, 22);
/* sample rate */
b.writeUInt32LE(sampleRate, 24);
/* byte rate (sample rate * block align) */
b.writeUInt32LE(sampleRate * 4, 28);
/* block align (channel count * bytes per sample) */
b.writeUInt16LE(4, 32);
/* bits per sample */
b.writeUInt16LE(16, 34);
/* data chunk identifier */
b.write('data', 36);
/* data chunk length */
//b.writeUInt32LE(40, samplesLength * 2);
b.writeUInt32LE(40, 0);
outStream.write(b.slice(0,50))
outStream.end()
For some reason, the first 8 bytes of the file are wrong :
hexdump -C zaz.wav
00000000 28 00 00 00 f0 07 00 00 57 41 56 45 66 6d 74 20 |(.......WAVEfmt |
00000010 10 00 00 00 01 00 02 00 44 ac 00 00 10 b1 02 00 |........D.......|
00000020 04 00 10 00 64 61 74 61 80 5a 57 ac ef 04 00 00 |....data.ZW.....|
00000030 18 57 |.W|
The first line should be :
00000000 52 49 46 46 24 00 ff 7f 57 41 56 45 66 6d 74 20 |RIFF$...WAVEfmt |
Update :
this line is at fault :
b.writeUInt32LE(40, 0);
it needs to be :
b.writeUInt32LE(0, 40);
The first 8 bits are written by b.writeUInt32LE(40, 0);.
= write (int) 40 (= 0x28) in Little Endian at offset 0.
I don't know exactly what you want but this is the problem.

What are those 32 bits near the start of a FLAC file?

I am trying to write a parser to extract information from the following FLAC file:
$ hd audio.flac | head -n 6
00000000 66 4c 61 43 00 00 00 22 12 00 12 00 00 00 00 00 |fLaC..."........|
00000010 00 00 0a c4 42 f0 00 78 9f 30 00 00 00 00 00 00 |....B..x.0......|
00000020 00 00 00 00 00 00 00 00 00 00 84 00 02 64 1f 00 |.............d..|
00000030 00 00 47 53 74 72 65 61 6d 65 72 20 65 6e 63 6f |..GStreamer enco|
00000040 64 65 64 20 76 6f 72 62 69 73 63 6f 6d 6d 65 6e |ded vorbiscommen|
00000050 74 10 00 00 00 12 00 00 00 54 49 54 4c 45 3d 52 |t........TITLE=R|
Now, according to the specification, the format should be as follow (numbers are in bits):
<32> "fLaC", the FLAC stream marker in ASCII
<16> The minimum block size (in samples) used in the stream.
<16> The maximum block size (in samples) used in the stream.
<24> The minimum frame size (in bytes) used in the stream.
<24> The maximum frame size (in bytes) used in the stream.
<20> Sample rate in Hz.
<3> (number of channels)-1. FLAC supports from 1 to 8 channels
<5> (bits per sample)-1. FLAC supports from 4 to 32 bits per sample.
<36> Total samples in stream.
<128> MD5 signature of the unencoded audio data.
So, I start to write my parser and, while testing, get very strange results. So I test with a "real" metadata extractor:
$ metaflac --list audio.flac
METADATA block #0
type: 0 (STREAMINFO)
is last: false
length: 34
minimum blocksize: 4608 samples
maximum blocksize: 4608 samples
minimum framesize: 0 bytes
maximum framesize: 0 bytes
sample_rate: 44100 Hz
channels: 2
bits-per-sample: 16
total samples: 7905072
MD5 signature: 00000000000000000000000000000000
From the numbers, I can deduce the following:
66 4c 61 43 00 00 00 22 12 00 12 00 00 00 00 00
~~~~~~~~~~~ ~~~~~~~~~~~ ~~~~~ ~~~~~ ~~~~~~~~ ~~
^ ^ ^ ^ ^ ^
| | | | | |
| | | | | + Etc.
| | | | + Minimum frame size
| | | + Maximum block size
| | + Minimum block size
| + What is that ?!?
+ FLAC stream marker
Where does those 32 bits come from? I see they represent the length of the header, but isn't it against the standard to put it there (Taking into account that we already know the length: (32+16+16+24+20+3+5+36+128)/8)?
The 0x22 (34) is indeed the header block size in bytes as part of the METADATA_BLOCK_HEADER which follows the fLaC marker in the stream. Of the first 8 bits (00), bit 7 indicates that there are more metadatablocks to follow, the next 7 bits indicate that it's a STREAMINFO block. The following 3 bytes (00 00 22) is the length of the contents of the block;
16 + 16 + 24 + 24 + 20 + 3 + 5 + 36 + 128 = 272 bits
272 bits / 8 = 34 (0x22) bytes.

Resources