I'm writing a client and server program with Linux socket programming. I'm confused about something. Although sizeof(char) is guaranteed to be 1, I know the real size of char may be different in different computer. It may be 8bits,16bits or some other size. The problem is that what if client and server have different size of char. For example client char size is 8bits and server char size is 16bits. Client call write(socket_fd, *c, sizeof(char)) and Server call read(socket_fd, *c, sizeof(char)). Does Client sends 8bits and Server wants to receive 16bits? If it is true, what will happen?
Another question: Is it good for me to pass text between client and server because I don't need to consider the big endian and little endian problem?
Thanks in advance.
What system are you communicating with that has 16bits in a byte? In any case, if you want to know exactly how many bits you have - use int8 instead.
#Basile is right. A char is always eight bits in linux. I found this in the book Linux Kernel Development. This book also states some other rules:
Although there is no rule that the int type be 32 bits, it is in Linux on all currently supported architectures.
The same goes for the short type, which is 16 bits on all current architectures, although no rule explicitly decrees that.
Never assume the size of a pointer or a long, which can be either 32 or 64 bits on the currently supported machines in Linux.
Because the size of a long varies on different architectures, never assume that sizeof(int) is equal to sizeof(long).
Likewise, do not assume that a pointer and an int are the same size.
For the choice of pass by binary data or text data through the network, the book UNIX Network Programming Volume1 gives the two solutions:
Pass all numeric data as text strings.
Explicitly define the binary formats of the supported datatypes (number of bits, big- or little-endian) and pass all data between the client and server in this format. RPC packages normally use this technique. RFC 1832 [Srinivasan 1995] describes the External Data Representation (XDR) standard that is used with the Sun RPC package.
The c definition of char as the size of a memory cell is different from the definition used in Unicode.
A Unicode code-point can, depending on the encoding used, require up to 6 bytes of storage.
This is a slightly different problem than byte order and word size differences between different architectures, etc.
If you wish to express complex structures (containing unicode text), it's probably a
good idea to implement a message protocol, that encode messages to a byte array, that can be send over any communication channel.
A simple client/server mechanism is to send a fixed size header containing the length of the following message. It's a nice exercise to build something like this in c... :-)
Depending on what you are trying to do, it may be worthwhile to look at existing technologies for the message interface; Look at Etch, Thrift, SWIG, *-rpc, asn1, soap, xml, json, corba, etc.
Related
I've been trying to read the implementation of a kernel module, and I'm stumbling on this piece of code.
unsigned long addr = (unsigned long) buf;
if (!IS_ALIGNED(addr, 1 << 9)) {
DMCRIT("#%s in %s is not sector-aligned. I/O buffer must be sector-aligned.", name, caller);
BUG();
}
The IS_ALIGNED macro is defined in the kernel source as follows:
#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
I understand that data has to be aligned along the size of a datatype to work, but I still don't understand what the code does.
It left-shifts 1 by 9, then subtracts by 1, which gives 111111111. Then 111111111 does bitwise-and with x.
Why does this code work? How is this checking for byte alignment?
In systems programming it is common to need a memory address to be aligned to a certain number of bytes -- that is, several lowest-order bits are zero.
Basically, !IS_ALIGNED(addr, 1 << 9) checks whether addr is on a 512-byte (2^9) boundary (the last 9 bits are zero). This is a common requirement when erasing flash locations because flash memory is split into large blocks which must be erased or written as a single unit.
Another application of this I ran into. I was working with a certain DMA controller which has a modulo feature. Basically, that means you can allow it to change only the last several bits of an address (destination address in this case). This is useful for protecting memory from mistakes in the way you use a DMA controller. Problem it, I initially forgot to tell the compiler to align the DMA destination buffer to the modulo value. This caused some incredibly interesting bugs (random variables that have nothing to do with the thing using the DMA controller being overwritten... sometimes).
As far as "how does the macro code work?", if you subtract 1 from a number that ends with all zeroes, you will get a number that ends with all ones. For example, 0b00010000 - 0b1 = 0b00001111. This is a way of creating a binary mask from the integer number of required-alignment bytes. This mask has ones only in the bits we are interested in checking for zero-value. After we AND the address with the mask containing ones in the lowest-order bits we get a 0 if any only if the lowest 9 (in this case) bits are zero.
"Why does it need to be aligned?": This comes down to the internal makeup of flash memory. Erasing and writing flash is a much less straightforward process then reading it, and typically it requires higher-than-logic-level voltages to be supplied to the memory cells. The circuitry required to make write and erase operations possible with a one-byte granularity would waste a great deal of silicon real estate only to be used rarely. Basically, designing a flash chip is a statistics and tradeoff game (like anything else in engineering) and the statistics work out such that writing and erasing in groups gives the best bang for the buck.
At no extra charge, I will tell you that you will be seeing a lot of this type of this type of thing if you are reading driver and kernel code. It may be helpful to familiarize yourself with the contents of this article (or at least keep it around as a reference): https://graphics.stanford.edu/~seander/bithacks.html
I am trying to convert a Ruby program to NodeJS, but I seem to be getting stuck with buffers.
I have
rounds = header_bytes[120..-1].unpack('L*').first
In Ruby, which headers a buffer (header_bytes), and get's 120-124 (or in this case -1, which is remaining). Then unpacks it into an unsigned 32 bit integer.
I am trying to do the same thing in JS, but it can't seem to get it to work. I have
rounds = header.slice(120,124).toString('ucs2');
I've tried all the different formats in toString and nothing returns the same result as Ruby.
Assuming that header is an instance of Node's Buffer then you have a variety of functions for reading from a buffer as various sizes of integer, including
buf.readUInt32LE
buf.readUInt32BE
These both take an offset from which to read the bytes. The ruby L specifier means native byte order so depending on where this code is running you might need either of those functions, depending on whether you're on a big or little endian platform. For example on an x86 machine you'd do
header.readUInt32LE(120)
Protocols normally specify big or little endian (traditionally network byte order is big endian)
You can check the platform endianness with os.endianness
I've worked with a few protocols, and written my own. I have written some message formats with only 1 char to identify the message, and some with 4 chars. I don't feel that I'm experienced enough to tell which is better, so I'm looking for an answer which describes in which scenario one might be better than the other.
For performance, you would imagine that sending 2 bytes (A%1i) is faster than sending 5 bytes (ABCD%1i). However, I have noticed that when writing the protocol with the 1 byte prefix, if you have a bug which causes your code to not read enough data from the socket, you might get garbage data comming into your system.
So is the purpose of a 4 byte prefix just to provide a guarentee that your message is clean? Is it worth it for the performance you sacrafice? Do you really sacrafice any performance at all? Maybe it's better to have 2 or 3 byte prefix?
I'm not sure if this question should be specific to TCP, or whether it applies to all transport protocols. Advice on this would be interesting.
Update: For interest, I will mention that Synergy uses 4-byte message prefixes, so for a mouse move delta the header is the same size as the actual data. Some have suggested just having a 1 or 2 byte prefix to improve efficiency. I wonder what drawbacks this would have?
Update: Also, I wonder if only the handshake really matters, if you're worried about garbage data. Synergy has a long handshake (a few bytes), so are the 4-byte message prefixes needed? I made a protocol recently that has only a 1 byte handshake, and that turned out to be a bad idea, since incompatible protocols were spamming the system with bad data (off the back of this, I might reccomend at least having a long handshake).
The purpose of the header is to make it easier to solve the frame synchronization problem ( byte aligning in serial communication ).
To synchronize, the receiver looks for anything in the data stream that "looks like" a start-of-message header.
If you have lots of different kinds of valid start-of-message headers, and all of them are 1 byte long, then you will inevitably get a lot of "false frame synchronizations" -- garbage from something that "looks like" a start-of-message header, but isn't.
It would be better to pick some other header that makes it "unlikely" that anything in the serial data stream "looks like" a valid start-of-message header.
It is inevitable that you will get garbage data coming into your system, no matter how you design the packet header.
Whatever you use to handle these other problems (such as occasional bit errors in the middle of the message) should also be adequate to handle the occasional "false frame synchronization" garbage.
In some systems, any bad data is quickly overwritten by fresh new good data, and if you blink you might never see the bad data.
Other systems need at least some sort of error detection in the footer to reject the bad data.
Yet other systems need to not only detect such errors, but somehow keep re-sending that message -- until both sides are convinced that an error-free version of that message has been successfully received.
As Oleksi implied, in some systems the latency is not significantly different between sending a single binary bit (100 ms) and sending 10 bytes (102.4 ms).
So the advantages of using a tiny header (2.4% less latency!) may not be worth it compared to the advantages of using a more verbose header (easier debugging; easier to make backward-compatible and forward-compatible; easier to test the effect of minor changes "in isolation" without upgrading both sides in lockstep to the new protocol which is completely incompatible with the old protocol).
Perhaps you could get the best of both worlds by (a) keeping the verbose, easy-to-debug headers on messages that are so rarely used that the effect of tiny headers is too small to measure (which I suspect is nearly all messages), and (b) introducing a "tiny header" format for any kind of message where the effect of tiny headers is "noticeably better" or at least at least measurable.
It looks like the Synergy protocol is flexible enough to add such a "tiny header" format in a way that is easily distinguishable from the other kinds of message headers.
I use Synergy between my laptop and a few desktop machines. I am glad someone is trying to make it even better.
The performance will depend on the content of the message you are sending. If your content is several kilobytes, it doesn't really matter how many bytes your header is. For now, I would choose the scheme that's easiest to work with, because the performance difference between sending one byte, or four bytes is going to be negligible compared to the actual data that you're sending.
RS-232 communication sometimes uses 9-bit bytes. This can be used to communicate with multiple microcontrollers on a bus where 8 bits are data and the extra bit indicates an address byte (rather than data). Inactive controllers only generate an interrupt for address bytes.
Can a Linux program send and receive 9-bit bytes over a serial device? How?
The termios system does not directly support 9 bit operation but it can be emulated on some systems by playing tricks with the CMSPAR flag. It is undocumented and my not appear in all implementations.
Here is a link to a detailed write-up on how 9-bit emulation is done:
http://www.lothosoft.ch/thomas/libmip/markspaceparity.php
9-bit data is a standard part of RS-485 and used in multidrop applications. Hardware based on 16C950 devices may support 9-bits, but only if the UART is used in its 950 mode (rather than the more common 450/550 modes used for RS-232).
A description of the 16C950 may be found here.
This page summarizes Linux RS-485 support, which is baked into more recent kernels (>=3.2 rc3).
9-bit data framing is possible even if a real world UARTs doesn't.
Found one library that also does it under Windows and Linux.
See http://adontec.com/9-bit-serial-communication.htm
basically what he wants is to output data from a linux box, then send it on let's say a 2 wire bus with a bunch of max232 ic's -> some microcontroller with uart or software rs232 implementation
one can leave the individual max232 level converter's away as long as there are no voltage potency issues between the individual microcontrollers (on the same pcb, for example, rather than in different buildings ;) up until the maximum output (ttl) load of the max232 (or clones, or a resistor and invertor/transistor) ic.
can't find linux termios settings for MARK or SPACE parity (Which i'm sure the hardware uarts actually do support, just not linux tty implementation), so we shall just hackzor the actual parity generation a bit.
8 data bits, 2 stop bits is the same length as 8 databits, 1 parity bit, 1 stop bit. (where the first stopbit is a logic 1, negative line voltage).
one would then use the 9th bit as an indicator that the other 8 bits are the address of the individual or group of microcontrollers, which then take the next bytes as some sort of command, or data, as well, they are 'addressed'.
this provides for an 8 bit transparant, although one way traffic, means to address 'a lot of things' (256 different (groups of) things, actually ;) on the same bus. it's one way, for when one would want to do 2 way, you'd need 2 wire pairs, or modulate at multiple frequencies, or implement colission detection and the whole lot of that.
PIC microcontrollers can do 9 bit serial communication with ehm 'some trickery' (the 9th bit is actually in another register ;)
now... considering the fact that on linux and the likes it is not -that- simple...
have you considered simply turning parity on for the 'address word' (the one in which you need 9 bits ;) and then either setting it to odd or even, calculate it so that the right one is chosen to make the 9th (parity) bit '1' with parity on and 8 bit 'data', then turn parity back off and turn 2 stop bits on. (which still keeps a 9 bit word length in as far as your microcontroller is concerned ;)... it's a long time ago but as far as i recall stop bits are just as long as data bits in the timing of things.
this should work on anything that can do 8 bit output, with parity, and with 2 stop bits. which includes pc hardware and linux. (and dos etc)
pc hardware also has options to just turn 'parity' on or off for all words (Without actually calculating it) if i recall correctly from 'back in the days'
furthermore, the 9th bit the pic datasheet speaks about, actually IS the parity bit as in RS-232 specifications. just that you're free to turn it off or on. (on PIC's anyway - in linux it's a bit more complicated than that)
(nothing a few termios settings on linux won't solve i think... just turn it on and off then... we've made that stuff do weirder things ;)
a pic microcontroller actually does exactly the same, just that it's not presented like 'what it actually is' in the datasheet. they actually call it 'the 9th bit' and things like that. on pc's and therefore on linux it works pretty much the same way tho.
anyway if this thing should work 'both ways' then good luck wiring it with 2 pairs or figuring out some way to do collission detection, which is hell a lot more problematic than getting 9 bits out.
either way it's not much more than an overrated shift register. if the uart on the pc doesn't want to do it (which i doubt), just abuse the DTR pin to just shift out the data by hand, or abuse the printer port to do the same, or hook up a shift register to the printer port... but with the parity trick it should work fine anyway.
#include<termios.h>
#include<stdio.h>
#include<sys/types.h>
#include<sys/stat.h>
#include<fcntl.h>
#include<unistd.h>
#include<stdint.h>
#include<string.h>
#include<stdlib.h>
struct termios com1pr;
int com1fd;
void bit9oneven(int fd){
cfmakeraw(&com1pr);
com1pr.c_iflag=IGNPAR;
com1pr.c_cflag=CS8|CREAD|CLOCAL|PARENB;
cfsetispeed(&com1pr,B300);
cfsetospeed(&com1pr,B300);
tcsetattr(fd,TCSANOW,&com1pr);
};//bit9even
void bit9onodd(int fd){
cfmakeraw(&com1pr);
com1pr.c_iflag=IGNPAR;
com1pr.c_cflag=CS8|CREAD|CLOCAL|PARENB|PARODD;
cfsetispeed(&com1pr,B300);
cfsetospeed(&com1pr,B300);
tcsetattr(fd,TCSANOW,&com1pr);
};//bit9odd
void bit9off(int fd){
cfmakeraw(&com1pr);
com1pr.c_iflag=IGNPAR;
com1pr.c_cflag=CS8|CREAD|CLOCAL|CSTOPB;
cfsetispeed(&com1pr,B300);
cfsetospeed(&com1pr,B300);
tcsetattr(fd,TCSANOW,&com1pr);
};//bit9off
void initrs232(){
com1fd=open("/dev/ttyUSB0",O_RDWR|O_SYNC|O_NOCTTY);
if(com1fd>=0){
tcflush(com1fd,TCIOFLUSH);
}else{printf("FAILED TO INITIALIZE\n");exit(1);};
};//initrs232
void sendaddress(unsigned char x){
unsigned char n;
unsigned char t=0;
for(n=0;n<8;n++)if(x&2^n)t++;
if(t&1)bit9oneven(com1fd);
if(!(t&1))bit9onodd(com1fd);
write(com1fd,&x,1);
};
void main(){
unsigned char datatosend=0x00; //bogus data byte to send
initrs232();
while(1){
bit9oneven(com1fd);
while(1)write(com1fd,&datatosend,1);
//sendaddress(223); // address microcontroller at address 223;
//write(com1fd,&datatosend,1); // send an a
//sendaddress(128); // address microcontroller at address 128;
//write(com1fd,&datatosend,1); //send an a
}
//close(com1fd);
};
somewhat works.. maybe some things the wrong way around but it does send 9 bits. (CSTOPB sets 2 stopbits, meaning that on 8 bit transparant data the 9th bit = 1, in addressing mode the 9th bit = 0 ;)
also take note that the actual rs232 line voltage levels are the other way around from what your software 'reads' (which is the same as the 'inverted' 5v ttl levels your pic microcontroller gets from the transistor or inverter or max232 clone ic). (-19v or -10v (pc) for logic 1, +19/+10 for logic 0), stop bits are negative voltage, like a 1, and the same lenght.
bits go out 0-7 (and in this case: 8 ;)... so start bit -> 0 ,1,2,3,4,5,6,7,
it's a bit hacky but it seems to work on the scope.
Can a Linux program send and receive 9-bit bytes over a serial device?
The standard UART hardware (8251 etc.) doesn't support 9-bit-data modes.
I also made complete demo for 9-bit UART emulation (based on even/odd parity). You can find it here.
All sources available on git.
You can easily adapt it for your device. Hope you like it.
I am creating a protocol to have two applications talk over a TCP/IP stream and am figuring out how to design a header for my messages. Using the TCP header as an initial guide, I am wondering if I will need padding. I understand that when we're dealing with a cache, we want to make sure that data being stored fits in a row of cache so that when it is retrieved it is done so efficiently. However, I do not understand how it makes sense to pad a header considering that an application will parse a stream of bytes and store it how it sees fit.
For example: I want to send over a message header consisting of a 3 byte field followed by a 1 byte padding field for 32 bit alignment. Then I will send over the message data.
In this case, the receiver will just take 3 bytes from the stream and throw away the padding byte. And then start reading message data. As I see it, he will not be storing the 3 bytes and the message data the way he wants. The whole point of byte alignment is so that it will be retrieved in an efficient manner. But if the retriever doesn't care about the padding how will it be retrieved efficiently?
Without the padding, the retriever just takes the 3 header bytes from the stream and then takes the data bytes. Since the retriever stores these bytes however he wants, how does it matter whether or not the padding is done?
Maybe I'm missing the point of padding.
It's slightly hard to extract a question from this post, but with what I've said you guys can probably point out my misconceptions.
Please let me know what you guys think.
Thanks,
jbu
If word alignment of the message body is of some use, then by all means, pad the message to avoid other contortions. The padding will be of benefit if most of the message is processed as machine words with decent intensity.
If the message is a stream of bytes, for instance xml, then padding won't do you a whole heck of a lot of good.
As far as actually designing a wire protocol, you should probably consider using a plain text protocol with compression (including the header), which will probably use less bandwidth than any hand-designed binary protocol you could possibly invent.
I do not understand how it makes sense to pad a header considering that an application will parse a stream of bytes and store it how it sees fit.
If I'm a receiver, I might pass a buffer (i.e. an array of bytes) to the protocol driver (i.e. the TCP stack) and say, "give this back to me when there's data in it".
What I (the application) get back, then, is an array of bytes which contains the data. Using C-style tricks like "casting" and so on I can treat portions of this array as if it were words and double-words (not just bytes) ... provided that they're suitably aligned (which is where padding may be required).
Here's an example of a statement which reads a DWORD from an offset in a byte buffer:
DWORD getDword(const byte* buffer)
{
//we want the DWORD which starts at byte-offset 8
buffer += 8;
//dereference as if it were pointing to a DWORD
//(this would fail on some machines if the pointer
//weren't pointing to a DWORD-aligned boundary)
return *((DWORD*)buffer);
}
Here's the corresponding function in Intel assembly; note that it's a single opcode i.e. quite an efficient way to access the data, more efficient that reading and accumulating separate bytes:
mov eax,DWORD PTR [esi+8]
Oner reason to consider padding is if you plan to extend your protocol over time. Some of the padding can be intentionally set aside for future assignment.
Another reason to consider padding is to save a couple of bits on length fields. I.e. always a multiple of 4, or 8 saves 2 or 3 bits off the length field.
One other good reason that TCP has padding (which probably does not apply to you) is it allows dedicated network processing hardware to easily separate the data from the header. As the data always starts on a 32 bit boundary, it's easier to separate the header from the data when the packet gets routed.
If you have a 3 byte header and align it to 4 bytes, then designate the unused byte as 'reserved for future use' and require the bits to be zero (rejecting messages where they are not as malformed). That leaves you some extensibility. Or you might decide to use the byte as a version number - initially zero, and then incrementing it if (when) you make incompatible changes to the protocol. Don't let the value be 'undefined' and "don't care"; you'll never be able to use it if you start out that way.