I have been working with node.js for the last 4 month and now wants to increase tcp receving and sending buffersize.
My purpose is to speed up my application and expermantation with buffersizes may increase preformance.
I have searched on google but haven't found anything useful except that you can change the default socket buffersizes on linux as example on this website:
http://www.cyberciti.biz/faq/linux-tcp-tuning/
Is there any way to change/set tcp sending and receiving buffersizes for node.js io?
stream_wrap has an allocation callback passed to libuv that is passed a suggested_size of the allocated memory to use in the receiving the data. Right now it passes 64KB as the suggested size, and there's no way to change this afaik.
Is this along the line of your question?
I found the stream_wrap on git:
git... src/stream_wrap src file
And if you navigate src/stream_wrap.cc in node.js src folder and looking up following code:
// If less than 64kb is remaining on the slab allocate a new one.
if (SLAB_SIZE - slab_used < 64 * 1024) {
slab = NewSlab(global, wrap->object_);
} else {
wrap->object_->SetHiddenValue(slab_sym, slab_obj);
}
}
Then you might be able to change the size.
#Trev Norris you know anything about this?
Related
I am trying to develop a socket congestion algorithm in diameter stack by comparing the socket send buffer size[default max size] and the actual bytes available in send buffer queue.
getsockopt(ainfo->socket,SOL_SOCKET,SO_SNDBUF,(void *)&n, &m); // Getting the max default size
retval = ioctl(ainfo->socket,TIOCOUTQ,&bytes_available); // Getting the actual bytes available
Testing scenario:
Start the client and server.
Once handshake is successful, start the traffic from client to server.
Block the packets at server's end using iptables.
Check the netstat output and check the send buffer size with help of ss command.
The send buffer size get stuck after sometime at a certain number. Example size of send buffer is 87090 [tb as shown in ss output]. The Send Q is stuck at some random number which is much smaller than the tb [for example : 54344]. Sometimes it increases till 135920. Ideally it should reach somewhere around 80k and then get stuck. Can somewhen explain me this unusual behavior ?
Any help is appreciated.
Thanks!
While I was trying to implement benchmark testware using netperf I happened to read its manual. Where I got this query
In the TCP_STREAM specific test there are an option to mention -s and -S to specify local(netperf client), remote(netperf server) socket buffer sizes respectively. Is that a regular BSD socket size? There is also an option to specify the local send message size -m and remote receive message size -M; Is this the total message size after all TCP/IP encapsulation? Can anybody throw some light on this. It would be great if you can illustrate using a use-case why we need these separate parameters as the BSD socket size appears to be the upper boundary here.
The socket buffer sizes (set via -s and -S) will control how much data may be outstanding on the connection at one time by affecting either the receiver's advertised window (which will be based on the SO_SNDBUF) or how much data the sender can hold waiting for ACKnowledgement (which will be based on SO_SNDBUF).
The send and receive message sizes (-m and -M) control how much data is presented in any one "send" (-m) or requested in any one "recv" (-M) call.
As TCP is a streaming protocol, it is perfectly legal/possible to make a send call with a number of bytes larger than the socket buffer(s). When the socket is blocking (as netperf uses) it simply means the send call will remain there until the last of its bytes have been put into the send socket buffer. On the receive side, one can as for more than a socket buffer's worth of data in a single receive, but the semantics are such that the call will return with however many bytes happen to be there at the time if there are any, and will return with however many bytes arrive if the socket buffer was empty at the time of the call (again because netperf uses blocking sockets/calls).
This question suggests that the Node.js Net module doesn't handle ip-fragmentation:
Node.js how to handle packet fragmentation with net.Server
I can almost not imagine it's true, but I can't find any documentation about this (please forgive me if it shouldn't be too hard to find information about this :-) ). Is it true?
If not: thank you, that's a real day-safer to me :-).
If it is true: how to handle this problem if I don't know how large the whole datagram is?
Situation:
I have a TCP connection with an embedded system (Wiznet W5500). The MSS (Maximum Segment Size, mostly equal to MTU - 40) will be set to 536 and data packages are a variable size and may be over 4kb in size. So the data package will be send in multiple segments. Will the 'on data' event be triggered once a segment is received or only when the whole package is received?
Side question: Am I right that the data segment (about what Wiznet is talking in the explanation of the MSS register) equel is to a ip-fragment?
So if I have to send 4000byte (ie payload) and the MSS is set to 536 I will receive consecutively:
segment1: 536bytes payload
segment2: 536bytes payload
segment3: 536bytes payload
segment4: 536bytes payload
segment5: 536bytes payload
segment6: 536bytes payload
segment7: 536bytes payload
segment8: 248bytes payload
May the 'on data' event only be triggered after segment 8 an will the 'data argument' contain the whole package or may it happen that the 'on data' event will be triggered after every separately received segment?
How can I make 100% sure that I get the whole package before I continue to process it?
Solution I thought of:
First 2 byte of the data package is the byte length of the whole length, I keep concattenating received data until I have received as many bytes. If I received more than the package size, I'll assume these successive bytes are the start of a subsequent data package.
I do believe this 'solution' is somewhat tricky and I hope it's not required.
Thanks in advance! If any information is missing: I'm sorry, please feel free to ask for it :-).
I am a big fan of nodeJS, but in this case you should use python with scapy :)
http://www.secdev.org/projects/scapy/
I'm using perl raw sockets to send a large number of arbitrary raw packets (nothing malicious!), but I only get about 14500 going out before my send() call fails with "No buffer space available".
I'm guessing that something in the kernel is keeping track of what I'm sending? How can I prevent that. I just want to send each packet, there is no necessity to wait for replies or remember anything about what got sent. (Update: I notice loads of "CLOSE_WAIT" entries in my "netstat" command - how do I tell the kernel not to do that?)
What is the "buffer space" it's referring to, and is there a way to check what space is available, so I can avoid sending more when there's no room? Alternatively - is there a way to "block" on "send()", so I never experience the error?
Alternatively - is there some other way to send out an ethernet packet (eg: a SYN), where the kernel knows not to enter anything into any tables?
If the CLOSE_WAIT connections are yours:
You are using TCP, not raw sockets.
You aren't closing your sockets.
I am writing an application server that processes images (large data). I am trying to minimize copies when sending image data back to clients. The processed images I need to send to clients are in buffers obtained from jemalloc. The ways I have thought of sending the data back to the client is:
1) Simple write call.
// Allocate buffer buf.
// Store image data in this buffer.
write(socket, buf, len);
2) I obtain the buffer through mmap instead of jemalloc, though I presume jemalloc already creates the buffer using mmap. I then make a simple call to write.
buf = mmap(file, len); // Imagine proper options.
// Store image data in this buffer.
write(socket, buf, len);
3) I obtain a buffer through mmap like before. I then use sendfile to send the data:
buf = mmap(in_fd, len); // Imagine proper options.
// Store image data in this buffer.
int rc;
rc = sendfile(out_fd, file, &offset, count);
// Deal with rc.
It seems like (1) and (2) will probably do the same thing given jemalloc probably allocates memory through mmap in the first place. I am not sure about (3) though. Will this really lead to any benefits? Figure 4 on this article on Linux zero-copy methods suggests that a further copy can be prevented using sendfile:
no data is copied into the socket buffer. Instead, only descriptors
with information about the whereabouts and length of the data are
appended to the socket buffer. The DMA engine passes data directly
from the kernel buffer to the protocol engine, thus eliminating the
remaining final copy.
This seems like a win if everything works out. I don't know if my mmaped buffer counts as a kernel buffer though. Also I don't know when it is safe to re-use this buffer. Since the fd and length is the only thing appended to the socket buffer, I assume that the kernel actually writes this data to the socket asynchronously. If it does what does the return from sendfile signify? How would I know when to re-use this buffer?
So my questions are:
What is the fastest way to write large buffers (images in my case) to a socket? The images are held in memory.
Is it a good idea to call sendfile on a mmapped file? If yes, what are the gotchas? Does this even lead to any wins?
It seems like my suspicions were correct. I got my information from this article. Quoting from it:
Also these network write system calls, including sendfile, might and
in many cases do return before the data sent over TCP by the method
call has been acknowledged. These methods return as soon as all data
is written into the socket buffers (sk buff) and is pushed to the TCP
write queue, the TCP engine can manage alone from that point on. In
other words at the time sendfile returns the last TCP send window is
not actually sent to the remote host but queued. In cases where
scatter-gather DMA is supported there is no seperate buffer which
holds these bytes, rather the buffers(sk buffs) just hold pointers to
the pages of OS buffer cache, where the contents of file is located.
This might lead to a race condition if we modify the content of the
file corresponding to the data in the last TCP send window as soon as
sendfile is returned. As a result TCP engine may send newly written
data to the remote host instead of what we originally intended to
send.
Provided the buffer from a mmapped file is even considered "DMA-able", seems like there is no way to know when it is safe to re-use it without an explicit acknowledgement (over the network) from the actual client. I might have to stick to simple write calls and incur the extra copy. There is a paper (also from the article) with more details.
Edit: This article on the splice call also shows the problems. Quoting it:
Be aware, when splicing data from a mmap'ed buffer to a network
socket, it is not possible to say when all data has been sent. Even if
splice() returns, the network stack may not have sent all data yet. So
reusing the buffer may overwrite unsent data.
For cases 1 and 2 - does the operation you marked as // Store image data in this buffer require any conversion? Is it just plain copy from the memory to buf?
If it's just plain copy, you can use write directly on the pointer obtained from jemalloc.
Assuming that img is a pointer obtained from jemalloc and size is a size of your image, just run following code:
int result;
int sent=0;
while(sent<size) {
result=write(socket,img+sent,size-sent);
if(result<0) {
/* error handling here */
break;
}
sent+=result;
}
It is working correctly for blocking I/O (the default behavior). If you need to write a data in a non-blocking manner, you should be able to rework the code on your own, but now you have the idea.
For case 3 - sendfile is for sending data from one descriptor to another. That means you can, for example, send data from file directly to tcp socket and you don't need to allocate any additional buffer. So, if the image you want to send to a client is in a file, just go for a sendfile. If you have it in memory (because you processed it somehow, or just generated), use the approach I mentioned earlier.