I'm implementing a tcp protocol in Node.
Full source:
https://github.com/roelandmoors/ads.js/blob/master/ads.js
specs:
http://infosys.beckhoff.com/content/1033/tcadsamsspec/html/tcadsamsspec_amstcppackage.htm?id=17754
The problem is that I use this to send a package:
this.tcpClient.write(buf);
If I send multiple commands, then multiple commands are combined into a single tcp packet.
This doesn't work.
There are more questions about this on SO, but they recommend using a delimeter.
But since I can't change the protocol this isn't an option.
Isn't there a simple solution to flush the socket?
socket.setNoDelay() doesn't help.
Edit: I also tried to use the drain event to send the next write, but the event is never called?
Update:
This seems to solve the problem, but is very uggly and I don't now if it always works.
Instead of writing it directly I write to a buffer:
this.writeFILO.push(buf);
Every cycle(?) I'm writing a package to the socket stream:
var sendCycle = function(ads) {
if (ads.writeFILO.length > 0) {
ads.tcpClient.write(ads.writeFILO.shift());
}
setTimeout(function() {
sendCycle(ads);
}, 0);
}
I refer to the socket.write(data, [encoding], [callback]) API:
The optional callback parameter will be executed when the data is finally written out - this may not be immediately.
So, set up a queue (array is fine) which holds messages to be sent.
When the above callback is being called, check the queue and send if needed..
This however does not guarantee what you're looking for, you'll have to test. Unfortunately the docs don't explicitly state when there's an acknowledgement from the remote end point that it actually received that message...
In the end, as you concluded, TCP is a stream.
An interesting idea which just came up to me now, however, if you're FORCED TO use an existing protocol, then open two TCP connections.
When one connection acknowledges (whatever the higher-level protocol is) receiving that message, send the next through the other one... and so forth..
Anyway, nice challenge :)
I was wrong. TCP is a stream and the protocol works like a stream, but I didn't handle it like a stream.
PS: sending seperate messages seemed to work with setImmediate()
I know that this is an old question, and I'm not 100% sure I understand what you are looking for, but there is a way to flush a socket in node. First you need to implement a Transform class.
See here for example: https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream.
Then you can take your stream and pipe it through your transform before piping it into your socket.
I do not own this node module but I have seen an example of this here: https://github.com/yongtang/clamav.js/blob/master/index.js#L8
Related
The TCP server I am hitting (trying to use the built in node TLS Socket) expects a handshaking process of send/receives in a certain order (send, on receive of success, send more messages, on success, send more, etc). The receive messages does not have anything to let me know which send it is responding to, so I am not able to easily use the streaming nature of the built in TCP Node library.
Any ideas of what the best way to handle this case in Node?
example (python), and this is example of the process:
s.send("hello")
s.send("send this 1")
reply = s.recv()
message = reply[0]
if message == OK:
print('Got OK for hello')
s.send("send this 2")
reply = s.recv()
message = reply[0]
if message == OK:
print('Got it')
else:
raise Exception('Failed to send hello')
When you have non-blocking I/O and you want to do something such as send data, read specific response from that send you need to set up some appropriate state so that when the next set of data come in, you know exactly what it belongs to and therefore you know what to do with it.
There are a number of ways to do that I can think of:
Create a general purpose state machine where you send data and read data and whenever you read data, you can tell what state the socket is in and therefore what you are supposed to do with the data you read.
Create a temporal set of listeners where you send data, then add a temporal listener (you can use .once()) for incoming data that is specially designed to process it the way you are expecting this response to be. When the data arrives, you make sure that listener is removed.
Your pseudo-code example does not show enough info for anyone to make a more concrete suggestion. TCP, by its very nature is stream driven. It doesn't have any built-in sense of a message or a packet. So, what you show doesn't even show the most basic level of any TCP protocol which is how to know when you've received an entire response.
Even your reply = s.recv() shown in some other language isn't practical in TCP (no matter the language) because s.recv() needs to know when it's got a complete message/chunk/whatever it is that you're waiting to receive. TCP delivers data in order sent, but does not have any sense of a particular packet of information that goes together. You have to supply that on top of the TCP layer. Common techniques used for delineating messages are :
Some message delimiter (like a carriage return or line feed or a zero byte or some other tag - all of which are known not to occur inside the message itself)
Sending a length first so the reader knows exactly how many bytes to read.
Wrapping messages in some sort of container where the start and end of the container are made clear by the structure of the container (note options 1 and 2 above are just specific implementations of such a container). For example, the webSocket protocol uses a very specific container model that includes some length data and other info.
I was thinking of showing you an example using socket.once('data', ...) to listen for the specific response, but even that won't work properly without knowing how to delineate an incoming message so one knows when you've received a complete incoming message.
So, one of your first steps would be to implement a layer on top of TCP that reads data and knows how to break it into discrete messages (knows both when a complete message has arrived and how to break up multiple messages that might be arriving) and then emits your own event on the socket when a whole message has arrived. Then, and only then, can you start to implement the rest of your state machine using the above techniques.
I am building a multithreaded async HTTP server on top of mio.
How should I handle events on client TcpStream connections?
For is_readable it's pretty straightforward: I read the request and write the response.
For other kinds I am not sure. Should I shutdown() the stream if I get is_error or is_hup? What about is_none()?
All things that you mention have very precise meaning and map directly to POSIX/BSD Socket API. It's up to you to decide.
is_hup on Read mean the other side hanged-up it's sending side. Meaning it won't send you anything again. However it might have kept the reading open, and you might still want to send some data to it.
shutdown closes Reading/Writing/Both https://doc.rust-lang.org/std/net/enum.Shutdown.html , so it's up to you what and when you want to do.
TcpStream internally holds FileDesc and that will close the fd when you drop it, so if you don't shutdown manually everything will be closed anyway, as soon as you remove given TcpStream from usage. https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/fd.rs#L217
many people said: you can use socket.on('end',...) to get all chunks, but I want the socket keep connecting, so the event 'end' never fired.
how do I know I reveived all data on socket.on('data',...) ?
It depends on the underlying protocol. You have to have some format/layout to the data so that you know how to parse it. For example, you might have newline-delimited messages or you might have a length-prefixed messages.
I have a ClearTextStream for a TLS connection and I want to check if "end" was already called. The actual problem is, that I'm trying to write something into the stream and I get an "write after end" error.
Now to avoid that, I just want to check if "end" was already called. I do have an "close" event, but it isn't fired in all cases.
I can't find it in the documentation and I couldn't find anything like that by googling.
I could check the error event (which is throwing "write after end" for me) and handle the situation there - but is there really no way to check this in the beginning?
Thanks!
If you get a write after end error, that means that you are trying to write data to a Writable stream that has been closed (ie. that can't accept anymore input data). When a writable stream closes, the finish event is emitted (see the documentation). On the other hand, the close event is emitted by a Readable stream, when the underlying resource is closed (for instance when the file descriptor you are reading is closed).
As a ClearTextStream is a Duplex stream, it can emit both close and finish events, but they don't mean the same thing. In your particular case, you should listen to the finish event and react appropriately.
Another solution would be to check the this.ended and this.finished booleans (see the source code), but I wouldn't recommend that as they are private variables and only reflect the implementation details, not the public API.
I'm sending various custom message structures down a nonblocking TCP socket. I want to send either the whole structure in one send() call, or return an error with no bytes sent if there's only room in the send buffer for part of the message (ie send() returns EWOULDBLOCK). If there's not enought room, I will throw away the whole structure and report overflow, but I want to be recoverable after that, ie the receiver only ever receives a sequence of valid complete structures. Is there a way of either checking the send buffer free space, or telling the send() call to do as described? Datagram-based sockets aren't an option, must be connection-based TCP. Thanks.
Linux provides a SIOCOUTQ ioctl() to query how much data is in the TCP output buffer:
http://www.kernel.org/doc/man-pages/online/pages/man7/tcp.7.html
You can use that, plus the value of SO_SNDBUF, to determine whether the outgoing buffer has enough space for any particular message. So strictly speaking, the answer to your question is "yes".
But there are two problems with this approach. First, it is Linux-specific. Second, what are you planning to do when there is not enough space to send your whole message? Loop and call select again? But that will just tell you the socket is ready for writing again, causing you to busy-loop.
For efficiency's sake, you should probably bite the bullet and just deal with partial writes; let the network stack worry about breaking your stream up into packets for optimal throughput.
TCP has no support for transactions; this is something which you must handle on layer 7 (application).