I am currently developing as system from using storm. Data will come in from a kafka spout to a few bolt the the exit will be to a botl which sets a tcp connection to a NodeJs server application.
There is no problem when testing with low input of data. The problem arises when I test it with a large amount of data which the stream seems to be unstable.
I tried two ways to set the connection: Setting the connection only once and setting the connection and close once done. When I set the connection once the stream is coming in so fast into my NodeJs application the data got messed up and hit a data format error.
Opening and closing a connection makes the stream into NodeJs really unstable, it is ok the first few kb but after a while it stop the stream the flood stop and flood thr stream.
I'm just wondering: is there is any other way other then setting a tcp connection from storm to NodeJs or is there a way to make a connection once to NodeJs and make the message que. I'm running out of ideas any help on this issue?
Related
I have a NodeJS software running on ubuntu on a mini-computer used as a streaming system - like a middleware, or a streaming agent between my stream server and a proxy - communicating the stream data from the stream server to the proxy through a websocket.
Assume the connection to the stream server is always OK, and the problem arises only when there is an issue in the websocket connection to the proxy, or the receiving data (from the stream server) is too much for the current sending rate to the proxy (hence back-pressure on the websocket).
I want to find a reliable way for monitoring the websocket, and based on the back-pressure it feels, decide whether to switch to my local recording system.
This back-pressure might occur very often since network is not stable, but how should I realize "NOW" is the time for switching to the local-record and not wait a bit more hoping for the network connection to get better soon?
// a small pseudo code:
listener.on('data-from-stream-server', (data) => {
// do a lot of stuff to the data
websocket.send(modifiedData);
});
// connection monitoring:
...
if (websocketDisconnected) {
// switch to local record
}
else if (backpressureOnTheWebesocket) {
// decide whether to switch to local record OR keep sending data at current LOW rate
}
...
else {
// all good - keep sending data through the websocket
}
...
It's possible for me to replace websocket with any other library having the same functionalities or some wrapper over them
I read some software monitoring solutions/tools such as dotcom-monitor, also this stackoverflow answer seemed to be helpful, but still I'm not sure what to decide when facing too much back-pressure(usually caused by network issues).
Note that I prefer not to keep sending data for so long while experiencing too much back-pressure.
Thanks in advance
I've got a rather vague question, but I'm hoping for some help.
I have icecast installed on an unbuntu server, with two server blocks setup using nginx.
Icecast is set to stream on https://stream.domain.com:8443/stream
I have a node.js app running on https://app.domain.com
Is there a way to "listen" to the icecast domain/stream/port using node and start a function when the stream starts, and stop it when the stream stops?
I'm not hoping for a full solution, just clues to point me in the right direction! Thank you in advance.
Make a request to
https://stream.domain.com:8443/status-json.xsl
it will return a JSON status of the server, then see the "source" key in this JSON - not empty "source" will indicate that there is a source connected to the server and Icecast is broadcasting something.
That JSON also provides a lot of additional info you can also use.
Also a basic check is to try to connect to port 8443 - if connection is not happening in, say, 5 seconds - that will indicate that Icecast is completely down.
I have tick based server and client in Unity3D. Server sending data to clients. Middleware is NodeJS server.
My question is how much data I can transfer every tick (I have 25 Ticks per second now) before server start unsync itself and clients starts to getting data late? I am sending just JSON strings.
Now I am sending about 1kB of data every tick. Its too much or its ok for NodeJs to server this to clients every tick?
I am counting that when I have 100 clients connected and 1kB/tick, I need fom NodeJs server to serve 2,44MB/s. I mean, internet connection is not problem, but is this possible?
This will likely be most dependent on the hardware you end up running the server on, if you can distribute the task among multiple processes/servers, and what protocol you're using to send the data.
The easiest way to test the hardware that you currently have would be running a simple benchmark.
I put together a quick project to do some benchmarking with Socket.io
https://github.com/briancw/socket-io-stress-test
You'll need a way to simulate connected clients. I have previously created a stress testing tool that may be useful for this: https://www.npmjs.com/package/m65
It uses headless browsers, so it should be able to make actual websocket connections so you can simulate very realistically.
I am trying to write internal transport system.
Data should be transferred from client to server using net sockets.
It is working fine except handling of network issues.
If I place firewall between client and server, on both sides I will not see any error, so data will continue to fill kernel buffer on client side.
And if I will restart app in this moment I will lose all data in buffer.
Question:
Do we have any way to detect network issues?
Do we have any way to get data back from kernel buffers?
Node js exposes the low level socket api to you very directly. I'm assuming that you are using a TCP socket to send and receive data.
One way to ensure that there is an active connection between the client and server is to send heartbeat signals back and forth. If you fail to receive a heartbeat from the server while sending data, you can assume that the connection failed.
As for the second part of your question: There is no easy way to get data back from kernel buffers. If losing the data will be a problem, I would make sure to write it to disk.
I try to make connection between Leap Motion and mobile devices in Unity3D, via node.js.
Here are two examples I found online using node.js, this and this.
Now, I have successfully receive data from Leap Motion in this procedure:
Receiving JSON-formatted messages from "localhost:6437" in node.js
Parsing and Writing received data on another port (I use "localhost:8000") in node.js
Reading stream from port: 8000 in Unity3D
However, I wonder if it's possible to receive data DIRECTLY from "localhost:6437"? then maybe node.js is not needed.
I have tried to write in C# using TcpClient, but when I check "NetworkStream.DataAvailable", it returns false.
Thanks for your help.
For that you would need a WebSocket client, not a TCP client.