I am using nodejs to call my tcp subscriber to read messages.
var zmq = require("zeromq"),
sock = zmq.socket("sub");
sock.connect("tcp://pubsub.besteffortc.com:7658");
sock.subscribe("/ASD/Travel");
console.log("Subscriber connected to port 3000");
sock.on("message", function(topic, message) {
console.log(
"received a message related to:",
topic.toString(),
"containing message:",
message.toString()
);
});
Output I am getting is the following. What might be the issue ?
- can it be that my IP address needs to be whitelisted to see actual data ?
Note: Topic and URL name is not the actual value its dummy values.
Q : What might be the issue ?
There is one sure point in the ZeroMQ - if a message gets delivered at all ( for which a Zen-of-Zero does not pose any warranty ), then it is sure to be an exact bit-by-bit copy of the message, that the originator ( the PUB in the case here ) was intending to .send()
If you .recv() anything ( be it by the native API level .recv()-method or the wrapper provided .on( "message", f(...){...} )-handler ), be sure it carries the 1:1 bitmap of the payload provided by the originator ( unless your language wrapper did a dirty job of obfuscating your use of the documented ZeroMQ API, producing some nasty garbage ( which I would not bet on, yet it might be an issue - try any other ZeroMQ message-source, that is under your own control, to see if this happens or not ) )
Q : can it be that my IP address needs to be whitelisted to see actual data ?
As explained in detail above, this item is not a ZeroMQ issue. If your app-level / ISO-OSI-above-5 behaviour depends on contracting some kind of abonnement, only after which you get "access-validation" ( be it by whitelisting or other kind ), then your problem stays in the business domain, not the programming one.
ZeroMQ has nothing to do with any of this. ZMQ/RFC-documentation is explicit on each ZeroMQ service-composition and leaves zero-space for random or undocumented behaviours.
Related
I am trying to implement a pub/sub broker with ZeroMQ where it is possible to restrict clients from subscribing to prefixes they are not allowed to subscribe to. I found a tutorial that tries to achieve a similar thing using the ZMQ_XPUB_MANUAL option. With zeromq.js it is possible to set this option:
import * as zmq from "zeromq";
// ...
const socket = new zmq.XPublisher({ manual: true });
After setting this option I am able to receive the subscription messages by calling .receive() on this socket:
const [msg] = await socket.receive();
But I have no Idea how to accept this subscription. Usally this is done by calling setSockOpt with ZMQ_SUBSCRIBE but I don't know how to do this with zeromq.js.
Is there a way to call setSockOpt with zeromq.js or is there another way to accept a subscription?
Edit
I tried user3666197's suggestion to call setSockOpt directly, but I am not sure how to do this. Rather than doing that, I took another look in the sources and found this: https://github.com/zeromq/zeromq.js/blob/master/src/native.ts#L617
It seems like setSockOpt is exposed to the TypeScript side as protected methods of the Socket class. To try this out, I created my own class that inherits XPublisher and exposed an acceptSubscription message:
class CustomPublisher extends zmq.XPublisher {
constructor(options?: zmq.SocketOptions<zmq.XPublisher>) {
super(options);
}
public acceptSubscription(subscription: string | null): void {
// ZMQ_SUBSCRIBE has a value of 6
// reference:
// https://github.com/zeromq/libzmq/blob/master/include/zmq.h#L310
this.setStringOption(6, subscription);
}
}
This works like a charm! But do not forget to strip the first byte of the subscription messages, otherwise your client won't receive any messages since the prefix won't match.
Q : "Is there a way to call setSockOpt() with zeromq.js or is there another way to accept a subscription?"
So, let me first mention Somdoron to be, out of doubts & for ages, a master of the ZeroMQ tooling.
Next comes the issue. The GitHub-sources, I was able to review atm, seem to me, that permit the ZMQ_XPUB-Socket-archetypes to process the native API ZMQ_XPUB_MANUAL settings ( re-dressed into manual-property, an idiomatic shift ), yet present no method (so far visible for me) to actually permit user to meet the native API explicit protocol of:
ZMQ_XPUB_MANUAL: change the subscription handling to manual...with manual mode subscription requests are not added to the subscription list. To add subscription the user need to call setsockopt() with ZMQ_SUBSCRIBE on XPUB socket./__ from ZeroMQ native API v.4.3.2 documentation __/
Trying to blind-call the Socket-inherited .SetSockOpt() method may prove me wrong, yet if successful, it may be a way to inject the { ZMQ_SUBSCRIBE | ZMQ_UNSUBSCRIBE } subscription-management steps into the XPUB-instance currently having been switched into the ZMQ_XPUB_MANUAL-mode.
Please test it, and if it fails to work via this super-class inherited method, the shortest remedy would be to claim that collision/conceptual-shortcomings directly to the zeromq.js maintainers ( it might be a W.I.P. item, deeper in their actual v6+ refactoring backlog, so my fingers are crossed for either case ).
The documentation for connect method says,
Connects to the socket at the given remote address and returns immediately. The connection will be made asynchronously in the background.
But, await does not seem to be applicable as shown in their example of subscriber code.
subscriber.js
const zmq = require("zeromq")
async function run() {
const sock = new zmq.Subscriber
sock.connect("tcp://127.0.0.1:3000") //Happens async; can we await this?
sock.subscribe("kitty cats")
console.log("Subscriber connected to port 3000")
for await (const [topic, msg] of sock) {
console.log("received a message related to:", topic, "containing message:", msg)
}
}
run()
Also, what error(s) maybe raised by the connect() method? I provided an 'obscene' port number such as, 8124000, to connect. I was hoping for some error messages to be raised.
Q : "what error(s) maybe raised by the connect() method?"
The Error(s) part
The ZeroMQ native API distinguishes ( unchanged since v2.1 ) these errors for this :
EINVAL
The endpoint supplied is invalid.
EPROTONOSUPPORT
The requested transport protocol is not supported.
ENOCOMPATPROTO
The requested transport protocol is not compatible with the socket type.
ETERM
The ØMQ context associated with the specified socket was terminated.
ENOTSOCK
The provided socket was invalid.
EMTHREAD
No I/O thread is available to accomplish the task.
Yet your actual observer is dependent on the zeromq.js re-wrapping these principal states, so the best next step is to re-read the wrapper source code, so as to see, how these native API error states get actually handled inside the zeromq.js-wrapper.
The remarks :
The following socket events can be generated. This list may be different depending on the ZeroMQ version that is used.
Note that the error event is avoided by design, since this has a special behaviour in Node.js causing an exception to be thrown if it is unhandled.
Other error names are adjusted to be as close to possible as other networking related event names in Node.js and/or to the corresponding ZeroMQ.js method call. Events (including any errors) that correspond to a specific operation are namespaced with a colon :, e.g. bind:error or connect:retry.
are nevertheless quite warning, aren't they?
The await part
The MCVE-code ( as-is ) is unable to reproduce the live-session, so best adapt the MCVE-code so as to get run-able and we can proceed further on this.
I am unable to get two users chatting to each other despite reducing the complexity and the potential code that could have caused the issue.
I am able to emit to all connected sockets so I have established it's not an issue in context of emit/on structure but rather; coming from the way i'm handling the private socket ids.
I have tried various versions of trying to send the private message to the correct socket id; I have tried older ways such as socket.to and the current way from the docs which is io.to(sockid).emit('event' message); all these variations have been unable to help me. I have consoled out the socket id I have on my Angular client side by printing console.log('THIS IS MY SOCKET '+this.socket.id) and comparing it to the value I have in Redis using redis-cli and they both match perfectly every time which doesn't give me too much to go on.
problem arises here:
if (res === 1) {
_active_users.get_client_key(recipient)
.then(socket_id => {
console.log('======='+io.sockets.name)
console.log('I am sending the message to: '+ recipient + 'and my socket id is'+ socket_id)
// socket.to(socket_id)socket.emit('incoming', "this is top secret"
io.of('/chat').to(socket_id).emit('incoming', "this is top secret")
})
.catch(error => {
console.log("COULD NOT RETRIEVE KEY: " + error)
})
Here is the link to the pastebin with more context:
https://pastebin.com/fYPJSnWW
The classes I import are essentially just setters and getters for handling the socket id you can think of them as just a worker class that handles Redis actions.
Expected: To allow two clients to communicate based on just their socket ids.
Actual:
am able to emit to all connected sockets and receive the expected results but the problem arises when trying to send to a specific socket id from a unknown reason.
Issue was coming from my front end.. I hope nobody has a headache like this! but here is what happened; when you're digging your own hole you often don't realise how deep you got yourself if you don't take the time to look around. I had two instances of the sockets. I instantiated both and used the one to connect and the other to send the message; which of course you cannnot do if you want things to work properly. So what I did was created only one instance of the socket in and and passed that ref of the socket around where I needed it which is ( sendMessage(username, socket) getMessage(socket)
ngOnInit(
this.socket = io.connect('localhost:3600',{
reconnection: true,
reconnectionDelay: 1000,
reconnectionDelayMax : 5000,
reconnectionAttempts: Infinity});
I'm trying to figure out basic websocket communication using node.js, the "ws"-package (which seems to be a very popular websocket package from npmjs.com) and the bitfinex.com (a cryptocurrency exchange) websocket API.
I want to read the public Ticker for a certain currency-pair, the docs are here: https://docs.bitfinex.com/v2/reference#ws-public-ticker
My result so far is working but is still much different from what I am supposed to get according to the docs.
I am working with this code snippet taken from the documentation linked above:
const ws = require('ws')
const w = new ws('wss://api.bitfinex.com/ws/2')
w.on('message', (msg) => {
console.log(msg)
})
let msg = JSON.stringify({
event: 'subscribe',
channel: 'ticker',
symbol: 'tBTCUSD'
})
w.on('open', () => {
w.send(msg)
})
Which works so far by outputting to the console the message from the subscribed channel:
[1,[14873,23.49464465,14874,61.09031263,1087,0.0789,14872,56895.20497085,15500,13891]]
But now, and here is the issue, in the docs the response looks different. How would I determine which number is what? I should be able to get all kinds of more information from the response, no?
The given example response looks like this:
// response - trading
{
event: "subscribed",
channel: "ticker",
chanId: CHANNEL_ID,
pair: "BTCUSD"
}
How does this relate to that array of numbers I get? How would I for example read the "pair:" field ("BTCUSD") or any of the other listed fields, like (BID, BID_PERIOD, VOLUME, HIGH, LOW etc.)? Am I missing something obvious?
I know this is a lot to ask at once but maybe someone knows one or two good examples or hints to enlighten me. Thanks in advance!
Kind regards,
s
The overall websocket scheme for this API is described in https://bitfinex.readme.io/v2/docs/ws-general If you haven't already read that page, now would be a good time to do it.
For your example program you should have seen info and subscribed events as the first two messages from the websocket. info should have been sent as soon as the websocket connection was established, and subscribed should have been sent in response to your subscribe request.
After that, you should see a ticker snapshot message followed by periodic ticker update messages for the channel that you subscribed to. These are the JSON arrays that you're seeing. The format of these messages for a public ticker channel is is described at https://bitfinex.readme.io/v2/reference#ws-public-ticker -- click the Snapshot and Update headings in the dark green 'details' bar to see the definitions. In this case snapshots and updates use the same format:
[ CHANNEL_ID,
[ FRR, BID, BID_PERIOD, BID_SIZE, ASK, ASK_PERIOD, ASK_SIZE,
DAILY_CHANGE, DAILY_CHANGE_PERC, LAST_PRICE, VOLUME, HIGH, LOW
]
]
with meanings as described in the 'Stream Fields' table at the above URL.
You can parse these messages as JSON strings and access the field values just as you would for any array.
It's a little strange that the API sends these as arrays rather than as objects with named attributes. I imagine they're looking to keep these messages compact because they make up the bulk of the traffic.
Currently, I'm testing my Node.js, Socket.io server on localhost and on devices connected to my router.
For testing purposes, I would like to simulate a delay in sending messages, so I know what it'll be like for users around the world.
Is there any effective way of doing this?
If it's the messages you send from the server that you want to delay, you can override the .emit() method on each new connection with one that adds a short delay. Here's one way of doing that on the server:
io.on('connection', function(socket) {
console.log("socket connected: ", socket.id);
// override the .emit() method
const emitFn = socket.emit
socket.emit = (...args) => setTimeout(() => {
emitFn.apply(socket, args)
}, 1000)
// rest of your connection handler here
});
Note, there is one caveat with this. If you pass an object or an array as the data for socket.emit(), you will see that this code does not make a copy of that data so the data will not be actually used until the data is sent (1 second from now). So, if the code doing the sending actually modifies that data before it is sent one second from now, that would likely create a problem. This could be fixed by making a copy of the incoming data, but I did not add that complexity here as it would not always be needed since it depends upon how the caller's code works.
An old but still popular question. :)
You can use either "iptables" or "tc" to simulate delays/dropped-packets. See the man page for "iptables" and look for 'statistic'. I suggest you make sure to specify the port or your ssh session will get affected.
Here are some good examples for "tc":
http://www.linuxfoundation.org/collaborate/workgroups/networking/netem