Solace via NodeJS Not Waiting for Success - node.js

I'm trying to get Solace (a queuing system) to create a session, then send a message on that session. Instead, it listens to my session creation, receives all the event handlers (I registered all of them), fails to create that session and fails to tell me why. I cannot get this to WAIT for completion. I suspect if it had a few more microseconds, the session would be completed. The promises I have are not being kept. Any awaits that I put in are dutifully ignored.
The Typescript code below is attempting to make a connection to Solace to put a message on a queue. At a high level, it works by getting an instance of the Solace module, then it creates a Session, then with that session, it sends the message. Session creation returns an actual Session and not a promise. That doesn't mean it actually works though. Instead, you have to register an event handler. Because I don't see any of the console.log()s, I believe the createSession event handlers are not being run. Despite registering an event handler for every error in the session handler, Solace neither made the connection, nor said why. As far as I can tell, there's no concept of getting the current state of the session either.
Please note, in previous attempts, I was getting a WaitingForDNS error on the send. It also runs relatively quickly, so I don't think it's doing very much. When I turned on tracing, the most I could tell is that eventually Solace decided to resolve the IP address.
Please see my wishes annotated below:
export class TopicPublisher {
public async connect() {
// Return me a Promise for the Session; either the Session is fully loaded
// loaded, or it's rejected
return new Promise<Session>((resolve, reject) => {
if (this.session !== null) {
this.log("Already connected and ready to publish");
reject();
}
try {
this.session = this.solace.SolclientFactory.createSession({
// solace.SessionProperties
url: this.hosturl,
vpnName: this.vpn,
userName: this.username,
password: this.pass,
connectRetries: 1,
});
} catch (error: any) {
this.log('Error on creating session: ' + error.toString());
reject(error);
}
//The UP_NOTICE dictates whether the session has been established
this.session.on(solace.SessionEventCode.UP_NOTICE, () => {
// *** At this point, return the session as a successfully completing promise ***
this.log("=== Successfully connected and ready to subscribe. ===");
resolve(this.session);
});
//The CONNECT_FAILED_ERROR implies a connection failure
this.session.on(solace.SessionEventCode.CONNECT_FAILED_ERROR, (sessionEvent: { infoStr: string; }) => {
this.log("Connection failed to the message router: " + sessionEvent.infoStr + " - check correct parameter values and connectivity!");
reject(`Check the settings in game-config.ts and try again!`);
});
// Register every event handler in vain attempt at getting Solace to tell me
// why it does not work
let otherErrors = [
solace.SessionEventCode.DOWN_ERROR,
solace.SessionEventCode.REJECTED_MESSAGE_ERROR,
solace.SessionEventCode.SUBSCRIPTION_ERROR,
solace.SessionEventCode.SUBSCRIPTION_OK,
solace.SessionEventCode.VIRTUALROUTER_NAME_CHANGED,
solace.SessionEventCode.REQUEST_ABORTED,
solace.SessionEventCode.REQUEST_TIMEOUT,
solace.SessionEventCode.PROPERTY_UPDATE_OK,
solace.SessionEventCode.PROPERTY_UPDATE_ERROR,
solace.SessionEventCode.CAN_ACCEPT_DATA,
solace.SessionEventCode.RECONNECTING_NOTICE,
solace.SessionEventCode.RECONNECTED_NOTICE,
solace.SessionEventCode.REPUBLISHING_UNACKED_MESSAGES,
solace.SessionEventCode.ACKNOWLEDGED_MESSAGE,
solace.SessionEventCode.UNSUBSCRIBE_TE_TOPIC_OK,
solace.SessionEventCode.UNSUBSCRIBE_TE_TOPIC_ERROR,
solace.SessionEventCode.MESSAGE,
solace.SessionEventCode.GUARANTEED_MESSAGE_PUBLISHER_DOWN
];
for (let errorCodeIndex = 0; errorCodeIndex < otherErrors.length; errorCodeIndex++) {
this.log('Registering error handler code: '+otherErrors[errorCodeIndex]);
this.session.on(otherErrors[errorCodeIndex], (sessionEvent: { infoStr: string; }) => {
this.log("Connection failed with error code : " + otherErrors[errorCodeIndex] + " " + sessionEvent.infoStr);
reject(`Check the config settings`);
});
}
//DISCONNECTED implies the client was disconnected
this.session.on(solace.SessionEventCode.DISCONNECTED, (sessionEvent: any) => {
this.log("Disconnected.");
if (this.session !== null) {
this.session.dispose();
//this.subscribed = false;
this.session = null;
}
});
try {
this.session.connect();
} catch (error: any) {
reject();
}
});
};
public async publish(topicName: string, payload: any) {
// This builds a message payload, it works fine
let solaceMessage = this.getSolaceMessage(topicName, payload);
try {
// *** It does *not* wait for the connection ***
console.log('##This point is reached');
let localSession = await this.connect();
// UP_EVENT ***SHOULD*** have happened, but it does not wait for any events
// or promises to be completed.
console.log('##This point is reached');
console.log('localSession =' + localSession);
localSession.send(solaceMessage);
} catch (error) {
}
};
}
let topicPublisher: TopicPublisher = new TopicPublisher(getInitializedSolaceModule(),
argumentParser.hosturl,
argumentParser.usernamevpn,
argumentParser.username,
argumentParser.vpn,
argumentParser.pass,
argumentParser.topicName);
topicPublisher.publish(argumentParser.topicName, readMessageFromFile(argumentParser.messageFileSpecification)).then(() => {
console.log('##This point is reached');
}, () => {
console.log('##BP10.5 Error handler on publish');
}
).catch(error => {
console.log('publish error' + error);
});
console.log('##This point is reached');
topicPublisher.disconnect();
console.log('##This point is reached');
Solace API documentation is at https://docs.solace.com/API-Developer-Online-Ref-Documentation/nodejs/index.html, but I'm not sure this is a Solace error.

I don't have great exposure to TypeScript - is it possible that the check 'this.session !== null' ends up rejecting the promise, and no session is created. An uninitialized value, if it holds undefined, a !== null check would fail. Maybe your log output sequence can shed light on this.
My apologies, this is a silly point, and not offering any direct help.

Related

GetDone in ibmmq (Node.js) doesn't stop listener of the queue

I'm using ibmmq module https://github.com/ibm-messaging/mq-mqi-nodejs
I need to get message by CorrelId and then stop listen to the queue.
async listen(queue: string, messageId?: string, waitInterval?: number) {
let mqmd = new mq.MQMD()
let gmo = new mq.MQGMO()
gmo.Options = this.MQC.MQGMO_NO_SYNCPOINT | this.MQC.MQGMO_WAIT | this.MQC.MQGMO_CONVERT | this.MQC.MQGMO_NO_PROPERTIES | this.MQC.MQGMO_FAIL_IF_QUIESCING
gmo.MatchOptions = this.MQC.MQMO_MATCH_CORREL_ID
mqmd.CorrelId = this.hexToBytes(messageId)
gmo.WaitInterval = this.MQC.MQWI_UNLIMITED
mq.Get(obj as mq.MQObject, mqmd, gmo, getCB)
}
And the getCB function:
getCB(err: mq.MQError, hObj: mq.MQObject, gmo: mq.MQGMO, mqmd: mq.MQMD, buf: Buffer, hConn: mq.MQQueueManager) {
if (err) {
...
} else {
...
console.log('GetDone:', hObj)
mq.GetDone(hObj, err => {
console.log('GetDoneError:', err)
})
}
}
I start listening to the queue. Then I put a message with the CorrelId there. The listener get it. I see 'GetDone' in the terminal.
And then I put a message with the same CorrelId. And I get that message and Error.
GetDoneError: MQError: GetDone: MQCC = MQCC_FAILED [2] MQRC = MQRC_HOBJ_ERROR [2019]
at Object.exports.GetDone (/home/apps/connector/node_modules/ibmmq/lib/mqi.js:2316:11)
at MqiConnector.getCB (/home/apps/connector/src/wmq-mqi-connector.js:206:20)
at /home/apps/connector/node_modules/ibmmq/lib/mqi.js:2263:14
at Object.<anonymous> (/home/apps/connector/node_modules/ffi-napi/lib/_foreign_function.js:115:9) {
mqcc: 2,
mqccstr: 'MQCC_FAILED',
mqrc: 2019,
mqrcstr: 'MQRC_HOBJ_ERROR',
version: '1.0.0',
verb: 'GetDone'
}
Looks like the loop with the function getCB didn't stop after GetDone.
I get messages with this CorrelId as many times as I send them. And every time I see this error. The listener is still running.
What am I doing wrong?
I suspect that you are calling GetDone twice and the second time hObj is invalid in the call to mq.GetDone.
mq.GetDone(hObj, err => {
console.log('GetDoneError:', err)
})
I think you have fallen foul of Node.js asynchronous nature and you have hit a timing issue. IE. the cleanup following GetDone has not completed around the same time as the next message is being retrieved.
The function GetDone seems to be synchronous and can be found in https://github.com/ibm-messaging/mq-mqi-nodejs/blob/3a99e0bbbeb017cc5e8498a59c32967cbd2b27fe/lib/mqi.js
The error appears to come from this snippet in GetDone -
var userContext = getUserContext(jsObject);
var err;
if (!userContext) {
err = new MQError(MQC.MQCC_FAILED,MQC.MQRC_HOBJ_ERROR,"GetDone");
} else {
deleteUserContext(jsObject);
}
First time through userContext is found and then deleted. Second time round userContext doesn't exist and the error is thrown.
The Sample in the repo - https://github.com/ibm-messaging/mq-mqi-nodejs/blob/72fba926b7010a85ce2a2c6459d2e9c58fa066d7/samples/amqsgeta.js
only calls GetDone in an error condition, ie. when there are either no messages on the queue or there has been a problem getting the next message off the queue.
function getCB(err, hObj, gmo,md,buf, hConn ) {
// If there is an error, prepare to exit by setting the ok flag to false.
if (err) {
if (err.mqrc == MQC.MQRC_NO_MSG_AVAILABLE) {
console.log("No more messages available.");
} else {
console.log(formatErr(err));
exitCode = 1;
}
ok = false;
// We don't need any more messages delivered, so cause the
// callback to be deleted after this one has completed.
mq.GetDone(hObj);
} else {
if (md.Format=="MQSTR") {
console.log("message <%s>", decoder.write(buf));
} else {
console.log("binary message: " + buf);
}
}
}
Whereas you are calling it when you have retrieved a message. You may need to create a guard that stops you calling it twice.
As for why the second message has been obtained, without an error, you might need to raise an issue on the ibmmq module.

Returning a value from a mix of async/sync function from a provider (different script) to express server

Apologies for asking this question - I know there are tons of information about async functions out there but I seem to have tried everything and cannot find a solution..
First of all let me outline the architecture of my program. There are two scripts: a main server script (node.js, express), which processes GET requests and provider script, which deals with the blockchain in the background to return some values. The server script is responsible for invoking a method that returns a value from the provider. The provider does all the work.
The snippet of the provider script:
getInfo(index, account, key) {
//Waiting on an asynchronous method, which does some work in the blockchain in the background; everything functions as it should be
try {
await this.getBlockchain
(
index
, account
, key
).then(result => {
// Here instead I invoke a SYNCHRONOUS method, which simply formats the response in a correct way
const reply = this.sendReply(result)
console.log(reply) //Logs the correct reply in the format in which the server is expecting it
return reply;
});
}
catch (error) {
return { error: 003, result: false };
}
}
The snippet of the server script:
server.get("/getAccount", async (req, res) => {
let index = req.query.index;
let account = req.query.account;
let key = req.query.key;
// Here I also check for the validity of the query values, irrelevant to this issue
// The provider class is imported as provider, hence, the provider.method (this has been tested many times before)
try {
await provider.getInfo(index, account, key).then(reply => {
const { error: infoError, result: infoValue } = reply
if (infoError == false) {
res.send(`${infoValue}`);
} else {
res.send(`${infoError}`);
};
});
}
catch (error) {
res.send("008");
}
}
);
I honestly have no idea how to approach this; I tried self-contained async function on the server side as well as different syntax but the reply is always undefined even though the reply from a synchronous call in the provider is correct.
Could someone help me to understand what I'm doing wrong? This is my first time working with async with numerous scripts and functions and I'm finding it very confusing.
Thank you so much!
With your current structure, you need to return the result of the await so that the top level of your function is returning something from the async function.
async getInfo(index, account, key) {
try {
let retVal = await this.getBlockchain(index, account, key).then(result => {
return this.sendReply(result);
});
return retVal;
} catch (error) {
return { error: 003, result: false };
}
}
But, really, it's a better coding style to not mix await and .then() and to just go with one style like this:
async getInfo(index, account, key) {
try {
let result = await this.getBlockchain(index, account, key);
return this.sendReply(result);
} catch (error) {
return { error: 003, result: false };
}
}
Note, this function never rejects because it's catching its own rejections and turning it into a resolved value. So, the caller cannot use .catch() to see errors. The caller must always check for the error property in the resolved object. This is not usually how you program with promises. It can be made to work, but often does not meet the expectations of the caller (as errors are usually communicated back via rejected promises).
This has to be a dup. but... Don't mix await and .then.
You simply try/catch around await.
try {
const reply = await provider.getInfo(index, account, key);
const { error: infoError, result: infoValue } = reply
if (infoError == false) {
res.send(`${infoValue}`);
} else {
res.send(`${infoError}`);
};
} catch (error) {
res.send(500);
}

Ensuring data sent via Node socket.write() is fully sent via OS before new data is asked to be sent

I have an application that is acting as a TCP client and will keep trying to connect with a TCP server. Upon establishment of a connection, predefined text messages are sent. The application seems to be working fine at the moment. However, the server expects to receive "one" "complete" message at a time. Everything is on local network on a single switch. All the messages are 500 bytes max roughly. So they will normally get to the server in one go. The issue that i am facing is that I keep sending messages from my NODE app one after the other to the kernel buffer using socket.write() and they seem to be arriving at the destination server together as one long string. Although I didn't send them in one go. Which I believe is OK because its upto TCP to send data as it pleases. But can I ensure somehow that i send a message only when previous message has actually been sent by TCP fully?
Following is the code that is handling socket events.
Thanks.
//Socket events-------------------------------------------------------------------------------------------------------------------------
const socketControl = (constants, vars, status) => {
//Handle any errors.
vars.client.on("error", function (error) {
console.log(`Connection Error : ${error}`);
//If this happens, close event will automatically be called.
});
//Close connection
vars.client.on("close", () => {
console.log("Connection closed by server.");
console.log(
`Attempting reconneciton with ${constants.REMOTE_SYSTEMNAME} at IP ${
constants.SERVER_IP
} at port ${constants.SERVER_PORT} after ${
constants.RECONNECTTmrPV / 1000
}s`
);
console.log("--");
status.Connected = false;
status.socketAvailavleToWrite = false;
status.waitingcallback = false;
status.STENReceived = false;
//Clear watchdog
clearInterval(vars.heartbeatTmr);
vars.watchDogCountCurrent = 0;
//Clear init timer.
clearInterval(vars.initiateTmr);
//Attempt conneciton after a certain time.
//This is a once of call, because this function will get called again if the connection fails.
setTimeout(
() => FL_Utils.connect(constants, vars, status),
constants.RECONNECTTmrPV
);
});
//Handle incoming data.
vars.client.on("data", function (data) {
//Add the data to inbound buffer.
vars.bufferIn = vars.bufferIn + data.toString();
FL_Utils.processData(constants, vars, status);
//Handle different types of requests from server here.
});
//Drain event, socket available again.
vars.client.on("drain", () => {
console.log(`Socket available to write now`);
status.socketAvailavleToWrite = true;
status.waitingcallback = false;
//Check if there are more messages to send.
//Keep calling this function till there is stuff to be sent in the messageOut buffer.
if (vars.messagesOut.length > 0) {
FL_Utils.sendMessage(constants, vars, status);
}
});
//Conneciton established.
vars.client.on("connect", () => {
FL_Utils.onConnect(constants, vars, status);
});
};
module.exports = { socketControl };
Following is from another module that sends messages.
const sendMessage = (constants, vars, status) => {
if (vars.messagesOut.length > 0) {
//Only then try to do anything.
//Check if the socket is available to write.
if (status.socketAvailavleToWrite && !status.waitingcallback) {
let result = false;
//console.log(`Buffer size at send: ${(vars.messagesOut.length)}`)
result = vars.client.write(vars.messagesOut[0], () =>
MessageSendResponse(constants, vars, status)
);
//console.log('Callback requested.')
status.waitingcallback = true;
if (!result) {
//The socket is full. Wait for drain event to kick in.
console.log("Socket is not available to write at the moment.");
status.socketAvailavleToWrite = false;
}
}
}
};
const MessageSendResponse = (constants, vars, status) => {
status.waitingcallback = false;
//console.log(`Buffer size at cb before message shift : ${(vars.messagesOut.length)}`)
//Message sent.
console.log(`${timestamp()} : Message out : ${vars.messagesOut[0]}`);
//Delete message.
vars.messagesOut.shift();
//console.log(`Buffer size at cb after message shift : ${(vars.messagesOut.length)}`)
//Enable the socket available.
status.socketAvailavleToWrite = true;
//Check if there are more messages to send.
//Keep calling this function till there is stuff to be sent in the messageOut buffer.
if (vars.messagesOut.length > 0) {
//console.log('Recursive call to send message.')
sendMessage(constants, vars, status);
}
};

res.send() is not sending current response, instead keeps last one

This is some of my code that I have in my index.js. Its waiting for the person to visit url.com/proxy and then it loads up my proxy page, which is really just a form which sends back an email and a code. From my MongoDB database, I grab the users order using the code, which contains some information I need (like product and the message they're trying to get). For some reason, it seems like its responding before it gets this information and then holds onto it for the next time the form is submitted.
The newline in my res.send(product + '\n' + message) isnt working either, but thats not a big deal right now.
But.. for example, the first time I fill out the form ill get a blank response. The second time, I'll get the response to whatever I filled in for the first form, and then the third time ill get the second response. I'm fairly new to Web Development, and feel like I'm doing something obviously wrong but can't seem to figure it out. Any help would be appreciated, thank you.
app.get('/proxy', function(req,res){
res.sendFile(__dirname+ "/views/proxy.html");
});
var message = "";
var product = "";
app.post('/getMessage', function(req,res)
{
returnMsg(req.body.user.code, req.body.user.email);
//res.setHeader('Content-Type', 'text/plain');
res.send(product + "\n" + message);
});
function returnMsg(code, email){
MongoClient.connect(url, function(err, db){
var cursor = db.collection('Orders').find( { "order_id" : Number(code) })
cursor.each(function(err, doc){
assert.equal(err, null);
if (doc!= null)
{
message = doc["message"];
product = doc["product"];
}
else {
console.log("wtf");
// error code here
}
});
console.log(email + " + " + message);
var document = {
"Email" : email,
"Message" : message
}
db.collection("Users").insertOne(document);
db.close();
});
}
You need to do lots of reading about your asynchronous programming works in node.js. There are significant design problems with this code:
You are using module level variables instead of request-level variables.
You are not correctly handling asynchronous responses.
All of this makes a server that simply does not work correctly. You've found one of the problems already. Your async response finishes AFTER you send your response so you end up sending the previously saved response not the current one. In addition, if multiple users are using your server, their responses will tromp on each other.
The core design principle here is first that you need to learn how to program with asynchronous operations. Any function that uses an asynchronous respons and wants to return that value back to the caller needs to accept a callback and deliver the async value via the callback or return a promise and return the value via a resolved promise. The caller then needs to use that callback or promise to fetch the async value when it is available and only send the response then.
In addition, all data associated with a request needs to stay "inside" the request handle or the request object - not in any module level or global variables. That keeps the request from one user from interfering with the requests from another user.
To understand how to return a value from a function with an asynchronous operation in it, see How do I return the response from an asynchronous call?.
What ends up happening in your code is this sequence of events:
Incoming request for /getMessage
You call returnMsg()
returnMsg initiates a connection to the database and then returns
Your request handler calls res.send() with whatever was previously in the message and product variables.
Then, sometime later, the database connect finishes and you call db.collection().find() and then iterate the cursor.
6/ Some time later, the cursor iteration has the first result which you put into your message and product variables (where those values sit until the next request comes in).
In working out how your code should actually work, there are some things about your logic that are unclear. You are assigning message and product inside of cursor.each(). Since cursor.each() is a loop that can run many iterations, which value of message and product do you actually want to use in the res.send()?
Assuming you want the last message and product value from your cursor.each() loop, you could do this:
app.post('/getMessage', function(req, res) {
returnMsg(req.body.user.code, req.body.user.email, function(err, message, product) {
if (err) {
// send some meaningful error response
res.status(500).end();
} else {
res.send(product + "\n" + message);
}
});
});
function returnMsg(code, email, callback) {
let callbackCalled = false;
MongoClient.connect(url, function(err, db) {
if (err) {
return callback(err);
}
var cursor = db.collection('Orders').find({
"order_id": Number(code)
});
var message = "";
var product = "";
cursor.each(function(err, doc) {
if (err) {
if (!callbackCalled) {
callback(err);
callbackCalled = true;
}
} else {
if (doc != null) {
message = doc["message"];
product = doc["product"];
} else {
console.log("wtf");
// error code here
}
}
});
if (message) {
console.log(email + " + " + message);
var document = {
"Email": email,
"Message": message
}
db.collection("Users").insertOne(document);
}
db.close();
if (!callbackCalled) {
callback(null, message, product);
}
});
}
Personally, I would use promises and use the promise interface in your database rather than callbacks.
This code is still just conceptual because it has other issues you need to deal with such as:
Proper error handling is still largely unfinished.
You aren't actually waiting for things like the insert.One() to finish before proceeding.

How to work around amqplib's Channel#consume odd signature?

I am writing a worker that uses amqplib's Channel#consume method. I want this worker to wait for jobs and process them as soon as they appear in the queue.
I wrote my own module to abstract away ampqlib, here are the relevant functions for getting a connection, setting up the queue and consuming a message:
const getConnection = function(host) {
return amqp.connect(host);
};
const createChannel = function(conn) {
connection = conn;
return conn.createConfirmChannel();
};
const assertQueue = function(channel, queue) {
return channel.assertQueue(queue);
};
const consume = Promise.method(function(channel, queue, processor) {
processor = processor || function(msg) { if (msg) Promise.resolve(msg); };
return channel.consume(queue, processor)
});
const setupQueue = Promise.method(function setupQueue(queue) {
const amqp_host = 'amqp://' + ((host || process.env.AMQP_HOST) || 'localhost');
return getConnection(amqp_host)
.then(conn => createChannel(conn)) // -> returns a `Channel` object
.tap(channel => assertQueue(channel, queue));
});
consumeJob: Promise.method(function consumeJob(queue) {
return setupQueue(queue)
.then(channel => consume(channel, queue))
});
My problem is with Channel#consume's odd signature. From http://www.squaremobius.net/amqp.node/channel_api.html#channel_consume:
#consume(queue, function(msg) {...}, [options, [function(err, ok) {...}]])
The callback is not where the magic happens, the message's processing should actually go in the second argument and that breaks the flow of promises.
This is how I planned on using it:
return queueManager.consumeJob(queue)
.then(msg => {
// do some processing
});
But it doesn't work. If there are no messages in the queue, the promise is rejected and then if a message is dropped in the queue nothing happens. If there is a message, only one message is processed and then the worker stalls because it exited the "processor" function from the Channel#consume call.
How should I go about it? I want to keep the queueManager abstraction so my code is easier to reason about but I don't know how to do it... Any pointers?
As #idbehold said, Promises can only be resolved once. If you want to process messages as they come in, there is no other way than to use this function. Channel#get will only check the queue once and then return; it wouldn't work for a scenario where you need a worker.
just as an option. You can present your application as a stream of some messages(or events). There is a library for this http://highlandjs.org/#examples
Your code should look like this(it isn`t a finished sample, but I hope it illustrates the idea):
let messageStream = _((push, next) => {
consume(queue, (msg) => {
push(null, msg)
})
)
// now you can operate with your stream in functional style
message.map((msg) => msg + 'some value').each((msg) => // do something with msg)
This approach provides you a lot of primitives for synchronization and transformation
http://highlandjs.org/#examples

Resources