I have a client in NodeJS using the MQTT module (mqtt.js) to communicate with a topic of WebSphere MQ, redirected to a queue : I need the result to be on a queue so I use a topic between the source and the destination (since MQTT protocol communicate only with topics).
So we have :
NodeJS (source) -> Topic of Websphere MQ (intermediate) -> Queue of Websphere MQ (destination)
The problem is that the received messages on the queue of WebsphereMQ are MQHRF2 format but I need MQSTR.
I did not find any property in mqtt.js side to specify the format.
I supposed we can force it with IBM Websphere MQ, but how ? Can I create a channel or anything for this use, that's to say convert the receiving format ?
Thanks in advance for your help !
Configuration :
MQTT.js v3
WebSphere MQ v7.5
Constrains :
WebSphere MQ must be v7.5 (so I can't upgrade to v8 to use mqlight)
Better stay with the module MQTT.js
It works correctly when I publish directly on a topic via the WebSphere MQ explorer. However, it fails when I publish from the NodeJS application.
I tried both methods listing in the comments but I found a weird thing : The result is different according to if I use MQExplorer (what I always used) or the MQ console :
On the MQ Explorer :
On the MQ Console, with the command : amqsbcg QUEUE_MQTT_VERIF MQTTVerification :
As you can see, in the console, the format field is empty, whereas on the MQExplorer it is MQHRF2. The messages are also different.
So the result on the MQConsole is right whereas the MQExplorer still shows a wrong result after the changes.
Note that the result is right only if changes has been done on both :
Subscription : ALTER SUB(APPLE.PRICES) PSPROP(NONE)
Queue : ALTER QLOCAL(PRICES) PROPCTL(NONE)
Maybe the source of the problem is just the display on MQExplorer software ?
EDIT:
The difference between MQExplorer and MQConsole was due to the parameter in MQ Explorer : Window -> Preferences -> WebSphereMQ Explorer -> Message properties -> Uncheck "Display message properties".
When redirecting publications made on a topic to a queue you probably have some definitions like this:-
DEFINE QLOCAL(PRICES)
DEFINE TOPIC(APPLES) TOPICSTR('Price/Fruit/Apples')
DEFINE SUB(APPLE.PRICES) TOPICOBJ(APPLES) DESTCLAS(PROVIDED) DEST(PRICES)
which redirects publications made on the topic string 'Price/Fruit/Apples' to the queue PRICES.
What you will find with such a set up however, is that the topic string is added to the message by the queue manager and thus adds an MQRFH2 header onto the front of your published message.
Message Descriptor (MQMD)
Report :00000000
Message Type :8 (Datagram)
Format :'MQHRF2 '
Priority :0
Persistence :0 (Not Persistent)
Message Id :414D51204D51473120202020202020202D77835720003702
'AMQ MQG1 -w.W .7.'
ReplyToQ :' '
ReplyToQMgr :'MQG1 '
[ 102 bytes] Message Content
<mqps>
<Top>
Price/Fruit/Apples
</Top>
</mqps>
Apples are $2/kilo
To remove this without changing the application reading from the queue, make this alteration to the subscription.
ALTER SUB(APPLE.PRICES) PSPROP(NONE)
This stops the queue manager even putting the topic string into the message in the first place.
From your MQ Explorer screenshot, it can be seen that the MQRFH2 is still present after the SUB is changed to PSPROP(NONE) because there is another property present - mqtt.clientid. Since there are other items in the MQRFH2 as well as the topic string, then the SUB change will not remove those - only the topic string added by the queue manager. In that case, you should try the following.
If alternatively you want to have the topic string there sometimes and only remove it for applications that don't want to see it, you can make a similar change to the queue, which can be over-ridden in application code to force the properties to be delivered to the application, but otherwise they won't be. This would also allow you to read any existing messages that had already been published (the change to the SUB isn't retrospectively applied to messages that are already on the queue).
ALTER QLOCAL(PRICES) PROPCTL(NONE)
This will mean that applications that don't specifically request MQGMO_FORCE_RFH2 will see the message without the MQRFH2. The amqsbcg sample is one such application.
If an application codes MQGMO_FORCE_RFH2 it will still be able to see the properties in an MQRFH2 header because it has explicitly asked for that. You cannot stop that by changing the queue.
Your problem is only now with the way MQ Explorer chooses to display the message. There is a parameter in MQ Explorer : Window -> Preferences -> WebSphereMQ Explorer -> Message properties -> Uncheck "Display message properties" which will stop it forcing them to be an MQRFH2, and then you'll be all good.
Related
TLDR: using python client library to subscribe to pulsar topic. logs show: 'broker notification of consumer closed' when something happens server-side. subscription appears to be re-established according to logs but we find later that backlog was growing on cluster b/c no msgs being sent to our subscription to consume
Running into an issue where we have an Apache-Pulsar cluster we are using that is opaque to us, and has a namespace defined where we publish/consume topics, is losing connection with our consumer.
We have a python client consuming from a topic (with one Pulsar Client subscription per thread).
We have run into an issue where, due to an issue on the pulsar cluster, we see the following entry in our client logs:
"Broker notification of Closed consumer"
followed by:
"Created connection for pulsar://houpulsar05.mycompany.com:6650"
....for every thread in our agent.
Then we see the usual periodic log entries like this:
{"log":"2022-09-01 04:23:30.269 INFO [139640375858944] ConsumerStatsImpl:63 | Consumer [persistent://tenant/namespace/topicname, subscription-name, 0] , ConsumerStatsImpl (numBytesRecieved_ = 0, totalNumBytesRecieved_ = 6545742, receivedMsgMap_ = {}, ackedMsgMap_ = {}, totalReceivedMsgMap_ = {[Key: Ok, Value: 3294], }, totalAckedMsgMap_ = {[Key: {Result: Ok, ackType: 0}, Value: 3294], })\n","stream":"stdout","time":"2022-09-01T04:23:30.270009746Z"}
This gives the appearance that some connection has been re-established to some other broker.
However, we do not get any messages being consumed. We have an alert on Grafana dashboard which shows us the backlog on topics and subscription backlog. Eventually it either hits a count or rate thresshold which will alert us that there is a problem. When we restart our agent, the subscription is re-establish and the backlog is can immediately be seen heading to 0.
Has anyone experienced such an issue?
Our code is typical:
consumer = client.subscribe(
topic='my-topic',
subscription_name='my-subscription',
consumer_type=my_consumer_type,
consumer_name=my_agent_name
)
while True:
msg = consumer.receive()
ex = msg.value()
i haven't yet found a readily-available way docker-compose or anything to run a multi-cluster pulsar installation locally on Docker desktop for me to try killing off a broker and see how consumer reacts.
Currently Python client only supports configuring one broker's address and doesn't support retry for lookup yet. Here are two related PRs to support it:
https://github.com/apache/pulsar/pull/17162
https://github.com/apache/pulsar/pull/17410
Therefore, setting up a multi-nodes cluster might be nothing different from a standalone.
If you only specified one broker in the service URL, you can simply test it with a standalone. Run a consumer and a producer sending messages periodically, then restart the standalone. The "Broker notification of Closed consumer" appears when the broker actively closes the connection, e.g. your consumer has sent a SEEK command (by seek call), then broker will disconnect the consumer and the log appears.
BTW, it's better to show your Python client version. And GitHub issues might be a better place to track the issue.
I created a topic "Messages" in my service bus instance, and added a new subscription to it. I can send messages to this topic and the Azure Function v3 trigger is activated just fine. The message is received and displayed in an instant.
When I add an sql filter to filter messages for a subscription it is not working anymore.
What I did so far.
Created an sql filter in the azure portal:
sys.Label = "Test" -> Not working as no messages are received anymore, even though I verified that the Label attribute is set.
sys.Label != "Test" Not working as no messages are received anymore, even though I verified that the Label attribute is set and not matching "Test"
sys.To = "Test" -> Not working, no messages are received. Verified that the messages contain the To member.
sys.Label is not Null -> This is strangely working.
What am I doing wrong here?
As stated in the comment, the solution to this question can be found in the Microsoft Q&A.
To prevent a loss of this solution I am going to post it here too:
The SQL filter value should be in single quotes 'test'
Example : sys.To = 'test'
Make sure that you are defining the message system properties while
sending the message to topic.
Source
If someone is caught off guard by the red single quotes. They are just red, but don't indicate that something is wrong with it. It will just work fine.
I am trying to use jms:publish-subscribe-channel to pub/sub on a single ActiveMQ topic. I am able to receive messages from ActiveMQ on the channel just fine, however when I publish to the channel the message body is null (when received by another application listening on the ActiveMQ topic). I was able to recreate the problem using spring-integration-samples->basic->jms. I modified outboundChannelAdapter.xml to use jms:publish-subscribe-channel instead of jms:outbound-channel-adapter. Is there another step needed in order to publish a simple string message? Here's my code change to outboundChannelAdapter.xml:
<stream:stdin-channel-adapter id="stdin" channel="stdinToJmsoutChannel"/>
<jms:publish-subscribe-channel id="stdinToJmsoutChannel" topic="requestTopic" />
<stream:stdout-channel-adapter id="stdout" channel="stdinToJmsoutChannel" append-newline="true"/>
I am not sure what you mean by "the message body is null".
I just made the exact same change to the sample and it worked fine for me...
Please type something and hit <enter>
foo
foo
I had to add -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* to the command line because activemq needs whitelisting for classes (the whole message is serialized in jms-backed channels).
I'm currently implementing a flow on a Spring Integration-based (ver. 3.0.1.RELEASE) application that requires to store messages on a JMS queue to be picked up later.
For that, I've been trying to use a Spring Integration JMS Inbound Channel Adapter with a custom selector, and then picking up the message from the queue by changing the JMS selector of the JMSDestinationPollingSource to some matching ID included as a header property.
One of the requirements for this is that I cannot add a new service or a JAVA method, so I've been trying to sort it out using a Control Bus, but keep receiving the same error when I send the message to set the messageSelector to something different.
Inbound Channel Adapter definition:
<int-jms:inbound-channel-adapter id="inboundAdapter"
channel="inboundChannel"
destinationName="bufferQueue"
connection-factory="connectionFactory"
selector="matchingID = 'NO VALUE'">
<int:poller fixed-delay="1000"/>
</int-jms:inbound-channel-adapter>
Message:
#'inboundAdapter.source'.setMessageSelector("matchingID = 'VALUE'")
Error:
EvaluationException: The method 'public void org.springframework.integration.jms.JmsDestinationPollingSource.setMessageSelector(java.lang.String)' is not supported by this command processor. If usign the Control Bus, consider adding #ManagedOperation or #ManagedAttribute.
Which, AFAIK, means that the JmsDestinationPollingSource class is not Control Bus manageable, as it's not passing the ControlBusMethodFilter.
Is this approach nonviable, or is there something I'm missing? Is there any way to set the selector dynamically using SI XML configuration files only?
First of all it is strange to use Java tool and don't allow to write code on Java...
But that is your choice, or as you said requirements.
Change the employer! ;-)
That's correct: Control Bus allows only #ManagedOperation and #ManagedAttribute method. Since JmsDestinationPollingSource.setMessageSelector. We can make it like that. But does it make so much sense if we can reach it a bit different approach?
<int:outbound-channel-adapter id="changeSelectorChannel"
ref"inboundAdapter.source method="setMessageSelector"/>
where a new selector expression should be as a payload of the Message to this channel.
I am polling for files for a service-activator, using a PseudoTransactionManager to move them into processed/failed directories.
If/when the move fails, I would like to log this, including the file name.
As the information being passed around the flows is the Message object, I tried enriching the file name onto its header, but as we make copies of it for each step, this won't work unless I can move the header-enricher between the inbound channel adapter and transaction manager.
In simplified form the main flow I now have is this:
inbound-channel-adapter -> a) header-enricher -> service-activator
Because I want the files moved to a processed or failed directory, there is a second flow:
inbound-channel-adapter -> b) pseudo transaction-manager -> logging-channel-adapter (in case of problems moving the processed file).
I think this follows because the transaction manager definition is nested within the channel adapter definition in the xml.
How can I pass this information through the example setup here to a logging channel adapter?
The transaction stuff only has access to the original message. You can add an error-channel to the poller; the default error channel (errorChannel) is a pub/sub channel and has a logging channel adapter subscribed to it.
When an exception occurs, an ErrorMessage is sent to the error channel (if configured); the payload is a MessagingException with cause and failedMessage properties. The failedMessage is the message at the point the failure occurred.
The default error flow will simply log the message so your "transaction" will "commit".
Instead, you need a custom error flow; log what you want and then re-throw the cause and your "transaction" will "rollback".