I am using the spring cloud stream event binder for Azure event hubs and follow the setup in this tutorial. Now I want to set a custom value to the id Header of the message produced.
I know that those headers are read only in the Spring Message interface
So if I do something like this
MessageBuilder.withPayload(myPayload)
.setHeader("id", "myCustomIdHeaderValue")
.build(),
I get
java.lang.IllegalArgumentException: 'id' header is read-only
I want to integrate with an existing system which requires the messages header id to hold specific values and specified by the producer - unfortunately I cannot use another custom header for this one.
As a fallback I could use the plain azure event hub SDK but would prefer using Spring Cloud Stream binder instead if possible. The restrictions are there for valid reasons and overriding them doesn't sound like a good idea, but is there a way to overcome this anyway ?
Related
I have an Azure Service Bus topic. Messages send to the topic contain application properties.
I want to set permissions on the topic based on values of one or more application properties, i.e. only when identity A sends a message with property e.g. Destination = 'service' the message is allowed to be added to the topic.
The reason behind this: assume a topic where there are multiple publishers and multiple subscribers to consume the messages from the publishers. But each message shall only go to specific consumers using filters. An actor with bad intentions or just by accident could choose a filter value and send the message therefore to wrong consumers. And I don't want to create multiple topics. I want to limit the publishers to send messages to certain consumers by limiting their permissions by message attributes with certain values on a certain topic.
Is this possible? If yes, how?
You can start a separate app which will peek the message in the queue
check for the properties and then abandon the messages.
To peek create a receiver and use receiver.ReceiveMessageAsync();
to peek the message. After this you can apply your logic to filter
the messages using the properties.
After that use AbandonAsync to abandon the message with wrong properties.
// create a reciever and peek the messages.
await using var client = new ServiceBusClient(connectionString);
//reciever
ServiceBusReceiver receiver = client.CreateReceiver(queueName) ;
// get the messages
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
// write you logic for filtering the message
//now we can abandon the message. Here "reason":" Wrong message" is optional
await receivedMessage.AbandonAsync(new Dictionary<string, object> { { "Reason", "wrong properties"} });
Refernce:
Refer the article by sfeldman
Peek messages
You can't use ServiceBus (or any message broker) the way you describe.
My advice is to consider messages only as signals to do something. But the service is entirely responsible to check if the message is legitimate or not (by calling the source service or another source of truth synchronously for example).
Imagine this scenario:
SourceApp send a DeletedEvent{CustomerId="42"}.
DestApp receives this message, and before deleting data related to this customer, makes an HTTP call to the customer service GET CustomerService/42.
If the customer is deleted => perfect
If not... something went wrong, what you described by an actor with bad intentions or just by accident
Also, if you use the same topic for a lot of applications consider using AAD authentication, with managed identity per app. Or at least forbid sharing SAS keys. Every app should have dedicated connection string, scoped to the subscription or the topic with only the access needed.
Scenario
When business transactions are performed, we're supposed to make that data available to end clients.
Current Design
Our web app publishes transaction messages are added to a topic on the Azure Service Bus.
We expose APIs to clients through which they can consume the data from those transactions.
Upon calling these APIs, we read the messages from the Subscription and return it to the client.
Problem
We want a guaranteed delivery - we want to make sure the client acknowledges the delivery of the data. So we don't want to remove the message from the subscription immediately. We want to keep it until the client acknowledges it.
So we only want to do a "Peek" instead of "Receive".
So the client calls the first API, to get the data, where we do a Peek.
And once the client has received the packets, the client would call a second API, to acknowledge.
At this point, we want to remove the message from the Subscription, making it Complete.
The current design of the Service Bus Message Receiver is that, a Complete can be performed only using the same Receiver instance that performed the Peek, as per the documentation, and we also observed the same when we tried it out.
Both the APIs, are two separate APIs and we cannot do the Peek and Complete using the same instance of the Receiver.
Thinking about options to somehow make the Receiver as a Singleton, across APIs within that App Service.
However this will be a problem when the App Service scales out.
Is there a different way to achieve what we're trying to do here ?
There is an option available in Azure Service Bus to defer messages. Once a message is deferred, it can be received with the help of it's sequence number.
The first client should receive the message and instead of completing it, it should defer it and return it.
The second client (which has sequence number) can receive the message from the Subscription. Refer here for more details.
Another option would be to not use a Service Bus Client on your backend and instead your clients could directly work with Service Bus using its Service REST API (assuming they can't use the AMQP client if I am understanding your scenario correctly).
There are APIs to
Peek-Lock
Renew Lock
Unlock
Delete (Complete)
You could also proxy these requests if you'd like using your backend itself or a service like APIM if you are already using it.
PS: Cross posting the answer for the same query on the MSDN forum
I must send events to Azure Event Hub directly using http protocol. How can I validate payload json schema of the event in Azure (same as validating in a web api) before the event ingestion?
Inspection of event payloads and indexing are not within the feature scope of Event Hubs (or Apache Kafka).
See the Note Section here:
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-retention
Is it possible to attach certain attribute to each message that is auto forwarded? Or perhaps when the message is finally received see from where it has been forwarded?
Not possible. Auto-forwarding is defined on an entity level, but solely as an entity to forward to. Properties modification (Actions) can be only done with Rules only (on subscriptions) on the broker side.
The servicebus client 2.1 now supports AMQP 1.0.
On MSDN there is an article about how to use QPID.AMQP.JMS with Azure ServiceBus.
However, although I can connect to a predefined Topic and a Subscription, post messages and receive all of them, I cannot change the message selector or create a new topic/subscription.
My goal is to be able to connect to a Topic and dynamically create subscriptions based on different filters using org.apache.qpid.amqp_1_0.jms.
Questions:
1/ How can I create a new topic.
Topic newTopic = (Topic) session.createTopic("newtopic");
session.createProducer(newTopic); // returns error
This maybe because JMS doesn't support administration of the topics though... although I'm sure I've read somewhere it should create a topic if it doesn't exist.?
2/ How can I create a new subscription with a different message selector via Jms?
// This still gives me all messages no matter what I put in the 'class' property.
TopicSubscriber subscriber = session.createDurableSubscriber(topic, "sub1", "class = 'boo'", false);
Thanks,
Phil.
The AMQP 1.0 protocol deliberately left entity creation / management outside of it's scope, you'll need to create your topics and subscriptions beforehand using provider-specific mechanics for this (This post has a good explanation on why).
In the case of Service Bus, you can either use the Azure SDK, the portal (in case you don't need to do it programmatically), or their REST API for this.
Hope it helps!