I am trying to implement a pub/sub broker with ZeroMQ where it is possible to restrict clients from subscribing to prefixes they are not allowed to subscribe to. I found a tutorial that tries to achieve a similar thing using the ZMQ_XPUB_MANUAL option. With zeromq.js it is possible to set this option:
import * as zmq from "zeromq";
// ...
const socket = new zmq.XPublisher({ manual: true });
After setting this option I am able to receive the subscription messages by calling .receive() on this socket:
const [msg] = await socket.receive();
But I have no Idea how to accept this subscription. Usally this is done by calling setSockOpt with ZMQ_SUBSCRIBE but I don't know how to do this with zeromq.js.
Is there a way to call setSockOpt with zeromq.js or is there another way to accept a subscription?
Edit
I tried user3666197's suggestion to call setSockOpt directly, but I am not sure how to do this. Rather than doing that, I took another look in the sources and found this: https://github.com/zeromq/zeromq.js/blob/master/src/native.ts#L617
It seems like setSockOpt is exposed to the TypeScript side as protected methods of the Socket class. To try this out, I created my own class that inherits XPublisher and exposed an acceptSubscription message:
class CustomPublisher extends zmq.XPublisher {
constructor(options?: zmq.SocketOptions<zmq.XPublisher>) {
super(options);
}
public acceptSubscription(subscription: string | null): void {
// ZMQ_SUBSCRIBE has a value of 6
// reference:
// https://github.com/zeromq/libzmq/blob/master/include/zmq.h#L310
this.setStringOption(6, subscription);
}
}
This works like a charm! But do not forget to strip the first byte of the subscription messages, otherwise your client won't receive any messages since the prefix won't match.
Q : "Is there a way to call setSockOpt() with zeromq.js or is there another way to accept a subscription?"
So, let me first mention Somdoron to be, out of doubts & for ages, a master of the ZeroMQ tooling.
Next comes the issue. The GitHub-sources, I was able to review atm, seem to me, that permit the ZMQ_XPUB-Socket-archetypes to process the native API ZMQ_XPUB_MANUAL settings ( re-dressed into manual-property, an idiomatic shift ), yet present no method (so far visible for me) to actually permit user to meet the native API explicit protocol of:
ZMQ_XPUB_MANUAL: change the subscription handling to manual...with manual mode subscription requests are not added to the subscription list. To add subscription the user need to call setsockopt() with ZMQ_SUBSCRIBE on XPUB socket./__ from ZeroMQ native API v.4.3.2 documentation __/
Trying to blind-call the Socket-inherited .SetSockOpt() method may prove me wrong, yet if successful, it may be a way to inject the { ZMQ_SUBSCRIBE | ZMQ_UNSUBSCRIBE } subscription-management steps into the XPUB-instance currently having been switched into the ZMQ_XPUB_MANUAL-mode.
Please test it, and if it fails to work via this super-class inherited method, the shortest remedy would be to claim that collision/conceptual-shortcomings directly to the zeromq.js maintainers ( it might be a W.I.P. item, deeper in their actual v6+ refactoring backlog, so my fingers are crossed for either case ).
Related
Goal
A request comes in and is handled by the Azure Functions run-time. By default it creates a Request entry, and a bunch of Trace entries in Application Insights. I want to add a custom dimension to that top level request item (on a per-request basis) so I can use it for filtering/analysis later.
Query for -requests- on Application Insights
Resulting list of requests including custom dimensions column
The Azure Functions runtime adds a few custom dimensions already. I want to add a few of my own.
Approach
The most promising approach I've found is show below (taken from here https://github.com/microsoft/ApplicationInsights-node.js/issues/392)
appInsights.defaultClient.addTelemetryProcessor(( envelope, context ) => {
var data = envelope.data.baseData;
data.properties['mykey'] = 'myvalue';
return true;
});
However, I find that this processor is only called for requests that I initialise within my function. For example, if I make an HTTP request to another service, then details of that request will be passed thru the processor and I can add custom properties to it. But the main function does not seem to pass thru here. So I can't add my custom property.
I also tried this
defaultClient.commonProperties['anotherCustomProp'] = 'bespokeProp2'
Same problem. The custom property doesn't arrive in application insights. I've played with many variations on this and it appears that the logging done by azure-functions is walled off from anything I can do within my code.
The best workaround I have right now, is to call trackRequest manually. This is okay, except I end up with each request logged twice in application insights, one by the framework and one by me. And both need to have the same operation_id otherwise I can't find the associated trace/error items. So I'm having to extract the operationId in a slightly hacky way. This may be fine, my knowledge of application insights is pretty naive at this point.
import { setup, defaultClient } from 'applicationinsights' // i have to import the specific functions, because "import ai from applicationinsights" returns null
// call this because otherwise defaultClient is null.
// Some examples call start(), I've tried with and without this.
// I think the start() function must be useful when you're adding application-insights to a project fresh, whereas I think the azure-functions run-time must be doing this already.
setup()
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
// Extract the operation id from the traceparent as per w3 standard https://www.w3.org/TR/trace-context/.
const operationId = context.traceContext.traceparent.split('-')[1]
var operationIdOverride = { 'ai.operation.id': operationId }
// Create my own trackRequest entry
defaultClient.trackRequest({
name: 'my func name',
url: context.req.url.split('?')[0],
duration: 123,
resultCode: 200,
success: true,
tagOverrides: operationIdOverride,
properties: {
customProp: 'bespokeProp'
}
})
The Dream
Our C# cousins seem to have an array of options, like Activity.Current.tags and the ability to add TelemetryInitializer. However it looks like what I'm trying to do is supported, I'm just not finding the right combination of commands! Is there something similar for javascript/typescript/nodejs, where I can just add a tag on a per-request basis? Along the lines of context.traceContext.attributes['myprop'] = 'myValue'
Alternative
Alternatively, instrumenting my code using my own TelemetryClient (rather than the defaultClient) using trackRequest, trackTrace, trackError etc, is not a very big job and should work well - that would be more explicit. Should I just do that? Is there a way to disable the azure functions tracking - or perhaps I just leave that as something running side-by-side.
I know I can use a Request guard. However, if I have a REST API with hundreds of handlers, not only it would be annoying to have to add an extra function param to all of them, but it kinda scares me that it could be easy to miss adding such a param here or there and therefore create a security hole. That's why I'd like to know if there is a way to do such a validation globally.
The documentation on Fairings mentions they can be used for global security policies:
As a general rule of thumb, only globally applicable actions should be implemented via fairings. For instance, you should not use a fairing to implement authentication or authorization (preferring to use a request guard instead) unless the authentication or authorization applies to the entire application. On the other hand, you should use a fairing to record timing and/or usage statistics or to implement global security policies.
But at the same time the docs on the on_request() callback say this:
A request callback can modify the request at will and Data::peek() into the incoming data. It may not, however, abort or respond directly to the request; these issues are better handled via request guards or via response callbacks.
So how am I supposed to return an error to the user in the case of an invalid token for example?
OK, I think I found a way...
First we create a "dummy" handler like this:
#[put("/errHnd", format = "json")]
fn err_handler() -> ApiResult {
// Here simply return an error
}
Then we attach a fairing like this:
rocket::custom(cfg)
.attach(AdHoc::on_request("OnReq", |req, _| {
// Here we validate the token and if it's not OK,
// forward the request to our "dummy" handler:
let u = Origin::parse("/errHnd").unwrap();
req.set_uri(u);
req.set_method(Method::Put);
}))
.mount("/", routes![err_handler, ...
I'm not sure that's the best way to do it, but I tested it and it seems to work. I'm open to other suggestions.
P.S. It may also be worth mentioning that if we wanted to have an exception, so as to skip the validation in the fairing, say, based on the URL, we could simply add something like this in it:
if req.uri().path() == "/let-me-in-please" {
return;
}
TL;DR below.
I am currently developing a React/Redux SPA that is driven by real-time data. I've decided to use ws, instead of socket.io since socket.io feels a bit high level for what I'm doing, I'd rather manage sockets myself.
In saying that, I'm struggling to find a way to manage the separation of updates/messages per view/route. Since I'm using client-side routing it's per express route won't really work...
Messages between the server and client via WebSockets are JSON with actions like GET_ITEMS then a response of GET_ITEMS_SUCCESS with an array of 'items' and for errors: ..._ERROR etc. This is all fine, since it's just 1 to 1 transaction. Though the problem arises when broadcasting (1 to all) to all relevant clients when the server receives an update.
So, I assume it best practice to limit these broadcasts to the clients that are viewing/want the data. So when viewing, for example, the Item page, there is no point in broadcasting updates to the User data since that is only used on the User page.
I haven't been able to find any common practices when dealing with this sort of situation, just a few small outdated/barely used wrappers for ws that just add a few basic functions to leave/join but don't offer much flexibility with implementation.
What I think MIGHT work is to have an object/array for each 'group'/'room', which stores the clients that are currently listening to updates from a given section. So a user would send an action to INIT_LISTEN (& ``) with a param of category, e.g. ITEM for updates and other actions related to items.
TL;DR
What my question really boils down to is: How do I store a reference to a single socket? (ws client object? ws client ID?) Then, can I store this in an object/array to iterate through like below.
const ClientRooms = {
Items: {
{
...ws
}
/* ...rest of the client */
}
}
or
const ClientRooms = {
Items: [ "xyz" ] /* Array of ws ids */
}
I have a "ping--pong" heartbeat function to keep clients active and prevent silent connection failures/disconnections. I can't find if ws.terminate() still fires the ws close event so I can iterate 'group'/'room' the object/array to find and remove instances of that client.
I'd want to use a login page to access different firewalls, so I need to get information about the firewall I'm logging in.
In my controller I'd use
$this->container->get('security.context')->getToken()->getProviderKey()
but as an anonymous user I don't have access to getProviderKey method.
I could also parse
_security.xxx.target_path
to get xxx firewall but I'm looking for a more general solution if it exists at all.
Any idea?
As of symfony 3.2, you can now get the current firewall configuration using the following:
public function indexAction(Request $request)
{
$firewall = $this->container
->get('security.firewall.map')
->getFirewallConfig($request)
->getName();
}
Ref: http://symfony.com/blog/new-in-symfony-3-2-firewall-config-class-and-profiler
For Symfony 3.4 I wrote this to avoid referencing the non-public "security.firewall.map" service:
$firewallName = null;
if (($firewallContext = trim($request->attributes->get("_firewall_context", null))) && (false !== ($firewallContextNameSplit = strrpos($firewallContext, ".")))) {
$firewallName = substr($firewallContext, $firewallContextNameSplit + 1);
}
(Referencing "security.firewall.map" on 3.4 will throw an exception.)
Edit: This will not work in a custom exception controller function.
I was doing a little research on this myself recently so that I could send this information in an XACML request as part of the environment.
As far as I can tell from GitHub issues like this one:
https://github.com/symfony/symfony/issues/14435
There is currently no way to reliably get the information out of Symfony except the dirty compiler pass hack suggested on the linked issue. It does appear from the conversation on these issues, they are working on making this available, however, the status is still open, so we will have to be patient and wait for it to be provided.
#Adambean's answer is pretty elegant, but I'd write it as a one-liner:
$firewallName = array_slice(explode('.', trim($request->attributes->get('_firewall_context'))), -1)[0];
The difference is that $firewallName will always be a string (which may be empty).
Also, please note that this answer (like #Adambean's) doesn't work for a firewall with a dot in its name.
The Azure Service Bus supports a built-in retry mechanism which makes an abandoned message immediately visible for another read attempt. I'm trying to use this mechanism to handle some transient errors, but the message is made available immediately after being abandoned.
What I would like to do is make the message invisible for a period of time after it is abandoned, preferably based on an exponentially incrementing policy.
I've tried to set the ScheduledEnqueueTimeUtc property when abandoning the message, but it doesn't seem to have an effect:
var messagingFactory = MessagingFactory.CreateFromConnectionString(...);
var receiver = messagingFactory.CreateMessageReceiver("test-queue");
receiver.OnMessageAsync(async brokeredMessage =>
{
await brokeredMessage.AbandonAsync(
new Dictionary<string, object>
{
{ "ScheduledEnqueueTimeUtc", DateTime.UtcNow.AddSeconds(30) }
});
}
});
I've considered not abandoning the message at all and just letting the lock expire, but this would require having some way to influence how the MessageReceiver specifies the lock duration on a message, and I can't find anything in the API to let me change this value. In addition, it wouldn't be possible to read the delivery count of the message (and therefore make a decision for how long to wait for the next retry) until after the lock is already required.
Can the retry policy in the Message Bus be influenced in some way, or can a delay be artificially introduced in some other way?
Careful here because I think you are confusing the retry feature with the automatic Complete/Abandon mechanism for the OnMessage event-driven message handling. The built in retry mechanism comes into play when a call to the Service Bus fails. For example, if you call to set a message as complete and that fails, then the retry mechanism would kick in. If you are processing a message an exception occurs in your own code that will NOT trigger a retry through the retry feature. Your question doesn't get explicit on if the error is from your code or when attempting to contact the service bus.
If you are indeed after modifying the retry policy that occurs when an error occurs attempting to communicate with the service bus you can modify the RetryPolicy that is set on the MessageReciver itself. There is an RetryExponitial which is used by default, as well as an abstract RetryPolicy you can create your own from.
What I think you are after is more control over what happens when you get an exception doing your processing, and you want to push off working on that message. There are a few options:
When you create your message handler you can set up OnMessageOptions. One of the properties is "AutoComplete". By default this is set to true, which means as soon as processing for the message is completed the Complete method is called automatically. If an exception occurs then abandon is automatically called, which is what you are seeing. By setting the AutoComplete to false you required to call Complete on your own from within the message handler. Failing to do so will cause the message lock to eventually run out, which is one of the behaviors you are looking for.
So, you could write your handler so that if an exception occurs during your processing you simply do not call Complete. The message would then remain on the queue until it's lock runs out and then would become available again. The standard dead lettering mechanism applies and after x number of tries it will be put into the deadletter queue automatically.
A caution of handling this way is that any type of exception will be treated this way. You really need to think about what types of exceptions are doing this and if you really want to push off processing or not. For example, if you are calling a third party system during your processing and it gives you an exception you know is transient, great. If, however, it gives you an error that you know will be a big problem then you may decide to do something else in the system besides just bailing on the message.
You could also look at the "Defer" method. This method actually will then not allow that message to be processed off the queue unless it is specifically pulled by its sequence number. You're code would have to remember the sequence number value and pull it. This isn't quite what you described though.
Another option is you can move away from the OnMessage, Event-driven style of processing messages. While this is very helpful you don't get a lot of control over things. Instead hook up your own processing loop and handle the abandon/complete on your own. You'll also need to deal some of the threading/concurrent call management that the OnMessage pattern gives you. This can be more work but you have the ultimate in flexibility.
Finally, I believe the reason the call you made to AbandonAsync passing the properties you wanted to modify didn't work is that those properties are referring to Metadata properties on the method, not standard properties on BrokeredMessage.
I actually asked this same question last year (implementation aside) with the three approaches I could think of looking at the API. #ClemensVasters, who works on the SB team, responded that using Defer with some kind of re-receive is really the only way to control this precisely.
You can read my comment to his answer for a specific approach to doing it where I suggest using a secondary queue to store messages that indicate which primary messages have been deferred and need to be re-received from the main queue. Then you can control how long you wait by setting the ScheduledEnqueueTimeUtc on those secondary messages to control exactly how long you wait before you retry.
I ran into a similar issue where our order picking system is legacy and goes into maintenance mode each night.
Using the ideas in this article(https://markheath.net/post/defer-processing-azure-service-bus-message) I created a custom property to track how many times a message has been resubmitted and manually dead lettering the message after 10 tries. If the message is under 10 retries it clones the message increments the custom property and sets the en queue of the new message.
using Microsoft.Azure.ServiceBus;
public PickQueue()
{
queueClient = new QueueClient(QUEUE_CONN_STRING, QUEUE_NAME);
}
public async Task QueueMessageAsync(int OrderId)
{
string body = JsonConvert.SerializeObject(OrderId);
var message = new Message(Encoding.UTF8.GetBytes(body));
await queueClient.SendAsync(message);
}
public async Task ReQueueMessageAsync(Message message, DateTime utcEnqueueTime)
{
int resubmitCount = (int)(message.UserProperties["ResubmitCount"] ?? 0) + 1;
if (resubmitCount > 10)
{
await queueClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
else
{
Message clone = message.Clone();
clone.UserProperties["ResubmitCount"] = ++resubmitCount;
await queueClient.ScheduleMessageAsync(message, utcEnqueueTime);
}
}
This question asks how to implement exponential backoff in Azure Functions. If you do not want to use the built-in RetryPolicy (only available when autoComplete = false), here's the solution I've been using:
public static async Task ExceptionHandler(IMessageSession MessageSession, string LockToken, int DeliveryCount)
{
if (DeliveryCount < Globals.MaxDeliveryCount)
{
var DelaySeconds = Math.Pow(Globals.ExponentialBackoff, DeliveryCount);
await Task.Delay(TimeSpan.FromSeconds(DelaySeconds));
await MessageSession.AbandonAsync(LockToken);
}
else
{
await MessageSession.DeadLetterAsync(LockToken);
}
}