Goal
A request comes in and is handled by the Azure Functions run-time. By default it creates a Request entry, and a bunch of Trace entries in Application Insights. I want to add a custom dimension to that top level request item (on a per-request basis) so I can use it for filtering/analysis later.
Query for -requests- on Application Insights
Resulting list of requests including custom dimensions column
The Azure Functions runtime adds a few custom dimensions already. I want to add a few of my own.
Approach
The most promising approach I've found is show below (taken from here https://github.com/microsoft/ApplicationInsights-node.js/issues/392)
appInsights.defaultClient.addTelemetryProcessor(( envelope, context ) => {
var data = envelope.data.baseData;
data.properties['mykey'] = 'myvalue';
return true;
});
However, I find that this processor is only called for requests that I initialise within my function. For example, if I make an HTTP request to another service, then details of that request will be passed thru the processor and I can add custom properties to it. But the main function does not seem to pass thru here. So I can't add my custom property.
I also tried this
defaultClient.commonProperties['anotherCustomProp'] = 'bespokeProp2'
Same problem. The custom property doesn't arrive in application insights. I've played with many variations on this and it appears that the logging done by azure-functions is walled off from anything I can do within my code.
The best workaround I have right now, is to call trackRequest manually. This is okay, except I end up with each request logged twice in application insights, one by the framework and one by me. And both need to have the same operation_id otherwise I can't find the associated trace/error items. So I'm having to extract the operationId in a slightly hacky way. This may be fine, my knowledge of application insights is pretty naive at this point.
import { setup, defaultClient } from 'applicationinsights' // i have to import the specific functions, because "import ai from applicationinsights" returns null
// call this because otherwise defaultClient is null.
// Some examples call start(), I've tried with and without this.
// I think the start() function must be useful when you're adding application-insights to a project fresh, whereas I think the azure-functions run-time must be doing this already.
setup()
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
// Extract the operation id from the traceparent as per w3 standard https://www.w3.org/TR/trace-context/.
const operationId = context.traceContext.traceparent.split('-')[1]
var operationIdOverride = { 'ai.operation.id': operationId }
// Create my own trackRequest entry
defaultClient.trackRequest({
name: 'my func name',
url: context.req.url.split('?')[0],
duration: 123,
resultCode: 200,
success: true,
tagOverrides: operationIdOverride,
properties: {
customProp: 'bespokeProp'
}
})
The Dream
Our C# cousins seem to have an array of options, like Activity.Current.tags and the ability to add TelemetryInitializer. However it looks like what I'm trying to do is supported, I'm just not finding the right combination of commands! Is there something similar for javascript/typescript/nodejs, where I can just add a tag on a per-request basis? Along the lines of context.traceContext.attributes['myprop'] = 'myValue'
Alternative
Alternatively, instrumenting my code using my own TelemetryClient (rather than the defaultClient) using trackRequest, trackTrace, trackError etc, is not a very big job and should work well - that would be more explicit. Should I just do that? Is there a way to disable the azure functions tracking - or perhaps I just leave that as something running side-by-side.
Related
I currently have an Azure ContainerApp architected as a BackgroundService, deployed to Azure. The Worker listens for ServiceBusMessages and processes them.
.NET6
Microsoft.ApplicationInsights.WorkerService 2.21.0
Application Insights is setup like this:
builder.Services.AddApplicationInsightsTelemetryWorkerService(opts =>
{
opts.DependencyCollectionOptions.EnableLegacyCorrelationHeadersInjection = true;
});
builder.Services.ConfigureTelemetryModule<DependencyTrackingTelemetryModule>((module, o) => { module.EnableSqlCommandTextInstrumentation = true; });
and I record the AI Operation in question by injecting a TelemetryClient and using it in the ServiceBusProcessor.ProcessMessageAsync handler like:
using (var op = _telemetryClient.StartOperation<RequestTelemetry>("ProcessAdapterMessage.ProcessSBMessage"))
{
// Process Data
}
My issue is that this operation happens a LOT, which is fine because I use AI sampling, but it's also duplicated. It's recorded once under the proper operation name "ProcessAdapterMessage.ProcessSBMessage", and once under the the operation name "<Empty>".
If I drill down into the "<Empty>" operation, it's actually just "ServiceBusProcessor.ProcessSBMessage" operations that wrap the same method as before. The reason I created the manual op is because looking at data named "Empty" isn't useful, so I'd rather keep my manual op, and make the "Empty" one go away. Any ideas on how to fix this?
This is what the "Empty" Operation details look like
This is what the "ProcessAdapterMessage.ProcessSBMessage" Operation details look like:
In Fastify.js you have at least to ways to register hooks: globally (via fastify.addHook()) or as a property inside the route declaration. In the example below I'm trying to use fastfy-multer to handle file uploading but the maximum amount of files must be limited by a setting associated with a "room". As the app has many rooms, most of the requests contain a reference to a room, and every time the request is being augmented with room settings by the preHandler hook.
import fastify from 'fastify'
import multer from 'fastify-multer'
const server = fastify()
server.register(multer.contentParser)
// For all requests containing the room ID, fetch the room options from the database
fastify.addHook('preHandler', async (request, reply) => {
if (request.body.roomID) {
const roomOptions = await getRoomOptions(request.body.roomID)
if (roomOptions) {
reuqest.body.room = roomOptions
}
else {
// handle an error if the room doesn't exist
}
}
})
server.post('/post', {
// Limit the maximum amount of files to be uploaded based on room options
preHandler: upload.array(files, request.body.room.maxFiles)
})
In order for this setup to work, the global hook must always be executed before the file upload hook. How can I guarantee that?
Summary: As #Manuel Spigolon said:
How can I guarantee that? The framework does it
Now we can take Manuel's word for it (SPOILER ALERT: they are absolutely correct), or we can prove how this works by looking in the source code on GitHub.
The first thing to keep in mind is that arrays in JavaScript are remain ordered by the way objects are pushed into them, but don’t take my word for it. That is all explained here if you want to dive a little deeper into the evidence. If that was not true, everything below doesn't matter and you could just stop reading now.
How addHook works
Now that we have established that arrays maintain their order, let look at how the addHook code is executed. We can start by looking at the default export of fastify in the fastify.js file located in the root directory. In this object if scoll down you'll see the addHook property defined. When we look into the addHook function implementation we can see that in that add hook call we are calling this[kHooks].add.
When we go back to see what the kHooks property is we see that it is a new Hooks(). When we go to take a look at the add method on the Hooks object, we can see that it just validates the hook that is being add and then [pushes] it to the array property on the Hooks object with the matching hook name. This shows that hooks will always be in the order which add was called for them.
How fastify.route adds hooks
I hope you're following to this point because that only proves the order of the addHook calls in the respective array on the Hooks object. The next question is how these interact with the calls of fastify.(get | post | route | ...) functions. We can walk through the fastify.get function, but they are all pretty much the same (you can do the same exercise with any of them). Looking at the get function, we see that the implementation is just calling the router.prepareRoute function. When you look into the prepareRoute implementation, you see that this function returns a call to the route function. In the route function there is a section where the hooks are set up. It looks like this:
for (const hook of lifecycleHooks) {
const toSet = this[kHooks][hook]
.concat(opts[hook] || [])
.map(h => h.bind(this))
context[hook] = toSet.length ? toSet : null
}
What this does is go through every lifecycle hook and turn it into a set of all the hooks from the Fastify instance (this) and the hooks in the options (opts[hook]) for that given hook and binds them to the fastify instance (this). This shows that the hooks in the options for the routes are always added after the addHook handlers.
How Fastify executes hooks
This is not everything we need though. Now we know the order in which the hooks are stored. But how exactly are they executed? For that we can look at the hookRunner function in the hooks.js file. We see this function acts as a sort of recursive loop that continues running as long as the handlers do not error. It first creates a variable i to keep track of the handler function it is currently on and then tries to execute it and increments the function tracker (i).
If the handler fails (handleReject), it runs a callback function and does not call the next function to continue. If the handler succeeds (handleResolve), it just runs the next function to try the same process on the following handler (functions[i++]) in the functions set.
Why does this matter
This proves that the hook handlers are called in the order that they were pushed into the ordered collection. In other words:
How can I guarantee that? The framework does it
I'm still a novice web developer, so please bear with me if I miss something fundamental !
I'm creating a backoffice for a Strapi backend, using react-admin.
React-admin library uses a 'data provider' to link itself with an API. Luckily someone already wrote a data provider for Strapi. I had no problem with step 1 and 2 of this README, and I can authenticate to Strapi within my React app.
I now want to fetch and display my Strapi data, starting with Users. In order to do that, quoting Step 3 of this readme : 'In controllers I need to set the Content-Range header with the total number of results to build the pagination'.
So far I tried to do this in my User controller, with no success.
What I try to achieve:
First, I'd like it to simply work with the ctx.set('Content-Range', ...) hard-coded in the controller like aforementioned Step 3.
Second, I've thought it would be very dirty to c/p this logic in every controller (not to mention in any future controllers), instead of having some callback function dynamically appending the Content-Range header to any fetchAll request. Ultimately that's what I aim for, because with ~40 Strapi objects to administrate already and plenty more to come, it has to scale.
Technical infos
node -v: 11.13.0
npm -v: 6.7.0
strapi version: 3.0.0-alpha.25.2
uname -r output: Linux 4.14.106-97.85.amzn2.x86_64
DB: mySQL v2.16
So far I've tried accessing the count() method of User model like aforementioned step3, but my controller doesn't look like the example as I'm working with users-permissions plugin.
This is the action I've tried to edit (located in project/plugins/users-permissions/controllers/User.js)
find: async (ctx) => {
let data = await strapi.plugins['users-permissions'].services.user.fetchAll(ctx.query);
data.reduce((acc, user) => {
acc.push(_.omit(user.toJSON ? user.toJSON() : user, ['password', 'resetPasswordToken']));
return acc;
}, []);
// Send 200 `ok`
ctx.send(data);
},
From what I've gathered on Strapi documentation (here and also here), context is a sort of wrapper object. I only worked with Express-generated APIs before, so I understood this snippet as 'use fetchAll method of the User model object, with ctx.query as an argument', but I had no luck logging this ctx.query. And as I can't log stuff, I'm kinda blocked.
In my exploration, I naively tried to log the full ctx object and work from there:
// Send 200 `ok`
ctx.send(data);
strapi.log.info(ctx.query, ' were query');
strapi.log.info(ctx.request, 'were request');
strapi.log.info(ctx.response, 'were response');
strapi.log.info(ctx.res, 'were res');
strapi.log.info(ctx.req, 'were req');
strapi.log.info(ctx, 'is full context')
},
Unfortunately, I fear I miss something obvious, as it gives me no input at all. Making a fetchAll request from my React app with these console.logs print this in my terminal:
[2019-09-19T12:43:03.409Z] info were query
[2019-09-19T12:43:03.410Z] info were request
[2019-09-19T12:43:03.418Z] info were response
[2019-09-19T12:43:03.419Z] info were res
[2019-09-19T12:43:03.419Z] info were req
[2019-09-19T12:43:03.419Z] info is full context
[2019-09-19T12:43:03.435Z] debug GET /users?_sort=id:DESC&_start=0&_limit=10& (74 ms)
While in my frontend I get the good ol' The Content-Range header is missing in the HTTP Response message I'm trying to solve.
After writing this wall of text I realize the logging issue is separated from my original problem, but if I was able to at least log ctx properly, maybe I'd be able to find the solution myself.
Trying to summarize:
Actual problem is, how do I set my Content-Range properly in my strapi controller ? (partially answered cf. edit 3)
Collateral problem n°1: Can't even log ctx object (cf. edit 2)
Collateral problem n°2: Once I figure out the actual problem, is it feasible to address it dynamically (basically some callback function for index/fetchAll routes, in which the model is a variable, on which I'd call the appropriate count() method, and finally append the result to my response header)? I'm not asking for the code here, just if you think it's feasible and/or know a more elegant way.
Thank you for reading through and excuse me if it was confuse; I wasn't sure which infos would be relevant, so I thought the more the better.
/edit1: forgot to mention, in my controller I also tried to log strapi.plugins['users-permissions'].services.user object to see if it actually has a count() method but got no luck with that either. Also tried the original snippet (Step 3 of aforementioned README), but failed as expected as afaik I don't see the User model being imported anywhere (the only import in User.js being lodash)
/edit2: About the logs, my bad, I just misunderstood the documentation. I now do:
ctx.send(data);
strapi.log.info('ctx should be : ', {ctx});
strapi.log.info('ctx.req = ', {...ctx.req});
strapi.log.info('ctx.res = ', {...ctx.res});
strapi.log.info('ctx.request = ', {...ctx.request});
ctrapi.log.info('ctx.response = ', {...ctx.response});
Ctx logs this way; also it seems that it needs the spread operator to display nested objects ({ctx.req} crash the server, {...ctx.req} is okay). Cool, because it narrows the question to what's interesting.
/edit3: As expected, having logs helps big time. I've managed to display my users (although in the dirty way). Couldn't find any count() method, but watching the data object that is passed to ctx.send(), it's equivalent to your typical 'res.data' i.e a pure JSON with my user list. So a simple .length did the trick:
let data = await strapi.plugins['users-permissions'].services.user.fetchAll(ctx.query);
data.reduce((acc, user) => {
acc.push(_.omit(user.toJSON ? user.toJSON() : user, ['password', 'resetPasswordToken']));
return acc;
}, []);
ctx.set('Content-Range', data.length) // <-- it did the trick
// Send 200 `ok`
ctx.send(data);
Now starting to work on the hard part: the dynamic callback function that will do that for any index/fetchAll call. Will update once I figure it out
I'm using React Admin and Strapi together and installed ra-strapi-provider.
A little boring to paste Content-Range header into all of my controllers, so I searched for a better solution. Then I've found middleware concept and created one that fits my needs. It's probably not the best solution, but do its job well:
const _ = require("lodash");
module.exports = strapi => {
return {
// can also be async
initialize() {
strapi.app.use(async (ctx, next) => {
await next();
if (_.isArray(ctx.response.body))
ctx.set("Content-Range", ctx.response.body.length);
});
}
};
};
I hope it helps
For people still landing on this page:
Strapi has been updated from #alpha to #beta. Care, as some of the code in my OP is no longer valid; also some of their documentation is not up to date.
I failed to find a "clever" way to solve this problem; in the end I copy/pasted the ctx.set('Content-Range', data.length) bit in all relevant controllers and it just worked.
If somebody comes with a clever solution for that problem I'll happily accept his answer. With the current Strapi version I don't think it's doable with policies or lifecycle callbacks.
The "quick & easy fix" is still to customize each relevant Strapi controller.
With strapi#beta you don't have direct access to controller's code: you'll first need to "rewrite" one with the help of this doc. Then add the ctx.set('Content-Range', data.length) bit. Test it properly with RA, so for the other controllers, you'll just have to create the folder, name the file, copy/paste your code + "Search & Replace" on model name.
The "longer & cleaner fix" would be to dive into the react-admin source code and refactorize so the lack of "Content-Range" header doesn't break pagination.
You'll now have to maintain your own react-admin fork, so make sure you're already committed into this library and have A LOT of tables to manage through it (so much that customizing every Strapi controller will be too tedious).
Before forking RA, please remember all the stuff you can do with the Strapi backoffice alone (including embedding your custom React app into it) and ensure it will be worth the trouble.
I am using NestJS (version 6.5, with Express platform) and I need to handle a request with a property that can either be a File or a String.
Here is the code I currently have, but I don't find a clean way to implement this.
MyAwesomeController
#Post()
#UseInterceptors(FileInterceptor('source'))
async handle(#UploadedFile() source, #Body() myDto: MyDto): Promise<any> {
//do things...
}
Am I missing something obvious or am I supposed to write my own interceptor to handle this case?
Design-wise, is this bad?
Based on the fact you're designing a REST API:
It depends what use case(s) you want to achieve: is your - client-side - flow designed to be performed in 2 steps o not ?
Can string and file params be both passed at the same time or is there only one of the two on each call ? (like if you want to update a file and its name, or some other non Multer related attributes).
When you pass a string as parameter to your endpoint call, is a file resource created / updated / deleted ? Or maybe not at all ?
Depending on the answer and the flow that you thought of, you should split both cases handling within two independent endpoints, or maybe it makes sense to handle both parameters at the same time.
If only one of the params can be passed at a time, I'd say go for two independent endpoints; you'll benefit from both maintenance and code readability.
If both params can be passed at the same time and they're related to the same resource, then it could make sense to handle both of them at once.
Hope this helps, don't hesitate to comment ;)
I'm using node-wit to develop a chatbot application.
This is working fine mostly, but I've run into a problem with the use of the context.
I'm using the runActions api :
this.witClient.runActions(customer._key, messageText, witContext).then((newContext => {}
)).catch(reject);
I have defined a number of actions, which set the context.
This is working fine, as long the context is taking place over one message.
For example, if I were to call an action called addProduct :
addProduct({sessionId, context, text, entities}) {
return new Promise((resolve, reject) => {
context.product = `myNewProduct';
resolve(context);
});
},
It will then show a message using the 'product' context key.
However, when I try to use it over 2 messages, it seems to have lost the context ( for example, when asking a multiple choice question, and then handling that response ).
If I understand how it's working correctly, then node-wit doesn't keep the context beyond messages ( I assumed this at first because I'm passing a session key ).
A solution I see is to store the resulting context ( newContext in this case) in a session/user specific way, and then restore it and pass it again when the user is sending his new message.
Meaning, something like this :
witContext = getContextFromSession();
this.witClient.runActions(customer._key, messageText, witContext).then((newContext => { setContextInSession(newContext) }
)).catch(reject);
Would this be the correct way of handling it ?
Off course you have to store your context state, you decide how to store it. But, take into account what is the most efficient way if you're gonna have a lot of users, and your reasources available.
As you can see in the official example for nodeJs, there's a method named findOrCreateSession on https://github.com/wit-ai/node-wit/blob/master/examples/messenger.js they get the session before the wit actions are called.
In my particular case, I am storing it in the database, so I get the session before the action is called, so I can send the context, then in the actions I query the session again to modify the resulting context and store it again, try the best implementation for your needs.