I am using Express to expose a REST API in my Node.js application. When a REST request comes in, I extract the user information from the HTTP headers. I would like this information to be available throughout the life of this request, no matter what function I am in. An obvious but kludgy way is to pass around the user information as parameters to all function calls. Is there a better way? So far I have found the following solutions, but I am not sure if they are ready for prime time:
StrongLoop Zone Library: Docs say "The zone library and documentation are still under development: there are bugs, missing features, and limited documentation."
Continuation-Local Storage: Not sure if this is slated to be part of Node.js. This issue at the end recommends looking at StrongLoop zone.
Node.js Domains: Does not look like this actually took off.
I'm not sure if you're still looking, but a while back I built express-http-context for this very purpose. It's an express middleware that sits on top of cls-hooks (formerly continuation-local-storage).
Full disclosure: this is an npm package that I built and maintain.
Related
Bots are amazing, unless you're Google Analytics
After many months of learning to host my own Discord bot, I finally figured it out! I now have a node server running on my localhost that sends and receives data from my Discord server; it works great. I can do all kinds of the things I want to with my Discord bot.
Given that I work with analytics everyday, one project I want to figure out is how to send data to Google Analytics (specifically GA4) from this node server.
NOTE: I have had success in sending data to my Universal Analytics property. However, as awesome as that was to finally see pageviews coming into, it was equally heartbreaking to recall that Google will be getting rid of Universal Analytics in July of this year.
I have tried the following options:
GET/POST requests to the collect endpoint
This option presented itself as impossible from the get-go. In order to send a request to the collection endpoint, a client_id must be sent along with the request itself. And this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up.
If you consider this option possible, please let me know why.
Install googleapis npm package
At first, I thought I could just install the googleapis package and be ready to go, but that idea fell on its face immediately too. With this package, I can't send data to GA, I can only read with it.
Find and install a GTM npm package
There are GTM npm packages out there, but I quickly found out that they all require there to be a window object, which is something my node server would not have because it isn't a browser.
How I did this for Universal Analytics
My biggest goal is to do this without using Python, Java, C++ or any other low level languages. Because, that route would require me to learn new languages. Surely it's possible with NodeJS alone... no?
I eventually stumbled upon the idea of actually hosting a webpage as some sort of pseudo-proxy that would send data from the page to GA when accessed by something like a page scraper. It was simple. I created an HTML file that has Google Tag Manager installed on it, and all I had to do was use the puppeteer npm package.
It isn't perfect, but it works and I can use Google Tag Manager to handle and manipulate input, which is wonderful.
Unfortunately, this same method will not work for GA4 because GA4 automatically excludes all identified bot traffic automatically, and there is no way to turn that setting off. It is a very useful feature for GA4, giving it quite a bit more integrity than UA, and I'm not trying to get around that fact, but it is now the Bane of my entire goal.
https://support.google.com/analytics/answer/9888366?hl=en
Where to go from here?
I'm nearly at the end of my wits on figuring this one out. So, either an npm package exists out there that I haven't found yet, or this is a futile project.
Does anyone have any experience in sending data from NodeJS to GA4? (or even GTM?) How did you do it?
...and this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up...
Why, of course you can. GA4 generates it pretty much the same as UA does. You don't need anything from google to do it.
Besides, instead of mimicking just requests to the collect endpoint, you may just wanna go the MP route right away: https://developers.google.com/analytics/devguides/collection/protocol/ga4 The links #dockeryZ gave, work perfectly fine. Maybe try opening them in incognito, or in a different browser? Maybe you have a plugin blocking analytics urls.
Moreover, you don't really need to reinvent the bicycle. Node already has a few packages to send events to GA4, here's one looking good: https://www.npmjs.com/package/ga4-mp?activeTab=readme
Or you can just use gtag directly to send events. I see a lot of people doing it even on the front-end: https://www.npmjs.com/package/ga-gtag Gtag has a whole api not described in there. Here's more on gtag: https://developers.google.com/tag-platform/gtagjs/reference Note how the library allows you to set the client id there.
The only caveat there is that you'll have to track client ids and session ids manually. Shouldn't be too bad though. Oh, and you will have to redefine the concept of a pageview, I guess. Well, the obvious one is whenever people post in the chan that is different from the previous post in a session. Still, this will have to be defined in the code.
Don't worry about google's bot traffic detection. It's really primitive. Just make sure your useragent doesn't scream "bot" in it. Make something better up.
I'm looking into node.js which really seem like a pretty nice environment. I've worked with a lot of different Technologies and for server, mainly php and Java (jsp), but dabbled in som RoR and Python.
I find node.js really easy to get up and running and it feels quite natural to work with, and I found some good entry level tutorials.
I just am missing some more intermediate resources. For example when creating bigger frameworks or api's how would you structure or architect it. I set up some smaller api's to try it out where it would go something like this:
I've made use of the Express framework to create a http server, listen to a port, set up an express object and bound some requests.
However these have been quite small, and the purpose has been learning, if I think about scaling up the size of the API for production, perhaps wanting to do other stuff like serve web-pages as well. I find it hard to see how the architecture would look.
It's vague as I am still new to node.js but I'm mainly thinking about things like if you typically keep all api in one file or if there are good ways to split it up into modules? And if anyone know any resource talking a bit more about how to design the architecture when working in node.js
Sorry for the vague question and thanks for reading.
In my opinion, Express is the good way to go if you want to build complex or big APIs.
It is among others easily testable (for instance with Mocha or Jasmine) and customizable, especially thanks to its middlewares.
For the directory structure, what I usually use is (at least) the following:
app.js : the main entrypoint. Will create the express application, indicate which controller to use for every route prefix, and assign the middlewares. Example from a previous project
controllers : will contain the controllers, the functions which will handle the requests, in the same style as in standard MVC frameworks (e.g. UserController, ...). Each controller would create an express Router object and export it. Inside the controllers, individual handlers are in charge of individual API requests, such as /api/users/list. It would use some library to access your data (e.g. Mongoose for MongoDB), and would then send the response to the client. Example (UserController.js)
models : will contain the models with all their attributes and methods. In my case, it would be the Mongoose models. Example (Song.js)
middlewares : will contain the various middlewares of the project. A practical example would be a middleware checking for an access token in the incoming request, and returning a 403 HTTP error if not. Example (AuthMiddleware.js)
helpers : various helpers
tests : unit tests of you API
This could be the minimal directory organization. On top of that, you may want to use a templating engine such as EJS to serve webpage. Take a look at « Use EJS to template your node application ».
This is only to give you an overview of what an express directory structure could look like, but there are of course plenty (better?) other possibilities. Hope that gives you a quick and useful insight :)
I am new to Node and web development. I am working on a simple application to get started and want to limit the use of my endpoints to server side calls, i.e. anyone else trying to access them gets a 403.
However, after researching, I found PassportJS with examples of how to protect endpoints, but they all seem to be overkill especially for a small application like mine.
Am I overthinking this, or should I be looking for a more basic solution?
How do you need to consume these end points yourself? I don't know if all you are looking to do is to call one file from the other. You need not expose them as restful end points in that case. Create the methods as module methods (as in node modules). Then just use require syntax to reference that file as a dependency.
You need restful end points for server side only if it needs to be called from another web application.
I inherited an existing API and I would like to document it with swagger, but I don't yet know the full scope of it. Can Swagger (or another middleware/tool) auto-magically generate the yaml (for swagger) based on the existing express routes?
For what I saw on other questions, it would appear that this is mostly a manual job, but I'm double-checking if someone here found a way around this.
I have experience in BOTH auto-generating the Swagger json and manually writing it out for an API that I helped build. Here are the pros/cons of both based on my experience.
Swagger AUTOMATIC Documentation Generation:
We used the swagger-node-express module in combination with swagger-ui.
https://www.npmjs.com/package/swagger-node-express
https://github.com/swagger-api/swagger-ui
Pros
Super easy to document. Just throw a few lines above the resource definition and the documentation (json) is automatically generated by the module.
Cons
You are no longer using straight up Express when you use this package. Your route definitions have to be defined through the Swagger module and this pulls you away from vanilla Express.
Swagger MANUAL Documentation Generation:
We just pulled swagger-ui into the project and wrote the documentation manually.
https://github.com/swagger-api/swagger-ui
Pros
This approach decouples the documentation from the Express framework. Express endpoints are written as they normally would be written and the Swagger documentation is defined separate from the Express framework. Allows you to write pure express.
Cons
Documentation changes become a little more tedious due to the fact that you are manually writing and changing the yaml or json yourself. It's a little bit harder than just updating a few lines of code above a resource. This approach is also a little more prone to documentation typos and errors due to the fact it is entirely manually typed.
If you are planning to manually write your swagger documentation use the swagger editor below to validate your manual docs.
http://editor.swagger.io/#/
Conclusion
For this API project, we started out by auto-generating the documentation using the swagger-node-express package. However, we realized that decoupling the swagger documentation from the express library was important to enable us to use all the features and functionality of Express. I recommend manually writing the docs to have full control over both the Swagger documentation and the Express web framework that your app will use.
There is an option: you can embed middleware that will analyse all requests and responses and generate specification for you: https://github.com/mpashkovskiy/express-oas-generator
Then you can use it through your's app Swagger UI like http://host:port/api-docs
Yes !!!. You can use this awesome project typescript-test. Here is sample app. Clone it, run npm i,npm run swagger and go to /dist/swagger.json. Done. Swagger yaml and json is generated based on express routes !
With express-sitemap-html you may automatically generate a minimalistic Open API definition (only including route parameters) and install a Swagger UI for all routes of an existing express app. You only need:
const sitemap = require('express-sitemap-html')
...
sitemap.swagger('Your app name', app) // given that app is an express instance
Instead of analyzing HTTP requests at runtime, this approach inspects express app instance and mounted routes.
PROs you don't need to perform ahead requests to get an updated list of available routes.
CONs it provides untyped parameters features.
Have a look to swagger-jsdoc. It's a different approach.
The docs stick to the code, and also lets the express code to remain pure.
Guides:
https://dev.to/acanimal/express-api-with-autogenerated-openapi-doc-through-swagger-7na
https://dev.to/akshendra/generating-documentation-on-the-fly-in-express-2652
I am trying to allow users to upload images to a site I have built with Meteor, which requires a server-side route I can POST data to. Is there a way to set up a route server side such that I can call Meteor code from it (for example, a route that I can call this.userId or Meteor.userId() from)?
Server side routing is on the roadmap, but not yet available the way you need it. (At that link, Server-side rendering is set for version 1.0).
In the interim, you can do some server side rendering with Tom Coleman's excellent meteor-router mrt package. It's unclear to me how much of this will make it into Meteor core.
More likely, however, for the file upload problem, this issue describes the problem, and it looks like people have had luck implementing imgur's xhr api or perhaps even better is this smart idea.
Hope this helps.