I'm trying to build a NodeJS REST API project based on the so called "micro architecture" (basically multiple smaller NodeJS projects that can run totally independent, but at the same time work together).
Currently users are able to upload images from the app, and my NodeJS backend then processes and saves them appropriately.
Now, what I want to do is the following:
User selects an image to upload from the app -> The app makes a request to the "Main API" endpoint -> The Main API endpoint then forwards this request to the "Image Service" -> Once the Image Service (which is a totally different server) has successfully finished, it should return the URL where the image is stored to the Main API server endpoint, which will then return the info back to the app.
My question is, how do I forward the image upload request from one server to another? Ideally, I don't want the Main API to store the image temporarily and then make a request to the Image Service.
What I'd like is try and forward the data the Main API receives straight to the Image Service server. I guess you could say I want to "stream" the data from one place to another without having to temporarily store on disk or memory. I literally just want it to "tunnel" from one server to another.
Is this possible and is this an efficient way? I just want 1 central point for the app to access, I don't want it to know about this Image Service server. I'd like the app to only ever make requests to the Main API, which will then call my other little services as required.
I'm using NodeJS, Express, Multer (for image uploads) and Digital Ocean hosting (if that should make any difference at all).
What you would basically be doing is setting up a proxy server that will pass requests straight through to another machine and back. There are a few libraries out there to help with this, and this article in particular http://blog.vanamco.com/proxy-requests-in-node-js/ will explain how to go about setting it up even though they are really just trying to get around HTTPS, the same concept applies here.
In short, you get the file upload POST, and then immediately just make that same request to another server and when the response is returned, immediately return it back to the front end. Your entry point can be set up as a hub, and you can proxy requests through to other servers or even just handle them on the same server if necessary.
Related
I've been working on the front-end so far, now I'm going to create my first full-stack application. I want to use node.js, express and AWS for this.
At the design stage, I already encountered a few problems. Therefore, I have a few questions and I am asking you for help:
Can I send a message (simple JSON or database value) from the server to all clients who have already opened my home page in a simple and cheap way?
I'm not talking about logged in users, but all who downloaded the main page (GET, '/')?
Using the admin panel ('www.xxxxxxxxx/admin'), I want to send a message to the server once a day. Then I want to change the HTML to display this message. I was thinking to use EJS for this and download this message from the database.
Can I make it better? If someone visits my home page (GET, '/'), EJS will download the message from the database each time! Even though its value is the same for 24 hours. Can I get the value once and then use it until the value is changed? How to store the message? As a JSON on the server? Or maybe in the .env file?
If the user refreshes the page, do I have to pay for calling all AWS functions to build the page each time? Even if nothing has changed in the files?
How to check if the page has new content and then send it to the user, instead of sending the unchanged page files: .html, .js, .css, etc.?
Can I send the user only the changed, dynamically created html file, and not send again unchanged .js and .css files?
Does every user who opens the home page (GET, '/') create a new connection to the server using WebSocket / socket.io?
I will try to answer some of your questions:
Can I send a message (simple JSON or database value) from the server to all clients who have already opened my home page in a simple
and cheap way? I'm not talking about logged in users, but all who
downloaded the main page (GET, '/')?
I guess you mean sending push notifications from the server to the user. This can be done with different services depending on what are you trying to build.
If you are planning to use GraphQL, you already have GraphQL subscriptions out of the box. If you are using AWS, go for Appsync, which is the AWS service for GraphQL.
If you are using REST and a WebApp (not a mobile app), go for AWS IoT using lambdas. Here is a good resource using Serverless Framework (API Gateway + lambdas + IoT) for unauthenticated users: https://www.serverless.com/blog/serverless-notifications-on-aws
If you are planning to use notifications on a mobile app, you can go for SNS, the "de facto" service for push notifications in AWS world.
Using the admin panel ('www.xxxxxxxxx/admin'), I want to send a message to the server once a day. Then I want to change the HTML to display this message. I was thinking to use EJS for this and download this message from the database. Can I make it better? If someone visits my home page (GET, '/'), EJS will download the message from the database each time! Even though its value is the same for 24 hours. Can I get the value once and then use it until the value is changed? How to store the message? As a JSON on the server? Or maybe in the .env file?
Yes, this is the way it's expected to work. The HTML is changed dynamically using frontend code in Javascript; which makes calls (using axios for example) to the backend every time you get into, i.e. "/" path. You can store this data in frontend variables, or even use state management in the frontend using REDUX, VUEX, etc. Remember the frontend code will always run in the browser of your users, not on your servers!
If the user refreshes the page, do I have to pay for calling all AWS functions to build the page each time? Even if nothing has changed in the files?
What you can do is store all your HTML, CSS, Javascript in an S3 bucket and serve from there (this is super cheap, even free till a certain limit). If you want to use Server Side Rendering (SSR), then yes, you'll need to serve your users every time they make a GET request for example. If you use lambda, the first million request per month are free. If you have an EC2 instance to serve your content, then a t2.micro is also free. If you need more than that, you'll need to pay.
How to check if the page has new content and then send it to the user, instead of sending the unchanged page files: .html, .js, .css, etc.?
I think you need to understand how JS (or frameworks like React, Vue or Angular) do this. Basically you download the js code on the client, and the js makes all the functionality to update backend and frontend accordingly. In order to connect frontend with backend, use Axios for example.
Can I send the user only the changed, dynamically created html file, and not send again unchanged .js and .css files?
See answer above. Use frameworks like React or Vue, will help you a lot.
Does every user who opens the home page (GET, '/') create a new connection to the server using WebSocket / socket.io?
Depends on what you code. But by default what happens is the user will make a new GET request everytime he accesses your domain, and that's it. (It's not establishing any connection if you don't tell the code to do so).
Hope this helps!! Happy coding!
I'm currently working on an analytics webapp with a react frontend and node (express) backend.
Describing the functionality in a nutshell:
A user will be able to login on the website, search for a YouTube username and then the YouTube API is called, the data will be stored in a mysql db via my express API and also the data will be used to calucalte some statistics which afterwards are displayed in a dashboard.
Now I was wondering if I should:
Call the YouTube API from the frontend, i.e. inside my react code, do the calculations display them and and then store it in the DB via my express API.
Or, from the react app call an endpoint in my express API that will then call the YouTube API, store the data in the DB and then pass on the data to my react app.
Are there any best practices or up-/downsides to either approach?
When answering questions like these, it's important to remember that the client-side is different for each and every user that visits your website, their internet speed, their GPU & CPU power, etc., but the server is most commonly held in a stable container and much more powerful than a client.
The proper way would be the following:
1. Obtain a search query from a client
Meaning you should get the user's search query from an input, or any other form of control (text area, checkbox, etc.), this way client is doing the least business logic, as it should. The client should always focus more on UI / UX rather than business logic.
2. Send query to the server
Let the server use the query you've just obtained from client, call the youtube api from the server (either explicitly using Axios, or find a node.js youtube library), and do all the necessary calculation on the backend
3. Send processed data to the client
Let client receive the data in the form which is ready for use (iterations, mappings, etc.) - again separating concerns, server - business logic, client - UI / UX
Now to be fair, the example you have will most commonly be done all on the client-side, since it is not as computationally heavy as other enterprise examples, but this is a good rule to follow for big projects, and no one would really complain if you did it this way, since it would be the proper way.
I've been scouring the internet on this one, but I can't seem to find a situation exactly like what I'm trying to do....
I've built a React app (using CRA) which is used to import products from an existing eBay-like site (Reverb.com), make some small modifications, and then post them to a server which, in turn, manages the products on multiple eCommerce platforms. Pulling the data in from the Reverb.com API is easy enough; the problem comes when I submit the data to my local server's API. I need to post the product images from Reverb.com via formData without the user having to manually choose the files via a form input, but I don't have access to fs.createReadStream(). How can I get from an image URL to a file posted via formData?
I'm making a REST backend for private use of our frontend, they will both be in the same server.
The problem is that I'm worried about security issues, I don't want a attacker to use the API directly, either by JS or by using other REST client.
Lets take this service as an example
http://myserver:8080/something/webresources/film
That's a service that allows to GET, PUT, POST, DELETE I want that only the frontend be able to use it, otherwise since anyone can see client-code it would be easy to get the endpoint and start putting or getting data. I do have BASIC AUTH so they would have to register and if they did something wrong I would be able to see who did it, but that doesn't solve the problem.
I could implement access control so that a user only could get/update/delete their own movies, but I would have to do that for every REST service(I have over 100 by now), plus I might need to actually get others movies
This is my first public project, I am really lost.
You can do it through your web server. I'm using Nginx. I have an if statement that checks the $http_referer. If it returns nothing, or the value returned is not my application/frontend page (meaning someone is trying to hit the api directly), it'll return a 403 forbidden page.
If your application doesn't send out emails to your users don't worry about the following: I added a block to allow access to my static images as the only exception, as my application sends out emails with images and I don't want them to break.
That's it. Problem solved. No one has access to my api except my frontend page/application, unless they can forge the $http_referer to match my domain which if they can do that then they deserve to break in.
Your only option to call the REST API from server side. You cannot hide from the users what's going on in their browser... You can have a layered application, so the frontend layer can call the backend layer on the server while the client can see only the frontend. (Check the layered system constraint.)
I have developed a front-end interface using Aja(AngularJS) and HTML5. Right now, I send an HTTP get request to my backend server which returns some data based on the GET parameters.
Since the URL is exposed in the Javascript file, I believe anyone could just use the URL to create there own API to fetch the data. How can I prevent such things ?
One way I could think of is that now instead of directly sending the request to the backend server, an application server could be used (hosting the HTML as well). The Ajax request would then be sent to this server (PHP script ?) which would in turn forward the request to the backend server and return the result to the UI. To prevent 3rd party services, I can disable cross origin requests on my application server.
Is this the correct way to solve my problem or are there better ways to do this? I am concerned that this would unnecessarily create another hop (internal though) for requests.
Note: The backend is running Apache Tomcat
In APIs that are not open to the world the user has to authenticate first in order to use it, see for example https://stripe.com/docs/api#authentication or http://dev.maxmind.com/geoip/geoip2/web-services/ -> Authorization