I'm implementing a full backend for a mobile app, and while I am competent implementing mobile apps and simple mode servers, I have never set up the infrastructure for servers, db, etc.
Basically, I require a simple RESTful api server for communication with my app using Node.js, but my app will have schedules notifications throughout the day that the user can set.
Is it better to have just one server (can scale up as needed but no we will not have many users to start) that takes in the api requests and schedules push notifications with a scheduler library of some sort (making sure of course to not block the execution thread too much when this happens), or is it better to have a lambda function that makes another call to another server dedicated towards sending these notifications?
Or is there another standard to doing something similar to this? Im just not sure what the standard is here. I plan on using AWS services fyi.
I couldn't find much help on the web, so any suggestions are extremely helpful and appreciated!
Thanks in advance!
Related
To preface I'd like to say I apologize if this question is too general or has been asked before. I'm struggling with some concepts here and truly don't know what I don't know and have no idea where to look. For context, i'm a full stack dev whose experience has mainly been with writing web servers using Node.js. In such applications implementing BullMQ, bull-board and error-handling is fairly straight forward since you can use middlewares and bull-board ships with server framework specific adapters.
Currently I am working on a project written in Node.js and Typescript. It listens to events emitted by the new Opensea Stream package (a socket based application) and accordingly adds jobs to the BullMQ queue. So on startup it connects to the database, connects to the Opensea socket and then acts accordingly to emitted events. In my mind I would classify this as a Job/Worker/Process sort of application. I've had a couple of issues building out this project and understanding how its made and its place in the web application architecture.
Consequently I have come up with a couple of questions:
What are these sort of applications called? I've had a hard time looking up answers for my questions since I don't know what to categorize this application as. When I try to Google 'Node.js Job Application', as expected it gives me links to job postings.
Am I using the wrong tools for the job? Is Node.js only supposed to be used to write web servers and not for such job based applications?
How does error-handling work here? With web servers if an error shows up in the server, the middleware catches the error and sends a response to the client. In such job applications what happens if an error is thrown? Is anything done with the error or is it just logged in errors, is the job reran/cancelled etc?
How do I implement bull-board here to graphically observe my queue? Since i'm not using a web framework here in this application how is the integration done? Is it even possible?
How different are socket based architectures to REST API servers? Is it a completely different use case/style of implementation or can they have the same server architecture and are just different ways of communicating data between processes? Are sockets more for microservices?
Are there any resources or books that detail how these applications are supposed to be built? Any guidance on best practices or architecture would be greatly appreciated.
I was making a thinking exercise about how could I deploy a chat service like WhatsApp or Slack (just wondering), so people could really use it. You need two main parts, the client software (e.g. the app running on the smartphones), and the server software. So how would you develop the server-side code and make it work?
The first idea that came to me was the classic hosting service, but it cannot be the simple "web hosting service", probably because something like this should be programmed at a lower level and not working with HTTP requests and responses. Maybe using specific server-side technology like Node.js (any other suggestion?) to manage different type of requests at lower level, let's say at the layer where TCP lives, would be a better solution.
So I heard about the Amazon Web Services (AWS), which is not classic hosting, it's a cloud computing service. The problem is that I don't know exactly how this works. Could I deploy a server-side application that works at that low level of networking and also makes requests to databases? Would it be difficult to offer this kinf of service using AWS?.
I would like to hear all your opinions about any aspect of this. Would you use other kinf of technology on the server? What do you think about AWS, and if you would think it's a good option, where can I get some info to learn how to use it?
Server Side Code
You can create a chat service backend using NodeJS + express(or Hapi) to cater input Http Requests.
For Hosting: Cloud servers are pretty available these days and allow you to scale if your app grows with time.
Database:
if you already have your DB available (cool) just use ORMs ( like (Sequelize) which provides easy interaction of Nodejs service with your DB. (I have used MySQL + Sql Server + Oracle)
If not, you can create a new DB (MySql- free on your hosting server (cloud?)
I used Microsoft Azure to host a Nodejs(+ Hapi.js) Backend Service ,to be consumed by my mobile application, over the internet.
Azure gives you $200 free credit which is sufficient for you to try and make your hands dirty with them. There are numerous tutorials available for MS-Azure Api App hosting which will guide you to a successful deployment.
I have not yet explored AWS as of now, but i trust that they will be similar in their learning curves.
I am planning to build a nodejs application as back end of a web application.
This server application will provide a rest api, websockets service and process to scrape some sites with a headless navigation (like zombie.js) each n hours to feed a database.
I would to ask if it's a good approache build all the things in one nodejs instance or if it's better use several nodejs applications for every task.
If you are having small size application (which doesnot require scale in future), you can keep all the stuff Rest, Socket, scraping on the same project).
Note: After scraping if you are processing HTML content, it will take some time to process that in Sync manner. So at that time event loop will be blocked. So Rest API will not handle any request. In this scenario, you can keep Rest and Socket combinely in one project and Scraping thing in another project.
If you are planning to scale in near future, I suggest to Keep Seperate Instance, considering Benefit of SOA in scaling.
In my opinion better way is using different Node.js applications. First one will server your API. Another one will work as a socket server on different port.
About scraper it can be PHP (as well as nodejs) script which will run as a cron job. How to setup cron job you can check this question:
https://askubuntu.com/questions/2368/how-do-i-set-up-a-cron-job
or this tutorial for Ubuntu server: https://help.ubuntu.com/community/CronHowto
I think it will be best approach.
I am trying to plan out the best way to approach developing a node application, and am not sure what would provide the best performance. A little bit of info on the overall plan: the entire project will involve a web app as well as a 'bot' app. The bot app in question is node-steam, which is quite a substantial application on its own. My question is whether I should run two separate node processes for each app (one for web server and one for node-steam), or code them into one combined node process?
Also please note that I will need for the web app to be able to communicate with node-steam. I am planning on integrating socket.io into node-steam to make calls to it via web app actions. Is this the best approach if I keep the apps as separate node processes?
EDIT: When I refer to letting the web app communicating with node-steam, I meant that there are functions which need to be triggered in node-steam when a user does something in the web app (namely they perform specific actions in the browser), so I am planning on doing this via socket directly to the node-steam app, rather than to the web app and then routing the calls on to node-steam. As far as I can tell this is the simplest way of doing it.
Any guidance is appreciated.
Thank you
If the bot app is quite substantial, you should code it for scalability. You probably will be spawning worker threads or even scale across multiple nodes in no time, for the bot app alone.
I'm making an app that will have:
iOS and Android apps
A web-based "dashboard" to display data gathered from the mobile apps
The app requires that end-users create an account with us (we mostly likely will NOT use Facebook/Twitter logins).
Everything is/will be hosted on AWS using EC2/RDS/S3 (All encapsulated in Elastic Beanstalk)
| Web Browser | <----> | sails.js app | <-------> |actionhero.js API|
⬆︎
⬆︎
| Mobile app(s) | <-------------------------------------/
So far, I've built most of the backing API in actionhero.js, hosted on AWS.
It made sense to me to separate the API and the web app, because there web app is only for a small subset of users -- I'd expect 50x the traffic from our mobile apps over the web app.. We could scale the API to server the mobile users without unnecessarily scaling the sails.js app.
My questions are:
(biuggest unknown) How should I handle authentication? The sails.js app needs to be able to make requests to the API, and so do the mobile applications.
I was looking at the oauth2orize node module for creating our own Auth server, but it is designed for Connect/Express, so I don't think I could leverage it in the actionhero.js-based API.
If the solution is to create an OAuth server, am I supposed to host that on its own EC2 instance?
(AWS-specific question) I don't fully understand the use case for creating what AWS describes as a "worker tier" enviornment. Would there be a reason that the API would fall into that category?
If I want to run a data querying and aggregation task, I would create a separate node process for that, correct? If so, would that background worker have to exist on its own EC2 instance?
Sails.js and Actionhero.js both provide heavy support for socket.io. Should communication between the Sails app and my API happen over a persistent WebSocket connection? Will that scale if I need to create new instances in the future?
This seems like a fairly typical pattern; I'd like to hear if there are any big red flags in this design, before I paint myself into a corner. :-) THANKS!
Bonus question (specific to AWS Elastic Beanstalk)
Will I create separate "Applications" for the sails.js server and the API server? It looks like that's the only way to set it up, anyhow, but I want to make sure.
We have used node and beanstalk for a couple of applications now. For authentication, you can create an account for the user when they first access the app, and store the account id on the device. If you want them to be able to log in from multiple devices, you'll need to provide some kind of way of them identifying themselves, which is either id/password, or using Facebook. It's not that tough to set that up. Use session to allow them to log in and stay logged in. We generally just store the user id in the session.
A worker tier is for something you want to decouple from your app, something that you want to do that you don't need to know whether it succeeded/failed. A notification server is a prime example. You send the info for the notification into an SQS queue, that then gets sent to the worker tier, that does the work. We are just trying to figure this out now.
A big aggregation process, yes, I'd take it elsewhere, so it's not eating up your production server(s). You might want to create some data aggregation ongoing, as transactions are saved, so it accumulates. Big rollups after the fact can be time consuming and fragile.
Sounds like yes, they would be seperate applications.
A good tip. We use grunt to create the zip files for the app. It's a node batch tool. We check the latest info out of SVN, clean it up by doing things like removing .svn directories, apply our configuration into the config files by doing simple string replacement, then zip up resulting output. This then gets loaded into beanstalk. This takes all the guess work and time out of actually doing a new deployment. We can get a new build up in minutes that way.
Beanstalk can be very frustrating. When it fails, it's not very good at telling you why.