I currently manage a cluster of VMs on a number of dedicated hosts to provide apache, nginx and node live and development servers. This of course requires constant and time consuming maintenance to ensure security and reliability. I've found more time is spent looking after this platform then coding new and exciting projects. So I've been looking into the Google App Engine to remove the need of managing any VMs but I'm struggling to work out how to get it to function for me!
Currently I find myself developing mostly in Angular (v4-5) for my frontend and nodejs for backend. My development nginx server powers my angular apps and routing to ng-serve and to a separate vm that runs my node apps. I use PM2 to manage the apps on both servers.
This works great! I can code locally push my changes via an rsync script to the servers, the app restarts and changes updated. More importantly, I can affectively code between the front and backend! When ready I can comfortably switch the code to the live servers with little effort - nice!
This is where I am struggling...
I can't seem to work how I would develop and publish versions of both the front and backend code in one App Engine project.
Is this possible? How would I go about deploying/publishing both aspects?
Would I be better having two projects such as example.com & api.example.com? If so, can I get the two projects to talk to one another when developing?
I have and can create a angular/nodejs app in the App Engine but I can't work the basics of front and backend development in this managed service.
I'd like to use the great features of the App Engine such as versioning, easy scaling and importantly deployment of apps and updates. Also, to move all my websites including some older ones in PHP to the App Engine.
Any help surrounding this would be much appreciated. Thanks!
As #Yandrak3 suggested, a microservices architecture is what you need. But keep in mind that that document relates to the App Engine Standard environment which does not support Node.js as a runtime environment. But keep the microservices architecture in mind when deploying to App Engine Flexible.
On backend and frontend
Frontend and backend are no longer used to describe the presentation layer and the data access layer of an App Engine application. The only reference in the documentation is here. The (VM) instances managing a service of your app which are configured with automatic scaling are considered part of the frontend infrastructure, while the ones configured with manual scaling are considered backend infrastructure.
The reason for this is that automatic scaling is one of App Engine's
great features [...] easy scaling,
automatically presenting your app's frontend in a manner scaled with the number of external requests incoming to your app.
Manual scaling is more suited for backend operations, where you might want to run operations dependant on the state of the memory over time, or other scenarios. You can find some more information on scaling types here. Keep in mind that this latter document is under App Engine Standard documentation and it includes basic scaling, a feature not available in the App Engine Flexible environment.
On services and versioning
In your case, your frontend and backend modules of your application will become two separate services in App Engine Flex. For each service you can deploy multiple versions. More, explained here.
Communication between services, in this case between your frontend and backend, can be done through HTTP requests between them.
If the next question is how HTTP requests from users reach the appropriate version of a service (or a service), check this document.
To deploy multiple services, you will use the same commands and you will separate each deployment and service through their afferent configuration file, app.yaml.
Your question requires a response with a pretty wide (and deep) spectrum of concepts. Hopefully, this answer is good to start with.
Related
I am planning to build a rest api in node that will be consumed by a react-native mobile app(using JWT tokens) and reactJS web app(admin UI using sessions/http-only cookies). Would you advise me to write this directly in express or use a framework like nestJS?
How should I structure/architect the project to meet the above requirements taking into consideration, ease of Management, autoscaling, hosting, etc?
Would you consider dockerizing the node application over running node directly on the server? is there any performance gains?
should I break the project into 3 repos ie:
rest api | backend webUI(reactJS) | mobile app (react-native)
or
rest api + backend webUI(reactJS) | mobile app (react-native)
Ideas/advises are welcome please, thank in advance.
I'll try to answer based on my work experience in both:
Structure/architecture of the project depends if you are working alone on the API or in teams (express gives you freedom in code however you like but it's a nightmare if your application grows and gets bigger in complexity with a big team trying to maintain it), NestJS has a structure to follow for you and your team (if MVC model gets the job done for your use case)
Docker use is for ease of deployment for both (if you know what you are doing) and can have better performance when running multiple instances. the catch is in the production server with a docker database backup and recovery are harder and if you need migration/restructure after deployment in prod welp...you gonna cry
I always separate (API | web UI | mobile app) so I can debug them easily separate domains of errors but talk to your team and provide docs for what they want in the Rest service if you are alone and the WebUI is not exposed to the same user base as the mobile app then it doesn't matter really
I've started to port a web app backend to Google App Engine for scaling. But I'm completely new to GAE and just reading into the concepts. Steep learning curve.
I'm 95% certain that at some point many millions or at another point at least hundreds of thousands of users will start using the web app through a GUI app that I'm writing. And they will be globals users, so at some point in the future I'm expecting a relatively stable flow of connection requests.
The GAE Standard Environment comes to mind for scaling.
But I also want the GUI app to react when user related data changes in the backend. Which suggests web sockets, which aren't supported in the Standard Environment, but in the Flexible Environment.
Here's my idea: The main backend happens in a Standard app, but the GUI listens to update notifications from a Flexible app through web sockets. The Standard app calls the Flexible app after noteworthy data changes have occurred, and the Flexible app notifies the GUI.
But is that even possible? Because sibling Flexible instances aren't aware of each other (or are they?), how can I trigger the persistent connections held by the Flexible instance with an incoming call from the Standard app to send out a notification?
(The same question goes for the case where I have only one Flexible app and no Standard app, because the situation is kind of the same.)
I'm assuming that the Flexible app can access the same datastore that the Standard app can. Didn't look this one up.
Please also comment on whether the Standard app is even a good idea at all in this case and I should just go with Flexible. These are really new concepts to me.
Also: Are there limits to number of persistent connections held by a Flexible app? Or will it simply start another instance if a limit is reached?
Which of the two environments end up cheaper in the long run?
Many thanks.
You can only have one App engine instance per project however you can have multiple flex services or standard services inside of an instance.
Whether if standard is a good idea it depends up to your arquitecture, I'm pretty sure you've looked at the comparison chart, from experience is that if your app can work okay with all the restrictions (code runtimes, no availability to do background process, no SSH debugging, among others) I will definitely go for standard since it has a very good performance when working with spikes of traffic deploys new services in just seconds, keep in mind that automatic scaling is needed for the best performance result.
There are multiple ways to connect between flex or standard services one would be to just send an HTTP request from one service to another, but some other options with GCP services like Pub/Sub.
In the standard environment, you can also pass requests between
services and from services to external endpoints using the URL Fetch
API.
Additionally, services in the standard environment that reside within
the same GCP project can also use one of the App Engine APIs for the
following tasks:
Share a single memcache instance.
Collaborate by assigning work
between services through Task Queues.
Regarding Data Store you can access the same datastore from different services here is a quickstart for flex and the quickstart for standard
Which of the two environments end up cheaper in the long run?
Standard pricing is based on instance hours
Flexible pricing is based on usage of vCPU, memory, and persistent disks
If your service run very hight performance process on short periods of time probably standard will be chepear, however if you run low performance process on long periods of time, flex will be chepear, but again it depends on each use case.
I was making a thinking exercise about how could I deploy a chat service like WhatsApp or Slack (just wondering), so people could really use it. You need two main parts, the client software (e.g. the app running on the smartphones), and the server software. So how would you develop the server-side code and make it work?
The first idea that came to me was the classic hosting service, but it cannot be the simple "web hosting service", probably because something like this should be programmed at a lower level and not working with HTTP requests and responses. Maybe using specific server-side technology like Node.js (any other suggestion?) to manage different type of requests at lower level, let's say at the layer where TCP lives, would be a better solution.
So I heard about the Amazon Web Services (AWS), which is not classic hosting, it's a cloud computing service. The problem is that I don't know exactly how this works. Could I deploy a server-side application that works at that low level of networking and also makes requests to databases? Would it be difficult to offer this kinf of service using AWS?.
I would like to hear all your opinions about any aspect of this. Would you use other kinf of technology on the server? What do you think about AWS, and if you would think it's a good option, where can I get some info to learn how to use it?
Server Side Code
You can create a chat service backend using NodeJS + express(or Hapi) to cater input Http Requests.
For Hosting: Cloud servers are pretty available these days and allow you to scale if your app grows with time.
Database:
if you already have your DB available (cool) just use ORMs ( like (Sequelize) which provides easy interaction of Nodejs service with your DB. (I have used MySQL + Sql Server + Oracle)
If not, you can create a new DB (MySql- free on your hosting server (cloud?)
I used Microsoft Azure to host a Nodejs(+ Hapi.js) Backend Service ,to be consumed by my mobile application, over the internet.
Azure gives you $200 free credit which is sufficient for you to try and make your hands dirty with them. There are numerous tutorials available for MS-Azure Api App hosting which will guide you to a successful deployment.
I have not yet explored AWS as of now, but i trust that they will be similar in their learning curves.
I'm just starting to learn more about the Node.js paradigm and finding it hard to grasp basic concepts. I'm familiar with front-end tools (HTML, CSS, JS) and have been using PHP with Apache server and mySQL db to deploy websites until now.
It seems to be that node is it's own server, and I would then need a SaaS platform like Heroku, or AWS (I'm not even sure if i'm understanding the purpose of these) if someone could explain the difference?
Is the database managed inside this service?
Is the website being hosted there?
In steps how would you get the node app to be served onto your domain name?
For Scalability purposes I understand how having dedicated big infrastructure can help, but if building a low traffic website with small number of members is there even a point in using node?
normal hosting services cost between $4-20 usd. per month and AWS or Heroku seem to start at a MUCH higher price. Is Node only to be used for large scale scaling business model?
Thank you for any answers or good recent external resources (websites or books) you could point me to.
You could easily host a low traffic website built with node.js absolutely for free on Heroku.
To see how easy that is, just go through the Getting Started With Node.js Heroku tutorial, in which you will do just that.
When you build your website with node.js, your own code that your write is the web server. You have no separate web server to configure and interact with (such as Apache). So what you see (or code...) is exactly what you get.
You will probably want to use a framework such as Express to build your web server functionality in your node.js app.
As for NoSQL databases, the way to do this on Heroku is to use an appropriate "add-on" from the Heroku Elements Marketplace. For example, you could easily add Heroku Redis or MongoLab. These are just some of the NoSQL "Database as a Service" options. That means that the Database is itself hosted somewhere in the cloud, and your app simply interacts with it. You don't need to worry about database maintenance, security upgrades etc. You just need to concentrate on your app's interaction with the DB.
Almost all add-ons in the Heroku Elements Marketplace feature a free-tier, that may suffice for your needs, at least initially. So you might be able to get your low-traffic website (including the DB) up and running completely for free, at least initially.
One thing you will need to understand is how Heroku free dyno hours work.
If you need your website to be continuously available 24/7, you may need to verify your Heroku account with a credit card (even though no charges would be incurred as long as you deploy only 1 free web dyno and are on a free-tier plan of your NoSQL DB as a Service). For further details, see this answer.
You also need to consider whether you can tolerate dyno sleeping in your low-traffic app. If not, you would need to prevent your web app from sleeping, which can also be done completely for free. For tips on how to do that see here.
As for serving your Heroku node.js app website from your own domain name, see here. Note that for this too you will need to verify your Heroku account with a credit card, although this too does not incur any charges.
Node.js is supported by many web hosting already, especially for those who use Plesk or cPanel as their web hosting control panel. Here is guide about how to setup a Node.js website via Plesk control, https://www.bisend.com/blog/how-to-set-up-a-node-js-site-in-plesk. As you said, it's very easy to host your website with a cheap shared web hosting.
Am I correct in the assumption that without access to the MongoDB server, there is not much point developing with Meteor?
Meteor is a great framework for building, packaging and deploy apps and sites. From a development POV, the templating and responsive DB work make prototyping so much easier than most MVC's.
I understand that underneath the hood, websockets and DDP provide the realtime sync'ing magic which means that you need access to the MongoDB server, something you don't have with PaaS solutions like GoogleAppEngine, Parse or Kinvey.
So, for the backend developer, they don't derive much benefit from Meteor since they need to maintain the server stack and scalability issues.
Is there a path to create and deploy products with Meteor without having to build and maintain the backend infrastructure? Heroku is still pretty close to the bone when it comes to managing infrastructure.
Wondering if there's a way to have CRUD operations through a REST driver that maps out to whatever PaaS you want and have the PaaS post log changes to a server that strictly handles websocket connections. Basically, pass the CRUD operation to a PaaS and maintain your own websocket server/s.
MeteorPedia has a page on deploying to PaaS: http://www.meteorpedia.com/read/Category:PaaS_providers
Recently, Google AppEngine has added support for custom VMs.
You can also use MongoHQ or similar for the database.