Background
Our backend is currently written in Grails. We would like to change the backend to NodeJs. We would like to execute the change in small iterations. We deploy everything on AWS.
Question
How to change the technology from Grails to NodeJs iteratively?
My opinion
Although we don't use Microservice Architecture (and none of us has any experience with it) I personally would:
build a NodeJs server before our Grails server (something like a Gateway API maybe?)
at first the NodeJs would just pass requests/responses to/from Grails
then we would move other functionality from Grails (request logging, validations, ...) until we moved everything needed. (Maybe we keep something on Grails but most of the logic should end up on NodeJs.)
We have done a successful migration from Ruby & Rails to API Gateway & Lambda based Microservices written in NodeJS. The same architecture you can use if you prefer NodeJS server (Without Microservices) or using Docker Containers Cluster with ECS.
Setup CloudFront as a Proxy which will be getting all the HTTP traffic to your application domain (You can map the DNS to CloudFront CName)
In CloudFront you can add the current Grails Application as the default origin and behavior which makes your application works as it is same as today.
Then you can setup your Microservices Architecture with API Gateway and Lambda or NodeJS Web Server or Docker Container Cluster with ECS seperately. (Note that if you use a Relational Database like MySQL, it also requires to do proper placement of new server code in Lambda, WebServer or Containers so that it can access the Database)
Afterwards, you can write the new feature logic and override one http subpath at a time from CloudFront pointing to the new application.
Following diagram shows the architecture in high-level.
Note: In the diagram, it uses DynamoDB for new Microservices and in the migration phase, you can also connect to the current Database with proper VPC, Subnet and Server placement.
In Addition you gets the benefits from CloudFront CDN in caching static assets to improve application performance and also you will be able to terminate the SSL handhshake in CloudFront with free SSL Certificates issued by Amazon.
Your approach is definitely possible. But I'd take another one and try using microservices. Then you will move parts of your code one by one to little cute microservices and eventually have microservice architecture. I like this approach because it allows very fast switching... Of everything. You can build your microservices with Java, Node, Go - everything you want. And if you suddenly discover that node.js is not up to your expectations (for example, if you have hardcore math modules) - just throw this microservice out and quickly impliment it in any other language and framework. The most important part is defining communication architecture. REST API is already a thing of the past and you will possibly want to use some message broker like RabbitMQ.
Related
I currently manage a cluster of VMs on a number of dedicated hosts to provide apache, nginx and node live and development servers. This of course requires constant and time consuming maintenance to ensure security and reliability. I've found more time is spent looking after this platform then coding new and exciting projects. So I've been looking into the Google App Engine to remove the need of managing any VMs but I'm struggling to work out how to get it to function for me!
Currently I find myself developing mostly in Angular (v4-5) for my frontend and nodejs for backend. My development nginx server powers my angular apps and routing to ng-serve and to a separate vm that runs my node apps. I use PM2 to manage the apps on both servers.
This works great! I can code locally push my changes via an rsync script to the servers, the app restarts and changes updated. More importantly, I can affectively code between the front and backend! When ready I can comfortably switch the code to the live servers with little effort - nice!
This is where I am struggling...
I can't seem to work how I would develop and publish versions of both the front and backend code in one App Engine project.
Is this possible? How would I go about deploying/publishing both aspects?
Would I be better having two projects such as example.com & api.example.com? If so, can I get the two projects to talk to one another when developing?
I have and can create a angular/nodejs app in the App Engine but I can't work the basics of front and backend development in this managed service.
I'd like to use the great features of the App Engine such as versioning, easy scaling and importantly deployment of apps and updates. Also, to move all my websites including some older ones in PHP to the App Engine.
Any help surrounding this would be much appreciated. Thanks!
As #Yandrak3 suggested, a microservices architecture is what you need. But keep in mind that that document relates to the App Engine Standard environment which does not support Node.js as a runtime environment. But keep the microservices architecture in mind when deploying to App Engine Flexible.
On backend and frontend
Frontend and backend are no longer used to describe the presentation layer and the data access layer of an App Engine application. The only reference in the documentation is here. The (VM) instances managing a service of your app which are configured with automatic scaling are considered part of the frontend infrastructure, while the ones configured with manual scaling are considered backend infrastructure.
The reason for this is that automatic scaling is one of App Engine's
great features [...] easy scaling,
automatically presenting your app's frontend in a manner scaled with the number of external requests incoming to your app.
Manual scaling is more suited for backend operations, where you might want to run operations dependant on the state of the memory over time, or other scenarios. You can find some more information on scaling types here. Keep in mind that this latter document is under App Engine Standard documentation and it includes basic scaling, a feature not available in the App Engine Flexible environment.
On services and versioning
In your case, your frontend and backend modules of your application will become two separate services in App Engine Flex. For each service you can deploy multiple versions. More, explained here.
Communication between services, in this case between your frontend and backend, can be done through HTTP requests between them.
If the next question is how HTTP requests from users reach the appropriate version of a service (or a service), check this document.
To deploy multiple services, you will use the same commands and you will separate each deployment and service through their afferent configuration file, app.yaml.
Your question requires a response with a pretty wide (and deep) spectrum of concepts. Hopefully, this answer is good to start with.
For a project i have to create website showing data from a database with data that changes frequently (sensor data: temperaturen, humidity,...). This by making an API to get the data. I created a REST api in Node.js to do this.
So Database -> API -> WEBSITE
What would be the best way to create the Dynamic website and what is the best practice to get the API to the outside using AWS? And in this solution, is it usefull to use NGINX in any way?
Other suggestions and literature about the subject are also welcome.
To create the dynamic site i would say use angular JS/Angular2 which provides you the capability of two way data binding and make life easy to implement the website in quick period of time. For REST API you have used NODEJS which you can host to AWS and provide a reverse proxy viz; NGINX. the use of reverse proxy is useful in case of high traffic sites;which also protects the NodeJS server from direct exposure to Internet traffic and allows you a great deal of flexibility in using multiple application servers, in load balancing across the servers, and in caching content. So if you think your app is having heavy traffic you can use NGINX
So technology stack for you can be something like MEAN (MongoDB -if you also need to choose db; ExpressJs,NODEJS and ANGULAR) which hosted on AWS EC2 server.
Apart from this in case you dont want to maintain server you can also think of serverless architecture on aws (with lamda). Hope my answer will give you good pointers further.
I am new to Spring-Boot(Cloud) and going to work on one new project.
Our project architect have designed this new application like this:
One front end Spring boot application(which is also microservice) with Angular-2.
One Eureka server, to which other microservices will connect.
ZUUL proxy server, which will connect to Front end and mircoservices.
Now, followings are the things I am confuse about and I can't ask him as he too senior to me:
Do I need to have separate ZUUL proxy server ? I mean, what is the pro and cons of using same front end application as ZUUL server?
How MicorService-1 will communicate with Node's MicroService-1? Some blogs suggesting the Sidecar. But again, why ? because I can directly invoke ReST api of NodeJS-1 from the Microservice-1.
(I know, this is very tough to guess but still asking) NodeJS services(which are not legacy services) are suppose to call some third party api or retrieve data from DB.
Now, what I am not getting is why we need NodeJS code at all? Why can't we do the same in the microservices written in Java?
Can anyone who have worked with similar kind of scenario shed some light on my doubts?
I do not have the full context of the problem that you are trying to solve, therefore answers bellow are quite general, but still may be useful:
Do I need to have separate ZUUL proxy server? I mean, what is the pro and cons of using same front end application as ZUUL server?
Yes, you gonna need a separate API Gateway service, which may be Zuul (or other gateways, e.g tyk.io).
The main idea here is that you can have hundreds or even thousands of microservices (like Amazon, Netflix, etc.) and they can be scattered across different machines or data centres. It would be really silly to enforce your API consumers (in your case it's Angular 2) to memorise all the possible locations of each microservice. Better to have one API Gateway that knows about all the services under it, so your clients can call your gateway and have access to the underlying services through one single place. Also having one in your system will decouple your clients from your services, so it's possible to evolve them independently.
Another benefit is that you can have access control, logging, security, etc. in one single place. And, BTW, I think that you are missing one more thing in your architecture - it's Authorization Server. A common approach in building security for microservices is OAuth 2.0.
How MicorService-1 will communicate with Node's MicroService-1? Some blogs suggesting the Sidecar. But again, why? because I can directly invoke ReST API of NodeJS-1 from the Microservice-1.
I think you could use Sidecar, but I have never used it. I suppose that the question 'why' is related to the Discovery Service (Eureka in your architecture).
You can't call microservice NodeJS-1 directly because there may be several instances of NodeJS-1, which one to call? Furthermore, you can't know whether service is down or alive at any given point in time. That's why we use Discovery Services like Eureka - they handle all of these. When any given service has started, it must register itself in Eureka. So if you have started several instances of NodeJS-1 they all will be registered in Eureka and whenever Microservice-1 wants to call NodeJS-1 service it asks Eureka for locations of live instances of NodeJS-1. The service then chooses which one to call.
(I know, this is very tough to guess but still asking) NodeJS services(which are not legacy services) are suppose to call some third party api or retrieve data from DB.
Now, what I am not getting is why we need NodeJS code at all? Why can't we do the same in the microservices written in Java?
I can only assume that the NodeJS has been chosen because it has an outstanding performance for IO operations, including HTTP requests that may come in hand when calling 3-rd party services. I do not have any other rational explanations for this.
In general, microservices give you a possibility to have your microservices written in different languages and it's cool indeed since each language solves some problems better than the other. On the other hand, this decision should be made with caution and answer the question - "do we really need a new language in our stack to solve problem X?".
Am I correct in the assumption that without access to the MongoDB server, there is not much point developing with Meteor?
Meteor is a great framework for building, packaging and deploy apps and sites. From a development POV, the templating and responsive DB work make prototyping so much easier than most MVC's.
I understand that underneath the hood, websockets and DDP provide the realtime sync'ing magic which means that you need access to the MongoDB server, something you don't have with PaaS solutions like GoogleAppEngine, Parse or Kinvey.
So, for the backend developer, they don't derive much benefit from Meteor since they need to maintain the server stack and scalability issues.
Is there a path to create and deploy products with Meteor without having to build and maintain the backend infrastructure? Heroku is still pretty close to the bone when it comes to managing infrastructure.
Wondering if there's a way to have CRUD operations through a REST driver that maps out to whatever PaaS you want and have the PaaS post log changes to a server that strictly handles websocket connections. Basically, pass the CRUD operation to a PaaS and maintain your own websocket server/s.
MeteorPedia has a page on deploying to PaaS: http://www.meteorpedia.com/read/Category:PaaS_providers
Recently, Google AppEngine has added support for custom VMs.
You can also use MongoHQ or similar for the database.
I'm making an app that will have:
iOS and Android apps
A web-based "dashboard" to display data gathered from the mobile apps
The app requires that end-users create an account with us (we mostly likely will NOT use Facebook/Twitter logins).
Everything is/will be hosted on AWS using EC2/RDS/S3 (All encapsulated in Elastic Beanstalk)
| Web Browser | <----> | sails.js app | <-------> |actionhero.js API|
⬆︎
⬆︎
| Mobile app(s) | <-------------------------------------/
So far, I've built most of the backing API in actionhero.js, hosted on AWS.
It made sense to me to separate the API and the web app, because there web app is only for a small subset of users -- I'd expect 50x the traffic from our mobile apps over the web app.. We could scale the API to server the mobile users without unnecessarily scaling the sails.js app.
My questions are:
(biuggest unknown) How should I handle authentication? The sails.js app needs to be able to make requests to the API, and so do the mobile applications.
I was looking at the oauth2orize node module for creating our own Auth server, but it is designed for Connect/Express, so I don't think I could leverage it in the actionhero.js-based API.
If the solution is to create an OAuth server, am I supposed to host that on its own EC2 instance?
(AWS-specific question) I don't fully understand the use case for creating what AWS describes as a "worker tier" enviornment. Would there be a reason that the API would fall into that category?
If I want to run a data querying and aggregation task, I would create a separate node process for that, correct? If so, would that background worker have to exist on its own EC2 instance?
Sails.js and Actionhero.js both provide heavy support for socket.io. Should communication between the Sails app and my API happen over a persistent WebSocket connection? Will that scale if I need to create new instances in the future?
This seems like a fairly typical pattern; I'd like to hear if there are any big red flags in this design, before I paint myself into a corner. :-) THANKS!
Bonus question (specific to AWS Elastic Beanstalk)
Will I create separate "Applications" for the sails.js server and the API server? It looks like that's the only way to set it up, anyhow, but I want to make sure.
We have used node and beanstalk for a couple of applications now. For authentication, you can create an account for the user when they first access the app, and store the account id on the device. If you want them to be able to log in from multiple devices, you'll need to provide some kind of way of them identifying themselves, which is either id/password, or using Facebook. It's not that tough to set that up. Use session to allow them to log in and stay logged in. We generally just store the user id in the session.
A worker tier is for something you want to decouple from your app, something that you want to do that you don't need to know whether it succeeded/failed. A notification server is a prime example. You send the info for the notification into an SQS queue, that then gets sent to the worker tier, that does the work. We are just trying to figure this out now.
A big aggregation process, yes, I'd take it elsewhere, so it's not eating up your production server(s). You might want to create some data aggregation ongoing, as transactions are saved, so it accumulates. Big rollups after the fact can be time consuming and fragile.
Sounds like yes, they would be seperate applications.
A good tip. We use grunt to create the zip files for the app. It's a node batch tool. We check the latest info out of SVN, clean it up by doing things like removing .svn directories, apply our configuration into the config files by doing simple string replacement, then zip up resulting output. This then gets loaded into beanstalk. This takes all the guess work and time out of actually doing a new deployment. We can get a new build up in minutes that way.
Beanstalk can be very frustrating. When it fails, it's not very good at telling you why.