I am new to node.js and docker as well as the microservices architecture.
I am trying to understand what microservices architecture actually is and theoretically I do understand what microservices arch is.Please see the following implementation
This is the index.js file:
var express = require("express");
var app = express();
var service1 = require("./service1");
var service2 = require("./service2");
app.use("/serviceonerequest",service1);
app.use("/servicetwo",service2);
app.listen(3000,function(){
console.log("listening on port 3000");
});
The file service1:
var express = require("express");
var router = express.Router();
router.use(express.json());
router.get("/",(req,res)=>{
//perform some service here
res.send("in the get method of service 1");
res.end();
});
router.post("/letsPost",(req,res)=>{
res.send(req.body);
res.end("in post method here");
})
module.exports = router;
The file service2:
var express = require("express");
var router = express.Router();
router.use(express.json());
router.get("/",(req,res)=>{
//perform some service here
res.end("in the GET method for service 2");
});
router.post("/postservice2",(req,res)=>{
res.send(req.body);
});
module.exports = router;
Does the above qualifies as 'micro service architecture'?Since there are two services and they can be accessed through the 'api-gateway' index.js?
I have read the basic tutorial of Docker.Is it possible to have the above three "modules" in separate containers?
If the above does not qualify as a microservice what should be done to convert the above sample into microservices?
This does not really qualify as a microservice architecture.
The whole code you provided is small enough to be considered one single microservice (containing two routes), but this is not an example of a microservice architecture.
According to this definition;
"Microservices are small, autonomous services that work together"
Building Microservices <-- tip: you should read this book
Both service1 and service2 to be considered a microservice should be autonomous, what is not happening when you place them together in the same express app. For example; you cant restart one without not-affecting the other. You cant upgrade version of service1 without also having to deploy service2. They are not distributed in the sense that they can leave in separate machines.
Actually I think you are missing the concept of microservice architecture. Your services must be independent and if they need to communicate with each other they must use a service discovery mechanism that will return a healthy instance of that service. Another pattern of microservices architecture is that every single service must have an endpoint (/health) that returns the health status of the service, having this your service discovery can check if that instance is healthy and return it as a healthy instance..
Microservices is not about technology it's about the concept and implementing the right patterns. Otherwise you will have a chaos architecture :D
If you want to understande the concepts I really recommend this book: http://shop.oreilly.com/product/0636920033158.do
Related
I am required to save logs into a MySQL database of each request and response made to the backend. The issue is that we are migrating to microservices architecture. The backend was made with NodeJS and Express, and it has a middleware that does this task. Currently, it has this middleware attached to each microservice.
I would like to isolate this middleware as its own microservice. The issue is that I don't know how to redirect the traffic this way. This is how I would like to manage it:
I would like to do it this way, because we can make changes or add features to the middleware without having to implement it in each microservice. This is the middleware's code:
const connection = require("../database/db");
const viewLog = (req, res, next) => {
const oldWrite = res.write,
oldEnd = res.end,
chunks = [],
now = new Date();
res.write = function (chunk) {
chunks.push(chunk);
oldWrite.apply(res, arguments);
};
res.end = function (chunk, error) {
if (chunk) chunks.push(chunk);
const bodyRes = Buffer.concat(chunks).toString("utf8");
connection.query("CALL HospitalGatifu.insertLog(?,?,?,?,?)", [
`[${req.method}] - ${req.url}`,
`${JSON.stringify(req.body) || "{}"}`,
bodyRes,
res.statusCode === 400 ? 1 : 0,
now,
]);
oldEnd.apply(res, arguments);
};
next();
};
module.exports = viewLog;
I think there might be a way to manage this with Nginx which is the reverse proxy that we are using. I would like to get an approach of how to change the logs middleware.
Perhaps you might want to take a look at the sidecar pattern which is used in microservice architectures for common tasks (like logging).
In short, a sidecar runs in a container besides your microservice container. One task of the sidecar could be intercepting network traffic and logging requests and responses (and a lot of other possible tasks). The major advantage of this pattern is that you don't need to change any code in your microservices and you don't have to manage traffic redirection yourself. The latter will be handled by the sidecar itself.
The disadvantage is that you are required to run your microservices containerized and use some kind of container orchestration solution. I assume this being the case since you are moving towards a microservices based application.
One question about the log service in between of the webapp and the NGNIX server. What if the logging services goes down for some reason, is it acceptable for the entire application to go down?
Let me give you not exactly what you requested but something to think about.
I can think on 3 solutions for the issue of logging in microservices, each 1 have its own advantages and disadvantages:
Create a shared library that handles the logs, I think its the best choice in must cases. An article I wrote about shared libraries
You can create API gateway, it is great solution for shared logic to all the requests. So it will probably be more work but then can be used for other shared logic. Further read (not written by me :) )
A third option (which I personally don't like) is create a log microservice that listens to LogEvent or something like that. Then from your MSs publish this event whenever needed.
My question could be flagged as "opinion based" but I am wondering which approach is the best for my application as I am able to do it in both ways.
I am building chat application in which users and conversations are saved in MongoDB. I will have my react application consuming API/APIs. The question is - is it better to have REST API and Socket.io applications running separate? For example:
Have REST API running on port 3005
Have Socket.io running on port 3006
React Application consuming these 2 separately and basically they will not know about each other. My endpoints in REST API endpoints and socket.io will be invoked only in front-end.
On the other hand, I can have my socket.io application and REST API working together in 1 big application. I think it is possible to make it working without problems.
To sum up, at first glance I would take the first approach - more cleaner and easy to maintain. But I would like to hear other opinions or if somebody had a similar project. Usually how the things are made in this kind of projects when you have socket.io and REST API?
I would check the pros and cons for both scenario. For example code and resource reusability is better if you have a single application and you don't have to care about which versions are compatible with each other. On the other hand one error can kill both applications, so from security perspective it is better to have separate applications. I think the decision depends on what pros and cons are important to you.
you can make a separate file for socket.io logic like this:
// socket.mjs file
import { Server } from "socket.io"
let io = new Server()
const socketApi = {
io: io
}
io.on('connection',(socket)=>{
console.log('client connected:', socket.id)
socket.join('modbus-room')
socket.on('app-server', data=>{
console.log('**************')
console.log(data)
io.to('modbus-room').emit('modbus-client', data)
})
socket.on('disconnect',(reason)=>{
console.log(reason)
})
})
export default socketApi
and add it to your project like this:
// index.js or main file
//...
import socketApi from "../socket.mjs";
//...
//
/**
* Create HTTP server.
*/
const server = http.createServer(app);
socketApi.io.attach(server);
//
I am trying to develop an node js application that acts as api-gateway / facade layer for Services(developed in Spring Boot).
Is it a good practice or not?
If yes, which nodejs framework should I use?(Async / co / Promise / Async-Await ) etc. I mean what is currently used mostly on production enviornemnt?
"Is it a good practice or not?"
What is your question related to? Using an API gateway/facade? Using spring boot? Using async/await...? What exactly is your problem?
I guess you want to develop a spring boot based microservice architecture with a nodeJS based api orchestrator as a frontcontroller and single entry point?
Don't confuse the technical side of naive routing (load balancing with nginx, round robin, reverse proxy, etc.) to increase capacity, speed, availability etc. with the semantic business integration of services through url path mapping.
An API Orchestrator addresses the semantic abstraction and integration of an underlying service landscape. API Gateway vs. API Orchestrator!
To my personal view, using an API Orchestrator is a acceptable solution in conjunction with microservices. It is the easiest and modest way to
integrate and componse an underlying service layer.
Just to state a few positive and negative aspects:
Single entry point for standard business cases such as
authentification, security issues, session mangament, logging etc.
Can also be started and managend as a microservice. Feel free to use
a 3-tier layered architecture for the API orchestrator microservice
Abstracts the complexity of an underlying microservice layer
Might become a god thing.
In context of microservices, the API Orchestrator performs to much
of business cases
High coupling, complexity...
Design trial of a nodeJS based API Orchestrator with HTTP Communication ...
Evaluate a (web) server (express.js, hapi.js, your-own-node-server)
Evaluate a http-request API (axios, node-fetch, r2, your-own-http-api). HTTP-API should resolve to a promise object!
Example of an express.js based API Orchestrator:
const express = require('express');
const http = require('http');
const path = require('path');
const app = express();
const port = 3000;
// define middleware plugins in express.js for your API gateway like session management ...
app.use(express.static(path.join(__dirname, 'public')));
// define relevant business/use case relevant semantic routes or commands e.g. /getAllUsers or REST-URL or /whatever
app.get('/whatever', (request, response) => {
//consumes whatever service
const getWhatEverToGet = () => {
return new Promise((resolve, reject) => {
//connection data should be read from a service registry or by configuration management (process level, file level, environemnt level)
http.get({
hostname: 'localhost',
port: 3001,
path: `/whatever_service_url`
}, (res) => {
// built-in HTTP-API http.get() uses streams, hence "onData"-event should be buffered, not done here!
res.on('data', (data) => {
resolve(data.toString());
});
});
});
}
// Here you can consume more services with the same code, when they are connected to each other use async/await to share data synchronized...
//consumes whatever2 service returns promise
//consumes whatever3 service returns promise
const respondWhatEverData = async () => {
let whatEver = await getWhatEverToGet();
response.send(whatEver)
}
// trigger service complete
respondWhatEverData();
})
app.listen(port, (err) => {
if (err) {
return console.log('Shit happens...', err)
}
console.log(`server listens on ${port}`)
})
TL;DR If your NodeJS application is only expected to forward the request to Spring Boot application, then NodeJS setup would probably not be worth it. You should look at Nginx revere proxy which can do all that efficiently.
Async / co / Promise / Async-Await are not frameworks. Promise / async-await are programming constructs in NodeJS; Async / co are convenience libraries to make manage asynchronous code manageable before Promises and async-await were introduced. That said there are multiple rest frameworks, that you could use to receive and pipe requests to your SpringBoot servers. Take a look at Express.JS, Restify, Sails.js all of them can add REST capabilities to NodeJS. You will also need a Rest Client library (like axios or request both of them then support Promises) to be able to forward your requests to target server.
For passed year I've started around 40 independent web apps using nodejs (each is running it's own server with custom port using express + socket.io). What really buggs me is that pm2 processes list have a vertical scroll )))
The question is: is it normal running so much node servers or there is a better way?
There is no issues running multiple node servers, in fact when it comes to micro services architecture the more you break down, the better. There are a lot of pros and cons to this and you need to figure out if the cons affect you system more than you're willing to sacrifice. I assume when you started out you had no idea of following the micro services architecture but since your application is now distributed over 40+ services the following is an article might give you some insight into managing it properly.
https://derickbailey.com/2016/12/12/making-the-quantum-leap-from-node-js-to-microservices/
If they all are independent then you have to create individual server. But if they are talking to each other (I think they are talking to each other by socket.io) then you can use RPC call. Their is some lib like grpc(Google) and tchannel(Uber). I hope they can solve your problem.
You can start several express and socket.io instances in one NodeJS process, if you wish, like this (based on Express Hello world example):
const express = require('express');
// First app
const app1 = express();
app1.get('/', function (req, res) {
res.send('Hello World!');
});
app1.listen(3000, function () {
console.log('Example app1 listening on port 3000!');
});
// Second app
const app2 = express();
app2.get('/', function (req, res) {
res.send('Hello World 2!');
});
app2.listen(3001, function () {
console.log('Example app2 listening on port 3001!');
});
But there are several aspects to consider:
Performance - if each app is handled by a separate NodeJS process, then apparently each of them have more CPU time and high load on one of them will not affect others.
Logging - if several apps are handled by one NodeJS process you will need to distinguish which of them outputs somehow. Otherwise output will be chaotic.
Аault tolerance - if one NodeJS process handles several apps, then in case of a critical failure in one app all of them will crush simultaneously.
Security - probably some security issues in one app could affect other apps handled by one NodeJS process.
So I run a bunch of a little chatbots written in node, nothing too exciting. However, I recently decided to give them their own little web page to display information in a graphical manner. To do this, I figured I'd just run express.
However, I'm running my bots with a wrapper file that starts each chatbot as a child process. Which makes using express a little tricky. Currently I'm starting the express server in the wrapper.js file like so:
var express = require("express");
var web = express();
web.listen(3001);
And then in the child processes, I'm doing this:
var express = require("express");
var web = express();
web.get("/urlforbot",function (req,res) {
res.send("Working!");
});
However, when I navigate to :3001/urlforbot, I get Cannot GET /urlforbot.
Any idea what I'm doing wrong and how to fix this?
Edit: This is my complete wrapper file: http://snippi.com/s/3vn56m2
Edit 2: This is what I'm doing now. I'm hosting each bot on it's own port, and storing that information in the configs. This is the code I'm using, and it appears to be working:
web.get("/"+cfg.route, function (req,res) { // forward the data
res.redirect('http://url.com:'+cfg.port+"/"+cfg.route);
});
Since your bots run as separate processes (any particular reason?), you have to treat each one as having to implement their own HTTP server with Express:
var express = require("express");
var web = express();
web.get("/urlforbot",function (req,res) {
res.send("Working!");
});
web.listen(UNIQUE_PORT_NUMBER);
Each bot process needs to listen on a unique port number, it can't be shared.
Next, you need to map requests coming in on port 3001 in the 'master' process to the correct child process' Express server.
node-http-proxy has a useful option called a ProxyTable with which to create such a mapping, but it requires the master process to know what the endpoint (/urlforbot in your terms) for each bot is. It also requires that the master knows on which port the bots are listening.
EDIT: alternatively, you can use child_process.fork to fork a new process for each of your bots, and communicate between them and the master process (port numbers and such, or even all the data required to generate the /urlforbot pages) using the comm channel that Node provides, but that still sounds like an overly complex setup.
Wouldn't it be possible to create a Bot class instead? You'd instantiate the class for each bot you want to run, and that instance loads its specific configuration and adds its routes to the Express server. All from the same process.