I'm into a very big project where we have already built some 50 to 70 Microservices in Nodejs. All these services imports some 5 to 10 core common modules. At this stage if there is any single line of change in common core module, we have to update, build and deploy all the artifacts. Is there any better way to handle this?
Thanks.
I agree with #skjagini core modules should be stable, they should truely be core. In general I advocate for sharing as little as possible between your microservices, they should be independently developed and deployable. They should not require deployment synchronisation where you need to co-ordinate deployments of all your microservices least you break something. If that is the issue you are finding you have yourself a distributed monolith not a microservice architecture.
I cant see any easy resolution to the issue you post. If common code does change then naturally any deployable unit that uses that common code needs to be rebuilt and redeployed. The only exception to this would be if the change is not required by a particular deployable, and in this case it probably means your modules are doing too much and don't have a clear purpose or are too large.
Related
I have Core project where I need to do some cryptographic operations, e.g. verification of SHA256. What can I do if it's Core project, so it shouldn't depend on anything? I have to write my own cryptographic functions that are resistant to e.g. side-channel attack? This causes security problems.
So what to do? Can my Core project depend on a nuget package if I use Clean Architecture?
The guideline regarding dependencies is to keep the core project as simple as possible so that most of its logic is about solving the business problem.
By keeping it simple, it's much easier to express which part of the business domain the classes solve. It's also easy to write focused tests that prove that the code can solve the correct part of the business problem.
To me, preventing attacks is not a part of that. It's something that should be done on inbound API calls before the domain is called. I would put that logic in application services. Those services can, of course, live in the Core project but not in any of the bounded contexts.
In Clean Architecture we try to keep the domain and application logic as independent from external libraries and frameworks as possible so that we do not depend on their future development.
Nevertheless the application logic will have to interact with external libraries, services and other IO which is achieved via "dependency inversion": the application logic defines an interface which is implemented by the outer layers (infrastructure).
This was the application logic remains "clean" and can focus on decision making while you can still reuse external libraries and services.
A more detailed discussion of this topic you can find here: http://www.plainionist.net/Implementing-Clean-Architecture-Frameworks/
I am working on a monolith system. All of it's code is in one repository (Web API and background workers). System is written in Nodejs and MongoDB (Mongoose) is used as a data store. My goal is to set a new path how project should evolve. At first I was wondering if I could move towards microservices based architecture.
Monolith architecture creates some problems:
If my background workers needs to scale. I have to deploy all the project to the server despite only using a small fraction of it.
All system must be redeployed when code changes. What if payment processor calls webhook while system is being redeployed?
Using microsevices advantages are quite obvious:
Smaller code base for individual microservice. Easier to reason about it.
Ability to select programming tools best for particular use case.
Easier to scale.
Looking at the current code I noticed that Mongoose ODM (Object Document Mapper) models are used across all the project to create, query and update models in database. As a principle of a good programming all such interactions with database should be abstracted. Business logic should not leak into other system layers. I could do that by introducing REPOSITORY pattern (Domain Driven Design). While code is still being shared across web api and it's background workers it is not a hard task to do.
If i decide to extract repositories into standalone microservices than all bunch of problems arise:
Some sort of query language must be introduced to accommodate complex search queries.
Interface must provide a way to iterate over search results (cursor based navigation) without returning all database documents over network.
Since project is in it's early stage and I am the only developer, going to microservices based architecture seems like an overkill. Maybe there are other approaches I should consider?
Extracting business logic and interaction with database into separate repository and sharing among services to avoid complex communication protocols between services?
Based on my experience with working in Microservices for last few years, it seems like an overkill in current scenario but pays off in long-term.
Based on the information stated above, my thoughts are:
Code Structure - Microservices Architecture (MSA) applying in above context means not separating DAO, Business Logic etc. rather is more on the designing system as per business functions. For example, if it is an eCommerce application, then you can shipping, cart, search as separate services, which can further be divided into smaller services. Read it more about domain-driven design here.
Deployment Unit - Keeping microservices apps as an independent deployment unit is a key principle. Hence, keep a vertical slice of the application and package them as Docker Image with Application Code, App Server (if any), Database and OS (Linux etc.)
Communication - With MSA, communication between services become a key and hence general practice is to remain with the message-oriented approach for communication (read about the reactive system and reactive programming for more insight).
PaaS Solution - There are multiple PaaS solutions available, which you can apply so that you don't need to worry about all the other aspects like container management, container orchestration, auto-scaling, configuration management, log management and monitoring etc. See following PaaS solutions:
https://www.nanoscale.io/ by TIBCO
https://fabric8.io/ - by RedHat
https://openshift.io - by RedHat
Cloud Vendor Platforms - AWS, Azure & Google Cloud all of them have specific support for Microservices App from the deployment perspective, which we can use as an alternative solution if you don't want to deploy PaaS solution in your organization.
Hope these pointers will have in understanding the overall landscape so that you can structure your architecture for future need.
I am working on a monolith system... My goal is to set a new path how project should evolve. At first I was wondering if I could move towards microservices based architecture.
In what ways do you need to evolve the project? Will it be mostly bugfixes, adding features, improving performance and/or scalability? Do you anticipate other developers collaborating in the future? Are you currently having maintenance issues? The answers to these questions (and many more) should be considered in guiding your choices.
You seem to be doing your homework around the pros and cons of a microservice architecture, so if you haven't asked yourself why you're even doing this in the first place, now would be good time to do so.
Maybe there are other approaches I should consider?
There's always the good old don't-break-what's-going ;)
In theory I understand how Microservices work and why they can be helpful in various cases but I still don´t get how it works in practice.
Let´s say there´s an online shop based on a CMS as a monolith application.
And there´s now the need to run the online shop in a MIcroservices architecture.
How would this Microservices architecture differ technically from the current, monolith, architecture?
For example, I pick out the productsearch.php. If i want to scale this function, normally I had to set up a new server and copy the whole CMS ressources folder to it for loadbalancing.
And with Microservices, productsearch.php would be a single Microservice I guess, and I would have to just copy this php file to scale without the need to copy other ressources?
I have tried to explain it using this diagram of a fictitious CMS. With micro services architecture, we can independently scale each micro service. Each micro service may be developed by a different team, they may be even developed using different technology. But we great flexibility comes great maintenance overhead, I believe it is worth it as most of it can be automated.
Put simply, each module in a molithic application is a potential candidate for microservice. Howerver, microservices can be more granular than a traditional module.
This provides a good job at explaining how to decompose your monolithic application. http://microservices.io/patterns/decomposition/decompose-by-business-capability.html
Technically and conceptually, a microservice is independent of other services (where in a monolith you'd have modules with inter-dependencies).
Technically, a microservice built on modern microservices platforms (such as Node.JS, Spring Boot or .NetCore) will be more easily able to take advantages of containerization systems (such as Docker), perhaps supported by service registry and configuration management technologies (such as Kubernetes, ZooKeeper, Eureka and so on).
The advantage of containerization is that it'll be easier to scale-out (add more containers). Going further, the whole microservice / containerization concepts, and related technologies, also help enable things like CI/CD.
I learned about microservices from here
Now, I want to use microservices architecture in my next sails.js project.
One way I could think of is:
Breaking my one sails.js application into multiple small sails.js sub-projects/repositories.
Having one controller-model in one sub-project. For example, If we consider simple eCommerce app with entities say User, Products, Orders, etc. then there will be separate sails.js repositories for each of them with respective sails.js model-controller. Then this single sub-repository will from my one microservice.
Each sub-repository then will obviously have its own configs.
These microservices will them communicate with each other using some HTTP node module.
Then writing my own API gateway for routing in node.js, which will be responsible for invoking methods/web-services from these sub-repositories depending on the request from clients.
Is this the best way OR is there alternative way to design your project using microservices architecture?
What will be the best way to implement inter-service communication, API gateway with sail.js? If one microservice designed with above mentioned approach get bigger, and if I have to split it up in 2, how sails.js model should be changed?
The most important aspect of designing microservices is the separation of concerns which means each microservice will have a defined boundary under which they need to work.
Each microservice is designed to do a defined work so, first you need to find the independent functionalities in you project and try to create a microservice for it.
The most important thing to note is you should first start with a monolithic architecture and if you identify that some functionalities needs to be separated then you can create a microservice out of it.
As far as sails is considered then it is a good candidate for MVC and if the project is monolithic but if the number of microservices is large then it is not a good choice because running large number of microservices with sails.js will consume more of your system RAM.Sails.js internally uses so many libraries which you will not need. You can make a simple microservice with just node.js core modules and they will consume less memory too.
Also when each microservices handles small functionalities so the amount of
code will be less and there is no need for mvc arcitecture. you can use less number libraries to create it.
Conclusion
If number of services is less and you don't worry about system RAM then go for multiple sails application.
If number of services going to be more then try to make your services without using sails
I agree with the previous answer and I would add that Sails is a great candidate for clustering and in an environment where you may wish to scale horizontally to improve availability. I do not believe sails is the right candidate for the micro service architecture, however it is most likely the focus for an application which requires the usage of multiple services in its own right.
I use a message service to glue together multiple applications, with sails consuming these messages in order to update a webpage. I probably see those applications as offering smaller services, with defined boundaries and my sails application as the front end, with the controller gluing what is necessary to satisfy the requirements of the end user.
Assume we have a couple of libs. What is the difference between Core and Common library? How should they be recognized and do we organize the responsibilites of both?
+Common
-Class1
+Core
-Class2
+Lib1 has : Common
+Lib2 has : Core, Common
Should Common be truely common (i.e. all libs use it)? Or is Common only for those who need it?
What is good practice when refactoring / creating a project?
I don't really understand the difference between Core and Common.
I think this depends a lot on your particular application. In a single centralized app, I do think there might be a little overlap between the Core and Common folders. But the most important thing is that it makes sense for your app. Don't feel that you need to have those folders just because you've seen it in other apps...
For me, having a Core and a Common folders makes a lot of sense in some scenarios - e.g. a web app with an API and a client. You may have your Core folder in the API side, where the core execution (the business logic) takes place, and then have a Common folder with some things you need in both the API and the client sides - e.g., Http requests validation or a Json converter.
Anyway, it may make sense to have a Core and a Common folder in other kinds of apps.
For example, the Core folder would contain those classes that are central for your app - the vast majority of business model classes would be there.
In the Common folder, on the other hand, you can have some other classes that are shared, but not central - e.g., a Logger or a MessageSender could be there...
As for your little draft of code structure, I think that your Core package is the one to be revised - why Lib1 doesn't use Core? If something is core, generally it's because everything else needs it in order to run. If you do not have code that is conceptually central, maybe you can remove your Core package and keep only Common?
As for your other question - I do not think the Common stuff must be shared by all other packages, but just with 2 or more packages sharing something, that can be considered common.