I would like to know what is the best approach in create several deploys from a big code base. The idea is to divide the big API into microservices (each one in it's own server/vm),
The first idea: I could simply create a folder with only the available routes for that microservice, but still using the "common" codebase...
I currently end up with this, and it's a running API in production (with staging environment in heroku with their pipeline):
and I was thinking that I could have something like:
can anyone point me to a good reference on ... where to start? how can I push multiple version of the same base code to a server?
for more detail on the used technologies, I'm using:
mocha and chai for tests
sequelize for mariaDb modeling and access
restify for server engine
When you divide the API into microservices, you have few options:
Make completely separate repos for all of them with some code duplication
Make completely separate repos but sharing common code as Node modules
Make one repo with multiple microservices, each as its own Node module
Make one repo with one big codebase and build multiple modules with needed parts from that
I'm sure you can do it in even more ways
Having a mismatch of the number of Node modules and code repos will cause some troubles but it may have some benefits in certain cases.
Having a 1-to-1 mapping of repos and modules will be easier to work with with some cases, like the ability to add private GitHub repos directy to dependencies in package.json.
If you want to factor out some common functionality then you can do it in several ways:
The npm supports organizations, scoped packages, private modules and private scoped packages with restricted access.
You can host a private npm registry
You can host a module on GitHub or GitLab or any other git server
For more info see:
Node.js: How to create paid node modules?
There are some nice frameworks that can help you with splitting your code base into microservices, like Seneca:
http://senecajs.org/
Or to a certain extent with Serverless if you're using AWS Lambda, Microsoft Azure, IBM OpenWhisk or Google Cloud Platform:
https://serverless.com/
Related
I'm currently developing a project in Node JS that uses microservices architecture, in which each service has it's own repository that contains both the code for the service itself (NodeJS express server), and also an SDK file that I publish for other services to use with methods that are available in this service and Typescript definitions.
So for instance I have a users-service that handles all of the user related actions, and a reports-service that handles all of the reports that users can CRUD.
users-service has a method called "deleteUser" that also goes to reports-service SDK in order to delete all of this user reports. On the other hand reports-service uses user-service SDK to "getUserById" for instance. So what happens is that user-service has reports-service-sdk as one of it's dependencies, and reports-service has users-service-sdk as one of its dependencies. Because the SDK is inside the same npm module with the service, I get users-service-sdk as one of the dependencies of users-service.
I thought of separating the SDK with a different package.json file, but I wanted to know if it's the right way to go or am I doing something really wrong in my architecture :)
Thanks.
This sounds like Circular Dependency which as you stated in the title is tough to deal with. Microservices are great but this sort of architecture sounds like a lot of extra unnecessary work without any added benefit.
You should look into running your services/packages/repositories as Cloud functions (or Firebase functions). AWS also has their own solution for microservices architecture. The reason being is each service can communicate with other services by using internal authorized calls or authorized REST API calls --- or you can make them totally public.
The great thing about these Google Cloud Functions is each function is automatically an Express end-point that accepts GET, POST, DELETE, PUT. Or if you use the internal call for Firebase, each function automatically contains relevant context from the frontend (such as the user's authentication details).
You also configure IAM permissions to only allow who and what service you want to be able to execute your cloud functions so that you have full control of permissions.
To answer your questions directly though, I would definitely avoid Package A having Package B as a dependency as Package B has Package A as a dependency. You absolutely can make that work but there's no upside and a lot of downside.
Here's an old thread which covers the topic.
I am working on a project that has a few different code-bases (mostly Meteor), but they all use some of the same API code (schemas, publications and methods).
It doesn't seem very intuitive to have one repository and use sub-trees/sub-modules in my case. Maybe I'm wrong and somebody could help clear it up.
An example structure of my project:
project-landing-page (meteor app for collecting leads and offering basic account management)
project-app (an angular/ionic/cordova/meteor app that is distributed via the app store)
project-worker (a set of cron-like scripts that are executed in the background to manage the data in the mongo instance)
They all share the same schemas, and the two meteor apps use the same methods and publications. It seems a bit cluttered to have one repo for all of this code. Making a branch for the app would also branch the code for the worker scripts. That just seems messy.
Would it be okay to have another repo called "project-apis", that provides the shared code an could be cloned into the other projects? What are the drawbacks? Other than having to run git pull when the "project-apis" repo is updated, I can't really see any.
Would any git-wizards be able to chime in?
Thanks!
I'm developing my first node.js app deploying to GAE.
It'll be organized as an API service and a front-end web app developed with Next.js
I'm looking at this architecture, and, although I have the app separated in two repositories I could have one merged repo to create two different microservices:
https://medium.com/this-dot-labs/node-js-microservices-on-google-app-engine-b1193497fb4b
For me, it seems overwork creating a new repo to merge them and deploy (doesn't it break one of the basic ideas of microservices to make isolated deploys?)
I have to discourage this because we need SEO in some of the parts, and We should use Next.js (or similar):
https://cloud.google.com/storage/docs/hosting-static-website
Another idea I've been working on is... create different GAE projects for front and API to deploy independently. For me, it seems like the best option, but I would like to know your opinion as GAE experts.
Which one should I use?
Thanks!
GAE doesn't care how is the code to be deployed into the services mapped to one or more VCS repositories (or no repositories at all). That's entirely up to you.
With a single repository you may encounter difficulties deploying from CI/CD pipelines - for example unnecessary deployments to one service when only the other one is changed.
Many examples out there focus on applications rather than services, but those are nothing more than the default services of those applications. Personally I like keeping the code for different services in separate directories, see the image captured in Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure? (it's no longer present on the updated documentation page). This also allows for easy mapping to multiple, separate VCS repositories
As for multiple projects vs multiple services, this might be of help: Advantages of implementing CI/CD environments at GAE project/app level vs service/module level?
The static website link you mentioned isn't part of GAE, it's part of GCS - a different GCP product. It's fine to use by itself - for a static website, but it might be difficult/impossible to:
communicate between a service running on it and one running on GAE - if you need that
make the 2 services appear as one (for example serve under the same custom domain name)
How can I share components across multiple react projects without having to publish them on a public package manager like NPM?
Option 1: You can use npm and use private packages so they're not external facing. There are also artifactories and scoped packages that usually represent company-wide projects that can be public or private. See https://docs.npmjs.com/private-modules/intro and https://docs.npmjs.com/misc/scope.
Option 2: Essentially, you can develop projects with a flattened structure. You can then import various projects and/or components into other projects or folders. This is entirely dependent on your codebase and configuration. With this model though, a lot of times publishing to npm comes fairly naturally since each folder may be its own project with its own package.json.
Updated:
Option 3: Bit focuses on the composability of components from everything from the little things like a button to the actual view and app itself—each target is its own package. Overall, it's an opinionated, yet customizable framework that can enable quicker development, managed dependencies, and organized code.
Option 4: RushJS is a monorepo manager built by Microsoft that allows for flexibility of different kinds of apps and services utilizing pnpm underneath (as opposed to yarn and npm), which alleviates problems that stem from dependency issues.
Check out Bit:
Bit is an open-source cli tool for collaborating on isolated components across projects and repositories. Use Bit to distribute discrete components from a design library or a project into a standalone reusable package and utilize it across applications.
You could also upload them to a private git repo such a Github and then pull them in from there.
Ryanve has a nice example over here: https://stackoverflow.com/a/28729646/1592783
You could create a repo of shared components and then have your Node.js start script call a shell script to do a git pull from that repo and the move the shared components from that directory to your project's directory. That way, every time you call run 'npm start' you will have the latest version of the shared components loaded into your project
I've got a VS2013 solution with a mix of NodeJS (using TypeScript) and C# class library projects (they're bound together by EdgeJS). Where the NodeJS projects are concerned, one can be considered a library (for a RabbitMQ bus implementation), two are applications which are meant to be hosted as part of a fourth project with both using the bus.
So, one project (host) which will depend on three projects (bus, app1 and app2) (it starts the bus, and passes it to app1 and app2).
Of course, I could just lump all these projects together and be done with it - but that's a horrible idea.
How do I package these projects up for proper reuse and referencing (like assemblies in traditional .NET)?
Is that best done with NPM? If so, does VS provide anything in this area? If not, then what?
Note that, aside from the Bus project, I'm not looking to release these publicly - I'm not sure if that changes anything.
In general, if something can be bundled together as an independent library, then it's best to consider this a Node package and thus, refactor that logic out to it's own project. It sounds like you've already done this to some extent, separating out your bus, app1, and app2 projects. I would recommend they each have their own Git repositories if they are separate packages.
Here's some documentation to get you started with Node packages:
https://www.npmjs.org/doc/misc/npm-developers.html
The host project, if it's not something you would package but instead deploy, probably does not need to be bundled as a Node package. I would instead just consider this something that would be pulled down from Git and started on some server machine.
With all that said, your last line is important:
I'm not looking to release these publicly
GitHub does have private repositories, but as of now npmjs.org does not have private repositories. There are options to create your own private repository (Sinopia and Kappa offer different ways of accomplishing this), but if you don't want this code available for everyone do not deploy it do npmjs.org. You can still package it up in the way I've outlined it above, just not deploy it as of yet.