I spent the last couple of days figuring out what development stack to use for the interactive student platform I'm planning to build.
I figured out that the MEAN stack may suit the job very well. However, I face a dilemma whether to use Node.js as backend technology for the application:
Reasons to consider Node
The backend will mainly consist of realtime components. E.g. collaboration tools, notifications, etc.
These components will handle this data concurrently
It will scale better than a conventional server-side programming language such as PHP
Exposing the data with REST for e.g. a mobile applications will be a breeze
Having one data format (JSON) in the front- and backend will speed up development.
Doubts
Some components require computation. Although not that complex, it may slow down the application.
Although the application is mostly a single page application, the application will (in a later stage have some features that Node seems not typically suited for. E.g. a payment workflow.
I already made the switch from a previous approach, so this time I want to be sure to choose the right approach. Will Node.js be the right choice for this application, or will a, for example, PHP backend with Laravel suit the job better as the application matures?
I think there's a whole range of possibilities you're not considering, for example it's a perfectly valid approach to use Node for some of the back-end (e.g. connections to third parties, managing the UI, handling concurrent users) while delegating some of the back-end to other components that are more suited (e.g. components that require heavy computation).
That said, I don't see anything you describe in your 'doubts' as being particularly non-nodish. The computation stuff you say will be lightweight, but my recommendation there is to treat it like any other async task, then if you decide later that it is a problem (e.g. slows down the app) it's pretty trivial to extract that out into either a separate Node process (therefore not blocking your main app's event loop) or use a component built in your language of choice (Java, .NET, C, Perl, whatever) as described above.
I don't understand why you suggest a workflow isn't something Node is suited for. I've seen and built a number of them in Node and other frameworks, it's no less suited for it than any other framework, and better than some.
Related
To speed up development for my next Node-API I was looking for a suitable Framework. In the past I was building my APIs with express only.
One Design pattern I always found useful is to completely seperate the business logic from route-handling in services. Those services only accept the required information (like a user id or data) and return a promise resolving the result of the operation.
This way it is easy to reuse these services in other routes, to combine them, test them, or call them based on schedules or other events - totally independent from endpoint-calls. Routing and Middleware take care of access-controll, error-handling and respondig.
Looking at the documentations of those frameworks (sailsjs, keystonejs, ...) I mostly see the business-logic tightly coupled to individual routes, directly accepting request objects and handling the responses. Only as an afterthought it seems there is sometimes offered a way to extract "often used code" into helper functions.
Am I missing something? How come this pattern seems to be the standard of API design? Is this a best practice for a reason?
It might have to do with Node.js services being smaller in size. If you're coming from an enterprise background, you're well aware mixing business-logic with controller code doesn't fly in the long run. Perhaps small projects can get away with defying that, but once the size increases, you can't avoid the laws of physics. It's best to separate concerns and keep the codebase maintainable.
I'd also add that below services, it's good to have a separate layer that handles talking to outside process boundaries. That way, you can test business logic in isolation by providing appropriate test doubles for integrations. Here's a longer explanation of how it would work in a Node project: Organize Node.js API project using 3-layer architecture.
i want to create a website based on node js and mysql , but i've read that there is a framework called express for node js , and i'm wondering if i must to use such kind of a framework to create a decent website or it is possible without it and just work with pure node js.
No framework is required. You can write a full-blown web server using only the http module or if you really want to write everything yourself, you can even do it with only the net module.
It's really about what is the most effective use of your time and skill as a developer. Except for academic or pure learning experience reasons, if you're just trying to accomplish a task as efficiently as possible and free, pre-existing, pre-tested code exists that makes your job easier, then that's a better way to go.
For example, if I need to do a file upload from a browser to my back-end and the data is coming in as the multipart/formdata content-type from the browser, I have zero interest in reading and learning the multipart/formdata RFC then writing my own code to parse the multipart/formdata content-type. Pre-existing, already tested code exists to do that for me and I'm adding no value to the goals of my project by re-implementing and then testing it all myself. Therefore, I'd like to use a pre-built module that does all that for me. I can just configure the right library on the right route and out plops my uploaded file in only the amount of time it takes to understand the interface to the 3rd party module and how to use it properly.
This is where Express comes in. Not only does it offer a useful set of features and architecture for configuring routes, installing middleware, sending responses, using template engines, handling errors, etc... but there are also thousands of third party modules that are built to hook into Express and it is easiest to use them if you're using Express as your core framework. Some of these modules could be used outside of Express, some cannot - it really depends upon how they're designed and what Express interfaces they do or don't use.
Also, Express is fairly "un-opinionated" and fairly "lightweight" which means it doesn't force you into a particular methodology. It just offers you easier ways to do things you were already going to have to write code for yourself.
Look at it this way. When you get node.js, there are thousands of APIs that offer lots of already tested things such as a TCP library, a file I/O library, etc... Those are frameworks (in a sense) too. You don't have to use them either. You could rewrite whatever functionality you need from scratch. But, you wouldn't even think about doing that because tested code already exists that solves your problem. So, you happily build on top of things that are already done.
One of the BIG advantages of coding with node.js is getting access to the tens of thousands of pre-built modules on NPM that already solve problems that many people have. Coding in node.js with a mindset that you will never use any outside modules from NPM is throwing away one of the biggest advantages of coding with node.js.
could you tell me what are the Routes used for in frameworks?
A route is a URL that you wish for your web server to respond to. So, if you want http://myserver.com/categories to be URL that your server responds to, then you create a route for /categories so that you can write code for what should happen when that URL is requested. A framework like Express allows you to create that route very simply with just a single statement such as:
app.get('/categories', function(req, res) {
// put code here to handle that request
});
This is just the tip of the iceberg for what Express supports. It allows you to use wildcards in route definitions, identify parameters in urls, create middleware that does prep work on lots of routes (such as check if the user is logged in), etc...
You don't have to use a framework but it is recommended to use one of them since frameworks like Express make your life easier in many ways. Check this: What is Express.js?
Yes you CAN write a Node.js-based backend without any back end implementation framework such as Express. And if you are using Node.js for the first time without any previous experience of asynchronous coding, I'd advise against using Express, KOA or other Node implementation frameworks for your simple learner apps (e.g. those needing things like register/login form processing, logout button, user preference updates to database, etc) because:
(1) Node.js is a core skill for JavaScript back ends.
Stupid analogies between server tasking and restaurant waiters are no use to a real web engineer. You must first know what exactly Node can/cannot do in the server CPU that makes it different to most other back end technologies. Then you have to see how the Node process actually does this. Using Express/KOA/Hapi/etc you are sometimes effectively removing the mental challenges that come with a Node back end. Any time-saving is achieved at the expense of gaining a proper working understanding of what Node is and how it really operates.
(2) Learning Node.js and its asynchronous coding is hard enough without the added complication of coding with an unknown framework like Express/KOA that assumes users' familiarity with JavaScript constructs like callback functions and Promises. It's always better to learn something in isolation so you get the essence of its individual effects, rather than the overall effects if used with other packages/frameworks. So many of these Node.js Express tutorials are the software equivalent of learning to make a cake by watching Momma do it. We can copy it but we don't know how or why it's working. Professional coders can't just be good copycats.
(3) Available learning tutorials using Express often drag in other technologies like MongoDB, Mongoose, Mustache, Handlebars, etc that make learning Node.js even more awkward still.
(4) A share of basic web apps can be written more efficiently with Node.js, custom JS and existing JS modules imported off the npm repository rather than with Express.
(5) Once asynchronous coding and the JavaScript constructs available to assist with it are understood clearly, pure Node.js apps for basic tasks aren't that hard.
(5) After you do get your head around Node.js and can get basic web app functionalities working using server-side JavaScript constructs, you can then judiciously start to explore Express/Hapi/KOA/etc and see what an implementation framework can do for your workflow when doing larger projects needing numerous functionalities. At this point you know what Express code should be doing and why it is done the way it is.
Node.js has become the back-end technology of choice for most small to medium scale web applications over the last 10 years. It is also the major reason why the JavaScript language has evolved from a mere front-end scripting tool with a limited set of Java-aping constructs to the innovative and comprehensive language that it is today. It is also the most popular language in use today. Investing time in understanding the Node server framework, and the latest JavaScript constructs used in Node, is time well spent. Implementation frameworks such as Express, KOA, Hapi, Sails, etc have great benefit when writing more elaborate back ends on the Node.js platform. But all these implementation frameworks are predicated on the behaviour patterns of Node.js. So unless Node itself is understood first, the full utility of Express/KOA/Sails/etc will never be enjoyed.
Try here for the pure Node.js.
I am working on a monolith system. All of it's code is in one repository (Web API and background workers). System is written in Nodejs and MongoDB (Mongoose) is used as a data store. My goal is to set a new path how project should evolve. At first I was wondering if I could move towards microservices based architecture.
Monolith architecture creates some problems:
If my background workers needs to scale. I have to deploy all the project to the server despite only using a small fraction of it.
All system must be redeployed when code changes. What if payment processor calls webhook while system is being redeployed?
Using microsevices advantages are quite obvious:
Smaller code base for individual microservice. Easier to reason about it.
Ability to select programming tools best for particular use case.
Easier to scale.
Looking at the current code I noticed that Mongoose ODM (Object Document Mapper) models are used across all the project to create, query and update models in database. As a principle of a good programming all such interactions with database should be abstracted. Business logic should not leak into other system layers. I could do that by introducing REPOSITORY pattern (Domain Driven Design). While code is still being shared across web api and it's background workers it is not a hard task to do.
If i decide to extract repositories into standalone microservices than all bunch of problems arise:
Some sort of query language must be introduced to accommodate complex search queries.
Interface must provide a way to iterate over search results (cursor based navigation) without returning all database documents over network.
Since project is in it's early stage and I am the only developer, going to microservices based architecture seems like an overkill. Maybe there are other approaches I should consider?
Extracting business logic and interaction with database into separate repository and sharing among services to avoid complex communication protocols between services?
Based on my experience with working in Microservices for last few years, it seems like an overkill in current scenario but pays off in long-term.
Based on the information stated above, my thoughts are:
Code Structure - Microservices Architecture (MSA) applying in above context means not separating DAO, Business Logic etc. rather is more on the designing system as per business functions. For example, if it is an eCommerce application, then you can shipping, cart, search as separate services, which can further be divided into smaller services. Read it more about domain-driven design here.
Deployment Unit - Keeping microservices apps as an independent deployment unit is a key principle. Hence, keep a vertical slice of the application and package them as Docker Image with Application Code, App Server (if any), Database and OS (Linux etc.)
Communication - With MSA, communication between services become a key and hence general practice is to remain with the message-oriented approach for communication (read about the reactive system and reactive programming for more insight).
PaaS Solution - There are multiple PaaS solutions available, which you can apply so that you don't need to worry about all the other aspects like container management, container orchestration, auto-scaling, configuration management, log management and monitoring etc. See following PaaS solutions:
https://www.nanoscale.io/ by TIBCO
https://fabric8.io/ - by RedHat
https://openshift.io - by RedHat
Cloud Vendor Platforms - AWS, Azure & Google Cloud all of them have specific support for Microservices App from the deployment perspective, which we can use as an alternative solution if you don't want to deploy PaaS solution in your organization.
Hope these pointers will have in understanding the overall landscape so that you can structure your architecture for future need.
I am working on a monolith system... My goal is to set a new path how project should evolve. At first I was wondering if I could move towards microservices based architecture.
In what ways do you need to evolve the project? Will it be mostly bugfixes, adding features, improving performance and/or scalability? Do you anticipate other developers collaborating in the future? Are you currently having maintenance issues? The answers to these questions (and many more) should be considered in guiding your choices.
You seem to be doing your homework around the pros and cons of a microservice architecture, so if you haven't asked yourself why you're even doing this in the first place, now would be good time to do so.
Maybe there are other approaches I should consider?
There's always the good old don't-break-what's-going ;)
I'm wondering if some of you veterans could provide some guidance to some of us noobs.
Is there any advantage to writing vanilla Node over using Express? More specifically, any particular scenario where using Express will not be an option? And lastly for the tasks that you can use Express for, how deep should you go within the layers of abstraction to find out what's going on?
I mean there are plenty of people who can create robust and powerful web applications and they have no clue what the hell Express is actually doing for them.
Much appreciated
If I were you, I'd use express.
To be honest, Express isn't much of a web framework. It's really barebones, and barely adds any functionality on top of Node core.
With this said, however, there are some benefits:
For better or worse, express has become the 'defacto' default web framework for Node developers. There's a lot of information about it.
Express provides some core things that are useful: a routing layer (to map functions to URLs), an 'application' object that you can bind variables to for settings, etc. -- and a simple middleware model that lets you add functionality to your app easily.
Because express is so close to 'barebones' node, you can still write raw node code to work with it. It isn't at all complicated like other 'larger' frameworks: django, rails, etc.
There are a TON of third-party express middlewares you can use which add all sorts of functionality to your site. This makes building your site easier.
Finally -- the biggest reason to use express is that it does almost nothing. It isn't significantly different from using raw node except that it provides some simple abstractions over lower level stuff.
Because express is so simple, it means you don't need to learn much to use it, and can write your app in whatever way you want (it doesn't enforce any sort of patterns).
I'd like to add a few things here that might help you out. One thing that I have come to realize with software engineering is that there is never a "catch all" answer to a lot of these types of questions. Since each application is different, it is a good idea to look at the challenges and figure out what is the best tool for the job.
If you look at some of the performance tests done on the frameworks you notice that vanilla node is ultra performant. But most of the time your application isn't going to need to handle 8000 requests per second, and even if it does you can always scale your server horizontally nowadays. So you trade off a little bit of speed to gain a few benefits, such as:
Much easier to write.
The code base, in my opinion, is much easier to maintain.
A lot of the little gotchas are taken care of by the framework so you don't have to be a Node.js god.
Middleware.
Now, that doesn't mean that every single application should use express. A lot do because it is about as barebones as it gets. For instance, Walmart created Hapi because they claim that it is easier to maintain their code base with a 'configuration first' approach. So maybe if you are going to have this monster backend and a monster team, Hapi might be a good choice. If implementing something with real-time you might want to use something like socketcluster.io to help you scale websockets. It is all about picking the right tool for the right job.
That being said, there are advantages to building a server with vanilla Node, especially when learning to develop node apps. Since frameworks abstract a lot of the lower level stuff it takes away the opportunity to pick up some cool node tricks here or there. Or a lot of times if a framework is insufficient or an npm package is doing something weird it is good to have that knowledge of vanilla node under your belt to really understand what is going on. Having the skill of knowing how Node.js works is such a benefit when working with any framework.
We are planning to start a fairly complex web-portal which is expected to attract good local traffic and I've been told by my boss to consider/analyse node.js for the serve side.
I think scalability and multi-core support can be handled with an Nginx or Cherokee in front.
1) Is this node.js ready for some serious/big business?
2) Does this 'event/asynchronous' paradigm on server side has the potential to support the heavy traffic and data operation ? considering the fact that 'everything' is being processed in a single thread and all the live connections would be lost if it got crashed (though its easy to restart).
3) What are the advantages of event based programming compared to thread based style ? or vice-versa.
(I know of higher cost associated with thread switching but hardware can be squeezed with event model.)
Following are interesting but contradicting (to some extent) papers:-
1) http://www.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html
2) http://pdos.csail.mit.edu/~rtm/papers/dabek:event.pdf
Node.js is developing extremely rapidly, and most of its functionality is sturdy and ready for business. However, there are a lot of places where its lacking, like database drivers, jquery and DOM, multiple http headers, etc. There are plenty of modules coming up tackling every aspect, but for a production environment you'll have to be careful to pick ones that are stable.
Its actually much MUCH more efficient using a single thread than a thousand (or even fifty) from an operating system perspective, and benchmarks I've read (sorry, don't have them on hand -- will try to find them and link them later) show that it's able to support heavy traffic -- not sure about file-system access though.
Event based programming is:
Cleaner-looking code than threaded code (in JavaScript, that is)
The JavaScript engine is extremely efficient with processing events and handling callbacks, and its easily one of the languages seeing the most runtime optimization right now.
Harder to fit when you are thinking in terms of control flow. With events, you can never be sure of the flow. However, you can also come to think of it as more dynamic programming. You can treat each event being fired as independent.
It forces you to be more security-conscious when programming, for the above reason. In that sense, its better than linear systems, where sometimes you take sanitized input for granted.
As for the two papers, both are relatively old. The first benchmarks against this, which as you can see, has a more recent note about these studies:
http://www.eecs.harvard.edu/~mdw/proj/seda/
It also cites the second paper you linked about what they have done, but refuses to comment on its relevance to the comparison between event-based systems and thread-based ones :)
Try yourself to discover the truth
See What is Node.js? where we cover exactly that:
Node in production is definitely possible, but far from the "turn-key" deployment seemingly promised by the docs. With Node v0.6.x, "cluster" has been integrated into the platform, providing one of the essential building blocks, but my "production.js" script is still ~150 lines of logic to handle stuff like creating the log directory, recycling dead workers, etc. For a "serious" production service, you also need to be prepared to throttle incoming connections and do all the stuff that Apache does for PHP. To be fair, Rails has this exact problem. It is solved via two complementary mechanisms: 1) Putting Rails/Node behind a dedicated webserver (written in C and tested to hell and back) like Nginx (or Apache / Lighttd). The webserver can efficiently serve static content, access logging, rewrite URLs, terminate SSL, enforce access rules, and manage multiple sub-services. For requests that hit the actual node service, the webserver proxies the request through. 2) Using a framework like "Unicorn" that will manage the worker processes, recycle them periodically, etc. I've yet to find a Node serving framework that seems fully baked; it may exist, but I haven't found it yet and still use ~150 lines in my hand-rolled "production.js".