Using BreezeJS, RequireJS, AngularJS with NodeJS and MongoDB as backend, I'm building a fat client application, with great success so far, as BreezeJS takes away the work to keep my domain model persisted. But it's growing and it takes now over five seconds to load all the files if they are not cached on localhost, catastrophical if you are trying to do a quick demo using a remote server..
R optimizer Warning:
bower_components/breezejs/breeze.debug.js has more than one anonymous define.
May be a built file from another build system like, Ender. Skipping
normalization.
Trying to run the compiled production file throws:
Uncaught Error: Mismatched anonymous define() module: function (){ return definition(global); }
(breeze.debug.js L10)
Has anyone gotten BreezeJS+RequireJS into production?
Take a look at the Todo-Require sample in the breeze.samples.js GitHub repo.
The Todo-KO-Require sample shows you how to code with require but it doesn't show you how to package things for production. You will suffer if you're asking require to download every individual file on demand.
You need to optimize with bundling and minification ... a topic outside of the breeze purview and not something we are in a hurry to produce. Perhaps you'd like to take that bull by the horns and share with the rest of us.
Why worry?
[update, 2 July 2014]
Let's take a step back and rediscover the point of all this. What is require doing for you?
I've used it with KO as a vehicle for dependency injection. That's its role in Durandal.
Angular comes with its own DI which reduces the role of require in an Ng app to asynchronous file loader. That's usually "meh" for me, in part because one soon encounters the file-loading-flurry that you describe. That leads to bundling which is a headache and can as easily be done with other tooling.
I see the value in a large applications with dynamically loaded modules. But Ng is woeful in this regard quite apart from the async file loading. Something they'll address in v.2.
I'm happy to leave you to a contrary opinion. So let's consider what would happen if we can't fix this problem. What if breeze cannot be optimized with r?
My instinct is that it isn't really optimal to bundle breeze with anything else anyway!
The minimized breeze is rather large in itself. It is not evident to me that you would gain any performance advantage at all by bundling it with your application assets. Sure you want to keep the number of server requests down. But are two requests with 1/2 the payload slower than one big request? Do you know for your target environment?
I'm not the kind of pedant who insists that every script be delivered by require. It's trivial to load BreezeJS separately with a script tag and then make it available to other require-aware modules (I shall assume you know how to do this). What would be horrible about that?
While we look forward to your repro sample (see my comment below), I may have difficulty justifying priority attention to this issue. Convince me otherwise.
I managed to compile my projecte leaving out breeze. With a small adjustment to the breeze mongo dataservice file header. Using r optimizer config
paths: {
'breeze': 'empty:',
'breeze-dataservice-mongo': 'empty:'
}
Breeze mongodataservice can be included as soon as it conforms like lib/breeze-angular.
(function () {
"use strict";
requirejs.config({
paths: {
'breeze': 'bower_components/breezejs/breeze.debug',
'breeze-dataservice-mongo': 'lib/breeze.dataService.mongo'
}
});
require(['angular', 'jquery', 'core/logger', 'fastclick', 'core/index', 'domready!'], function (angular, $, logger, fastClick) {
logger.info('iaGastro client is booting');
fastClick.attach(document.body);
angular.bootstrap(document, ['iaApp']);
});
})();
Leaving out SaveQueueing completely, I think I can find a different solution for my concurrent save error..
#Ward:
RequireJS does static file loading, like my domain classes, also templates and json files. Now it also concatenates all my files and minifies them with one more parameter. It's probably the docs, which are not the best, because I feel I'm not the only one sometimes misunderstanding RequireJS..
Also it's error messages can be frustrating (circular dependencies..).
Related
We are integrating Amazon's node.js SDK into our project and while I do not think it matters due to require's cache and the fact that everything is compiled, I could not find a site that definitively states that multiple requires will not affect performance in run time.
Obviously it depends on what files you are requiring, the contents of those files, and whether or not they could block the event loop or have other code inside of them to slow performance.
I prefer to structure code based on functionality rather than just having a 10000+ line file that does not really relate to the task at hand. I just want to make sure I'm not shooting myself in the foot by break out functionality into separate modules and then requiring on an as needed basis.
Well, require() is a synchronous operation so it should ONLY be used during server initialization, never during an actual request. Therefore, the performance of require() should only affect your server startup time, not your request handling time.
Second, require() does have a cache behind it. It matches the fully resolved path of the module you are attempting to load. So, if you call require(somePath) and a module at that same path has previously been loaded, then the module handle is just immediately returned from the cache. No module is loaded from disk a second time. The module code is not executed a second time.
Obviously it depends on what files you are requiring, the contents of those files, and whether or not they could block the event loop or have other code inside of them to slow performance.
If you are requiring a module for the first time, it WILL block the event loop while loading that module because require() uses blocking, synchronous I/O when the module is not yet cached. That's why you should be doing this at server initialization time, not during a request handler.
I prefer to structure code based on functionality rather than just having a 10000+ line file that does not really relate to the task at hand. I just want to make sure I'm not shooting myself in the foot by break out functionality into separate modules and then requiring on an as needed basis.
Breaking code into logical modules is good for ease of maintenance, ease of testing and ease of reuse, so it's definitely a good thing.
I have seen people go too far where there are so many modules each with only a few lines of code in them that it backfires and makes the project unwieldly to work on, find things in, design test suites for, etc... So, there is a balance.
I'm writing a crawler module which is calling it-self recursively to download more and more links depending on a depth option parameter passed.
Besides that, I'm doing more tasks on the returned resources I've downloaded (enrich/change it depending on the configuration passed to the crawler). This process is going on recursively until it's done which might take a-lot of time (or not) depending on the configurations used.
I wish to optimize it to be as fast as possible and not to hinder on any Node.js application that will use it.I've set up an express server that one of its routes launch the crawler for a user defined (query string) host. After launching a few crawling sessions for different hosts, I've noticed that I can sometimes get real slow responses from other routes that only return simple text.The delay can be anywhere from a few milliseconds to something like 30 seconds, and it's seems to be happening at random times (well nothing is random but I can't pinpoint the cause).I've read an article of Jetbrains about CPU profiling using V8 profiler functionality that is integrated with Webstorm, but unfortunately it only shows on how to collect the information and how to view it, but it doesn't give me any hints on how to find such problems, so I'm pretty much stuck here.
Could anyone help me with this matter and guide me, any tips on what could hinder the express server that my crawler might do (A lot of recursive calls), or maybe how to find those hotspots I'm looking for and optimize them?
It's hard to say anything more specific on how to optimize code that is not shown, but I can give some advice that is relevant to the described situation.
One thing that comes to mind is that you may be running some blocking code. Never use deep recursion without using setTimeout or process.nextTick to break it up and give the event loop a chance to run once in a while.
What is the best way to use NodeJS's require function? By this, I'm referring to the placement of the require statement. Isf it better to load all dependencies from the beginning of the script, as you need them, or does it not make a notable difference whatsoever?
This article has a lot useful information regarding how require works, though I still can't come to a definitive conclusion as to which method would be most efficient.
Assuming you're using node.js for some sort of server environment, several things are generally true about that server environment:
You want fast response time to any given request.
The code that runs for processing requests should not use synchronous I/O operations because that seriously lessens the scalability of the server.
Server startup time is generally not something you need to optimize for (within reason) so if you're going to pay an initialization cost somewhere, it is usually better paid once at server startup time.
So, given that require() uses synchronous I/O when the module has not yet been cached, that means you really don't generally want to be doing require() operations inside a request handler. And, you want fast response times for your request handlers so you don't want require() calls inside your handler anyway.
All of these leads to a general rule of thumb that you load necessary modules at startup time into a module level variable that you can reuse from one request to the next and you don't load modules inside your request handlers.
In addition to all of this, if you put all your require() statements in a block near the top of your module, it makes your module a lot more self-documenting about what other modules it depends on and how it initializes those modules. If require() statements are sprinkled all over the code, then it makes it a lot harder for a developer to see what this module is using without a lot more study of the code.
It depends what performance characteristics you're looking for.
require() is not cheap; it has to read the JS file from disk, parse it, and execute any top-level code (and do all of that recursively for all files require()d by that file).
If you put all of your require()s on top, your code may take more time to start, but it won't suddenly slow down later. (note that moving the require() further down in the synchronous top-level code will make no difference beyond order of execution).
If you only require() other modules when first used asynchronously, your first request may be noticeably slower, as Node parses all of your dependencies. This also means that any errors from dependencies won't be caught until later. (note that all require() calls are cached, so the second request won't do any more work)
The other disadvantage to scattering require() calls throughout your code is that it makes it less readable; it's very nice to easily see exactly what each file depends on up top.
I'm using RequireJS for my web application. I'm using EmberJS for the application framework. I've come to a point where, I think, I should start bundling my application into a single js file. That is where I get a little confused:
If I finally bundle everything into one file for deployment, then my whole application loads in one shot, instead of on demand. Isn't bundling contradictory to AMD in general and RequireJS in particular?
What further confuses me, is what I found on the RequireJS website:
Once you are finished doing development and want to deploy your code for your end users, you can use the optimizer to combine the JavaScript files together and minify it. In the example above, it can combine main.js and helper/util.js into one file and minify the result.
I found this similar thread but it doesn't answer my question.
If I finally bundle everything into one file for deployment, then my whole application loads in one shot, instead of on demand. Isn't bundling contradictory to AMD in general and RequireJS in particular?
It is not contradictory. Loading modules on demand is only one benefit of RequireJS. A greater benefit in my book is that modularization helps to use a divide-and-conquer approach. We can look at it in this way: even though all the functions and classes we put in a single file do not benefit from loading on demand, we still write multiple functions and multiple classes because it helps break down the problem in a structured way.
However, the multiplicity of modules we create in development do not necessarily make sense when running the application in a browser. The greatest cost of on-demand loading is sending multiple HTTP requests over the wire. Let's say your application has 10 modules and you send 10 requests to load it because you load these modules individually. Your total cost is going to be the cost you have to pay to load the bytes from the 10 files (let's call it Pc for payload cost), plus an overhead cost for each HTTP request (let's call it Oc, for overhead cost). The overhead has to do with the data and computations that have to occur to initiate and close these requests. They are not insignificant. So you are paying Pc + 10*Oc. If you send everything in one chunk you pay Pc + 1*Oc. You've saved 9*Oc. In fact the savings are probably greater because (since compression is often used at both ends to reduce the size of the data transmitted) compression is going to provide greater benefits if the entire data is compressed together than if it is compressed as 10 chunks. (Note: the above analysis omits details that are not useful to cover.)
Someone might object: "But you are comparing loading all the modules in separately versus loading all the modules in one chunk. If we load on demand then we won't load all the modules." As a matter of fact, most applications have a core of modules that will always be loaded, no matter what. These are the modules without which the application won't work at all. For some small applications this means all modules, so it make sense to bundle all of them together. For bigger applications, this means that a core set of modules will be used every single time the application runs, but a small set will be used only on occasion. In the latter case, the optimization should create multiple bundles. I have an application like this. It is an editor with modes for various editing needs. A good 90% of the modules belong to the core. They are going to be loaded and used anyway so it makes sense to bundle them. The code for the modes themselves is not always going to be used but all the files for a given mode are going to be needed if the mode is loaded at all so each mode should be its own bundle. So in this case a model with one core bundle and a series of mode bundles makes sense to a) optimize the deployed application but b) keep some of the benefits of loading on demand. That's the beauty of RequireJS: it does not require to do one or the other exclusively.
While developing you want to have single-focused, small files. This causes their number to increase. When running in production, many HTTP requests really harm performance. Then again you do not want to load the entire application upfront - this is also not optimal.
To address this, I have created a small project in GitHub, require-lazy, you can call it plugin to the builder - r.js. It can lazy load parts of your application with a simple syntax and then create separately donloadable bundles during the build process; so if your application consists of 2 views that need to be independently loaded, require-lazy will (ideally) build 3 js files: (1) the bootstrap code and common libraries, (2) view 1 with all its private scripts and (3) view 2 with all its private scripts.
Lazy loading is simply defined as:
define(["lazy!view1"], function(view1) { .... });
And view1 must be accessed with a promise:
view1.get().done(function(realView1) {
...
});
The project is available through npm, the build process through grunt and there is a bower component.
Comments are more than welcome.
What is currently the best practice for loading models (and goes for all required files I guess)?
I'm thinking these two ways to achieve the solution (nonsense code to illustrate follows):
var Post = require('../models/post');
function findById(id) {
return new Post(id);
}
function party() {
return Post.getParty();
}
vs
function findById(id) {
return new require('../models/post')(id);
}
function party() {
return require('../models/post').getParty();
}
Is one of these snippets preferred? Are there considerable memory and time tradeoffs? Or is it just a premature optimization?
It's a premature optimization (calls to require() are cached and idempotent), but I'd personally call your first style better (loading dependencies during initialization rather than subsequent processing) since it's easier to get your head around what you're doing. Loading everything at the start will slightly slow down your startup (which is hardly ever an issue) in return for making most requests run slightly faster (which you shouldn't worry about unless you've identified a bottleneck and done some hardcore profiling).
You should definititely use the version with the single call to requireat the beginning. Although it does not make any difference regarding to how often the modules are loaded (they are only loaded once however you do it), there are performance issues in the second way of doing it.
The problem is that require is one of only a few functions in Node.js that is blocking. That means, as long as it runs, Node.js is not able to fulfill any incoming requests. On startup this is no problem: It only takes a while until your application is up and running.
But for sure you don't want to have blocking moments while your application is already running for a while.
So, if you do not have VERY special reasons for the second option, go with the first one.
I believe the second case is useful to avoid circular module dependencies, since the require() happens at run-time rather than load time.
Otherwise, I believe the first is (slightly?) faster and to me, quite a bit more readable.