I'm working with Node.js and Puppeteer, I'm using Puppeteer to scrape multiple websites. I'm thinking of organizing the code by separating each site implementation to its own file. So I will have like example.com.js example2.com.js etc
each of this will will require the same module(puppeteer/jsdom) and will export an object that contains variables and functions. This way, I can import all of these modules in the index.js file.
I've read about Nodejs caching modules and that this way of organizing of code is standard, however, in my case, if say I have 100 different website that I want to scrape, so I will have 100 module that will require (puppeteer/jsdom) in each one of them, so if Nodejs is caching the modules, I'm still using 100 different variables to store the modules.
const puppeteer = require('puppeteer');
So I'll have this line of code in 100 files, even if Nodejs is caching Puppeteer module, I'm still creating 100 puppeteer variable where I could have created just one if I wrote all code in a single file.
Am I missing something here ?
From the official document here
Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.
You're correct about "Nodejs is caching Puppeteer module", but "I'm still creating 100 puppeteer variable" isn't true. So, I would say, don't worry :), you can continue to organize your code in multiple files.
For example, if this is your setup:
// a.js
const puppeteer = require('puppeteer');
// b.js
const puppeteer = require('puppeteer');
// c.js
const a = require('./a.js');
const b = require('./b.js');
The following will happen;
File a.js is required
The module puppeteer is being parsed and referenced by the variable puppeteer inside the file.
File b.js is required
Also creates a reference to puppeteer, but the module has already been parsed and executed, so it is 'cached'
Doing this won't cause a performance issue, regarding the import of the same module multiple times.
Regarding your statement
I'm still creating 100 puppeteer variable
As mentioned above, you're only creating references, the contents won't be deeply cloned.
Related
I was trying to run some tests, but needed test data, so i created a generation file which created dummy html. When I attempt to run it though, it gives me a Reference Error: HTMLDivElement is not defined.
Is there something I need to import such that Node knows what HTMLDivElement is? I am not rendering anything, but just want correct data to pipe into follow on code.
I run my file through TSC, and then run it through node.
Sample Code:
const main = () => {
let root = new HTMLDivElement();
}
main();
Edit:
I was trying to just bypass it with: let root = document.createElement("div"); but node does not understand what document is, so I cant seem to get that running either as a fallback.
Indeed, Node.js doesn't come with a DOM implementation. If you want to run tests that use the DOM, you'll need to either load a Node.js-compatible DOM implementation such as jsdom, or if that doesn't meet your requirements, switch to a browser-based testing environment such as Selenium.
This is probably a duplicated question, but I couldn't find anything.
Since I'm learning NodeJS, I think that I'm not using the right words to search, so it's hard to find an answer.
Here is the situation:
I'm currently following an online course about NodeJS and coding an API.
In the current step we are using Winston library to log errors. The instructor, have configured on Index,js, which is the entry point of the app, like this:
File: index.js
const winston = require('winston');
const errorHandler = require(./middleware/error.js);
//(...) some other imports
app.use(errorHandler);
winston.add(winston.transports.File,{filename:'logFile.log'});
And in other module we've created in the course to handle errors, he requires winston and simply call to log the error. Something like this:
File: error.js
const winston = require('winston');
function errorHandler(err,req,res,next){
winston.error(err.message,err);
res.status(500).send("something failed");
}
module.exports = errorHandler;
After doing a test, the error is correctly written to the file, and my question is: How it works? How a setting made on the 'required version' of winston at index.js is visible from the other required version at error.js?
From index.js we are importing error.js too, so i can imagine somehow this two modules are sharing this winston object, but again, I don't understand how or where is it shared.
Again, please excuseme if I'm not using the right terms to refer anything here, I'll accept any advice.
Thanks.
When a module is loaded in node.js, it is cached by the require() sub-system. So, when you then require() it again, that means you'll get the exact same module as the previous one.
So ... if you initialized the module after you first loaded it and the module stores some state that represents that intialization, then subsequent use of that module will be using the same (already initialized) module.
And in other module we've created in the course to handle errors, he requires winston and simply call to log the error.
It gets the same instance of the winston module that was already initialized/configured previously.
After doing a test, the error is correctly written to the file, and my question is: How it works? How a setting made on the 'required version' of winston at index.js is visible from the other required version at error.js?
Module caching as describe above. There's only one winston module that all are sharing so if it's initialized/configured in one place, all will use that configuration.
Hi I'm new to nodeJs and currently developing a Rest API using node.I'm planning to develop it with a good folder structure, so I can scale it up easily. There I'm using several route files according to the business logic.
ex :- authRoutes,profileRoutes,orderRoutes ......
Currently in every single route file I had to include following codes
var express = require('express');
var router = express.Router();
var jwt = require('jsonwebtoken');
var passport = require('passport');
My problem is , Is it totally fine to use above code segments in all the route files( I'm concerning about the code optimisation/coding standards and execution speed ) or is there any better way to do this.
It's better if you can explain the functionality of require() function.
Thanks
In my experience, this is very standard to do. I suggest reading this question and answer to learn more about the way it affects the speed of your programs.
TL;DR of require()
When your lines of code are ran that include a variable that exists because of a require(), for instance
var https = require('https');
https.get(url, function(response) {...});
The compiler reads it and goes into the https module folder, and looks for the .get function.
However, if you are trying to require() a certain JavaScript file, such as analysis.js, you must navigate to that file from the file you are currently in. For instance, if the file you want is on the same level as the file you are in, you can access it like this:
var analysis = require('./analysis.js');
//Let analysis have a function called analyzeWeather
analysis.analyzeWeather(weather_data);
These lines of code are a little different from above. In this require() statement, we are saying grab the .js file with this name, analysis. Once you require it, you can access any public function inside of that analysis.js file.
Edits
Added require() example for .js file.
I'm building an isomorphic React/React-Router/Redux/Webpack application and I'm attempting to implement server side rendering.
My directory looks like:
/client
/actions
/components
/containers
/server
/server.js
In my webpack config, I have aliases set up for all the folders inside client:
var path_base = path.resolve(__dirname, '..');
const resolve = path.resolve;
const base = function() {
var args = [path_base];
args.push.apply(args, arguments);
return resolve.apply(resolve,args);
};
const resolve_alias = base.bind(null, 'src/client');
const aliases = [
'actions',
'components',
'constants',
'containers',
'middleware',
'reducers',
'routes',
'store',
'styles',
'utils',
'validation'
];
so that inside the code that gets bundled by webpack, I can do:
import { Widget } from 'components';
and that import gets resolved by webpack.
Now in my server code, in order to do the rendering, I have to import some of my client files, like routes/index.js. The problem I'm running into when I import my routes file, it's using a webpack alias to another file, say components or containers so naturally, the node js require system can't resolve it.
How do I fix something like that? I looked at this question and it talks about essentially setting up the same aliases that exist in webpack with mock-require. But then the issue becomes that my routes file imports all my components which then all import things like stylesheets, images, etc. Should I then be using something like webpack-isomorphic-tools?
The guides I've been looking at (this for example) are all great at showing how server side rendering is accomplished but none of them really talk about how to resolve all the requires and whatnot.
After battling with this issue for 2 days I settled on babel-plugin-webpack-alias.
What you need to do to resolve paths with that is:
$ npm install --save-dev babel-plugin-webpack-alias
Add the plugin to your .babelrc
Add the aliases to your webpack.config (make sure you use path.join())
Refer to this post if you have problems loading styles
The other option I tried was universal-webpack but I found it to be a bit verbose. If you want to see roughly how the whole server-side loading works, you can check out this video.
If you really want them, run your server side code through babel and use this plugin: https://www.npmjs.com/package/babel-plugin-module-alias which will let you do the same thing as webpack.
Edit: This one works a lot better: https://github.com/jagrem/babel-resolve-relative-module it allows multiple paths
Try to use NODE_PATH. Node will always look for a module in this path during require calls. It allows to short cut your relative paths as you want.
// turn this
import {Widget} from '../../components';
// into this
import {Widget} from 'components';
See Node.js docs for more information.
P.S. this thing is very sensitive, so use it carefully. Now your code tightly depends from the environment and may break somewhere.
If you use webpack-isomorphic-tools then it'll take your webpack config into account for your server side which will make all your aliases work.
https://www.npmjs.com/package/webpack-isomorphic-tools
I wrote a simple npm module to precompile my handlebars templates when using django compressor to do post-processing for some client side components and found that I need to ship the npm module with a few js files.
Currently I just assume no one is installing this with the global flag because I've "hard coded" the path to these dependencies in the npm module itself
example layout of my npm module
/
* /bin
* /lib/main.js
* /vendor/ember.js
Now inside main.js I want to use the ember.js file ... currently my hard coded approach looks like this
var emberjs = fs.readFileSync('node_modules/django-ember-precompile/vendor/ember.js', 'utf8');
Again -this only works because I assume you install it local but I'd like to think node.js has a more legit way to get locally embedded files
Anyone know how I can improve this to be more "global" friendly?
What you can do is get the directory of the current file and make your file paths relative to that.
var path = require('path')
, fs = require('fs');
var vendor = path.join(path.dirname(fs.realpathSync(__filename)), '../vendor');
var emberjs = fs.readFileSync(vendor + '/ember.js', 'utf8');
Hope that helps!
One of the great strengths of Node.js is how quickly you can get up and running. The downside to this approach is that you are forced to fit the design patterns it was build around.
This is an example where your approach differs too much from Nodes approach.
Node expects everything in a module to be exposed from the modules exports, including templates.
Move the readFileSync into the django-ember-precompile module, then expose the returned value via a module export in lib/main.js.
Example:
package.json
{
"name": "django-ember-precompile",
"main": "lib/main.js"
}
lib/main.js
module.exports.ember = readFileSync('vendor/ember.js')
vendor/ember.js
You obtain your template via
var template = require('django-ember-precompile').ember
This example can be refined, but the core idea is the same.