Fatal error when scraping dynamic page content with x-ray and x-ray-phantom - node.js

I am getting this error in the command line when I run app.js on node:
FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope
Abort trap: 6
This is what am app.js looks like:
var phantom = require('x-ray-phantom');
var Xray = require('x-ray');
var fs = require('fs');
x = new Xray().driver(phantom());;
x('http://www.bbc.co.uk/news', '.most-popular__list panel-read li', [{
content: ''
}])(function (err, results) {
fs.writeFile('results.json', JSON.stringify(results, null, '\t'));
})
My package.json
{
...
"dependencies": {
"phantomjs": "^1.9.19",
"x-ray": "^2.0.2",
"x-ray-phantom": "^1.0.1"
}
...
}
My node version is v5.3.0, Npm version is 3.3.12, Phantomjs version is 1.9.8. I am on a Mac El Capitan 10.11.1

Check if this works:
x = new Xray().driver(phantom({
dnodeOpts: {
weak: false
}
}));
I use phantom instead of x-ray-phantom and I encountered the same issue. It appears that the module node-weak is broken (https://github.com/TooTallNate/node-weak/issues/65), and bypassing this module as this link suggests has solved my case.

Related

FS - Use FS for Electron on Webpack 5

I recently upgraded my Quasar framework to v2 which uses Vue3, Electron 16 and Webpack 5.
When I tried to run my application, I keep getting this error:
Module not found: Can't resolve imported dependency "fs"
I understand that webpack 5 does not automatically do polyfills so I added these in my quasar.conf.js:
chainWebpack (chain) {
const nodePolyfillWebpackPlugin = require('node-polyfill-webpack-plugin')
chain.plugin('node-polyfill').use(nodePolyfillWebpackPlugin)
}
But this doesn't support polyfills for the fs module.
When I tried adding this:
if (!cfg.resolve.fallback) {
cfg.resolve.fallback = {}
cfg.resolve.fallback.fs = false
}
Now I get an error:
TypeError: fs.readFileSync is not a function
I also tried adding this in my package.json, same behavior (as expected):
"browser": {
"fs": false
}
But then.. How can I use fs in my electron application?
Was able to figure it out now.
Adding it here in case anyone might need it in the future. I see a lot of people asking this question but got no resolution.
So.. I added this to electron-preload.js, since fs / built-in node modules are accessible via the renderer:
import { contextBridge } from 'electron'
const fs = require('fs')
contextBridge.exposeInMainWorld('electronFs', {
readFileSync: fs.readFileSync,
existsSync: fs.existsSync
// Other fs methods here
})
And use it like this in Vue / JS / modules:
const fs = window.electronFs
const jsonContent = JSON.parse(fs.readFileSync(SOME_PATH))
You can solve Can't resolve imported dependency "fs" error by adding the code below to "build" section in quasar.conf.js:
extendWebpack(cfg) {
cfg.resolve.fallback = {
fs: false
}
}
quasar.conf.js:
module.exports = configure(function (ctx) {
return {
build: {
extendWebpack(cfg) {
cfg.resolve.fallback = {
fs: false
}
}
}
}
});
In addition, if there are more Can't resolve imported dependency "crypto", "Buffer", "os", "path", "stream", "assert" and so on, you can add them as shown below:
quasar.conf.js:
module.exports = configure(function (ctx) {
return {
build: {
extendWebpack(cfg) {
cfg.resolve.fallback = {
fs: false,
crypto: false,
Buffer: false,
os: false,
path: false,
stream: false,
assert: false,
dns: false,
net: false,
tls: false,
http2: false,
https: false,
http: false,
zlib: false
}
}
}
}
});
"node-polyfill-webpack-plugin" cannot cover all Can't resolve imported dependency "xxx" errors and if you fully use my solution above, you don't need to use "node-polyfill-webpack-plugin". In other words, you can use "node-polyfill-webpack-plugin" to cover some parts of Can't resolve imported dependency "xxx" errors then for the rest of Can't resolve imported dependency "xxx" errors, you use my solution above so you can use 2 solutions at the same time.
There is a new flag in Webpack 5 to target the build for Electron, which includes the correct support.
Simply add target = 'electron-main' to your webpack config.
See more here: https://webpack.js.org/configuration/target/

npm bin return me "/usr/local/bin/X: 1: /usr/local/bin/X: Syntax error: "(" unexpected" when calling it

So i'm just trying to create an npm bin who create a file in the current directory.
// ./index.js
const program = require('commander');
const fs = require('fs');
const path = require('path');
program
.command('c <name> <content>')
.action((name, content) => {
fs.writeFile(path.resolve(process.cwd(), name), content, err => err ? console.error(err) : console.log('Success'));
});
program.parse(process.argv);
This is not because of fs, even if i replace the writeFile by a console.log i still have to same error.
Here's my package.json :
{
"name": "test-crayzzit",
"dependencies": {
"commander": "^2.19.0"
},
"bin": {
"testcc": "./index.js"
},
"version": "1.0.3"
}
Everything's work well if i do something like node index.js test.txt hello
But if i install the package with npm : sudo npm i -g test-crayzzit
And do testcc c test.txt hello
It return me an error : /usr/local/bin/testcc: 1: /usr/local/bin/testcc: Syntax error: "(" unexpected
You can try by your self with the package : https://www.npmjs.com/package/test-crayzzit
Looks like you're missing the shebang. The first line of index.js should read as follows:
#!/usr/bin/env node
Moreover, the file should have LFline endings to be read properly on MacOS, Linux and Windows if you care about using the package on different platforms.
EDIT: I have tested your package (same error for me on Linux). Adding the shebang as described above works for me.
See also: Appropriate hashbang for Node.js scripts

Error when trying to use CanvasJS via RequireJS

I'm trying to use CanvasJS inside my project. I'm using RequireJS to manage the modules, and have this in the main script:
define(['domReady',"canvasjs","common-functions"], function(domReady,CanvasJS) {
domReady(function () {
window.CanvasJS = CanvasJS;
init_page_select();
});
});
This is what I have in my requireJS config file for the path:
"paths": {
// other stuff here
"canvasjs": "node_modules/canvasjs/dist/canvasjs.min"
},
I can see the canvasjs.min.js file being grabbed fine - but then I get this weird error:
ReferenceError: intToHexColorString is not defined[Learn More] canvasjs.min.js:7:7042
[33]</n.prototype.render https://www.test.org/2018/js/lib/node_modules/canvasjs/dist/canvasjs.min.js:7:7042
[28]</n.prototype.render https://www.test.org/2018/js/lib/node_modules/canvasjs/dist/canvasjs.min.js:5:14150
n/this.render https://www.test.org/2018/js/lib/node_modules/canvasjs/dist/canvasjs.min.js:8:17771
init_page_select https://www.test.org/2018/js/lib/spot_view_stats.js:83:2
<anonymous> https://www.test.org/2018/js/lib/spot_view_stats.js:4:3
domReady https://www.test.org/2018/js/lib/domready.js:105:13
<anonymous> https://www.test.org/2018/js/lib/spot_view_stats.js:2:2
execCb https://www.test.org/2018/js/lib/require.js:5:12859
check https://www.test.org/2018/js/lib/require.js:5:6575
enable/</< https://www.test.org/2018/js/lib/require.js:5:9031
bind/< https://www.test.org/2018/js/lib/require.js:5:812
emit/< https://www.test.org/2018/js/lib/require.js:5:9497
each https://www.test.org/2018/js/lib/require.js:5:289
emit https://www.test.org/2018/js/lib/require.js:5:9465
check https://www.test.org/2018/js/lib/require.js:5:7169
enable/</< https://www.test.org/2018/js/lib/require.js:5:9031
bind/< https://www.test.org/2018/js/lib/require.js:5:812
emit/< https://www.test.org/2018/js/lib/require.js:5:9497
each https://www.test.org/2018/js/lib/require.js:5:289
emit https://www.test.org/2018/js/lib/require.js:5:9465
check https://www.test.org/2018/js/lib/require.js:5:7169
enable https://www.test.org/2018/js/lib/require.js:5:9358
init https://www.test.org/2018/js/lib/require.js:5:5716
h https://www.test.org/2018/js/lib/require.js:5:4287
completeLoad https://www.test.org/2018/js/lib/require.js:5:12090
onScriptLoad https://www.test.org/2018/js/lib/require.js:5:13014
I'm invoking it with:
var chart = new CanvasJS.Chart("thegraph",
{
title:{
text: impressionText
},
theme: "theme2",
axisX: {
valueFormatString: "MMM-DD-YYYY",
labelAngle: -50
},
axisY:{
valueFormatString: "#0",
title: impressionText
},
data: [
{
type: "line",
showInLegend: true,
legendText: legendText,
dataPoints: dataPoints
}
]
});
chart.render();
Interestingly, if I tell it to load canvasjs.js instead of canvasjs.min.js, I get another error:
ReferenceError: intToHexColorString is not defined[Learn More]
OK so the problem seemed to be my version. For some reason "npm install canvasjs" was installing 1.8.1, but 2.2 was out. As per their request, I updated it to 2.2 and it sorted the problem. It seems weird npm is running such an outdated version though

bundle.js bundle.js:1 Uncaught Error: Cannot find module '/node_modules\mongodb-core\lib\topologies/../../package.json'

I am getting the below error:
bundle.js:1 Uncaught Error: Cannot find module '/node_modules\mongodb-ore\lib\topologies/../../package.json'
Here is my version detial:
OS: Windows10
MongoDB: 2.2.16
MongoDB-core: 2.1.2
Node: 6.9.2
I have used npm install bson-ext and changed \node_modules\mongodb-core\node_modules\bson-ext\ext\index.js to
try {
// Load the precompiled win32 binary
if(process.platform == "win32" && process.arch == "x64") {
bson = require('bson');
} else if(process.platform == "win32" && process.arch == "ia32") {
bson = require('bson');
} else {
bson = require('bson');
}
} catch(err) {
console.log(err)
// Attempt to load the release bson version
try {
bson = require('bindings')('bson.node');
} catch (err) {
throw new Error("js-bson: Failed to load c++ bson extension, using pure JS version");
}
}
while the original is:
bson = require('./win32/x64/bson');
because when I try browserify range.js > bundle.js, it cannot find bson-ext module in mongoDB-core.
I am not sure whether this kind of operation may cause the above error.
Here is my package.json file :
"dependencies": {
"browserify": "^13.1.1",
"bson": "^1.0.1",
"d3": "^4.4.0",
"express": "^4.14.0",
"hbs": "^4.0.1",
"jsdom": "^9.9.1",
"mongodb": "^2.2.16",
"mongodb-core": "^2.1.2"
}
I haven't been able to confirm this yet, but I think the problem is that MongoDB's JavaScript (Node.js) driver is not intended for use in a browser, for security reasons. Not clear if the problem in the OP is due to Browserify incorrectly resolving relative paths or something else, but regardless the preferred technique is to proxy requests to your MongoDB instance via a Node server.
Mongo lists off-the-shelf solutions for this here:
https://docs.mongodb.com/ecosystem/tools/http-interfaces/
Note also the --rest option, which allows an application to read directly from the DB via a URL schema:
https://docs.mongodb.com/ecosystem/tools/http-interfaces/#simple-rest-api
As the docs mention, this is not good practice for security concerns, but may help with prototyping.

Meteorjs with ta-lib npm module (server side)

I would like to use npm talib module within a meteorjs application.I followed the instruction here and I added this to the packages.json file:
{
"talib" : "0.6.0"
}
After running meteor the module talib.node is there in the node_modules/talib/build/Release/ folder.
Then I wrote the following:
if (Meteor.isServer) {
var Talib = Meteor.npmRequire('talib');
var talib = new Talib();
}
The Talib object is created and can be logged, like this:
{ version: '0.6.0',
functions:
[ { name: 'ADD',
group: 'Math Operators',
...
}
],
functionUnstIds:
{ TA_FUNC_UNST_ADX: 0,
TA_FUNC_UNST_ADXR: 1,
...
}
}
The line var talib = new Talib() gives instead this error:
TypeError: object is not a function
What I am doing wrong?

Resources