I've written a small program using Playwright and I'm trying to make it use worker_threads. Now I've written a test for it but for some reason I'm getting the above error in my test.
I tried tweaking the tsconfig settings, but that also didn't work. What I read most about is using commonjs as the module, which I'm doing. I also installed ts-node, but that also didn't do anything.
Below is the function I wrote to create a worker.
import { Worker } from "worker_threads";
const create_worker = (file: string, tasksData: {}) =>
new Promise((resolve, reject) => {
const worker = new Worker(file, {
workerData: tasksData,
});
worker.on("message", (data) => {
resolve(data);
});
worker.on("error", (msg) => {
reject(`An error occurred: ${msg}`);
});
});
export default create_worker;
And below is my test.
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
const search_grid = await construct_search_grid();
const chunks = chunkify_array(search_grid, 4);
const workerPromises: unknown[] = [];
for (let i = 0; i < 5; i++) {
const worker = await create_worker(
"./src/adapters/playwright.ts",
{
search_grid: chunks![i],
}
);
workerPromises.push(worker);
}
const thread_results = await Promise.all(workerPromises);
Now I believe everything checks out so I'm not really sure what to look for now. Does anyone know?
It feels like you don't have tsconfig.json "module" configured?
I have mine set to "commonjs". Without it, I also get that same error.
I was able to overcome this error by following this guide: https://wanago.io/2019/05/06/node-js-typescript-12-worker-threads/
Related
I'm using Node.js v18.12.1 and Discord.js v14. for developing the Discord bot. I need to read some data from Firebase. I'm confused because I'm used to how Java with Hibernate fetches data differently. Here, I need to use onValue() listener.
My onValue() acts strange. Instead of just reading the data from Firebase, it skips entirely, then it triggers multiple times, each time skipping the body block of its code, and then it actually does the code after.
I've read somewhere on this forum that this can happen because there are more onValue() listeners that are subscribed and they are all fired up. Someone mentioned I need to use the off() function somewhere "before" the onValue(). This confuses me because I'm using this listener in many locations. I need it in each command file, in execute(interaction) functions. You know, when you need to execute slash commands in Discord. I have it something like this:
async execute(interaction) {
const infographicRef = ref(db, '/infographics/arena/' + interaction.options.getString("arena-team"));
var imageUrl = null;
var postUrl = null;
onValue(infographicRef, (snapshot) => {
imageUrl = snapshot.child("image-url").val();
interaction.reply(imageUrl);
})
},
And I planned for each command, in each command.js file to have onValue(). I'm not sure exactly what to do.
Also, I tried to work around this with once() method, I see it in Firebase documentation, but I got the error: ref.once() is not a function.
It seems that after first triggering of onValue method when the body is not executed, my code in interactionCreate.js is triggered as well, it points for a command to be executed again:
const { Events } = require('discord.js');
module.exports = {
name: Events.InteractionCreate,
async execute(interaction) {
if (!interaction.isChatInputCommand()) return;
const command = interaction.client.commands.get(interaction.commandName);
if (!command) {
console.error(`No command matching ${interaction.commandName} was found.`);
return;
}
try {
await command.execute(interaction);
} catch (error) {
console.error(`Error executing ${interaction.commandName}`);
console.error(error);
}
},
};
my bot.js (which is in my case an index file)
const fs = require('node:fs');
const path = require('node:path');
const { Client, Collection, Events, GatewayIntentBits } = require('discord.js');
const { token } = require('./config.json');
const client = new Client({ intents: [GatewayIntentBits.Guilds] });
const eventsPath = path.join(__dirname, 'events');
const eventFiles = fs.readdirSync(eventsPath).filter(file => file.endsWith('.js'));
for (const file of eventFiles) {
const filePath = path.join(eventsPath, file);
const event = require(filePath);
if (event.once) {
client.once(event.name, (...args) => event.execute(...args));
} else {
client.on(event.name, (...args) => event.execute(...args));
}
}
client.commands = new Collection();
const commandsPath = path.join(__dirname, 'commands');
const commandFiles = fs.readdirSync(commandsPath).filter(file => file.endsWith('.js'));
for (const file of commandFiles) {
const filePath = path.join(commandsPath, file);
const command = require(filePath);
client.commands.set(command.data.name, command);
}
client.once(Events.ClientReady, () => {
console.log('Ready!');
});
client.on(Events.InteractionCreate, async interaction => {
if (!interaction.isChatInputCommand()) return;
const command = client.commands.get(interaction.commandName);
if (!command) return;
try {
await command.execute(interaction);
} catch (error) {
console.error(error);
await interaction.reply({ content: 'There was an error while executing this command!', ephemeral: true });
}
});
client.login(token);
The onValue function registers a realtime listener, that continues to monitor the value on the database.
If you want to read a value once, that'd be done with get() function in v9 (which is the equivalent of the once method in earlier SDK versions). Have a look at the code sample in the documentation on reading data once.
I'm trying to crawl several web pages to check broken links and writing the results of the links to a json files, however, after the first file is completed the app crashes with no error popping up...
I'm using Puppeteer to crawl, Bluebird to run each link concurrently and fs to write the files.
WHAT IVE TRIED:
switching file type to '.txt' or '.php', this works but I need to create another loop outside the current workflow to convert the files from '.txt' to '.json'. Renaming the file right after writing to it also causes the app to crash.
using try catch statements for fs.writeFile but it never throws an error
the entire app outside of express, this worked at some point but i trying to use it within the framework
const express = require('express');
const router = express.Router();
const puppeteer = require('puppeteer');
const bluebird = require("bluebird");
const fs = require('fs');
router.get('/', function(req, res, next) {
(async () => {
// Our (multiple) URLs.
const urls = ['https://www.testing.com/allergy-test/', 'https://www.testing.com/genetic-testing/'];
const withBrowser = async (fn) => {
const browser = await puppeteer.launch();
try {
return await fn(browser);
} finally {
await browser.close();
}
}
const withPage = (browser) => async (fn) => {
const page = await browser.newPage();
// Turns request interceptor on.
await page.setRequestInterception(true);
// Ignore all the asset requests, just get the document.
page.on('request', request => {
if (request.resourceType() === 'document' ) {
request.continue();
} else {
request.abort();
}
});
try {
return await fn(page);
} finally {
await page.close();
}
}
const results = await withBrowser(async (browser) => {
return bluebird.map(urls, async (url) => {
return withPage(browser)(async (page) => {
await page.goto(url, {
waitUntil: 'domcontentloaded',
timeout: 0 // Removes timeout.
});
// Search for urls we want to "crawl".
const hrefs = await page.$$eval('a[href^="https://www.testing.com/"]', as => as.map(a => a.href));
// Predefine our arrays.
let links = [];
let redirect = [];
// Loops through each /goto/ url on page
for (const href of Object.entries(hrefs)) {
response = await page.goto(href[1], {
waitUntil: 'domcontentloaded',
timeout: 0 // Remove timeout.
});
const chain = response.request().redirectChain();
const link = {
'source_url': href[1],
'status': response.status(),
'final_url': response.url(),
'redirect_count': chain.length,
};
// Loops through the redirect chain for each href.
for ( const ch of chain) {
redirect = {
status: ch.response().status(),
url: ch.url(),
};
}
// Push all info of target link into links
links.push(link);
}
// JSONify the data.
const linksJson = JSON.stringify(links);
fileName = url.replace('https://www.testing.com/', '');
fileName = fileName.replace(/[^a-zA-Z0-9\-]/g, '');
// Write data to file in /tmp directory.
fs.writeFile(`./tmp/${fileName}.json`, linksJson, (err) => {
if (err) {
return console.log(err);
}
});
});
}, {concurrency: 4}); // How many pages to run at a time.
});
})();
});
module.exports = router;
UPDATE:
So there is nothing wrong with my code... I realized nodemon was stopping the process after each file was saved. Since nodemon would detect a "file change" it kept restarting my server after the first item
My Discord bot continuously sends [Object Promised] when I use
const DabiImages = require("dabi-images");
const DabiClient = new DabiImages.Client();
DabiClient.sfw.real.random().then(json => {
console.log(json);
}).catch(error => {
console.log(error);
});
I was wondering how I can fix this, as I'm not sure.
I think the comments on this already answer the question but you could..
const DabiImages = require('dabi-images')
const DabiClient = new DabiImages.Client()
const main = async () => {
const dabiJson = await DabiClient.sfw.real.random()
console.log(dabiJson)
}
main()
Also worth mentioning, that the sfw doesn't exist
I've created a script that scrapes information from a webpage and writes it to a Google Sheet. This is working great on my local machine, but I'd like to schedule this on GCP.
It sounds like Cloud Functions are the way to go, but when I deploy my function I'm getting this error:
Function failed on loading user code. Error message: Node.js module defined by file working.js is expected to export function named run
I'm not sure what I should be using as the "Function to execute". Here's the function I've uploaded:
const puppeteer = require('puppeteer');
const jsonexport = require('jsonexport');
const GoogleSpreadsheet = require('google-spreadsheet');
const creds = require('./google-generated-creds.json');
const fs = require('fs');
var doc = new GoogleSpreadsheet('1qaFi0xnhaCZEduylUvGXWpyMJv00Rz6Y9qqyFR1E9oI');
function run() {
return new Promise(async (resolve, reject) => {
try {
const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
const page = await browser.newPage();
const urls = [
"https://www.marksandspencer.com/pure-cotton-long-sleeve-jumpsuit/p/p60258655?image=SD_01_T42_6701_XB_X_EC_90&color=INDIGO&prevPage=plp",
"https://www.marksandspencer.com/cotton-rich-striped-3-4-sleeve-t-shirt/p/p60210598?prevPage=plp",
"https://www.marksandspencer.com/high-neck-long-sleeve-blouse/p/p60260040?image=SD_01_T43_5168_HD_X_EC_90&color=LIGHTDENIM&prevPage=plp",
"https://www.marksandspencer.com/pure-cotton-printed-short-sleeve-t-shirt/p/p60263529?image=SD_01_T41_8030Z_Z4_X_EC_90&color=WHITEMIX&prevPage=plp",
"https://www.marksandspencer.com/pure-cotton-button-detailed-denim-mini-skirt/p/p60260145?image=SD_01_T57_4004_QP_X_EC_90&color=DARKINDIGO&prevPage=plp",
"https://www.marksandspencer.com/pure-cotton-long-sleeve-shirt-midi-dress/p/p60258654?image=SD_01_T42_6703_HP_X_EC_90&color=DENIM&prevPage=plp",
"https://www.marksandspencer.com/mid-rise-skinny-leg-ankle-grazer-jeans/p/p60220155?prevPage=plp",
"https://www.marksandspencer.com/pure-cotton-long-sleeve-shirt/p/p60260208?image=SD_01_T43_5181_HP_X_EC_90&color=DENIM&prevPage=plp",
"https://www.marksandspencer.com/long-sleeve-shirt-mini-dress/p/p60258652?image=SD_01_T42_6704_HP_X_EC_90&color=DENIM&prevPage=plp",
"https://www.marksandspencer.com/wide-fit-suede-lace-up-trainers/p/p60216277?prevPage=plp",
"https://www.marksandspencer.com/suede-ankle-boots/p/p60226911?prevPage=plp",
"https://www.marksandspencer.com/leather-buckle-hip-belt/p/p60186701?prevPage=plp",
"https://www.marksandspencer.com/cross-body-bag/p/p60215352?prevPage=plp"
];
const productsList = [];
for (let i = 0; i < urls.length; i++) {
const url = urls[i];
await page.goto(url);
let products = await page.evaluate(() => {
let product = document.querySelector('h1[itemprop=name]').innerText;
let results = [];
let items = document.querySelectorAll('[data-ttip-id=sizeGridTooltip] tbody tr td label');
items.forEach((element) => {
let size = element.getAttribute('for');
let stockLevel = "";
let nearest_td = element.closest('td');
if (nearest_td.classList.contains('low-stock')) {
stockLevel = "Low stock"
} else if (nearest_td.classList.contains('out-of-stock')) {
stockLevel = "Out of stock"
} else {
stockLevel = "In stock"
}
results.push({
product: product,
size: size,
stock: stockLevel
})
});
return results
})
productsList.push(products)
}
browser.close();
function flatten(arr) {
return arr.reduce(function(flat, toFlatten) {
return flat.concat(Array.isArray(toFlatten) ? flatten(toFlatten) : toFlatten);
}, []);
}
var flatProducts = flatten(productsList)
flatProducts.forEach(function(row) {
// Authenticate with the Google Spreadsheets API.
doc.useServiceAccountAuth(creds, function(err) {
// Get all of the rows from the spreadsheet.
doc.addRow(1, row, function(err, rows) {
console.log(row);
});
});
});
} catch (e) {
return reject(e);
}
})
}
run().then(console.log).catch(console.error);
I've never used Cloud Functions before so unsure how much I'd need to modify my script.
You can't just upload any script to run. You have to define a function using either the Cloud tools (via gcloud) or the Firebase tools and SDK. You will also have to figure out how you want to trigger it. When the function is triggered, then you can arrange to have your code executed.
I would say that it's mostly non-trivial to just port an existing script to Cloud Functions. You will have to take time to learn about how the system works in order to make effective use of it.
What that errors is referring to is that Cloud Functions can't find a function to run in that file (working.js) because you haven't exported one. For example, if you create a Cloud Function named run, then you must export a function in the script by assigning it to exports.run in your module:
exports.run = (event, callback) => {
callback(null, `Hello ${event.data.name || 'World'}!`);
};
There's more examples in the documentation, but it's likely that other changes will be necessary in your script for authentication, etc, to work from GCP.
I just wrote a script to release a build of one of the products I'm working on. The script does the job, but I don't really like the code itself, looks like spaghetti code and callback hell combined.
Is there a cleaner way to do this? I'd like to be able to run commands in series, log the outputs (stdout.on('data')) and when the task is finished. (easier for further debug and when waiting for the task to be done, reassuring to know what's happening on the background)
Maybe using Promises would help clean the mess a bit, but still, I feel like there should be a cleaner way to deal with multiple commands.
Some explanation about what the code does:
Create a tag with the commit you want and the tag version you want, i.e: git tag 1.2.5.
Build the release file with gulp build.
Create a folder doc/<tag>.
Convert doc/doc_reader.odt to doc/<tag>/documentation.pdf. (Open it and export as PDF)
Copy build/reader.js and doc/changelog.txt in the created folder.
Zip the 3 files.
Commit everything with commit message: Release 1.2.11 (for example)
Push.
Create a new release on GitHub using the commit you just pushed and the same tag.
Here is the code, as an example. (ES5, Node 4.6.0+)
var mkdirp = require('mkdirp');
var fs = require('fs-extra');
var path = require('path');
var spawn = require('child_process').spawn;
var zip = new require('node-zip')();
var package = require('../package.json');
var version = package.version;
var releaseDirectory = 'doc'
var gitTagProcess = spawn('git', ['tag', version]);
var gulpBuildProcess = spawn('gulp', ['build']);
console.log(`Running "git tag ${version}"...`);
gitTagProcess.stdout.on('data', function (chunk) {
console.log(chunk.toString('utf8'));
});
gitTagProcess.on('close', function () {
console.log('Tag created.')
console.log('Running "gulp build"...');
gulpBuildProcess.stdout.on('data', function (chunk) {
console.log(chunk.toString('utf8'));
});
gulpBuildProcess.on('close', function () {
console.log('"gulp build" done.')
console.log(`Creating "${releaseDirectory}/${version}" directory.`)
mkdirp(`${releaseDirectory}/${version}`, function () {
console.log('Directory created.');
var docFile = `${releaseDirectory}/doc_reader.md`;
console.log(`Converting ${docFile} to pdf ...`);
var docBuildProcess = spawn('npm', ['run', 'build:doc']);
docBuildProcess.stdout.on('data', function (chunk) {
console.log(chunk.toString('utf8'));
});
docBuildProcess.on('close', function () {
console.log('Doc created.');
console.log('Copying changelog.txt ...');
fs.copySync('doc/changelog.txt', `doc/${version}/changelog.txt`);
console.log('changelog.txt copied.');
console.log(`Copying "build/reader.js" to "doc/reader-${version}.js" and "doc/reader.js" ...`);
fs.copySync('build/reader.js', `doc/${version}/reader.js`);
fs.copySync('build/reader.js', `doc/${version}/reader-${version}.js`);
console.log('reader.js copied.');
console.log('Zipping all files ...');
zip.file('changelog.txt', fs.readFileSync(`doc/${version}/changelog.txt`));
zip.file('doc_reader.pdf', fs.readFileSync(`doc/${version}/doc_reader.pdf`));
zip.file('reader.js', fs.readFileSync(`doc/${version}/reader.js`));
zip.file(`reader-${version}.js`, fs.readFileSync(`doc/${version}/reader-${version}.js`));
var data = zip.generate({ base64: false, compression: 'DEFLATE' });
var zipFilename = `doc/${version}/HTML5Reader_${version}.zip`;
fs.writeFileSync(zipFilename, data, 'binary'); // it's important to use *binary* encode
console.log(`${zipFilename} created.`);
console.log(`\nRelease ${version} done. Please add generated files and commit using:`);
console.log(`\n\tgit add * && git commit -m "Release ${version}"`);
console.log(`\n\nDon't forget to push and create a new release on GitHub at https://github.com/$domain/$product/releases/new?tag=${version}`);
});
});
});
});
Edit:
Here is the implementation using async/await (node 7.8.0)
I used special mkdirp and exec modules, that allow usage with await. But I couldn't find an equivalent for spawn.
const mkdirp = require('async-mkdirp');
const fs = require('fs-extra');
const spawn = require('child-process-promise').spawn;
const exec = require('mz/child_process').exec;
const zip = new require('node-zip')();
const c = require('chalk');
const error = c.bold.red;
const warn = c.yellow;
const info = c.cyan;
const info2 = c.magenta;
const version = require('../package.json').version;
const releaseDirectory = 'doc'
async function git_tag() {
async function exec_git_tag() {
return await exec(`git tag ${version}`);
}
console.log(info(`Creating git tag ${version}`));
return exec_git_tag()
.then(() => {
console.log(info(`Git tag created for ${version}`))
})
.catch((err) => {
console.log(warn('warn', err));
})
// Finally
.then(() => {
console.log(info(`"git tag ${version}" - Completed`))
});
};
async function gulp_build() {
async function exec_gulp_build() {
const promise = spawn('gulp', ['build'])
const childProcess = promise.childProcess;
childProcess.stdout.on('data', (data) => {
console.log(info2(data.toString()));
});
childProcess.stderr.on('data', (data) => {
console.log(error(data.toString()));
});
return promise
.catch((err) => {
console.error(error(err));
})
// Finally
.then(() => {
console.log(info('"gulp build" - Completed'))
});
}
console.log(info('Running "gulp build"...'))
return exec_gulp_build()
}
async function create_dir() {
const dirPath = `${releaseDirectory}/${version}`;
console.log(info(`Creating "${dirPath}" directory.`))
await mkdirp(`${dirPath}`);
console.log(info(`Directory ${dirPath} created.`));
}
async function build_doc() {
const docFile = `${releaseDirectory}/doc_reader.md`;
console.log(info(`Converting ${docFile} to pdf ...`));
async function exec_build_doc() {
return await exec(`npm run build:doc`);
}
return exec_build_doc()
.catch((err) => {
console.error(error(err));
})
.then(() => {
console.log(info(`Doc "${docFile}" created.`));
})
}
function copy_files() {
console.log(info('Copying changelog.txt ...'));
fs.copySync('doc/changelog.txt', `doc/${version}/changelog.txt`);
console.log(info('changelog.txt copied.'));
console.log(info(`Copying "build/reader.js" to "doc/reader-${version}.js" and "doc/reader.js" ...`));
fs.copySync('build/reader.js', `doc/${version}/reader.js`);
fs.copySync('build/reader.js', `doc/${version}/reader-${version}.js`);
console.log(info('reader.js copied.'));
}
function zip_files() {
console.log(info('Zipping all files ...'));
zip.file('changelog.txt', fs.readFileSync(`doc/${version}/changelog.txt`));
zip.file('doc_reader.pdf', fs.readFileSync(`doc/${version}/doc_reader.pdf`));
zip.file('reader.js', fs.readFileSync(`doc/${version}/reader.js`));
zip.file(`reader-${version}.js`, fs.readFileSync(`doc/${version}/reader-${version}.js`));
const data = zip.generate({ base64: false, compression: 'DEFLATE' });
const zipFilename = `doc/${version}/HTML5Reader_${version}.zip`;
fs.writeFileSync(zipFilename, data, 'binary'); // it's important to use *binary* encode
console.log(info(`${zipFilename} created.`));
}
async function release() {
await git_tag();
await gulp_build();
await create_dir();
await build_doc();
copy_files();
zip_files();
console.log(`\nRelease ${version} done. Please add generated files and commit using:`);
console.log(`\n\tgit add . && git commit -m "Release ${version}"`);
}
release();
There is an mz module that can be very helpful here. See:
https://www.npmjs.com/package/mz
This, combined with async/await will allow you to write code like this:
let exec = require('mz/child_process').exec;
(async () => {
let version = await exec('node --version');
console.log(version);
let result = await exec('some other command');
console.log(result);
// ...
})();
This is a simple example but you can use all functions from the child_process, fs and many other modules that way.
What's important here is that this code is still asynchronous and non-blocking.
Note that you can only use await inside of a function created with the async keyword. For more info, see:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await
For support in browsers, see:
http://caniuse.com/async-functions
For support in Node, see:
http://node.green/#ES2017-features-async-functions
In places where you don't have native support for async and await you can use Babel:
https://babeljs.io/docs/plugins/transform-async-to-generator/
or with a slightly different syntax a generator based approach like in co or Bluebird coroutines:
https://www.npmjs.com/package/co
http://bluebirdjs.com/docs/api/promise.coroutine.html