I have a gulp task that downloads a few JSON files from GitHub, then prompts the user for values to replace in those files. For example, I have an .ftpconfig that gets download, and then the user is asked to enter hostname, username, password, and path.
Because the file first needs to be downloaded before it can be configured, and each file needs to be configured sequentially, I'm using quite a few nested callbacks. I'd like to change this "callback hell" system so that it utilizes async/await and/or promises instead, but I'm having a lot of difficulty understanding exactly why my code isn't working; it seems that promises fire their .then() functions asynchronously, which doesn't make sense to me.
My goals are as follows:
Download all config files asynchronously
Wait for all config files to finish downloading
Read existing settings from the config files
Prompt the user for changed settings in each config file synchronously
I've tried a number of approaches, none of which worked. I discarded the code I've used, but here's a rough recreation of the things I've tried:
Attempt #1:
return new Promise((resolve) => {
// download files...
}).then((resolve) => {
// configure first file...
}).then((resolve) => {
// configure second file...
}).then((resolve) => {
// configure thrid file...
});
Attempt #2:
const CONFIG_FILES = async () => {
const bs_download = await generate_config("browsersync");
const ftp_download = await generate_config("ftp");
const rsync_download = await generate_config("rsync");
return new Promise(() => {
configure_json("browsersync");
}).then(() => {
configure_json("ftp");
}).then(() => {
configure_json("rsync");
});
};
I'm sure I'm doing something very obviously wrong, but I'm not adapt enough at JavaScript to see the problem. Any help would be great appreciated.
My gulp task can be found here:
gulpfile.js
gulp-tasks/config.js
Thanks to #EricB, I was able to figure out what I was doing wrong. It was mostly a matter of making my functions return promises as well.
https://github.com/JacobDB/new-site/blob/d119b8b3c22aa7855791ab6b0ff3c2e33988b4b2/gulp-tasks/config.js
Related
I've been working on a Guilded Bot that automatically runs a function after x amount of MS. My goal is to automate this function to check a website for new posts. The issue I'm encountering is when trying to import the function and call on it within another file. None of the recommended methods I've found seem to work. Below is my code.
//relay.ts under ./automations/
async function patchNotes(message:Message) {
}
export { patchNotes }
//The main file in src its called index.ts
import path from "path";
import { BotClient, Client, Message } from "#guildedjs/gil";
const { token, token2 } = require('./config.json');
import { patchNotes } from './automations/relay';
const client = new BotClient({
token: token,
prefix: "/",
});
client.once('ready', () => console.log('Ready! Shut down using "ctrl+c"'));
client.login();
process.on("unhandledRejection", console.log)
//setTimeout(() => console.log(client.commands), 600);
// Automations
patchNotes
setInterval(() => patchNotes, 6000);
Currently, this method doesn't return console errors for both Types and other stuff. But it also doesn't run the code at all? I've tried other methods too but none have worked so far. Below are what packages I'm using.
ts-node "10.8.1"
typescript "4.7.4"
It's running Node.js and all files are written in TS. If you need any more details, I'd be happy to give them. Really hoping to get past this issue instead of just putting the function in my main file.
So I've actually just found the answer. So it seems I can use setInterval with async tasks. Below is the code I use to achieve this.
setInterval(async () => {
await function();
}, delay)
As for my other issue. I've figured out that I could just write client.messages.send instead of putting message. in front of it. Reason I didn't follow the advice of the recent comment is because this function shouldn't have any values returning. The reason I added message: Message is because there is a line in my code that uses "message". Which is the one mentioned above. Shoulda added that to this thread. Thanks for the response though. Resolved.
I'm trying to use the unzipper node module to extract and process a number of files (exact number is unknown). However, I can't seem to figure out how to know when all the files are processed. So far, my code looks like this:
s3.getObject(params).createReadStream()
.pipe(unzipper.Parse())
.on('entry', async (entry) => {
var fileName = entry.path;
if (fileName.match(someRegex)) {
await processEntry(entry);
console.log("Uploaded");
} else {
entry.autodrain();
console.log("Drained");
}
});
I'm trying to figure out how to know that unzipper has gone through all the files (i.e., no more entry events are forthcoming) and all the entry handlers have finished so that I know I've finished processing all the files I care about.
I've tried experimenting with the close and finish events but when I have, they both trigger before console.log("Uploaded"); has printed, so that doesn't seem right.
Help?
Directly from the docs:
The parser emits finish and error events like any other stream. The parser additionally provides a promise wrapper around those two events to allow easy folding into existing Promise-based structures.
Example:
fs.createReadStream('path/to/archive.zip')
.pipe(unzipper.Parse())
.on('entry', entry => entry.autodrain())
.promise()
.then( () => console.log('done'), e => console.log('error',e));
I have an endpoint on my Hapi server which builds a (sometimes) large zip file.
I want to return that file back to the frontend, but I can't seem to figure out how to make that work in the current (18) version of Hapi. There were some MAJOR api changes recently and all the examples I can find are completely out of date.
Some code that writes a test zip is below:
handler: async (request, h) => {
let zip = new JSZip();
zip.file('test.txt', 'derp derp');
let stream = zip.generateNodeStream({streamFiles:true})
.pipe(fs.createWriteStream('out.zip'))
.on('finish', function () {
// JSZip generates a readable stream with a "end" event,
// but is piped here in a writable stream which emits a "finish" event.
console.log("out.zip written.");
});
but I cant seem to figure out what to do after. I have tried all of the following so far:
// vanilla hapi
return zip;
// from Inert:
return h.file(zip);
// from Toys
return Toys.stream(zip);
// vanilla Hapi, take 2:
return h.reply(zip).type('application/zip');
But no luck. Ideally I'd like it to stream as the file is being built, which I have seen examples of people doing, but they were all either an outdated version of Hapi or another framework altogether.
Thanks in advance!
I wrote a function that work like this
onNewZipFileRequested
{get all the necessary data}
.then{download all the files}
.then{create a zipfile with all those file}
.then{upload that zipfile} (*here is the problem)
.than{update the database with the signedUrl of the file}
Here is the relevant code
[***CREATION OF ZIP FILE WORKING****]
}).then(() =>{
zip.generateNodeStream({type:'nodebuffer',streamFiles:true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log("zip written.");
return bucket.upload(tempPath, { //**** problem****
destination: destinazionePath
});
});
}).then(()=>{
const config = {
action:'read',
expires:'03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato=>{
const daSalvare ={
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
On the client side, as soon as the app see the status change and the new Url, a download button (pointing to the new url) appears
Everything is working, but if I try to download the file immediately... there is no file yet!!!
If I wait same time and retry the file is there.
I noted that the time I have to wait depend on the size of the zipfile.
The bucket.upload promise should resolve on the end of the upload, but apparently fires too early.
Is there a way to know exactly when the file is ready?
I may have to make same very big file, it's not a problem if the process takes several minutes, but I need to know when it's over.
* EDIT *
there was a unnecessary nesting in the code. While it was not the error (results are the same before and after refactoring) it was causing some confusion in the answers, so i edited it out.
Id' like to point out that i update the database only after getting the signed url, and i get that only after the upload (i could not otherwise), so to get any result at all the promise chain MUST work, and in fact it does. When on the client side the download button appears (happens when 'status' become 'fatto') it is already linked to the correct signed url, but if i press it too early the file is not there (Failed - No file). If i wait some second (the bigger the file the longer i have to wait) then the file is there.
(English is not my mother language, if i have been unclear ask and i will try to explain myself better)
It looks like the problem could be that the braces are not aligned properly, causing a then statement to be embedded within another. Here is the code with the then statements separated:
[***CREATION OF ZIP FILE WORKING****]}).then(() => {
zip.generateNodeStream({type: 'nodebuffer', streamFiles: true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log('zip written.')
return bucket.upload(tempPath, {
destination: destinazionePath
})
})
}).then(() => {
const config = {
action: 'read',
expires: '03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato => {
const daSalvare = {
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
Ok, this is kinda driving me batty. I am trying to run a mocha test on an app I'm writing that's supposed to select and copy a file based on project parameters (a dockerfile). The test is working for other cases, and in fact it's green for this case, except that it shouldn't be.
The test uses fs.readFileSync to store the file contents that should be there with the file contents that are actually there, to determine if the right file is copied.
Problem is, there's no file in the location being checked yet (as I haven't written the code to put it there, and I have validated by printing out the directories the test is using and then navigating there myself), but the test passes. Even more strange, as far as I can tell no code executes after the readFileSync operation. But it also doesn't time out...
Here's the test code in question:
it('should create a Dockerfile from ./data/dockerfiles/war.docker', () => {
let projectDocker = path.join(project, 'Dockerfile');
let warDocker = path.join(__dirname, '..', 'data','dockerfiles','war.docker');
select(project, (err, result) => {
let correct = fs.readFileSync(warDocker, 'utf-8');
console.log('projectDocker');
console.log('checking for projectDocker');
let prj = fs.readFileSync(projectDocker, 'utf-8');
console.log('Read file sync for project has completed'); // This line never fires
expect(prj).to.equal(correct);
expect(err).to.not.exist;
expect(result).to.exist;
expect(result).to.have.property('relPath', project);
expect(result).to.have.property('prepared', true);
});
});
select() looks suspiciously async, in which case your test should be async as well:
it('should create a Dockerfile from ./data/dockerfiles/war.docker', done => {
...
select(project, (err, result) => {
if (err) return done(err);
...
done();
});
});
Otherwise you run a big risk of exceptions being swallowed, as #Martin also suggests.