Jobs doesn't get picked up by #Processor in Nest.js BullQueue - node.js

I'm trying to implement a queue using the BullQueue package in nest.js. I have two queues, one is a file queue and another is an email queue.
I use the file queue to regenerate thumbnails in the background after the thumbnail is uploaded. And email queue to send the email. The problem is that my file queue is working completely fine but the email queue I have created gives me an inconsistent result. Sometimes the jobs are getting picked up by the #Process('JobName') function and sometimes it doesn't.
More importantly, I have noticed that when the problem occurs, I tried removing the dist folder and restarting the server which regenerated the dist directory. It worked until it didn't after a few trials.
This is my folder structure.
my-nest-app/
├─ src/
│ ├─ bull-queue/
│ │ ├─ bull-queue.module.ts
│ │ ├─ file-queue/
│ │ │ ├─ file-queue-producer.service.ts
│ │ │ ├─ file-queue-consumer.service.ts
│ │ ├─ email-queue/
│ │ │ ├─ email-producer.service.ts
│ │ │ ├─ email-processor.service.ts
bull-queue.module.ts
export enum BullQueue {
FileQueue = 'file-queue',
EmailQueue = 'email-queue',
}
#Global()
#Module({
imports: [
BullModule.forRoot({
redis: {
host: process.env.REDIS_HOST,
port: parseInt(process.env.REDIS_PORT) || 6379
}
}),
BullModule.registerQueue(
{ name: BullQueue.FileQueue },
{ name: BullQueue.EmailQueue }
),
ProjectModule,
FileModule
],
providers: [
BullQueueService,
FileQueueConsumerService,
FileQueueProducerService,
EmailProcessorService,
EmailProducerService
],
controllers: [],
exports: [FileQueueProducerService, EmailProducerService]
})
export class BullQueueModule {}
email-producer.service.ts
export enum EmailJobs {
PasswordReset = 'password_reset',
}
#Injectable()
export class EmailProducerService {
constructor(#InjectQueue(BullQueue.EmailQueue) private queue:Queue){}
async password_reset_email(payload: PasswordResetPayload){
await this.queue.add(EmailJobs.PasswordReset, payload);
}
}
email-processor.service.ts
#Processor(BullQueue.EmailQueue)
export class EmailProcessorService {
constructor(
private readonly loggerService: LoggerService,
private readonly helperService: HelperService
) {}
#Process(EmailJobs.PasswordReset)
async password_reset(job: Job<PasswordResetPayload>) {
console.log('password_reset Job Processor')
}
}
I was wondering if I violated any design principle or if this was just an issue with the queue package manager.

Related

Generating uuid v5 in node.js erros out with no information?

I am using this npm package:
https://www.npmjs.com/package/uuid
I want to egenrate a v5 uuid.
I can generate a v4 no problem by requiring the module:
const { v4: uuidv4 } = require('uuid');
and then running:
console.log(`uuidv4: ${uuidv4()}`);
So then I try to generate a v5:
const { v5: uuidV5 } = require('uuid');
const MY_NAMESPACE = 'f709b20b-3353-4c32-8df9-66bc48e91ea9';
var v5uuid = uuidV5('hello', MY_NAMESPACE);
console.log(`userUUID: ${v5uuid}`);
However, the app gets to line var v5uuid = uuidV5('hello', MY_NAMESPACE); and then goes straight to the catch error. In the variables error says:
'uuidV5 is not a function'
running npm ls uuid:
├─┬ nodemon#1.3.3
│ └─┬ update-notifier#0.1.10
│ └─┬ configstore#0.3.2
│ └── uuid#2.0.3
├─┬ request#2.88.2
│ └── uuid#3.3.2 deduped
├─┬ sequelize#6.14.1
│ └── uuid#8.3.2
└── uuid#3.3.2
What am I doing wrong?
Below solution worked for me!
If your using package.json, add the following to package.json
{
"type": "module"
...
}
Now use can use import with node js
// index.js
import { v5 as uuidv5 } from "uuid";
const MY_NAMESPACE = "1b671a64-40d5-491e-99b0-da01ff1f3341";
uuidv5("Hello World", MY_NAMESPACE); // ⇨ 'a572fa0f-9bfa-5103-9882-16394770ad11'
Check your output using
node index.js

How to import all express router files from multiple directories in nodejs?

I'm building a REST API with versioning support. Here is my directory structure.
.
├── src
│ ├── api
│ │ ├── v1
│ │ │ ├── modules ─ ...
│ │ │ ├── routers
│ │ │ │ ├─── auth.router.js
│ │ │ │ ├─── posts.router.js
│ │ ├── v2
│ │ │ ├── modules ─ ...
│ │ │ ├── routers ─ ...
├── app.js
I want the router files imported to app.js. I've looked for the solution for hours but all I found is how to import each file manually through app.use(). This is doable but as the version numbers and router files keep increasing, this can lead to redundant work. I need a way to import these files with the least manual lines of code possible.
It is not possible to directly do this with Express, generally people mange modules manually with NodeJS, as it doesn't take a lot of work to do at all. In terms of version numbers, you could specify a version setting or constant somwhere, and import depending on that number.
For instance:
// routes.js
const apiVersion = "v2";
module.exports = {
require(`./${apiVersion}/auth.route`),
}
If this is not ideal, one hacky way to manage this is by grabbing all of the route files with the fs module, and importing them automatically. This is quite a hacky way of doing it, but I came up with something like this:
// router.js
const fs = require("fs/promises");
const { Router } = require("express");
const router = Router();
const apiVersion = "v2";
const loadRoutes = async () => {
// grab all the route files from a directory using fs
// use require to grab them from the source files
}
const routes = loadRoutes();
routes.forEach(route => {
router.use(route);
})
// app.js
const router = require("./path/to/router");
// ...boilerplate
app.use(router);

PM2 & Puppeteer Watch Restarting

I have a puppeteer script which I am using to produce an export from a reporting tool that I use (called pivot.js):
const fs = require('fs');
const path = require('path');
const events = require('events');
const puppeteer = require('puppeteer');
let eventEmitter = new events.EventEmitter();
const directoryPath = "./storage/"; /* A path to the storage of exported files */
((directoryPath) => {
fs.mkdir(path.resolve(path.resolve(),
directoryPath.replace(/^\.*\/|\/?[^\/]+\.[a-z]+|\/$/g, '')), { recursive: true }, error => {
if (error) console.error(error);
});
})(directoryPath); /* Creating a storage folder for exported files (if such a folder doesn't exist yet) */
(async () => {
eventEmitter.once('reportcomplete', () => {
/*
All changes should be made within this function.
Available methods:
- setReport (https://www.flexmonster.com/api/setreport/)
- exportTo (https://www.flexmonster.com/api/exportto/)
The exportTo method takes two parameters: type and params.
Callback function will be ignored.
Possible destination types:
- plain (the file will be saved by the path defined as a value of the "directoryPath" variable)
- server (the file will be exported to the server)
Available events (use "eventEmitter" to manage events):
- ready (https://www.flexmonster.com/api/ready/)
- reportcomplete (https://www.flexmonster.com/api/reportcomplete/)
- exportcomplete (https://www.flexmonster.com/api/exportcomplete/)
Additional methods and events can be added using the template.
*/
eventEmitter.once('reportcomplete', () => { /* Exporting when the report is ready */
exportTo("csv");
exportTo("html");
exportTo("pdf");
exportTo("image");
exportTo("excel");
});
let exportCount = 0;
eventEmitter.on('exportcomplete', () => {
exportCount++;
if (exportCount == 5) browser.close(); /* Closing the browser when all the exports are complete */
});
setReport({
dataSource: {
filename: 'https://cdn.flexmonster.com/data/data.json'
}
});
});
const browser = await puppeteer.launch(); /* Launching the headless browser */
const page = await browser.newPage(); /* Creating a new page */
/* A function to set a report for the component dynamically */
function setReport(report) {
page.evaluate(report => {
flexmonster.setReport(report);
}, report)
}
/* This code is responsible for the export itself. It supports five export formats:
.html, .xlsx, .pdf, .csv, and .png. */
function exportTo(type, params) {
page.evaluate((type, params) => {
type = type.toLowerCase();
if (params) {
if (params.destinationType != "plain" && params.destinationType != "server")
params.destinationType = "plain";
}
else params = { destinationType: "plain" };
if (!params.filename) params.filename = "pivot";
flexmonster.exportTo(type, params, (result) => {
switch (type) {
case "pdf":
result.data = result.data.output();
break;
case "excel":
result.data = Array.from(result.data);
break;
case "image":
result.data = result.data.toDataURL();
break;
}
exportHandler(result);
});
}, type, params);
}
await page.exposeFunction('exportHandler', (result) => {
switch (result.type) {
case "excel":
result.data = Buffer.from(result.data);
result.type = "xlsx";
break;
case "image":
result.data = Buffer.from(result.data.replace(/^data:image\/\w+;base64,/, ""), 'base64');
result.type = "png";
break;
}
fs.writeFile(`${directoryPath}${result.filename}.${result.type}`, result.data, 'utf8', error => {
if (error) console.log(error);
});
});
/* This code adds functions to emit ready, reportcomplete, and exportcomplete events for the browser
when called. This approach allows us to handle the component's events in the browser's scope. */
await page.exposeFunction('onReady', () => {
eventEmitter.emit('ready')
});
await page.exposeFunction('onReportComplete', () => {
eventEmitter.emit('reportcomplete')
});
await page.exposeFunction('onExportComplete', () => {
eventEmitter.emit('exportcomplete')
});
/* Reading the file with the component and setting it as the browser page's contents */
await page.setContent(fs.readFileSync('index.html', 'utf8'));
/* This code runs in the page's scope, subscribing the browser window to the component's ready,
reportcomplete, and exportcomplete events */
await page.evaluate(() => {
window.addEventListener('ready', () => window.onReady());
window.addEventListener('reportcomplete', () => window.onReportComplete());
window.addEventListener('exportcomplete', () => window.onExportComplete());
});
})();
I am then using PM2 to watch the file, allowing me to swap out the code used to produce different reports, using:
pm2 start pivot.js --watch
The issue I have is that whenever I delete the contents of the storage folder (which the script writes into), a new export appears straight away. Almost as if the script is continuously being called, or PM2 is being restarted.
Both of the logs for PM2 are completely blank. But after running:
pm2 show 0
I receive the following:
│ status │ stopping │
│ name │ pivot │
│ namespace │ default │
│ version │ 1.0.0 │
│ restarts │ 1957 │
│ uptime │ 0 │
│ script path │ C:\Users\admin\Documents\Windows Puppeteer\pivot.js │
│ script args │ N/A │
│ error log path │ C:\Users\admin\.pm2\logs\pivot-error.log │
│ out log path │ C:\Users\admin\.pm2\logs\pivot-out.log │
│ pid path │ C:\Users\admin\.pm2\pids\pivot-0.pid │
│ interpreter │ node │
│ interpreter args │ N/A │
│ script id │ 0 │
│ exec cwd │ C:\Users\admin\Documents\Windows Puppeteer │
│ exec mode │ fork_mode │
│ node.js version │ 14.15.1 │
│ node env │ N/A │
│ watch & reload │ ✔ │
│ unstable restarts │ 0 │
│ created at │ 2020-11-30T01:24:27.461Z │
I hope you can help.
The issue is that the puppeteer script (named pivot.js) dumps the returning file in a folder called "storage". Storage is within the same directory as pivot.js, meaning that whilst PM2 monitors that directory, it is creating an infinite loop. The solution is to use the ignore watch option.
Creating a ecosystem file as such:
module.exports = {
apps : [{
script: 'pivot.js',
watch: '.',
ignore_watch : ["node_modules", "storage"]
}],
...
};
Or using:
pm2 start pivot.js --watch --ignore-watch="storage"
In my examples above will resolve the problem.

Aliasing modules using NodeJS

Some context here: It's not that I cannot use Webpack, it's that I do not want to use Webpack. I would like to keep everything as "vanilla" as possible.
Currently when creating modules in a project you have to require them using either a relative or absolute path, for example in the following directory..
project/
├── index.js
├── lib/
│ ├── network/
│ │ request.js
│ │ response.js
├── pages/
│ ├── foo.js
Considering we're in index.js we would import request via
var networkRequest = require('./lib/network/request.js')
and if we're in foo.js we would import request via
var networkRequest = require('../lib/network/request.js')
What I'm wondering is that if there's any way to perhaps, set a local alias in Package.json or anywhere else like so:
localPackages = [
{ name: 'network-request', path: './lib/network/request.js' }
];
In which you could just do
var networkRequest = require('network-request')
From any file and it will provide the correct path.
Yep, that's what npm link is for. Native and out of the box.
You can also set local paths in package.json
{
"name": "baz",
"dependencies": {
"bar": "file:../foo/bar"
}
}

clearing cloudflare cache programmatically

I am trying to clear the cloudflare cache for single urls programmatically after put requests to a node.js api. I am using the https://github.com/cloudflare/node-cloudflare library, however I can't figure out how to log a callback from cloudflare. According to the test file in the same repo, the syntax should be something like this:
//client declaration:
t.context.cf = new CF({
key: 'deadbeef',
email: 'cloudflare#example.com',
h2: false
});
//invoke clearCache:
t.context.cf.deleteCache('1', {
files: [
'https://example.com/purge_url'
]
})
How can I read out the callback from this request?
I have tried the following in my own code:
client.deleteCache(process.env.CLOUDFLARE_ZONE, { "files": [url] }, function (data) {
console.log(`Cloudflare cache purged for: ${url}`);
console.log(`Callback:${data}`);
})
and:
client.deleteCache('1', {
files: [
'https://example.com/purge_url'
]
}).then(function(a,b){
console.log('helllllllooooooooo');
})
to no avail. :(
Purging Cloudflare cache by url:
var Cloudflare = require('cloudflare');
const { CF_EMAIL, CF_KEY, CF_ZONE } = process.env;
if (!CF_ZONE || !CF_EMAIL || !CF_KEY) {
throw new Error('you must provide env. variables: [CF_ZONE, CF_EMAIL, CF_KEY]');
}
const client = new Cloudflare({email: CF_EMAIL, key: CF_KEY});
const targetUrl = `https://example.com/purge_url`;
client.zones.purgeCache(CF_ZONE, { "files": [targetUrl] }).then(function (data) {
console.log(`Cloudflare cache purged for: ${targetUrl}`);
console.log(`Callback:`, data);
}, function (error) {
console.error(error);
});
You can lookup cloudflare zone this way:
client.zones.browse().then(function (zones) {
console.log(zones);
})
Don't forget to install the current client version:
npm i cloudflare#^2.4.1 --save-dev
I wrote a nodejs module to purge cache for a entire website. It scan your "public" folder, build the full url and purge it on cloudflare:
You can run it using npx:
npm install -g npx
npx purge-cloudflare-cache your#email.com your_cloudflare_key the_domain_zone https://your.website.com your/public/folder
But, you can install it and run using npm too:
npm install -g purge-cloudflare-cache
purge your#email.com your_cloudflare_key the_domain_zone https://your.website.com your/public/folder
For a public/folder tree like:
├── assets
│ ├── fonts
│ │ ├── roboto-regular.ttf
│ │ └── roboto.scss
│ ├── icon
│ │ └── favicon.ico
│ └── imgs
│ └── logo.png
├── build
│ ├── main.css
│ ├── main.js
├── index.html
It will purge cache for files:
https://your.website.com/index.html
https://your.website.com/build/main.css
https://your.website.com/build/main.js
https://your.website.com/assets/imgs/logo.png
https://your.website.com/assets/icon/favicon.ico
https://your.website.com/assets/fonts/roboto.css
https://your.website.com/assets/fonts/roboto-regular.ttf
This is probably happening because my mocha tests don't wait for the callback to return.
https://github.com/mochajs/mocha/issues/362

Resources