How would you implement a Cron Job that can send A lot of emails at once in NEST JS - cron

I need to Implement a Cron Job that runs once a day and check if a user meets the requirements, the thing is that the platform this will be implemented in has already 3000+ users, I am Using AWS SES and NODEMAILER,
This is my mailer config
transport: {
pool: true,
rateDelta: 1000,
rateLimit: 13,
socketTimeout: 10000,
maxConnections: 5,
maxMessages: 100,
host: process.env.SMTP_MAIL_HOST,
port: parseInt(process.env.SMTP_MAIL_PORT),
secure: false, // upgrade later with STARTTLS
auth: {
user: process.env.SMTP_MAIL_USER,
pass: process.env.SMTP_MAIL_PASS,
},
},
and this is my Cron Job
// Every day at 12:00:00 America/Atikokan (EST)
#Cron('0 0 18 * * *', {
name: 'ReferralsPromo',
timeZone: 'America/Atikokan', // <-- EST
})
async handleCron() {
this.logger.log('Referral Cron job executed');
const candidates =
await this.candidatesService.FindReferralPromoCandidates();
this.logger.log(`Found ${candidates.length} candidates`);
candidates.forEach(async (candidate) => {
try {
if (condition) {
const data =
await this.referralUniqueIdService.getCandidateUniqueReferralId(
candidate.User.Email,
);
const { UniqueId } = data;
this.mailService.sendReferralPromoteEmail(candidate.User, UniqueId);
this.candidatesService.updateCandidateReferralPromo(candidate.Id);
}
} catch (error) {
this.logger.error(error);
}
});
}
I would really appreciate some advice

Related

Knex migration command result in error: Timeout acquiring a connection

I'm building a backend project with Docker as my environment plus
Node 17+
Knex ^1.0.1
PG ^8.7.1
PostgreSQL 14.1-alpine
I was able to connect to my DB via a GUI as Table Plus and DB Beaver.
I also was able to use the knex migrate:make command to create the migration.
I followed some online and here suggestions to avoid this issue but I was unable to make it work.
Looks like I did a code mistake maybe.
No clue what's wrong right now.
Now the issue is with the command to migrate knex migrate:latest resulting as error
Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
at Client_PG.acquireConnection (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/lib/client.js:307:26)
at async Runner.ensureConnection (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/lib/execution/runner.js:294:28)
at async Runner.run (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/lib/execution/runner.js:30:19)
at async listCompleted (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/lib/migrations/migrate/migration-list-resolver.js:12:3)
at async Promise.all (index 1)
at async Migrator.latest (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/lib/migrations/migrate/Migrator.js:63:29)
at async Command.<anonymous> (/Users/jakub/Desktop/jakeDev/test/node_modules/knex/bin/cli.js:232:32)
The way I configured Knex
const development = {
client: 'pg',
connection: {
host: config.POSTGRES_HOST,
database: config.POSTGRES_DB,
user: config.POSTGRES_USER,
password: config.POSTGRES_PASSWORD,
},
pool: {
min: 0,
max: 50,
createTimeoutMillis: 3000,
acquireTimeoutMillis: 30000,
idleTimeoutMillis: 30000,
reapIntervalMillis: 1000,
createRetryIntervalMillis: 100,
propagateCreateError: false,
},
migrations: {
directory: './src/database/migrations',
},
};
And my migration setup
import tableNames from '../../constants/tableNames';
import { createTableName } from '../../utils/tableUtils.js';
/**
* #param { import("knex").Knex } knex
* #returns { Promise<void> }
*/
export async function up(knex) {
await Promise.all([
createTableName(knex, tableNames.pokemons),
createTableName(knex, tableNames.restoredPokeDex),
]);
}
/**
* #param { import("knex").Knex } knex
* #returns { Promise<void> }
*/
export async function down(knex) {
await Promise.all(
[tableNames.pokemons, tableNames.restoredPokeDex].map((tableName) =>
knex.schema.dropTableIfExists(tableName)
)
);
}
I'm importing the following part above
function tableDefaultColumns(table) {
table.text('name').notNullable().unique();
table.text('type_1').notNullable();
table.text('type_2');
table.integer('total').notNullable();
table.integer('hp').notNullable();
table.integer('attack').notNullable();
table.integer('defense').notNullable();
table.integer('sp_atk').notNullable();
table.integer('sp_def').notNullable();
table.integer('speed').notNullable();
table.integer('generation').notNullable();
table.boolean('legendary').defaultTo(false);
}
function createTableName(knex, tableName) {
return knex.schema.createTableName(tableName, (table) => {
table.increments().notNullable();
tableDefaultColumns(table);
});
}
export { tableDefaultColumns, createTableName };

The best practice for RabbitMQ consumers with NestJS and PM2

I use PM2 to handle each worker (consumer) in a separate process, e.g. ecosystem.config.js:
module.exports = {
apps: [
{
name: "Nest App",
script: "./dist/main.js",
instances: "3",
autorestart: false,
watch: true,
max_memory_restart: "1G",
exec_mode: "cluster",
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
},
},
{
name: "Consumer",
script: "./dist/message-queue/consumer.service.js",
instances: 1,
},
],
};
I also created a message-queue module which includes message-queue.service.ts that is used to setup the queue and publish messages, no problem with that.
My structure looks like:
src
message-queue
consumer.service.ts
message-queue.module.ts
message-queue.service.ts
The problem comes when we deal with the consumer (worker), channel consumed correctly, but I can't get any data from the database or call any model/service, my consumer.service.ts looks like:
import * as amqp from 'amqp-connection-manager';
import { Channel } from 'amqplib';
import { Constants } from '../utils/constants';
// consume messages from RabbitMQ
async function testConsumer(): Promise<void> {
const connection: amqp.AmqpConnectionManager = await amqp.connect('');
const channel: amqp.ChannelWrapper = await connection.createChannel({
setup: function (channel: Channel) {
return Promise.all([channel.prefetch(2)]);
},
});
connection.on('connect', function () {
console.log('Connected from test consumer');
});
return new Promise((resolve, reject) => {
channel.consume(Constants.MessageQueues.TEST, async function (msg) {
// parse message
// ...
});
});
}
testConsumer();
Can I query to get any data from the database, call any service (e.g. authService), or call any database model (e.g. userModel), and how?

NodeJS VM2 proper way to access console when set to 'redirect'

I'm using the VM2 package to run user code. I'm trying to intercept console output and have set the NodeVM object's console property to 'redirect':
// Create a new sandbox VM for this request
const vm = new NodeVM( {
console: 'redirect',
timeout: 30000,
sandbox: { request, state, response },
require: {
external: true
}
});
According to the documentation that redirects console output to 'events'. I'm new to NodeJS, how do I hook into those events to capture the console.log messages executed inside the Sandbox?
After digging through the source code, I found this file where the event emit is occuring:
sandbox.js
if (vm.options.console === 'inherit') {
global.console = Contextify.readonly(host.console);
} else if (vm.options.console === 'redirect') {
global.console = {
log(...args) {
vm.emit('console.log', ...Decontextify.arguments(args));
return null;
},
info(...args) {
vm.emit('console.info', ...Decontextify.arguments(args));
return null;
},
warn(...args) {
vm.emit('console.warn', ...Decontextify.arguments(args));
return null;
},
error(...args) {
vm.emit('console.error', ...Decontextify.arguments(args));
return null;
},
dir(...args) {
vm.emit('console.dir', ...Decontextify.arguments(args));
return null;
},
time: () => {},
timeEnd: () => {},
trace(...args) {
vm.emit('console.trace', ...Decontextify.arguments(args));
return null;
}
};
}
All you need to do to listen to these events is to bind an event listener on the vm you've created:
// Create a new sandbox VM for this request
const vm = new NodeVM( {
console: 'redirect',
require: {
external: ['request']
}
});
vm.on('console.log', (data) => {
console.log(`VM stdout: ${data}`);
});
Likewise, you can bind to console.log, console.info, console.warn, console.error, console.dir, and console.trace. Hopefully this will save someone else some time.

My task isn't working of ActionHerojs

exports.sayHelloAction = {
name: 'sayHelloAction',
description: '',
outputExample: {},
version: 1,
inputs: {},
run: function (api, data, next) {
// Enqueue the task now, and process it ASAP
// api.tasks.enqueue(nameOfTask, args, queue, callback)
api.tasks.enqueue("sayHello", null, 'default', function (error, toRun) {
next(error)
});
}
};
and my task is like this, but when I run my task from my action y cant see the log(">>>>>>>>>>") in my console :(
const sayHello = {
name: 'sayHello',
description: 'I say hello',
queue: "default",
plugins: [],
pluginOptions: [],
frequency: 1000,
run: function(api, params, next){
console.log(">>>>>>>>>>>>>>>>>>>>>>>>>>")
next(true);
}
};
exports.task = sayHello
versions: Nodejs: 7.7, ActionHerojs 17
You are enquing a task, not running it. You need to enable some workers on your server.

Nightwatch not terminating after browser.end()

I'm running Nightwatch after launching a child process that starts up my local servers. Nightwatch runs the tests, they complete successfully, and the browser windows all close, but the nightwatch process continues to run after printing the message "OK. 10 total assertions passed.".
I thought it may have something to do with how I'm watching events on the nightwatch process, but as far as I can tell I am watching all events that would indicate Nightwatch is exiting.
The method shutdown() in runner.js is never called. How do I get Nightwatch to terminate when the tests finish?
Update
If I remove the last test in sign-in.js then Nightwatch exits as expected.
runner.js
import spawn from 'cross-spawn'
// 1. start the dev server using production config
process.env.NODE_ENV = 'testing'
let servers
function shutdown (result) {
console.log('HERE', result)
try {
// Passing a negative PID to kill will terminate all child processes, not just the parent
if (servers) process.kill(-servers.pid)
} catch (e) {
console.error('Unable to shutdown servers, may need to be killed manually')
}
if (result) {
console.error(result)
process.exit(1)
} else {
process.exit(0)
}
}
function watch (child) {
child.on('close', shutdown)
child.on('disconnect', shutdown)
child.on('error', shutdown)
child.on('exit', shutdown)
child.on('uncaughtException', shutdown)
}
try {
servers = spawn('yarn', ['run', 'dev-all'], { cwd: '..', stdio: 'inherit', detached: true })
watch(servers)
// 2. run the nightwatch test suite against it
// to run in additional browsers:
// 1. add an entry in test/e2e/nightwatch.conf.json under "test_settings"
// 2. add it to the --env flag below
// or override the environment flag, for example: `npm run e2e -- --env chrome,firefox`
// For more information on Nightwatch's config file, see
// http://nightwatchjs.org/guide#settings-file
var opts = process.argv.slice(2)
if (opts.indexOf('--config') === -1) {
opts = opts.concat(['--config', 'e2e/nightwatch.conf.js'])
}
if (opts.indexOf('--env') === -1) {
opts = opts.concat(['--env', 'chrome'])
}
var runner = spawn('./node_modules/.bin/nightwatch', opts, { stdio: 'inherit' })
watch(runner)
watch(process)
} catch (error) {
shutdown(error)
}
nightwatch.conf.js
require('babel-register')
var config = require('../../frontend/config')
// http://nightwatchjs.org/guide#settings-file
module.exports = {
src_folders: ['e2e/specs'],
output_folder: 'e2e/reports',
custom_assertions_path: ['e2e/custom-assertions'],
selenium: {
start_process: true,
server_path: 'node_modules/selenium-server/lib/runner/selenium-server-standalone-3.0.1.jar',
host: '127.0.0.1',
port: 4444,
cli_args: {
'webdriver.chrome.driver': require('chromedriver').path
}
},
test_settings: {
default: {
selenium_port: 4444,
selenium_host: 'localhost',
silent: true,
globals: {
devServerURL: 'http://localhost:' + (process.env.PORT || config.dev.port)
}
},
chrome: {
desiredCapabilities: {
browserName: 'chrome',
javascriptEnabled: true,
acceptSslCerts: true
}
},
firefox: {
desiredCapabilities: {
browserName: 'firefox',
javascriptEnabled: true,
acceptSslCerts: true
}
}
}
}
sign-in.js (one of the tests)
import firebase from 'firebase-admin'
import uuid from 'uuid'
import * as firebaseSettings from '../../../backend/src/firebase-settings'
const PASSWORD = 'toomanysecrets'
function createUser (user) {
console.log('Creating user', user.uid)
let db = firebase.database()
return Promise.all([
firebase.auth().createUser({
uid: user.uid,
email: user.email,
emailVerified: true,
displayName: user.fullName,
password: PASSWORD
}),
db.ref('users').child(user.uid).set({
email: user.email,
fullName: user.fullName
}),
db.ref('roles').child(user.uid).set({
instructor: false
})
])
}
function destroyUser (user) {
if (!user) return
console.log('Removing user', user.uid)
let db = firebase.database()
try { db.ref('roles').child(user.uid).remove() } catch (e) {}
try { db.ref('users').child(user.uid).remove() } catch (e) {}
try { firebase.auth().deleteUser(user.uid) } catch (e) {}
}
module.exports = {
'Sign In links exist': browser => {
// automatically uses dev Server port from /config.index.js
// default: http://localhost:8080
// see nightwatch.conf.js
const devServer = browser.globals.devServerURL
browser
.url(devServer)
.waitForElementVisible('#container', 5000)
browser.expect.element('.main-nav').to.be.present
browser.expect.element('.main-nav a[href^=\'https://oauth.ais.msu.edu/oauth/authorize\']').to.be.present
browser.expect.element('.main-nav a[href^=\'/email-sign-in\']').to.be.present
browser.end()
},
'Successful Sign In with Email shows dashboard': browser => {
const devServer = browser.globals.devServerURL
firebase.initializeApp(firebaseSettings.appConfig)
let userId = uuid.v4()
let user = {
uid: userId,
email: `${userId}#test.com`,
fullName: 'Test User'
}
createUser(user)
browser.url(devServer)
.waitForElementVisible('.main-nav a[href^=\'/email-sign-in\']', 5000)
.click('.main-nav a[href^=\'/email-sign-in\']')
.waitForElementVisible('button', 5000)
.setValue('input[type=text]', user.email)
.setValue('input[type=password]', PASSWORD)
.click('button')
.waitForElementVisible('.main-nav a[href^=\'/sign-out\']', 5000)
.end(() => {
destroyUser(user)
})
}
}
After the tests complete successfully, I see the following:
grimlock:backend egillespie$ ps -ef | grep nightwatch
501 13087 13085 0 1:51AM ttys000 0:02.18 node ./node_modules/.bin/nightwatch --presets es2015,stage-0 --config e2e/nightwatch.conf.js --env chrome
I was not explicitly closing the Firebase connection. This caused the last test to hang indefinitely.
Here's how I am closing the connection after doing test cleanup:
browser.end(() => {
destroyUser(user).then(() => {
firebase.app().delete()
})
})
The destroyUser function now looks like this:
function destroyUser (user) {
if (!user) return Promise.resolve()
let db = firebase.database()
return Promise.all([
db.ref('roles').child(user.uid).remove(),
db.ref('users').child(user.uid).remove(),
firebase.auth().deleteUser(user.uid)
])
}
In my case (nightwatch with vue/vuetify) after each test like so:
afterEach:function(browser,done){
done();
}
AfterAll(async()=>{
await closeSession();
await stopWebDriver();
}
place this in the config file #Erik Gillespie
Nightwatch still have this issue with the browser.end()
If you run Nightwatch with node.js you can stop the process
by doing something like that:
browser.end(() => {
process.exit();
});
It will close the browser and end the process.
I have tried the following method:
In "nightwatch.conf.js",
"test_settings" {
"default" {
"silent": true,
...
},
...
}
I set "silent" from true to false.
It lead to becoming verbose in the console. And the chromedriver.exe will exit peacefully after running the tests
I was using the vue template from: https://github.com/vuejs-templates/pwa
My platform:
Windows 7 (64bit)
node v8.1.3
"nightwatch": "^0.9.16",
"selenium-server": "^3.6.0",
"chromedriver": "^2.33.1"

Resources