Dynamic Crons are executed many times at the same time - cron

I'm creating new CronJobs and scheduling them to run in the future, but when the execution time arrives, the same Job is fired three times.
After the execution of the job I am removing it from the registry and even so it does not avoid the tripling of the job.
localhost it's triggered onnly once
published it's triggered thrice
we have three pods behind kubernetes. i guess is something related with that.
const date = dateFns.addMinutes(new Date(), 10);
const job = new CronJob({
cronTime: date,
start: true,
onTick: async () => {
await this.sendEmail(params);
}
});
this.schedulerRegistry.addCronJob('job01', job);

Based on your reply in the comment, you are using clusters. So for every instance that is running, a cron is created. If you have three instances in the cluster, you will get three crons. What you need to do is assign a name to your cluster. I'll give you an example of the setup we have.
We are running instances based on max cpu cores.
We assigned a name to the instances, and gave one of them the name primary. And set it to run on one core.
The remaining instances doesn't matter what you name them, but we set the count to -1. This way we utilize all cores on the machine.
Here's an example of the ecosystem.config.js
module.exports = {
apps: [
{
name: 'nest-primary',
script: './dist/src/main.js',
instances: '1',
exec_mode: 'cluster',
time: true,
combine_logs: true,
max_memory_restart: '3500M',
max_old_space_size: 3000,
log_date_format: 'HH:mm YYYY-MM-DD Z',
log_type: 'json',
merge_logs: true,
env_local: {
NODE_ENV: 'local',
HOST: 'localhost',
PORT: 3000,
DATABASE_URL: 'mysql://user:password#localhost:3306/himam',
DATABASE_URL_PG: 'postgresql://postgres:password#localhost:5432/himam',
},
env_development: {
NODE_ENV: 'development',
PORT: 3000,
HOST: '0.0.0.0',
DATABASE_URL: 'mysql://user:password#localhost:3306/himam',
DATABASE_URL_PG: 'postgresql://postgres:password#localhost:5432/himam',
},
},
{
name: 'nest-replica',
script: './dist/src/main.js',
instances: '-1',
exec_mode: 'cluster',
time: true,
combine_logs: true,
max_memory_restart: '3500M',
max_old_space_size: 3000,
log_date_format: 'HH:mm YYYY-MM-DD Z',
log_type: 'json',
merge_logs: true,
env_local: {
NODE_ENV: 'local',
HOST: 'localhost',
PORT: 3000,
DATABASE_URL: 'mysql://user:password#localhost:3306/himam',
DATABASE_URL_PG: 'postgresql://postgres:password#localhost:5432/himam',
},
env_development: {
NODE_ENV: 'development',
PORT: 3000,
HOST: '0.0.0.0',
DATABASE_URL: 'mysql://user:password#localhost:3306/himam',
DATABASE_URL_PG: 'postgresql://postgres:password#localhost:5432/himam',
},
},
When I launch the cluster, I pass --env production
pm2 start ecosystem.config.js --env production
The most important part, in your crons, you need to check the name of the instance. You can do this by adding the names you used in the config above to your .env
PM2_PRIMARY_NAME=nest-primary
PM2_REPLICA_NAME=nest-replica
Finally, in your code when you want to run the cron, check the name of the process, like this:
async handleCron() {
if (process.env.name !== this.configService.get('PM2_PRIMARY_NAME')) {
return;
}
// do your cron logic here.
This ensures that your cron will run only once, because your primary instance is only running on 1 core, and you won't have duplicate triggers. Please do update us.

thanks for the help!
I was able to resolve this using cron from the yaml file:
cronjob:
use: true
schedule: '*/5 * * * *'
env:
CRON: 1
and call that on bootstrap app like that:
appBootstrapInstance
.bootstrap()
.then((app) => {
return app;
})
.then(async (app) => {
const env = app.get(EnvService);
if (env.getAsBoolean('CRON')) {
await new Promise((resolve) => setTimeout(resolve, CRON_TIMEOUT));
const task = app.get(MyTaskService);
await task.doSomething();
await new Promise((resolve) => setTimeout(resolve, CRON_TIMEOUT));
});

Related

TypeORM createConnection hangs in Jest

When I'm trying to create a TypeORM connection to a local postgres database in a beforeAll Jest hook, TypeORM's createConnection keeps hanging for indefinite amount of time.
I don't want to have it globally because the majority of the tests don't need this database connection.
jest.config.ts
/** #type {import('ts-jest/dist/types').InitialOptionsTsJest} */
module.exports = {
preset: 'ts-jest',
testEnvironment: 'node',
};
example.spec.ts
let conn;
describe('GET /healthz', () => {
beforeAll(async () => {
conn = await createConnection({
name: 'default',
type: 'postgres',
host: 'localhost',
port: 5433, // <- not a typo, I tested on both 5433 and 5432
database: 'test-local',
username: 'user',
password: 'pwd',
synchronize: true,
logging: true,
});
});
afterEach(async () => {
// omitted, but truncates all tables after every test
});
afterAll(async () => {
await conn.close();
});
it('should be true', () => {
expect(true).toBe(true);
});
});
Output of running jest with --detectOpenHandles:
However when I copy exactly these connection options in my normal application, it works correctly without any errors. And also in my jest it doesn't throw any errors so I'm pretty lost on what's going on here. I tried it in globalSetup before, but even there it just hangs. It just doesn't get past the createConnection. Any ideas or suggestions is much appreciated!
Altough the --detectOpenHandle pointed at the createDbConnection of TypeORM, it was actually a totally different thing that hanged.
It was very misleading, but I started cronjobs somewhere in the express app, which were hanging instead of the TypeORM createConnection.

When the server is restarted, previous cron jobs keep it working correctly (Node.js,Bull.js and Redis)

How can I do that cron jobs work in the background? I mean when the server is restarted, previous cron jobs keep it working correctly.
const queue = new Queue('make_recurring',{
redis: {host: "127.0.0.1", port: 6379}
});
queue.add(order,{
repeat:{
cron: date // 38,40 12 12,13,14,15 3 *
},
removeOnComplete: true
});
queue.process(async(job,done) => {
console.log(job.data);
}

Strapi CMS, Heroku error: no pg_hba.conf entry for host

Three months ago, I created an Strapi App that has deployed on Heroku, and everything works fine. I used macOS 10.13.6 and node 14.15.4 for the local environment
The configuration of database was created inside a file named database.js which located in rootApp/config/env/production/database.js
The following are everything config inside these file (database.js):
const parse = require('pg-connection-string').parse;
const config = parse(process.env.HEROKU_POSTGRESQL_MAROON_URL);
module.exports = ({ env }) => ({
defaultConnection: 'default',
connections: {
default: {
connector: 'bookshelf',
settings: {
client: 'postgres',
host: "ec2-35-169-184-61.compute-1.amazonaws.com",
port: 5432,
database: "d3d9tcukxxx",
username: "mwtwuvkwxxxx",
password: "42f0337xxxxx",
},
options: {
ssl:true,
},
},
},
});
But after 3 months (right now), I checked from heroku logs --tail then these app getting an error and the message was:
error error: no pg_hba.conf entry for host "3.86.36.125", user "mwtwuvkwtrqpir", database "d3d9tcukrk5fgh", SSL off
I used Strapi 3.2.5 , and I was deployed on Heroku Postgres with Plan free (Hobby).
I hope everyone helping me for this questions, and hope helping others for same case.
Thank you
We had the same issue on our Heroku instances and just recently found a fix.
Adding rejectUnauthorized to the database config appears to work.
config/database.js
module.exports = ({ env }) => ({
defaultConnection: 'default',
connections: {
default: {
connector: 'bookshelf',
settings: {
client: 'postgres',
host: env('DATABASE_HOST', 'localhost'),
port: env.int('DATABASE_PORT', 5432),
database: env('DATABASE_NAME', 'strapi'),
username: env('DATABASE_USERNAME', 'strapi'),
password: env('DATABASE_PASSWORD', 'strapi'),
schema: env('DATABASE_SCHEMA', 'public'), // Not Required
ssl: {
rejectUnauthorized: env.bool('DATABASE_SSL_SELF', false), // For self-signed certificates
},
},
options: {
ssl: env.bool('DATABASE_SSL', false),
},
},
},
});
I cannot take full credit however, it was this post on the Strapi forum that led me to the answer:
https://forum.strapi.io/t/error-no-pg-hba-conf-entry-for-host-ssl-off/3409
subsequently this link:
https://strapi.io/documentation/developer-docs/latest/setup-deployment-guides/configurations.html#database

How to set up postgresql config with heroku and nodejs?

This is my first time try to host nodeJS application - built with hapi.js, typeorm and postgresql - on heroku. I've create two apps on heroku - for "staging" (server-staging) and "production" (server-prod) - that using same code but will use different configuration. Why different configuration? because each application on heroku will use different postgres credential, as it's attached as an add-ons.
objective
My objective/main question is How and where I have to set the database config for my application?
I use .env file (which I ignore in .gitignore - I don't want to put the credential in my repo) to connect the application to my local database. Here is how the .env looks like:
HOST=localhost
PORT=3001
TYPEORM_CONNECTION=postgres
TYPEORM_HOST=localhost
TYPEORM_USERNAME=postgres
TYPEORM_PASSWORD=password
TYPEORM_DATABASE=database
TYPEORM_PORT=5432
TYPEORM_SYNCHRONIZE=true
TYPEORM_LOGGING=false
In the application, I never do/write code such process.env.TYPEORM_USERNAME since its done by the typeorm node_modules. What I do to start the connection is by doing this:
const server = new Hapi.Server({
port: process.env.PORT,
host: process.env.HOST,
routes: {
cors: Cors,
},
});
await server.register(Plugins);
server.route(Router);
await createConnection();
await server.start();
And my application automatically connected to the specified database as defined in the .env. Now, in heroku, the credential is lies here:
All information lies there, but, [Q1] I don't know how to tell my application (of course, without store the credential in my code/repo) that I have to use the config as defined in above picture? Also, as stated in above image, "Heroku rotates credentials periodically and updates applications where this database is attached.". Does it means the credentials will changed periodically? [Q2] If yes, is there any way to make my application auto recognise the new credential?
Sorry if my explanation make confused. If you did not understand what I am trying to achieve, please ask things that you don't understand, so I can fix/update my question to make it understandable.
Anyway, I found this example first-example and second-example. But, they are using process.env.DATABASE_URL, which contain credential. I think, it means that they not ignore their .env file in their repo?
*) Note: Q1 means Question 1, and so for the rest
Write a ormconfig.js file in the root of your repo. This way you can access the environment variables like the url provided from heroku and you don't have credentials in your repo.
require('dotenv').config();
module.exports = [
{
name: 'default',
type: 'postgres',
url: process.env.DATABASE_URL,
synchronize: false,
logging: false,
entities: ['dist/entities/*.*'],
migrations: ['dist/database/migrations/**/*.js'],
subscribers: ['dist/database/subscribers/**/*.js'],
cli: {
entitiesDir: 'dist/entities',
migrationsDir: 'dist/database/migrations',
subscribersDir: 'dist/database/subscribers',
},
},
{
name: 'development',
type: 'postgres',
host: process.env.POSTGRES_HOST,
port: process.env.POSTGRES_PORT,
username: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: process.env.POSTGRES_DB,
synchronize: true,
logging: true,
entities: ['src/entities/*.*'],
migrations: ['src/database/migrations/**/*.ts'],
subscribers: ['src/database/subscribers/**/*.ts'],
cli: {
entitiesDir: 'src/entities',
migrationsDir: 'src/database/migrations',
subscribersDir: 'src/database/subscribers',
},
},
];
With this configuration you can then get a specific configuration in javascript/typescript:
let connectionOptions: ConnectionOptions;
if(process.env.NODE_ENV ==='development') {
connectionOptions = await getConnectionOptions("development");
} else {
connectionOptions = await getConnectionOptions("default");
}
await createConnection(connectionOptions);

res.render doesn't render correctly after 1023 characters

I have a parent Express app, and a Ghost app as a child app, using Ghost as an npm module here.
I routed Ghost to be rendered at http://localhost:9000/blog. All the configuration works fine (Ghost will throw an error if the basic configuration isn't being provided correctly).
Here is my Ghost startup code
ghost({
config: path.join(__dirname, '/config/ghost.config.js')
}).then(function (ghostServer) {
app.use(ghostServer.config.paths.subdir, ghostServer.rootApp);
ghostServer.start(app);
});
here is my Ghost config
// # Ghost Configuration
// Setup your Ghost install for various [environments](http://support.ghost.org/config/#about-environments).
// Ghost runs in `development` mode by default. Full documentation can be found at http://support.ghost.org/config/
var path = require('path'),
config;
config = {
// ### Production
// When running Ghost in the wild, use the production environment.
// Configure your URL and mail settings here
production: {
url: 'http://my-ghost-blog.com',
mail: {},
database: {
client: 'sqlite3',
connection: {
filename: path.join(__dirname, '/content/data/ghost.db')
},
debug: false
},
server: {
host: '127.0.0.1',
port: '2368'
}
},
// ### Development **(default)**
development: {
// The url to use when providing links to the site, E.g. in RSS and email.
// Change this to your Ghost blog's published URL.
url: 'http://localhost:9000/blog/',
// Example mail config
// Visit http://support.ghost.org/mail for instructions
// ```
// mail: {
// transport: 'SMTP',
// options: {
// service: 'Mailgun',
// auth: {
// user: '', // mailgun username
// pass: '' // mailgun password
// }
// }
// },
// ```
// #### Database
// Ghost supports sqlite3 (default), MySQL & PostgreSQL
database: {
client: 'sqlite3',
connection: {
filename: path.join(__dirname, '../blog/data/ghost-dev.db')
},
debug: false
},
// #### Server
// Can be host & port (default), or socket
server: {
// Host to be passed to node's `net.Server#listen()`
host: '127.0.0.1',
// Port to be passed to node's `net.Server#listen()`, for iisnode set this to `process.env.PORT`
port: '9000'
},
// #### Paths
// Specify where your content directory lives
paths: {
contentPath: path.join(__dirname, '../blog/')
}
},
// **Developers only need to edit below here**
// ### Testing
// Used when developing Ghost to run tests and check the health of Ghost
// Uses a different port number
testing: {
url: 'http://127.0.0.1:2369',
database: {
client: 'sqlite3',
connection: {
filename: path.join(__dirname, '/content/data/ghost-test.db')
}
},
server: {
host: '127.0.0.1',
port: '2369'
},
logging: false
},
// ### Testing MySQL
// Used by Travis - Automated testing run through GitHub
'testing-mysql': {
url: 'http://127.0.0.1:2369',
database: {
client: 'mysql',
connection: {
host : '127.0.0.1',
user : 'root',
password : '',
database : 'ghost_testing',
charset : 'utf8'
}
},
server: {
host: '127.0.0.1',
port: '2369'
},
logging: false
},
// ### Testing pg
// Used by Travis - Automated testing run through GitHub
'testing-pg': {
url: 'http://127.0.0.1:2369',
database: {
client: 'pg',
connection: {
host : '127.0.0.1',
user : 'postgres',
password : '',
database : 'ghost_testing',
charset : 'utf8'
}
},
server: {
host: '127.0.0.1',
port: '2369'
},
logging: false
}
};
module.exports = config;
So basically, when I go to http://localhost:9000/blog, it isn't being rendered at all. Nothing. I was using Chrome and also testing it using Safari. Also tested those two without JavaScript turned on.
And then I try to do curl http://localhost:9000/blog, and try using a requester app (like Postman) and they returned the correct html string. I also tried to do a curl using the user agent as Chrome and as Safari, it also returns the correct html.
I traced down to ghost node_modules, and the renderer is in ghost > core > server > controllers > frontend > index.js in this line res.render(view, result)
I changed the res.render to be like this
res.render(view, result, function(err, string) {
console.log("ERR", err);
console.log("String", string);
res.send(string);
})
and there is no error, it logs the current string, but it doesn't render anything on the browser.
I tried curl, postman, works, but browser doesn't work.
then I tried to send a hello world string, it works, the browser rendered it.
Then I add the string length one by one, and it turns out, any str.length < 1023 will be able to be rendered by the browser, but once it get past that, it doesn't.
And I tried in my parent Express app, it is able to send string which length is more than 1023, and if I use the ghost module as a standalone, it also able to send string more than 1023.
So something must have happened between those two, but I don't know how to debug this.
Please help

Resources