How to serialize jobs in job queue of BullMQ?
I need that the next job is scheduled for execution only when the previous job is finished.
I am using BullMQ 3.
You can use the FlowProducer to create this type of behaviour.
An example would be:
const flowProducer = new FlowProducer({connection: {
host: "host_here",
port: 6379
}});
const jobChain = await flowProducer.add({
name:'example',
data: { field: 'value'},
"queueName",
children: [{
name: 'example2', //can be same/different name
data: {field: 'otherValue'},
"queueName", // can be a different queuename
children: ...
}],
});
Here the the "example" job would only be executed when "example2" is successfully processed. This has to do with both the FlowProducer behaviour & general understanding of the job lifecycle.
Lifecycle info: https://docs.bullmq.io/guide/architecture
Flows info: https://docs.bullmq.io/guide/flows
Related
I have one collection called Audit Trail. This collection is to collect all the sender and admin actions. As for now has more than 50K data
const auditSchema = mongoose.Schema({
who: {
type: String
},
what: {
type: String
},
when: {
type: Date
},
who_id: {
type: Schema.Types.ObjectId
},
who_type: {
type: String
}
});
I notice that some of the admins may accidentally search the audit trail from day one until now. So this will make my server down due to the huge data.
Is that any way to let MongoDB auto stop the query when the query has run more than 1 minute?
So far I have a solution for the frontend. Is that any solution that I can apply in backend?
Node JS: v12.22.5
MongoDB: v3.6.8
maxTimeMS
The maxTimeMS() method sets a time limit for an operation. When the operation reaches the specified time limit, MongoDB interrupts the operation at the next interrupt point.
Terminate a Query
From
mongosh
, use the following method to set a time limit of 60000 milliseconds is equivalent to 1 minute for this query:
db.location.find( { "town": { "$regex": "(Pine Lumber)",
"$options": 'i' } } ).maxTimeMS(60000)
Terminate a Command
Consider a potentially long running operation using distinct to return each distinct collection field that has a city key:
db.runCommand( { distinct: "collection",
key: "city" } )
You can add the maxTimeMS field to the command document to set a time limit of 45 milliseconds for the operation:
db.runCommand( { distinct: "collection",
key: "city",
maxTimeMS: 45 } )
prehistory
I have a nodejs server which instances are running on multiple machines and every instance runs cron job once per day at the same time.
While one instance is running or has just finished its job, another instances should skip executing logic inside of jobs.
I've already had mongodb connection so I decided to save state of runned job and its time to DB and check/change it inside of every job's callback.
The document model I chose to save job state in collection is:
interface JobDao {
_id: ObjectId;
type: string;
state: 'locked' | 'failed' | 'completed';
updatedDate: DateISO;
}
I use package "mongodb": "^3.6.3" to make queries.
After some tries, I wonder if I can implement described bellow behavior or not. Also maybe somebody can suggest another solution to sync running jobs for multiple machines.
So solution I try to implement and I ask for help with:
When cron starts, get job from DB.
check state of job with such conditions:
if it's locked and not expired -> skip logic (note: I use one hour expiration to prevent some unexpected issues when server was broken while running)
if it's locked and expired -> change state of job to locked
if it's not locked but was updated till last 5 minutes -> change state of job to locked
execute logic due to condition above
"unlock" job (update job's state in document)
But here's the issue I met. As there's no concurrency. Between getting and updating the document in one machine, another machines can get or update stale document with not relevant data.
I've tried such solutions as:
findOneAndUpdate
tried to add aggregation (here's the proplem to compare the exipration and it looks to be impossible).
trtransactions
bulk
But nothing had worked. I start thinking to change DB but maybe somebody can say how to implement it with mongodb or can recomend some more suitable solution?
After a small rest I decided to start from the scratch and I finally found a solution.
Here's my code example. It's not perfect, so I'm planning to refactore it, hovever, it works and solves my issue! Hope it'll help sb.
Small description of its logic
The "mediator" is the public method scheduleJob. Logic order:
when we schedule job, it creates new document for the type in DB if it doesn't exist.
unlocks job if it's stale (it's stale if has been locked more than for a half an hour). Server can fall down while running job what cause infinite lock of job but checking stale job should help
next step is locking unlocked job, othervise, finish the logic. It's possible when one instance finishes job just before next instance starts, so I added finishing of the job if the same job was running for last 5 minutes. It's important that such condition restricts frequency as jobs can't bet runned every 5 minutes but in my case it's suitable solution
CronJobDao and CronJobModel are the same and represent the document in DB:
export interface CronJobDao {
type: CronJobTypeEnum;
isLocked: boolean;
updatedAt: Date;
completedAt: Date;
}
Service with scheduleJob method:
import { inject, injectable } from 'inversify';
import { Job, scheduleJob } from 'node-schedule';
import { CronJobTypeEnum } from '../core/enums/cron-job-type.enum';
import { CronJobRepository } from './cron-job.repository';
#injectable()
export class CronJobService {
readonly halfHourMs = 30 * 60 * 1000;
readonly fiveMinutesMs = 5 * 60 * 1000;
constructor(
#inject(CronJobRepository) private cronJobRepository: CronJobRepository,
) {}
scheduleJob(type: CronJobTypeEnum, timeRule: string, callback: Function): Job {
this.cronJobRepository.registerJob(type).then();
return scheduleJob(
type,
timeRule,
async () => {
await this.unlockStaleJob(type);
const lockedJob = await this.cronJobRepository.lockJob(type);
if (!lockedJob) {
console.warn('Job has already been locked');
return;
}
if ((new Date().getTime() - lockedJob.completedAt?.getTime()) < this.fiveMinutesMs) {
await this.cronJobRepository.unlockJob(type);
console.warn('Job has recently been completed');
return;
}
console.info('Job is locked');
callback();
await this.cronJobRepository.completeJob(type);
console.info('Job is completed');
},
);
}
private async unlockStaleJob(type: CronJobTypeEnum): Promise<void> {
const staleJob = await this.cronJobRepository.unlockIfTimeExpired(type, this.halfHourMs);
if (!staleJob) {
return;
}
console.warn('Has stale job: ', JSON.stringify(staleJob));
}
}
Class for communication with DB:
import { inject, injectable } from 'inversify';
import { Db } from 'mongodb';
import { CronJobDao, mapCronJobDaoToModel } from '../core/daos/cron-job.dao';
import { CronJobTypeEnum } from '../core/enums/cron-job-type.enum';
import { CronJobModel } from '../core/models/cron-job.model';
import { AbstractRepository } from '../core/utils/abstract.repository';
#injectable()
export class CronJobRepository extends AbstractRepository<CronJobDao> {
constructor(#inject(Db) db: Db) {
super(db, 'cron_jobs');
}
async registerJob(type: CronJobTypeEnum) {
const result = await this.collection.findOneAndUpdate(
{ type },
{
$setOnInsert: {
type,
isLocked: false,
updatedAt: new Date(),
},
},
{ upsert: true, returnOriginal: false },
);
return result.value;
}
async unlockIfTimeExpired(type: CronJobTypeEnum, expiredFromMs: number): Promise<CronJobModel | null> {
const expirationDate = new Date(new Date().getTime() - expiredFromMs);
const result = await this.collection.findOneAndUpdate(
{
type,
isLocked: true,
updatedAt: { $lt: expirationDate },
},
{
$set: {
updatedAt: new Date(),
isLocked: false,
},
});
return result.value ? mapCronJobDaoToModel(result.value) : null;
}
async lockJob(type: CronJobTypeEnum) {
return this.toggleJobLock(type, false);
}
async unlockJob(type: CronJobTypeEnum) {
return this.toggleJobLock(type, true);
}
private async toggleJobLock(type: CronJobTypeEnum, stateForToggle: boolean): Promise<CronJobModel | null> {
const result = await this.collection.findOneAndUpdate(
{
type,
isLocked: stateForToggle,
},
{
$set: {
isLocked: !stateForToggle,
updatedAt: new Date(),
},
},
);
return result.value ? mapCronJobDaoToModel(result.value) : null;
}
async completeJob(type: CronJobTypeEnum): Promise<CronJobModel | null> {
const currentDate = new Date();
const result = await this.collection.findOneAndUpdate(
{
type,
isLocked: true,
},
{
$set: {
isLocked: false,
updatedAt: currentDate,
completedAt: currentDate,
},
},
);
return result.value ? mapCronJobDaoToModel(result.value) : null;
}
}
i would recommend you to use some locking mechanism to sync between multiple services.
you can use basic mutex in case you want that only one service will write/read in your critical section.
im not sure exactly what you want in case one service trying to read while other service performing some changes (wait, skip or something else).
you can use some shared component such as redis to store the locking key.
When I create a topic using the sns.createTopic (like the code below) it won't accept the booleans and say 'InvalidParameterType: Expected params.Attributes['FifoTopic'] to be a string', even though the docs say to provide boolean value, and when I provide it with a string of 'true' it still doesn't set the topic type to be FIFO, anyone knows why?
Here's the code:
const TOPIC = {
Name: 'test.fifo',
Attributes: {
FifoTopic: true,
ContentBasedDeduplication: true
},
Tags: [{
Key: 'test-key',
Value: 'test-value'
}]
};
sns.createTopic(TOPIC).promise().then(console.log);
Used aws-sdk V2
I sent FifoTopic and ContentBasedDeduplication as strings.
The below code works fine for me
const TOPIC = {
Name: 'test.fifo',
Attributes: {
FifoTopic: "true",
ContentBasedDeduplication: "true"
},
Tags: [{
Key: 'test-key',
Value: 'test-value'
}]
};
let sns = new AWS.SNS();
let response3 =await sns.createTopic(TOPIC).promise();
console.log(response3);
Note: Make sure your lambda has correct permissions.
You will be getting attributes like FifoTopic and ContentBasedDeduplication when performing the getTopicAttributes.
let respo = await sns.getTopicAttributes({
TopicArn:"arn:aws:sns:us-east-1:XXXXXXX:test.fifo"}
).promise();
please find the screenshot
I have afterSubmit function that I wrote that will iterate through all related transactions connected with the same CTG_ID value (which is a custom parent record) and the script will actually update just one field on all of these values.
My problem is because this is a really slow method, more transactions I have connected to the same parent more time the user needs to wait after clicking the "Save" button. Script execution time is terrible.
Is there any faster / better way to update a certain field on a group of records?
My function for updating those transactions:
function afterSubmit(context) {
const newRecord = context.newRecord;
const ctgId = newRecord.getValue({ fieldId: 'custbody_ctgid' });
const currentCustomerPo = newRecord.getValue({ fieldId: 'custbodyctg_customer_po'})
search.create({
type: 'transaction',
filters: [['custbody_ctgid', 'anyof', ctgId],
"AND",
['mainline','is','T']],
columns: ['custbodyctg_customer_po']
}).run().getRange({start: 0, end:100}).forEach((result,line) => {
const ctgPo = result.getValue('custbodyctg_customer_po') as string;
const recType = result.recordType;
const recId = result.id;
let rec = record.load({
type: recType,
id: recId,
isDynamic: true
})
rec.setValue({
fieldId: 'custbodyctg_customer_po',
value: currentCustomerPo,
ignoreFieldChange: true
})
rec.save();
})
}
Thanks to Brian Duffy's answer, this is working a lot better!
I modified the script so now I iterate through results with each instead of forEach function. I'm using record.submitFields function instead of record.load
function afterSubmit(context) {
const newRecord = context.newRecord;
const oldRecord = context.oldRecord;
const ctgId = newRecord.getValue({fieldId: 'custbody_ctgid'});
const currentCustomerPo = newRecord.getValue({fieldId: 'custbodyctg_customer_po'})
const oldCustomerPo = oldRecord.getValue({fieldId: 'custbodyctg_customer_po'})
if (oldCustomerPo !== currentCustomerPo) {
search.create({
type: 'transaction',
filters: [['custbody_ctgid', 'anyof', ctgId],
"AND",
['mainline', 'is', 'T']],
columns: ['custbodyctg_customer_po', 'type']
}).run().each((result) => {
if (result.recordType !== 'salesorder') {
const recType = result.recordType;
const recId = result.id;
record.submitFields({
type: recType,
id: recId,
values: {
custbodyctg_customer_po: currentCustomerPo
},
options: {
enableSourcing: false,
ignoreMandatoryFields: true
}
});
}
return true;
})
}
}
Still, after testing few transactions with an average of 5 linked transactions to them this is running like 5-7 seconds. Stil slow for me. If anyone has suggestions it would be AWESOME!
I would try using run.each instead of getRange for your search results and use record.sumbitFields instead of loading and saving the record.
If speed of saving the current record is key, consider making your logic asynchronous. The current record will save nearly as fast as normal, just taking the time to schedule a Map/Reduce or Scheduled script.
The user event script would just send as parameters the current record's Id, the value in CTG_ID and the value in custbodyctg_customer_po. The asynchronous script would search for all tranx with that same Id except for the one that triggered it (or skip any records with a matching value while looping thru results), then submit the custbodyctg_customer_po for each of those.
It's a heavier solution, so you must weigh priorities.
Alternatively, look into record.submitFields.promise(options). It seems a limitation is it cannot update Select fields (ie List/Record)
Lets say we scripted the scenarios following way for our evolving servers
Actual server v1 response
response = { id: "1", name: "karate" }
Mocking client v1 schema
schema = { id: "#string", name: "#string }
* match response == schema
Actual server v2 response
response = { id: "2", name: "karate", value: "is" }
Mocking client v2 schema
schema = { id: "#string", name: "#string, value: "#string" }
* match response == schema
Actual server v3 response
response = { id: "3", name: "karate", value: "is", description: "easy" }
Mocking client v3 schema
schema = { id: "#string", name: "#string", value: "#string", description: "#string" }
* match response == schema
Similarly for backward compatibility testing of our evolving servers, we script the scenarios following way
Actual server v3 response
response = { id: "3", name: "karate", value: "is", description: "easy" }
Mocking client v1 schema
schema = { id: "#string", name: "#string }
* match response contains schema
Actual server v2 response
response = { id: "2", name: "karate", value: "is" }
Mocking client v1 schema
schema = { id: "#string", name: "#string }
* match response contains schema
Actual server v1 response
response = { id: "1", name: "karate" }
Mocking client v1 schema
schema = { id: "#string", name: "#string }
* match response contains schema
Proposal is to be able to use some kind of flag in match statement that dynamically decides the kind of match we do during testing.
Lets say that the name of flag is SOMEFLAG and we provide the kind of match we want to do during testing (set in karate-config.js for global effect)
var SOMEFLAG = "contains";
OR
var SOMEFLAG = "==";
Now in scenario we do following
# Depending on what is set in karate-config.js, it will use either contains or == for verification.
* match response SOMEFLAG schema
Is it possible to do this in karate ?
Also note that success of this idea really depends on https://github.com/intuit/karate/issues/826 due to ability match nested object using contains match.
Personally, I am strongly against this idea because it will make your tests less readable. It is a slippery slope once you start this. For an example of what happens when you attempt too much re-use (yes, re-use can be a bad thing in test automation, and I really don't care if you disagree :) - see this: https://stackoverflow.com/a/54126724/143475
What I would do is something like this:
* def lookup =
"""
{
dev: { id: "#string", name: "#string },
stage: { id: "#string", name: "#string, value: "#string" },
preprod: { id: "#string", name: "#string", value: "#string", description: "#string" }
}
"""
* def expected = lookup[karate.env]
* match response == expected
EDIT - I have a feeling that the change we made after this discussion will also solve your problem - or at least give you some new ideas: https://github.com/intuit/karate/issues/810