I'm trying to schedule some job on nodejs using node-sched (I've also tried node-cron-tz which supports timezones), on my current dev machine it works correctly and works fine.. on customer test/prod env it executes it 2 hours later.
I need to perform this job at the time present on a DB and it's normally 0 8 * * *
The code I used in my latest test is
scheduleService.addSqlTask = function (item) {
if (item == null)
return;
logger.info('Adding job {Item} to schedule', {Item:item.stored_procedure});
//logger.info("Adding {Item} : {Cron} to tasks", { Item: item.stored_procedure, Cron : item.cron_tab });
cron.schedule(item.cron_tab,
async () => {
try {
logger.info("Executing {Item} to schedule", { Item: item.stored_procedure });
await dataStore.executeSql(item);
} catch (e) {
logger.error(e.message);
}
},
{
scheduled: true,
timezone: "Europe/Rome"
}
);
}
In this case, it works wrong even on my machine since it doesn't consider saving light...
How can I handle this?
Related
I know we can retry failed jobs in Bull using the backoff attribute in Bull like below
var Queue = require('bull');
var myQueue = new Queue('myQueue');
runQueryQueue
.add(data, {
jobId: uid,
attempts: 10,
backoff: {
type: 'exponential',
delay: 2000,
},
stackTraceLimit: 100,
})
.then((result) => {})
.catch(console.log);
But I want to retry only when the failure reasons are worth retrying - example timeouts, rate limits, etc and not retry when the errors are related to user inputs, etc.
How can I check the failure error message and decide to retry?
You can mark a job as complete, e.g. do not throw errors, but write error data to a result of job completion. Result can be processed different way, if there is any sign of error.
async function jobProcess(job) {
if (doNotRetryError) {
return doNotRetryError
} else if (anyOtherError) {
throw new Error('retry')
} else {
return {success: true}
}
}
async function jobCompleted(job, result) {
if (result instanceof Error) {
// some error happened, but job shouldn't be retried
} else {
// job completed
}
}
Keep setting your backoff config. If the job is not worth retrying, set opts.attempts to attemptsMade + 1 or smaller inside the processor.
Your changed opts will not be saved, but your job will be moved to the failed list instantly.
async process(job: Job): Promise<void> {
try {
await doStuff(job);
} catch (error) {
if (!isWorthRetrying(error)) {
// Move job to failed list instantly by setting attemps to attemptsMade + 1 or smaller
job.opts.attempts = job.attemptsMade + 1;
}
throw error;
}
}
This answer is for bull-mq so would be useful if you're willing to upgrade.
There is now an UnrecoverableError you can throw and bull-mq will take care of not retying the job.
https://docs.bullmq.io/guide/retrying-failing-jobs#stop-retrying-jobs
bull has a job.discard() function that should allow you to mark a job as non-retriable in the process.
I'm writing a Windows Node.js server app (using ES6 btw).
The first thing I want to do - in the top-level code - is sit in a while loop, calling an async function which searches for a particular registry key/value. This function is 'proven' - it returns the value data if found, or else throws:
async GetRegValue(): Promise<string> { ... }
I need to sit in a while loop until the registry item exists, and then grab the value data. (With a delay between retries).
I think I know how to wait for an async call to complete (one way or the other) before progressing with the rest of the start-up, but I can't figure out how to sit in a loop waiting for it to succeed.
Any advice please on how to achieve this?
(I'm fairly new to typescript, and still struggling to get my head round all async/await scenarios!)
Thanks
EDIT
Thanks guys. I know I was 'vague' about my code - I didn't want to put my real/psuedo code attempts, since they have all probably overlooked the points you can hopefully help me understand.
So I just kept it as a textual description... I'll try though:
async GetRegValue(): Promise<string> {
const val: RegistryItem = await this.GetKeyValue(this.KEY_SW, this.VAL_CONN);
return val.value
}
private async GetKeyValue(key: string, name: string): Promise<RegistryItem> {
return await new Promise((resolve, reject) => {
new this.Registry({
hive: this.Hive, key
}).get(name, (err, items) => {
if (err) {
reject(new Error('Registry get failed'));
}
else {
resolve( items );
}
});
})
.catch(err => { throw err });
}
So I want to do something like:
let keyObtained = false
let val
while (keyObtained == false)
{
// Call GetRegValue until val returned, in which case break from loop
// If exception then pause (e.g. ~100ms), then loop again
}
}
// Don't execute here till while loop has exited
// Then use 'val' for the subsequent statements
As I say, GetRegValue() works fine in other places I use it, but here I'm trying to pause further execution (and retry) until it does come back with a value
You can probably just use recursion. Here is an example on how you can keep calling the GetRegValue function until is resolves using the retryReg function below.
If the catch case is hit, it will just call GetRegValue over and over until it resolves successfully.
you should add a counter in the catch() where if you tried x amount of times you give up.
Keep in mind I mocked the whole GetRegValue function, but given what you stated this would still work for you.
let test = 0;
function GetRegValue() {
return new Promise((resolve, reject) => {
setTimeout(function() {
test++;
if (test === 4) {
return resolve({
reg: "reg value"
});
}
reject({
msg: "not ready"
});
}, 1000);
});
}
function retryReg() {
GetRegValue()
.then(registryObj => {
console.log(`got registry obj: ${JSON.stringify(registryObj)}`)
})
.catch(fail => {
console.log(`registry object is not ready: ${JSON.stringify(fail)}`);
retryReg();
});
}
retryReg();
I don't see why you need this line:
.catch(err => { throw err });
The loop condition of while isn't much use in this case, as you don't really need a state variable or expression to determine if the loop should continue:
let val;
while (true)
{
try {
val = await GetRegValue(/* args */);
break;
} catch (x) {
console.log(x); // or something better
}
await delay(100);
}
If the assignment to val succeeds, we make it to the break; statement and so we leave the loop successfully. Otherwise we jump to the catch block and log the error, wait 100 ms and try again.
It might be better to use a for loop and so set a sensible limit on how many times to retry.
Note that delay is available in an npm package of the same name. It's roughly the same as:
await new Promise(res => setTimeout(res, 100));
I took one of the sample functions from the Firestore documentation and was able to successfully run it from my local firebase environment. However, once I deployed to my firebase server, the function completes, but no entries are made in the firestore database. The firebase function logs show "Deadline Exceeded." I'm a bit baffled. Anyone know why this is happening and how to resolve this?
Here is the sample function:
exports.testingFunction = functions.https.onRequest((request, response) => {
var data = {
name: 'Los Angeles',
state: 'CA',
country: 'USA'
};
// Add a new document in collection "cities" with ID 'DC'
var db = admin.firestore();
var setDoc = db.collection('cities').doc('LA').set(data);
response.status(200).send();
});
Firestore has limits.
Probably “Deadline Exceeded” happens because of its limits.
See this. https://firebase.google.com/docs/firestore/quotas
Maximum write rate to a document 1 per second
https://groups.google.com/forum/#!msg/google-cloud-firestore-discuss/tGaZpTWQ7tQ/NdaDGRAzBgAJ
In my own experience, this problem can also happen when you try to write documents using a bad internet connection.
I use a solution similar to Jurgen's suggestion to insert documents in batch smaller than 500 at once, and this error appears if I'm using a not so stable wifi connection. When I plug in the cable, the same script with the same data runs without errors.
I have written this little script which uses batch writes (max 500) and only write one batch after the other.
use it by first creating a batchWorker let batch: any = new FbBatchWorker(db);
Then add anything to the worker batch.set(ref.doc(docId), MyObject);. And finish it via batch.commit().
The api is the same as for the normal Firestore Batch (https://firebase.google.com/docs/firestore/manage-data/transactions#batched-writes) However, currently it only supports set.
import { firestore } from "firebase-admin";
class FBWorker {
callback: Function;
constructor(callback: Function) {
this.callback = callback;
}
work(data: {
type: "SET" | "DELETE";
ref: FirebaseFirestore.DocumentReference;
data?: any;
options?: FirebaseFirestore.SetOptions;
}) {
if (data.type === "SET") {
// tslint:disable-next-line: no-floating-promises
data.ref.set(data.data, data.options).then(() => {
this.callback();
});
} else if (data.type === "DELETE") {
// tslint:disable-next-line: no-floating-promises
data.ref.delete().then(() => {
this.callback();
});
} else {
this.callback();
}
}
}
export class FbBatchWorker {
db: firestore.Firestore;
batchList2: {
type: "SET" | "DELETE";
ref: FirebaseFirestore.DocumentReference;
data?: any;
options?: FirebaseFirestore.SetOptions;
}[] = [];
elemCount: number = 0;
private _maxBatchSize: number = 490;
public get maxBatchSize(): number {
return this._maxBatchSize;
}
public set maxBatchSize(size: number) {
if (size < 1) {
throw new Error("Size must be positive");
}
if (size > 490) {
throw new Error("Size must not be larger then 490");
}
this._maxBatchSize = size;
}
constructor(db: firestore.Firestore) {
this.db = db;
}
async commit(): Promise<any> {
const workerProms: Promise<any>[] = [];
const maxWorker = this.batchList2.length > this.maxBatchSize ? this.maxBatchSize : this.batchList2.length;
for (let w = 0; w < maxWorker; w++) {
workerProms.push(
new Promise((resolve) => {
const A = new FBWorker(() => {
if (this.batchList2.length > 0) {
A.work(this.batchList2.pop());
} else {
resolve();
}
});
// tslint:disable-next-line: no-floating-promises
A.work(this.batchList2.pop());
}),
);
}
return Promise.all(workerProms);
}
set(dbref: FirebaseFirestore.DocumentReference, data: any, options?: FirebaseFirestore.SetOptions): void {
this.batchList2.push({
type: "SET",
ref: dbref,
data,
options,
});
}
delete(dbref: FirebaseFirestore.DocumentReference) {
this.batchList2.push({
type: "DELETE",
ref: dbref,
});
}
}
I tested this, by having 15 concurrent AWS Lambda functions writing 10,000 requests into the database into different collections / documents milliseconds part. I did not get the DEADLINE_EXCEEDED error.
Please see the documentation on firebase.
'deadline-exceeded': Deadline expired before operation could complete. For operations that change the state of the system, this error may be returned even if the operation has completed successfully. For example, a successful response from a server could have been delayed long enough for the deadline to expire.
In our case we are writing a small amount of data and it works most of the time but loosing data is unacceptable. I have not concluded why Firestore fails to write in simple small bits of data.
SOLUTION:
I am using an AWS Lambda function that uses an SQS event trigger.
# This function receives requests from the queue and handles them
# by persisting the survey answers for the respective users.
QuizAnswerQueueReceiver:
handler: app/lambdas/quizAnswerQueueReceiver.handler
timeout: 180 # The SQS visibility timeout should always be greater than the Lambda function’s timeout.
reservedConcurrency: 1 # optional, reserved concurrency limit for this function. By default, AWS uses account concurrency limit
events:
- sqs:
batchSize: 10 # Wait for 10 messages before processing.
maximumBatchingWindow: 60 # The maximum amount of time in seconds to gather records before invoking the function
arn:
Fn::GetAtt:
- SurveyAnswerReceiverQueue
- Arn
environment:
NODE_ENV: ${self:custom.myStage}
I am using a dead letter queue connected to my main queue for failed events.
Resources:
QuizAnswerReceiverQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:provider.environment.QUIZ_ANSWER_RECEIVER_QUEUE}
# VisibilityTimeout MUST be greater than the lambda functions timeout https://lumigo.io/blog/sqs-and-lambda-the-missing-guide-on-failure-modes/
# The length of time during which a message will be unavailable after a message is delivered from the queue.
# This blocks other components from receiving the same message and gives the initial component time to process and delete the message from the queue.
VisibilityTimeout: 900 # The SQS visibility timeout should always be greater than the Lambda function’s timeout.
# The number of seconds that Amazon SQS retains a message. You can specify an integer value from 60 seconds (1 minute) to 1,209,600 seconds (14 days).
MessageRetentionPeriod: 345600 # The number of seconds that Amazon SQS retains a message.
RedrivePolicy:
deadLetterTargetArn:
"Fn::GetAtt":
- QuizAnswerReceiverQueueDLQ
- Arn
maxReceiveCount: 5 # The number of times a message is delivered to the source queue before being moved to the dead-letter queue.
QuizAnswerReceiverQueueDLQ:
Type: "AWS::SQS::Queue"
Properties:
QueueName: "${self:provider.environment.QUIZ_ANSWER_RECEIVER_QUEUE}DLQ"
MessageRetentionPeriod: 1209600 # 14 days in seconds
If the error is generate after around 10 seconds, probably it's not your internet connetion, it might be that your functions are not returning any promise. In my experience I got the error simply because I had wrapped a firebase set operation(which returns a promise) inside another promise.
You can do this
return db.collection("COL_NAME").doc("DOC_NAME").set(attribs).then(ref => {
var SuccessResponse = {
"code": "200"
}
var resp = JSON.stringify(SuccessResponse);
return resp;
}).catch(err => {
console.log('Quiz Error OCCURED ', err);
var FailureResponse = {
"code": "400",
}
var resp = JSON.stringify(FailureResponse);
return resp;
});
instead of
return new Promise((resolve,reject)=>{
db.collection("COL_NAME").doc("DOC_NAME").set(attribs).then(ref => {
var SuccessResponse = {
"code": "200"
}
var resp = JSON.stringify(SuccessResponse);
return resp;
}).catch(err => {
console.log('Quiz Error OCCURED ', err);
var FailureResponse = {
"code": "400",
}
var resp = JSON.stringify(FailureResponse);
return resp;
});
});
I'm not sure of how to adequately achieve my desired control flow using promises/bluebird.
Essentially I have a database with X 'tasks' stored and each needs to be loaded and executed sequentially. I don't want to run more than one task concurrently and the entire code must continue executing indefinitely.
I have achieved this with the following code so far:
export default function syncLoop() {
getNextTaskRunner().then((taskRunner) => {
if (taskRunner) {
taskRunner.startTask()
.then(syncLoop)
.catch((error) => {
throw new Error(error);
});
} else {
syncLoop();
}
});
}
getNextTaskRunner() simply loads and resolves with the next task from the database (calc'd based on timestamps). Or it resolves with null (no task avail).
taskRunner.startTask() resolves with null when the full task has completed.
I've been advised that the way it is structured (recursive /w promises) could lead to stack issues after it has been running for some time.
What I've thought about doing is to restructure it to something like:
let running = false;
setInterval(() => {
if (!running) {
running = true;
getNextTaskRunner().then((taskRunner) => {
if (taskRunner) {
taskRunner.startTask()
.then(() => {
running = false;
})
.catch((error) => {
log.error(error);
});
} else {
running = false;
}
});
}
}, 5000);
Or as yet another possibility, using event emitters in some form?
task.on('complete', nextTask());
Thoughts and advice will be greatly appreciated!
What stack issues? The way you've written your code is perfectly fine as long as getNextTaskRunner is truly async (i.e. it gives control back to the main loop at some point, e.g. if it does async io). There is no recursion in your code in that case. Whoever told you that is mistaken.
Though you might want to add a setTimeout somewhere so you won't flood your db with requests. Plus it will help you if getNextTaskRunner will no longer be sync (due to for example in memory caching):
export default function syncLoop() {
setTimeout(() => {
getNextTaskRunner().then((taskRunner) => {
if (taskRunner) {
taskRunner.startTask()
.then(syncLoop)
.catch((error) => {
throw new Error(error);
});
} else {
syncLoop();
}
});
}, 2000);
}
has anyone integrated QUnit tests with ZombieJS? I have a script so i want to pass a "tests.html" file and poll until the tests complete, then read the results. Similar to what i'm doing with PhantomJS and it works perfectly fine. I'd mostly like to compare Phantom and Zombie with respect to performance. Also we have a lot of Qunit tests already created so i don't want to just dump them and rewrite everything from scratch in the Zombie environment (if i decide to go for it in the first place :)
the issue i am facing is that my tests never complete, so Qunit is always in running state. Haven't debugged anything in detail yet, but just wanted to make sure i am not missing something obvious.
var Browser = require("zombie");
var assert = require("assert");
// Load the page from localhost
browser = new Browser();
function waitFor(testFx, onReady, timeOutMillis) {
var maxtimeOutMillis = timeOutMillis ? timeOutMillis : 100000, //< Default Max Timout is 3s
start = new Date().getTime(),
condition = false,
interval = setInterval(function() {
if ( (new Date().getTime() - start < maxtimeOutMillis) && !condition ) {
// If not time-out yet and condition not yet fulfilled
condition = (typeof(testFx) === "string" ? eval(testFx) : testFx()); //< defensive code
} else {
if(!condition) {
// If condition still not fulfilled (timeout but condition is 'false')
console.log("'waitFor()' timeout");
process.exit(1);
} else {
// Condition fulfilled (timeout and/or condition is 'true')
console.log("'waitFor()' finished in " + (new Date().getTime() - start) + "ms.");
typeof(onReady) === "string" ? eval(onReady) : onReady(); //< Do what it's supposed to do once the condition is fulfilled
clearInterval(interval); //< Stop this interval
}
}
}, 100); //< repeat check every 250ms
};
browser.visit("tests.html", function () {
waitFor(function(){
return browser.evaluate(function(){
var el = browser.document.getElementById('qunit-testresult');
if (el && el.textContent.match('completed')) {
return true;
}
return false;
});
}, function(){
var failedNum = browser.evaluate(function(){
var el = browser.document.getElementById('qunit-testresult');
console.log(el.textContent);
try {
return el.getElementsByClassName('failed')[0].innerHTML;
} catch (e) { }
return 10000;
});
process.exit((parseInt(failedNum, 10) > 0) ? 1 : 0);
});
});