This question already has answers here:
Schedule query in BigQuery
(3 answers)
Closed 4 years ago.
What I want to do is run a daily query and insert that daily data into another table.
I am a beginner to big query and the jobs part confuses me a bit since it cannot be done through the GUI.
I read here that it should be done programmatically.
https://cloud.google.com/bigquery/docs/jobs-overview
The problem here is that this all looks like one time jobs which cannot be automatically rescheduled. Is there any way to do this in nodejs or do I need to setup a chronjob or something to use for scheduling?
Does anybody have a good example or some pointers to get me started correctly.
Thanks
You can schedule a job in node with something like this:
node-cron
For this the node server will have to be running all time so that it can execute the job at the required time.
Alternatively you can create a cron job that calls a script which would perform the job.
Related
This question already has answers here:
Run a cron job only once after a set time
(2 answers)
Firebase Functions, run after 15 minutes of user inactivity
(1 answer)
Run a Cron Job every 30mins after onCreate Firestore event
(3 answers)
How to run a background Cloud Function and schedule a change in database?
(1 answer)
Is it ok to use setTimeout in Cloud Functions?
(1 answer)
Closed 10 months ago.
I've searched the internet but haven't found anything that works for me.
I describe what I need.
I have a function in firebase that is executed every time I add a new node to my Realtime Database
exports.makeMagic = functions.database.ref("/users/{uid}")
.onCreate((snap, context) => {
const iterate = snap.val();
if(iterate1.loops <= 5){
// run 5 minutes later
myFunction(iterate1,iterate2,iterate3);
}
});
What I need is for myFunction to make a change to the database. But it must be strictly 5 minutes after the node is created. Not before and not after. I also don't want a cron job that is always running checking.
I have also read about using settimeout
But this would block my "makeMagic" function, until the 5 minutes were up. And I think maximum can last 1 minute, I'm not sure about that.
Any help would be nice to keep trying
Thank you so much!
You can use Cloud Tasks, as explained in this article.
You will need to write another Cloud Function, of type HTTPS, which implements the desired business logic.
I have a DAG with one task that fetches the data from the API. I want that task to fetch the data only for certain time interval and marks itself as SUCCESS so that the tasks after that starts running.
Please note that the tasks below are dependent on the tasks which I want to mark SUCCESS. I know I can mark the task SUCCESS manually from CLI or UI but I want to do it automatically.
Is it possible to do that programmatically using python in Airflow?
You can set status of task using python code, like this:
def set_task_status(**kwargs):
execution_date = kwargs['execution_date']
ti = TaskInstance(HiveOperatorTest, execution_date)
ti.set_state(State.SUCCESS)
How to get list of next materialized times for oozie coordinator if we have scheduled it frequency using CRON.
In HUE i can see the next materialized time only, I want a list of all the time it will run.
Thank you.
Oozie doesn't give you that list nor provide tools to calculate it.
I usually convert my expression CRON syntax and check with this http://www.cronmaker.com/. This is not a solid answer to your question but at least gives you the execution frequency.
rest syntax
curl http://www.cronmaker.com/rest/sampler?expression={expression}
sample with rest api
http://www.cronmaker.com/rest/sampler?expression=0%200/2%20*%201/1%20*%20?%20*
I am planning for a nodeJS Application where the server has to handle time-based jobs. For example:
Job 1: Every 2 hours
Job 2: Every 5 minutes
Job 3: Every 10 days
...
These jobs are saved in a database and there will be a lot of jobs - it could be a millions of these.
So now is my question which is the right way in nodeJS to handle this ? There are a lot cron packages for cron like processes but I thought its maybe "to much" for this.
I was thinking about .... Timeouts! Right - timeouts.
This is the schema I was thinking of:
var job = getJob();
var executeInMs = getMSUntilExecute(job);
setTimeout(function(){handle();}, executeInMs);
With this schema it really could be that there are a million timeouts running at the same time in my application.
The question now is: Is this a Server-Killer or is this the right way to handle this ?
My objective is, I need to get the current timestamp using Syncsort if one OPC job(existing Job) run fine in production. In my case I can not interpret my new job after existing OPC job. Is there any facility to check the existing job ran fine in production ?
I mean any reference table to have production job details with status for each day ?
Please help anyone to move.
There are commercial packages that track jobs and job status. CA (computer associates) is one such vendor.
However, these packages cost a lot. A simple, home grown solution, is to have a dataset known to both jobs and write a one line record into that data set when job1 completes and the second job2 can read the dataset to "KNOW" if the job ran. IF this is what you are trying to do, it is not exactly clear from your question. But any solution along these lines works, until management wants to cough up $50K (or whatever) for a commercial package.