I have job(script) which is written in nodejs.
I have another API which writes data (Id and time-t1) to the Cloud Spanner. As soon as the API is a hit I want to run the same job at given time-t1 and pass id as parameter
Can I write some code in my API which will trigger the job at a given time (Note - for a single hit on API job should run job only once). I tried searching on the net but could only find periodic scheduler.
In order to schedule a task for a specific dynamic time you can use Google Cloud Task and Google Cloud Functions
Read it here:
https://cloud.google.com/tasks/docs/tutorial-gcf
Related
I am new to cloud and Azure, so I do not know how logical my thoughts seem. Please tell me if there are better approaches to solve the issue.
There is a SharePoint list of jobs with their corresponding Cron expressions. The jobs all execute same tasks but with different parameters and schedules.
What already is done is a logic app that is scheduled to run every 5 minutes to scan the list and run the jobs which match the current timestamp but I do not find it really logical.
I was thinking of something like having a script which creates separate jobs for each list item (like dynamic DAG creation in Airflow based on a template); however, I cannot find any resources to learn how to do it.
Can I create separate logic apps by using JSON representations of tasks for each item in SharePoint list with their corresponding parameters and deploying them on Azure using a Python script?
Below is something that you can follow to achieve your requirement, I have created the same flow as yours over logic apps expect for the part where I have added Azure function which creates separate jobs for each list item through HTTP trigger. Below is the flow of my logic app.
I'm using the below sample code in my Function.
import logging
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
body = req.params.get('name')
return func.HttpResponse(f"{body}")
However, you can write your custom code inside it.
IN SHAREPOINT FILE
RESULTS:
I'm guessing this has been asked before, but I haven't been able to find the answer... Is there a way to look at a trigger to see which functions are being started by it?
We are building functions that will be run on different timed triggers. We'd like to be able to see which ones set to the hourly trigger or the daily trigger (as an example).
We were finally able to find an answer to this question... Inside your DataFactory...
Go to Manage
Select Triggers
View the code for the trigger
The associated functions are shown under the pipelines as pipelineReferences
Timer Trigger function is a non Http-Triggered Function & lets you run a function on a schedule based on the CORN Expression.
You can look at the CRON expression which was declared in the function.json file of a particular function.
You can also refer the below documentation to get more understanding about the corn expressions.
Alternatively, you can look at the logs of the function app based on the logs you validate which function is running at hourly basis & daily basis.
You can refer the below SO thread for more information about how to fetch the logs of a particular function under functionapp.
I am totally new to Lambda (or AWS) and am still to build knowledge and experience around it.
Now, I was building an app where in it requires to fetch data from twitter Hashtag.
If I got it correctly, Twitter restricts number of API calls we make every minute(?) hence we need to have a backend and needs to have oAuth2 authentication.
In a simple express app, I would have done an API call in the global scope to get the data and use setInterval to hit that API after every x minute so as to not exceed number of limits.
Now based on the very vague understanding, I guess Lambda runs function when we need it, Hence is it right to assume that we can't use lambda for such use cases?
The old-school way of doing this is to run a cron job that fires a particular script every so often. The AWS way of running code periodically is using CloudWatch scheduled events. You can configure how often you want to run a given target, and set the target as a lambda function.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
I want to build a nodejs application to scrape data from a website every 20mins and store it in firebase. Can you please tell me which product of google( compute engine, app engine or cloud functions ) is effective for this requirement as below are the things i am expecting to do,
1. Run Nodejs, cheerio to scrape data from website and store in firebase
2. Schedule it to run 20mins initially later may change it to 30mins or 1hr.
After reading the docs, i know that there are too many ways to implement this, but i am looking for a cost/resource effective way.
Pointers and ideas would be good.
Host the Node.js application within the App Engine[1] as Cloud Functions are event-driven[2]. You can use App Engine standard[3] or App Engine flexible[4] environment. For the scheduling part, Google Cloud Platform has a Cron Service[5] and you can create a cron job for your task hitting App Engine[6]. You can find a sample design here[7].
It depends on how much time your script spends waiting on requests. During that time the script is idle but you're getting charged at a super-high rate.
If you're doing a lot of concurrency then I would say do it with cloud functions.Another pro of doing it that way is your ip won't get blocked because it will be different every time.
Regarding scheduling, I'm not sure if Google lets do that, but I know AWS does.
A cost effective/simple way would be to use cronjob.org and have it send an http request to your cloud functions url to trigger it. If you're worried about other people triggering it, tell your cronjob to send an http header w/ an api key. Check this api key in your cloud function code to verify cronjob.org sent the request. I don't think it gets any more easy/cheap than this.
Is it possible to use dynamic content in the POST body for a scheduled job in Azure scheduler?
I am writing a logic app that I would like to be able to pass a start time and a look back minute count to so that a failed invocation can be re-run across the same underlying data by passing the same parameters. I'm wondering if there are functions or operations similar to what can be found in logic apps for values such as utcNow()
We do not support dynamic content in Scheduler, you may find some timestamp in the request header in the calls Scheduler made though.
Why are you not using Logic Apps when it can perform what you need?