Hi i am having an application which needs to be scheduled in order to perform a task continuously, but i am doing the development on IBM Bluemix cloud and after like 10 days of research i have not been able to find a correct solution to implement a QUARTZ or CRON scheduler in bluemix, in a web application.
whatever i have found out is a service inside the BLuemix called as Workload scheduler but no success has been obtained so far from the steps mentioned there.
Secondly i have found a blog where there are some steps posted to implement scheduler in bluemix, but no success from that as well, the link is mentioned below.
Link: http://sureshgarrepalli.blogspot.in/2015_08_01_archive.html
If anyone here can help me regarding this would be a lot of help. Thanks.
I am using java as my technology and would like to prefer QUARTZ scheduler over CRONJOB.
If someone would be having a snippet of working code, it will be of great help. Thanks
have you considered using OpenWhisk, which has the capability for triggering logic on a scheduled fashion?
See here for details: https://console.ng.bluemix.net/docs/openwhisk/openwhisk_alarms.html#openwhisk_catalog_alarm
can you add more details on what you want to do? If you want one of your restful services to be invoked, the workload scheduler service will help.
If you want to have a workflow with steps spanning across different systems, monitor the execution of these steps from a user interface, be notified if anything goes wrong etc etc, the workload scheduler service will be your first choice.
Thanks, Umberto
Related
I'm moving my first steps with prefect, and I'm trying to see what its degrees of freedom are. To this end, I'm investigating whether prefect supports running different tasks on different schedules in the same python process. For example, Task A might have to run every 5 minutes, while Task B might run twice a day with a Cron scheduler.
It seems to me that schedules are associated with a Flow, not with a task, so to do the above, one would have to create two distinct one-task Flows, each with its own schedule. But even as that, given that running a flow is a blocking operation, I can't see how to "start" both flows concurrently (or pseudo-concurrently, I'm perfectly aware the flows won't execute on separate threads).
Is there a built-in way of getting the tasks running on their independent schedules? I'm under the impression that there is a way to achieve this, but given my limited experience with prefect, I'm completely missing it.
Many thanks in advance for any pointers.
You are right that schedules are associated with Flows and not Tasks, so the only place to add a schedule is a Flow. Running a Flow is a blocking operation if you are using the open source Prefect core only. For production use cases, it's recommended running your Flows against Prefect Cloud or Prefect Server. Cloud is the managed offering and Server is when you host it yourself. Note that Cloud has a very generous free tier.
When using a backend, you will use an agent that will kick off the flow run in a new process. This will not be blocking.
To start with using a backend, you can check the docs here
This Prefect Discourse topic discusses a very similar problem and shows how you could solve it using a flow-of-flows orchestrator pattern.
One way to approach it is to leverage Caching to avoid recomputation of certain tasks that require lower-frequency scheduling than the main flow.
I have a python code which reads data from one cloud system via rest api using the requests module and then writes data back to another cloud system via rest api . This code runs anywhere from 1 to 4 hours every week. Is there a place in Google Cloud Platform , I can execute this code on a periodic basis. Sort of like a scheduled batch job . Is there a serverless option to do this in App Engine . I know about the App engine cron service but seems like it is only for calling a URL regularly . Any thoughts ? Appreciate your help.
Google Cloud Scheduler could be the tool you are looking for. As it is mentioned in its documentation:
Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations, and more. You can automate everything, including retries in case of failure to reduce manual toil and intervention.
Here you have the quickstart for Cloud Scheduler, and also another tutorial for Cron jobs.
You can use the Google Genomics API pipelines.run endpoint to run a long-running job on a Google Compute Engine virtual machine and then it will destroy the machine when it's done. If your job will run for less than 24 hours and it can handle a failure, then you can use a Preemptible VM to save cost.
Pipelines: Run
https://cloud.google.com/genomics/reference/rest/v2alpha1/pipelines/run
Preemptible Virtual Machines
https://cloud.google.com/preemptible-vms/
You could use Cloud Scheduler to kick off the job
Pipelines may be preferred to trying to use one of the serverless technologies because they don't tend to handle the long running jobs as well.
You can use AI Platform Training to run any arbitrary Python package — it doesn’t have to be a machine learning job.
Which is the best job scheduler to hit REST endpoints which also has UI??
One option is IBM Workload Scheduler / HCL Workload Automation, an enterprise level workload automation solution available on-premise and on cloud.
If you just need to run few jobs you can consider using the SaaS version: IBM Workload Automation on Cloud: https://www.ibm.com/us-en/marketplace/workload-automation/details
If you want a light-weight solution, you can try our open-source and free QuartzDesk Executor webapp available on GitHub. It is a simple, yet powerful Java webapp based on Quartz. Out-of-the-box it comes with a UrlInvokerJob which is a generic Quartz job implementation that can do a POST to an arbitrary URL specified in the job data map (a map containing job configuration parameters).
QuartzDesk Executor does not have a UI, but for that, you can maybe use our QuartzDesk product. There are 3 editions available, one being completely free. Alternatively you can use any of the existing Quartz scheduler management solutions (google for "quartz scheduler manager or gui").
I have a question about job schedular, can use crontab(linux) command replace java quartz?
I want to know the advantage of quartz, someone could give some advice.
Depending on the scale of the problem being solved, using the cron scheduler provided by Linux will work well for many problems (on a single host). When you would like some fail over capability quartz is going to be the solution. Quartz can act as a clustered scheduler. Configured correctly, one node could be brought down for patching while the jobs running on quartz continue to process. There are also features of quartz that cron does not provide. Persistence and disallowing concurrent execution are two that I am using for a project. Those are some of the features that stand out to me. It would be best to check the documentation and look at some of the examples provided.
Cron is available by default on any unix based os. Quarz is simply a Java API with (more scheduling options). If you wish to schedule tasks within a Java application, Quartz is the way to go. If you wish to schedule adhoc os commands, unless you feel like writing your own generic scheduler, cron is the way to go.
Can anyone tell me, is there a way to run a process in IIS shared hosting service.
Suppose, the scenario is like "I want to send emails to a list of email id's after everywhere 3 hrs", so the challenge here is the process should not be invoked by a HTTP link. It should be automatic.
I think we can do this by IIS worker processes.
Also this all will be happening on a shared server(like GoDaddy) in IIS7, .NET 3.5
Please anyone give me a direction.
Thanks in advance.
This question was asked ages ago, but for what it's worth - I ended up using Hangfire to handle my long running tasks in my ASP app.
You can easily configure it for shared hosting and then for a dedicated server if you can scale up / out according to your needs.
It's super easy to use, just follow the doc step by step.
Cheers,
Andrew
You should write and run this as a Windows service, assuming you have access to install a service.
You could run a background worker thread from your asp.net code-behind but the problem is that IIS will terminate the thread after it is idle for a relatively short period of time. I tried going this route, trying to geocode a list of addresses (800+, from a SharePoint list) and IIS kept timing out my thread and stopping it. We ended up going with adding events to the SharePoint list that would geocode when the item was changed/added to the list.
One other option you could look into is using Windows Workflow API, it was designed for this kind of thing.