I am looking for a front end for our operator to run our Talend jobs. We do not want him to have the ability to delete or modify jobs. Only to run them and monitor their results. Any suggestions for tools for doing this?
Thanks
The subscription version of Talend (called Talend Integration Suite, or TIS) has precisely that. It's a web-based console called Talend Administration Center (TAC) and it allows an operator to run jobs and monitor their results -- among many other things. The permissions can be set in the way you described, so that the operator is not able to delete or modify the jobs.
In my company we use SOS Job Scheduler, which you can use with TOS out-of-the-box as a general diy scheduling solution. Just these days I'm working on a Talend Custom Component to integrate deeply between the first and the latter (like log-propagation, context parameter sharing and so on. Basically it let you use SOS API from a TOS job). I can speed-up and put on github in a few days, if you need :)
You can export job as shall script (.sh) and anyone can execute that and will not able to delete those jobs.
To see the execution details put some tlog component into your talend jobs.
This is just a work around not the solution if you only have open studio (free version).
I'm using Rundeck for manual and scheduled execution of Talend jobs . I find the job setup and scheduling to be far easier in Rundeck than in SpagoBI (which I frequently use for reports and BI).
You could also use something like automic (uc4) automation engine. It has fine grained security and is a commercial product . I've used it in the past for similar work.
Related
I'm trying to find a tool where you can easily monitor cronjobs for the company I work at. Rundeck seems like the perfect tool for this but I can't figure out if it's possible to integrate the existing cronjobs into Rundeck. It's too much work to do this manually because there are hundreds of them.
If you now another tool that can do this feel free to recommend!
I'd rather want it to be open source but if it's paid and it works correct I'm open to it.
Rundeck works as a "very enhanced cron replacement" maybe the best approach in your case is to migrate your cron jobs to Rundeck and use the notification for monitoring. This looks like a good starting point for you, don´t forget to visit the official documentation.
What Mega mentions is correct, rundeck can be used as a replacement for cron.
Even better, there is no need to configure all of those manually: Rundeck allows you to import job definitions via API call.
Steps:
Get rundeck installed
Set up a job to run one of your cron jobs manually
Export that job
Use a script to make many copies of that file, that each contain a different item from cron*
Import all those files via API call
*You'll need to change at least the name and workflow (called sequence commands file - you'll be able to see based on the workflow step you configured)
I have a VM machine that i would like to shutdown/power off at a certain time and then restart at a certain time. I have tried this in task scheduler and obviously i can shutdown at a given time but cant then set the restart time
I would like the VM machine to shutdown at 10pm and restart at 5am and run a task scheduler task i have that restarts key services (that side of it works)
I have played around with automation tasks within azure but run into a variety of RMLogin issues
i just want the simplest way to schedule this
there is no auto-startup as far as I'm aware, so you'd have to use some sort of Automation. There is an official solution from Microsoft. Which is somewhat overkill, but should work (never tried it, tbh). There are various other scripts online that work with Azure Automation. They are easily searchable (like so).
If you go to my blog you can also find an example script that does the same, and an example of a runbook that you can trigger manually to start\stop vms
I would assume you would have gone through the below mentioned suggestion, The automation book https://learn.microsoft.com/en-us/azure/automation/automation-solution-vm-management is the way to achieve this. You can achieve auto shutdown via the portal but not restart and start.
Please check this links that talks about Start and Shut down role of the VM through REST API. You can wire up the end point with Azure Function, Puppet, or Chef to automate this process
VM - Start/Shut down Role(s): https://learn.microsoft.com/en-us/previous-versions/azure/reference/jj157189(v=azure.100)
If anything doesn't work for you I would suggest to leave your feedback.
So to simply answer your question, no, there is not a more simple way to achieve this.
If you want, you can add your feedback for this feature suggestion here
https://feedback.azure.com/forums/34192--general-feedback
We have built a project in Enterprise Guide for the purpose of creating a easy understandable and maintainable code. The project contain a set of process flows which run should be done in specific order. This project we need to run on a Linux Server machine, where the SAS Metadata Server is running.
Basic idea is to extract this project into SAS code, which we would be able to run from command line in Linux as a batch job.
Question 1:
Is there any other way to schedule a batch job in Linux-hosted SAS Server? I have read about VBS scripting for scheduling/running batch jobs, but in order this to be done on Linux Server, a installation of WINE is required, which on a production machine which already runs a number of other important applications, is almost completely out of question.
Is there a way to specify a complete project export into SAS code, provided that I give the specific order of running process flows? I have tried out ordered list, which is able to make you a list of tasks to run in order (although there is no way to choose a whole process flow as a single task), but unfortunately, this ordered list itself is later not possible to be exported as a SAS code.
Current solution we do is the following:
We export each single process flow of the SAS EG project into SAS code, and then create another SAS code with %include lines to run all the extracted codes in order that we want. This is of course a possible solution, but definitely not the most elegant one.
Question 2:
Since I don't know how exactly the code is being exported afterwards, are there any dangers I should bear in mind with the solution I chose.
Is there any other, more elegant way?
You have a couple of options from what I'm familiar with, plus I suspect if Dom happens by he'll know more. These answers are based on EG 6.1, which is the current version (ships with 9.4); it's possible some of these things may not be true in earlier versions.
First, if you're running Enterprise Guide from Windows, you can schedule the job locally (on any Windows machine with Enterprise Guide). You're not scheduling the server directly, you schedule Windows to launch an EG process that connects to the server and does its magic. That's how I largely interact with scheduling (because I have a fairly 'light' scheduling need).
Second, from the blog post "Four Ways to Schedule SAS Tasks", options 3 and 4 may be helpful for you. The SAS Platform Suite is designed in part for scheduling, and the options using SAS Management Console to schedule via operating system tools, are both very helpful.
Third, you may want to look into SAS Stored Processes, which should be schedulable. A process flow can be converted into a stored process.
For your specific questions:
Question 1: When you export a process flow or a project, at least in 6.1 you have the option to change the order in which the programs are exported. It's manual, so it's probably not perfect, but it does give you that option. (The code seems to be by default in creation order, which is sub-optimal.) The project export does group process flows together, but you don't have the option of manipulating the order of process flows - you have to move each program around, which would be tedious. It also of course gives you less flexibility if you need to multiply run programs.
Question 2: As Stig Eide points out in comments, make sure your System Option LRECL is > 256 (the default) or you run some risk of code being cut off. In 9.2+ this is modifiable; just place LRECL=32767in your config.sas file.
I want to create a scheduled job in SQLyog (or suggest me if any) that will run a SQL query every 30 minutes.
I want to run the query -
UPDATE `db`.`table` SET sessiontime = CEILING(sessiontime/6)*6 WHERE id>1000
Please help me to figure out this.
Thanks
First, SQLyog is a windows tool for managing MySQL databases. Windows usually don't have a cron, instead it has the Windows Task Scheduler.
SQLyog has something build in which is called the Job Agent, which allows you to execute queries and generate, format and send personalized mails with results. These jobs are saved as XML files, which can be executed with SQLyog. This is used to use the Windows Task Scheduler for performing regular querying (as a matter of fact, you can either put them there if you want to or use SQLyog to do this for you).
Basically, use the wizard to put in all the data required (database connection, smtp options, query), test it, then use the Job Agent to put this into the Windows Task Scheduler.
Keep in mind, the feature Job Agent is available in SQLyog Enterprise and Ultimate only. The free edition doesn't have it unlocked.
I am new to Essbase. Here I am creating an Application by using Essbase Administration Services (EAS), Planning and Workspace.
If I am able to create and manage an application using the above services alone, then what are the additional features that MaxL offers..?
There is excellent documentation on MaxL that Oracle provides. It is located here.
If you like working from the command line, you might prefer MaxL to EAS and Workspace because there aren't any mouseclicks involved. Secondly, you are connecting directly to the Essbase Server, so that makes it all that much faster as well.
Raam is correct that you can automate tasks using MaxL. For instance, if you have four calculation scripts that you need to run in a row, you can forego kicking them off in EAS sequentially and instead write a MaxL script that runs them all in a row for you.
Another benefit of scripting the process is being able to take advantage of some nice logging features Essbase in a directory that is convenient for you instead of buried deep within the Hyperion or EPM system directories.
However, since you're new to Essbase, you should know that MaxL does not provide all the functionality you have in the Hyperion Planning and Workspace UIs. There are command line utilities for planning and workspace, but they are not based on MaxL.
MAXL is a command line tool for administration of Essbase. While EAS is a GUI based tool. As with all command line tools, MAXL can be used to automate tasks by creating scripts. For example you can create a data load script and run that as and when needed instead of doing the same from the UI.