I'm trying to make an alert in splunk that checks every minute for query results. For testing purposes I set the interval to 361s to ensure that I get plenty of results and see if the results from one check will also still be there in the subsequent 5-6 checks.
This is what I configured: (sorry for posting german screenshot, but that is our splunk is installed)
I'm only getting an e-mail at the full hour. Considering my 361s interval, a successfull trigger are 11h00 should also result in a successfull trigger at 11h01, 11h02 etc., but there is no e-mail. And there definitely are query results every few minutes.
isn't "0 * * * *" the corret cron expression for "every minute"? And what is "Ablauf" anyways? Is this the issue maybe, I can't find any documentation what it is for.
I just noticed that the first digit for the cron expression is for minutes and not for seconds. I guess that explains it.
Related
UPDATE:
the actual problem is different from what I've described. I'll provide and update/edit to this ticket once we'll resolve the issue. More details may be found at this thread - https://techcommunity.microsoft.com/t5/Azure-Log-Analytics/Reliably-trigger-alerts-for-Log-Analytics-log-entries/m-p/319315/highlight/false#M1224
Original question:
We use Azure Monitor to create alerts based on logs in Log Analytics. For this we choose our Log Analytics account as a "RESOURCE", then choose "Custom log search" signal name for "CONDITION". Alert logic - "Number of results greater than 0".
Sample query:
search *
| where ResourceProvider == "MICROSOFT.DATAFACTORY" and status_s == "Failed"
For Period and Frequency lets set 15 minutes. All looks simple, but...
The issue: described above setup does not work (it works sometimes), because alerts are fired only sometimes, a lot of them are missed which is completely unacceptable behavior.
If we set Period = Frequency = 5 minutes we basically miss almost every event. Period = Frequency = 15 minutes works better, but still a lot of events are missing. Period = Frequency = 30 works even better, but all this looks weird.
Important notice - logs are collected from Data Factory V2 into Log Analytics. I suspect that alert misses are due to the fact that logs are delivered to Log Analytics with some delay (up to several minutes). So when Azure Monitor evaluates alert query for the last 15 minutes (Period=15) it might be that most resent log entries are still not in Log Analytics. When next query evaluation is executed in 15 minutes it will miss the logs that were ingressed with a delay for prev 15 minutes interval. Is this assumption correct? If so, this is very weird - how then we supposed to configure Period and Frequency values? If I set Period > Frequency (e.g. Period = 30 and Frequency = 5, which means "evaluate expression every 5 minutes, take data for last 30 minutes from current time") then we get multiple duplicated alerts because Period is larger than Frequency so there is a big chance of log search query returning the same log entries every 5 minutes - this is highly undesirable behavior.
Issue happened to be with a buggy bahavior of ARM template creating alerts. Thanks to Stanislav Zhelyazkov it has been nailed down and resolved - I use alternative ARM API now and it seems to work fine. More details on the topic may be found here - https://techcommunity.microsoft.com/t5/Azure-Log-Analytics/Reliably-trigger-alerts-for-Log-Analytics-log-entries/m-p/309610.
I have a website that is live. I have a cron job that executes every 24 hours. the cron job fetches and analyzes the data from a database table.
The problem is that the website gets very slow during the time when cron job is running. And gets back to normal after that. It gives me error Too many connections during this time.
I set the maximum allowed connections to 500 in mysql. The number of active connections that I checked in mysql were less than limit during that time.
I am unable to find any relevant help or even a clue to think in a particular direction.
Update:
I noticed one thing. the number of mysql connection continuously increases in this time. Although still less than the maximum limit.
nice command can change priority of a process. You want to lower the priority of the background process so it will try not to execute be executing while the website is being busy. E.g.
0 3 * * * nice -n 20 myjob arg arg
to execute myjob arg arg with lowered priority every day at 3am.
EDIT: Although, if the job is spending most of its time in database queries, this will not affect it much. MySQL has LOW_PRIORITY flag for INSERT and UPDATE statements that will do kind of the same thing for those queries.
I want to make schedules that depend on entries of a database to schedule cron jobs. Like if there's an entry in database with a timestamp 2:00 PM, 3rd of Apr, I want to send a mail to users on 2nd of Apr. I also want to send notifications at 1:55 PM 3rd of Apr.
So, this means I have to look into the database, find the entries after the current times tamp, see if they suit the criteria for notification (like 5 minutes to time stamp or 1 day to time stamp) and send the notification or mail. I'm only worried that every one minute seems like too much overload. Are the AWS web workers built for this sort of thing?
Any suggestions on how this can be accomplished?
I don't think crontab will be the best choice but if you're familiar with it, it's fine.
First you should estimate how frequently your entries are created. If, let's say, only a couple of hundred a day. My suggestion is to create the crontab job right after the entry is created. But if more than a hundred a minutes, pooling will be fine.
But there are also side effects, like canceling or updating the cron job .
I think it's better to use a proper MQ.
I am searching for some data on splunk for a 5 minute time range. I want this query to run after every 5 minutes in splunk on it's own. How can this be done? I tried finding it on splunk but all I can see is how to schedule alerts and reports. And after the query is activated, how can we access the produced results generated by the query?
Technically you can have a scheduled search, but it only makes sense to talk about a report or an alert. Your scheduled approach is actually the best-practice (as there is also the possibility for a real-time search of the last 5 minutes).
If you just want a report, you tell Splunk to email it to you either as an HTML table or as a PDF document.
If you only want to be alerted if some condition matches (i.e. more than X results) then you want to set up an alert.
Scheduled searches are available, but they are a bit tricky to access (imho)
In the alerts/reports schedule options you have to set the following:
Earliest: -6m#m
Latest: -1m#m
Cron expression: */5 * * * *
Don't forget to set some trigger condition (for an alert) or a delivery method (for the report) ;)
i need to run a job 'x' times a day. job timing is every (say 'y' hours) between 8am and 8pm.
i read the documentation of cron between could not figure out how to place the "between times". any suggestion or a good tutorial should be really helpful.i could figure out this much.
Found this in a tutorial. I believe this will serve the requierment
SimpleTrigger simpleTrigger = new SimpleTrigger("simpleTrigger", "triggerGroup-s1");
simpleTrigger.setStartTime(d);
simpleTrigger.setRepeatInterval(1000*60*60*24);
simpleTrigger.setRepeatCount(15);
simpleTrigger.setEndTime(new Date(ctime + 60000L));
simpleTrigger.setPriority(10);
scheduler.scheduleJob(jobDetail, simpleTrigger);
scheduler.start();
how could i modify the expression to serve my purpose.
You have a * in your example where you need to put the start/end hours:
0 8-20/y * * *
Where you replace y with the number of hours you want between runs should be fine. If you have some strange time like "1 hour and 15 minutes between runs", it's going to be a pain - probably just easier to calculate each time and enter it explicitly.
I don't know you can express this in cron expression.
However, you can use multiple Quartz triggers to serve this purpose.