I'm currently working on a requirement for a real-time dashboard using steam analytics with a power bi output. The user would like to see a cumulative sum of an amount for the current hour. This would revert back to 0 at the beginning of the next hour.
From my understanding a Tumbling window should do this but the value doesn't change in the Power BI dashboard.
Any help would be appreciated!
Example:
SELECT System.Timestamp AS Timestamp,
SUM(Revenue) AS Revenue
INTO [TotalRevenuePerHour]
FROM [Input]
GROUP BY TUMBLINGWINDOW(hour,1)
Per my understanding of your requirements,you want to sum some value every one hour.And the value need to be refreshed in real-time during this one hour. If my words is right,i think your direction is right. However, you could know about Hopping Windows, not Tumbling Windows.
You could get know about differences between them with below two pictures:
Hopping Windows
Tumbling Windows
Actually,tumbling window is simply a hopping window whose ‘hop’ is equal to its ‘size’.
Please see the example of Hopping Windows from above link:
SELECT System.Timestamp() AS WindowEnd, TollId, COUNT(*)
FROM Input TIMESTAMP BY EntryTime
GROUP BY TollId, HoppingWindow(Duration(hour, 1), Hop(minute, 5), Offset(millisecond, -1))
Related
I'm continuously streaming the difference between two sensor values. I'd like to display the sum of those differences across a shift live while also logging the number at the end of the shift as well.
I've looked into tumbling windows but it seems I can't specify an actual time for those. I can specify the amount of time but not an actual time to start the window. If I could specify a time I could at least capture the historical values across shifts.
For the live data, I'd need to sum the data from start of tumbling window to the current data point, which to my knowledge is also not an option. Only the final aggregate is output after the window has ended.
I also took a look at using LAG() for this but that also does not allow specifying a timestamp and it would only allow comparing two values, not aggregate values.
Based on the documentation these are the two functions that are closest to having the functionality I need but neither accomplish the job.
Is there any way to aggregate all values from a certain start time to a current time (keep in mind the start time changes at the beginning of each shift)?
EDIT1: download file with 2 days of real data
My home automation controller collects data from several 4-in-1 motion sensors in different rooms of my house. The sensor prioritizes motion, sending motion reports every few seconds, but also independently reports temperature, humidity, and illuminance. I am trying to determine if the temp and humidity reports are sent frequently enough to automate control of heaters and exhaust fans.
Sensors independently report each category to the controller, which sends data to excel. Sample data below, but without motion reports that clutter up the real data.
A pivot table generated from the raw data:
Answering the question of frequency takes me several manual steps. Sorting/filtering the dataset for temp/humidity by room, then manually adding a time diff column
where time diff = (<current Date-Time cell> - <prev Date-Time cell>)*24*60. I then calculate the average and stdev of minutes between reports by manually selecting, in turn, each room/category subset in the time diff column; once for the average and once for the stdev.
After a few more manual steps, I end up with this desired result:
BUT I have to do it all over every time new data is added to the table. I'm certain excel can do this automatically, but I didn't find a solution through pivots, power pivots, slicing, or queries. I'm hoping one of you excel gurus can help. Thanks!
While using Azure Stream Analysis, I can create a kind of moving average by using AVG and group them by the HoppingWindow like the code shown below.
However, this will create a moving average of the points in the last 5 seconds. Is there a way to create a moving average of the last n datapoint? I understand that I can adjust the windows size to make n points coming into the window, but is there a way to measure the exact last n points like the one in MySQL and PostgresSQL?
SELECT System.TimeStamp AS OutTime, AVG (value)
INTO
[output]
FROM [input]
GROUP BY HoppingWindow(second,5,1)
Today ASA windows are only based on time. However you can use the LAG function to get previous events.
For your query, if you want to make the average in the 3 latest events, it will be something like this:
SELECT System.TimeStamp AS OutTime,
AvgValue= ( LAG(value,1) OVER (LIMIT DURATION(minute, 5))
+ LAG(value,2) OVER (LIMIT DURATION(minute, 5))
+ LAG(value,3) OVER (LIMIT DURATION(minute, 5))
)/3
FROM input
Sorry for the inconvenience.
Thanks,
JS
I am pretty new to Kibana.
I am logging ssh access hits and I want to compare the access hit counts during night time vs day time. How can I get this data? Also, How can I visualize this?
Also, what if I want to compare hits on weekends vs weekdays?
I can only see continuous time-line on X-axis in the visualization tab.
Any help is appreciated.
Hi Your question is very useful & important for time based analysis in Kibana.The answer is based on Kibana 4.1.
For example you want to create visualizations for day vs week:-
1. Click on Visualize tab.
2. Select Line Chart & select from a new search.
3. Select Count in Y-Axis Metric
4. Select Date Range in X-Axis agregation, select date field in Field option, in From field option input the range such as :- now-1W & corresponding mention in To field option as :now-1d
5. Then click Split Lines & select terms and its field to display top N results for that time range.
Hope it answers your query.
You can use hits chart, or trends. Try use trends with 12 hours difference - and that would be day and night for example. Also you can tag your log entries with spec tags depending on time you sent them to server.
Is it possible to calculate a percentage change in a Cognos Report?
Currently, I have a crosstab that has years as the columns and widgets in the rows with a calculation of total revenue. I am looking to calculate the annual % increase. New to Cognos, but I could accomplish this using other tools.
Any help will be very much appreciated.
Sorry for the late answer to your question but perhaps it will help others who see it.
It's kind of annoying, but it can be done. If you have one Query which uses an input parameter to select the year, perhaps it is a filter that says [Year] = ?YearPrompt?, then you can make a second Query which uses the filter [Year] = ?YearPrompt? - 1. Then, you can join these two queries, and the third query (made by joining the first two) will have both this year's revenue, and last year's revenue, available to it for a calculation, such as what the percent change is.
You need to create a query calculation item as one of the columns and in it you take Year2 - Year1(You can choose the exact columns you already have in the report using the Data Items tab when you are making the query calc) Also make sure you make this query calc a % in its format.