does anyone know of a good statistics service for a widget I'm developing?
my requirements are
1.have the ability to get hundreds of thousands of events per day.
2.API to get results and registering events.
3.near real time results.
thnx
michael
Try www.getclicky.com, it meets all your requirements.
Related
We're transitioning to a managed service provider for our IT service desk and deskside and we're working out the details of their SLAs. Many of the SLAs are based on ticket status. An example of this is the following:
"Measures the amount of time it takes to assess, schedule, test, and package application packages before they are available for User Acceptance Testing."
My first thought was to try using SLAs to measure this, as they neatly tie together calendars and priorities, but I'm having a really hard time finding any information about how I could do this.
Now I'm looking into using the TKSTATUS.STATUSTRACKING attribute on tickets, but I believe this just tracks straight 24/7 time instead of taking into consideration any calendars.
Has anyone tried this before? Any suggestions?
We are in a similar process, but we are measuring time on site of vendors for our work orders. Opposed to using any SLAs we have diffrent statuses which mark different events. Then when we want to know how long it took for an event to finish, we look to see the time required to change status in the wostatus table.
Ever since I got my Nest all I've wanted is to be able to aggregate my energy usage over a long period of time to look for patterns and mainly appease my curiosity. Sure there are the monthly usage reports, and the web and device only goes back 10 days.
Does the API expose that data, or would I need to somehow calculate it myself? I'm not seeing on the API reference anything like 'is_running'. Or is there a chance a future "Nest Data API" is going to come out?
The Nest API only provides realtime data, not historical. Nest has an online community where they accept product suggestions for future enhancements.
Their API does not give you a way to access the data.
https://developer.nest.com/documentation/api#thermostats
If you've noticed they only store 10 days of data on the thermostat anyway (supposedly). The user agreement for the API also seems to forbid developers from providing software that will aggregate the data over time. I guess for your personal use it might be OK, but you wouldn't be able to distribute it.
Say I have a bunch of webservers each serving 100's of requests/s, and I want to see real time stats like:
Request rate over last 5s, 60s, 5 min etc
Number of unique users seen again per time window
Or in general for a bunch of timestamped events, I want to see real-time derived statistics - what's the best way to go about it?
I've considered having each GET request update a global counter somewhere, then sampling that at various intervals, but at the event rates I'm seeing it's hard to get a distributed counter that's fast enough.
Any ideas welcome!
Added: Servers are Linux running Apache/mod_wsgi, with a Python (Django) stack.
Added: To give a sense of the event rates I want to track stats for, they're coming in at over 10K events/s. Even incrementing a distributed counter at that rate is a challenge.
You might like to help us try out the beta of our agent for application performance monitoring in Python web applications.
http://newrelic.com
It delves more into the application performance rather than just the web server, but since any bottlenecks aren't generate going to be the web server, but your application then that is going to be more useful anyway.
Disclaimer. I work for New Relic and this is the project I am working on. It is a paid product, but the beta means it is free for now with all features. Later when that changes, if you didn't want to pay for it, their is still a Lite subscription level which is free and which gives you basic web metrics reporting which still covers some of what you are after. Anyway, right now would be a great opportunity to make use of it to debug your performance while you can.
Virtually all good servers provide this kind of functionality out of the box. For example, Apache has the mod_status module and Glassfish supports JMX. Furthermore, there are many commercial packages for monitoring clusters, such as Hyperic and Zenoss.
What web or application server are you using? It is difficult to provide a solution without that information.
Look at using WebSockets, their overhead is much smaller than a HTTP request, they are very well suited to real-time web applications. See: http://nodeknockout.com/ for Node based websocket examples.
http://en.wikipedia.org/wiki/WebSocket
You will need to run a daemon if you want to run it on your apache server.
Also take a look at:
http://kaazing.com/ if you wan't less hassle, but are willing to fork out some cash.
On the Windows side, Perfmonance monitor is the tool you should investigate.
As Jared O'Connor said, you should precise what kind of web server you want to monitor.
I have been looking at the "_layouts/SpUsageSite.aspx" logs for my site, but they are giving erroneous results (eg 0 unique visitors when I know at least I have been on the site)
What is the best way to see these logs in a better way than the ootb functionality?
Did you enable the usage processing and the usage logging for the site in question?
You can enable them in you central admin under:
Operations -> Usage analysis processing
It may also be that the processing is limited to a speciffic timespan
I have come across a bug with the Usage analysis processing to do with UTC date conversion which resulted in the processed numbers being erroneous. This is apparently fixed in SP2, but we have not been able to implement this quite yet.
The alternative is a bit onerous as you need to copy the usage logs from each front end server to a location and configure the log parser to store the information in a data base.
Serge van den Oever steps through this quite well here.
I don't really recommend this as a regular process as it takes a lot of effort, but it does give you a huge amount of information for when you wish to take a detailed look at usage on a particular point of your SharePoint farm.
Ideally we would have a solution to parse the logs automagically using the log parser utility and provide that information in SSRS reports.
We patched to sp2 and it all started working again like magic.
I wonder how high traffic websites handle traffic logging, for example a website like myspace.com receives a lot of hits, I can imagine it would take a lot of space to log all those requests, so, do they log every single request or how do they handle this?
If you view source on a MySpace page, you get the answer:
<script type="text/javascript">
var pageTracker = _gat._getTracker("UA-6293770-1");
pageTracker._setDomainName(".myspace.com");
pageTracker._setSampleRate("1"); //sets sampling rate to 1 percent
pageTracker._trackPageview();
</script>
That script means they're using Google Analytics.
They can't just gauge traffic using IIS logs because they may sell ads to third parties, and third parties won't take your word for how much traffic you get. They want independent numbers from a separate company, and that's where Google Analytics comes in.
Just for future reference - whenever you've got a question about how a web site is doing something, try viewing the source. You'd be amazed at what you can find there in plain view.
We had a similar issue with out Intranet which is used by hundreds of people. The disk activity was huge and performance was being hurt.
The short answer is Asynchronous non-blocking logging.
probably like google analytics.
Use Javascript to load a page on a difference server, etc.
Don't how they track it since I don't work there. I am pretty sure that they have enough storage to record every little thing about their user if they wanted.
If I were them, I would use AwStats if I just wanted to know basic stuff about my users.
It is more likely that they have developed their own scripts for tracking their users. Stuff they would log
-ip_address
-referrer
-time
-browser
-OS
and so on. Then a script to see different data about the user varying by day, weeks, or months. As brulak said, something along the line of Analytics, but since they have access to actual database, they can learn much more about their users.
ZXTM traffic shaping and logging, speaking from experience here
I'd be extremely surprised if they didn't log every single request, yes, and operations with particularly high traffic volumes usually roll their own log-management solutions against the raw server logs, in some form or other -- sometimes as simple batch-type processes, sometimes as complete subsystems.
One company I worked for, back in the dot-com heyday, got upwards of twenty million pageviews a day; for that site (actually a set of them, running across a few dozen machines in all, as I recall), our ops team wrote a quite sophisticated, clustered solution in C that parsed, translated (into relational storage), compressed and distributed the logs daily. Log files, especially verbose ones, pile up fast, and the commercial solutions available at the time just couldn't cut it.
If by logging you mean for collecting server related information (request and response times, db and cpu usage per request etc) I think they sample only the 10% or 1% of the traffic. That gives the same results (provide developers with auditing information) without filling in the disks or slowing the site down.