I was thinking about monitoring the evolution of a single value (nope, this is not the SO rep :-p), and I'd like to have some nice histograms about it. My needs are simple:
daily / weekly / monthly / yearly evolution histograms;
daily / weekly / monthly / yearly calculation for max, min and average value.
Ideally the product should be scriptable so I can feed it with the result of the script.
Something simple like: set it up, set cron (or, if it has a daemon, even better), set input, enjoy output.
If it does not exist, do you feel like you would be interested in such a tool. Because I could end up coding it eventually.
EDIT :
I am not looking for a lib or a language but an app.
I'm not sure if they are what you are looking for, but I would look at the R statistics packages and the various Python numerical and scientific libraries, like numpy and scipy. They are probably overkill, but they might have graphical components that you can leverage.
How about RRDtool?
From a Java program you can create charts with JFreeChart
Related
I am new here, so sorry in advance if I make any mistakes!
Problem: I need to analyze this music .wav file, particularly for its frequency, amplitude, and pitch over specific intervals of time.
Is there any easy to use software and steps I can take that can help me accomplish this?
I have tried audacity, sonicvisualizer, and sigview, but I am unsure how to utilize these softwares appropiately to achieve my specific goal.
Thanks in advance!
Praat is good for these kinds of things. It has been specifically designed for speech research, but it can (and has) been used for analysing music as well.
It has a scripting language that allows for automation, and can analyse the things you mention for specific intervals or for the whole sound. Take a look at the documentation, specifically the sections on Pitch, Intensity, and spectral analysis.
Sigview is super easy to use and is your best bet if you want to be the most scientific about it (without investing too much $$ or time).
To use SigView, drag & drop an audio file into sigview. Highlight the portion of the waveform you're interested in and right click, select the option 'open selection in new window'. When you're looking at the portion of the waveform you're interested in, hit ctrl-f and it will perform an FFT on the segment. Right click and select show peaks so see the amplitude & frequency of each peak.
If I have a large sample of data (a vector of 10000 numbers or so), is there some tool that will look at the data and try to fit a number of common statistical distributions to the data and report on how good a fit each one is? If I have to specify the list of possible distributions myself, that's ok.
If you are looking for Statistics tools ...
R would probably be the first choice - have a look at: www.r-project.org
Grab RStudio as an IDE install package Deducer and off you go with your analysis.
Ah, and by the way it's all Open-Source / Freeware ...
Greetings, Walter
My realtime app generates a data log: 100 words of data #10Khz. I need to analyze it and produce some plots of the results. There are intermediate calculations involved - I need to take some differences, averages, etc. Excel would work fine, except for:
the 32000 item limit on graph data series is too small - that's only 3 seconds of data.
the glacial speed at which it processes changes to graphs containing large data series is unbearable.
What are good alternatives to Excel for manipulating and plotting large quantities of data? I'm looking for something interactive, not a library.
For this sort of stuff we typically roll our own, but I know that isn't the solution you want. Can you use a good quality database (eg Oracle) to do the manipulation, then maybe put the summarized data back into Excel for the plotting? I believe Excel will link to databases these days, so you could make it quite automated.
Otherwise there are statistical tools like [SAS][1], but get your cheque book out first.
[1]: http://www.sas.com/technologies/analytics/statistics/stat/index.html SAS
There are also several free tools for analysing and plotting (see below). But I am not sure whether they have components to handle data in real-time.
R (similar to SAS) for statistical computations
octave (similar to Matlab) for mathematical computations
R (for data manipulation) and its ggplot2 module for creating sexy graphs. Incredibly useful.
If you need real-time graphics, then I'd look at building something using matplotlib. It's a Python module, and you can link it to R using rpy2 if required.
In particle and nuclear physics the big tool is ROOT, which I have seen used in a "update every two seconds as the data comes in" mode with a lot of data and a modest amount of intermediate processing.
Mind you, the student who wrote that module was a very slick programmer, and it took a while to shake the bugs out, even so.
ROOT is available for free, and provides all kinds of tools and support.
I'm looking for something containing similar functions to Matlab’s financial and financial derivatives toolbox but don’t have the cash to spend on matlab. I would appreciate any info on free or open source libraries or programs that will let me easily calculate interest rates, risk etc.
How about JQuantLib or QuantLib?
How about the Octave financial functions?
http://www.gnu.org/software/octave/doc/interpreter/Financial-Functions.html#Financial-Functions
I'm not familiar with the Matlab toolbox, so you'll have to judge for yourself.
GNU Octave is a high-level language,
primarily intended for numerical
computations. It provides a convenient
command line interface for solving
linear and nonlinear problems
numerically, and for performing other
numerical experiments using a language
that is mostly compatible with Matlab.
It may also be used as a
batch-oriented language.
Exactly what functions do you need? How advanced? You have some financial functions in .Net
Im sure it doesnt cover everything, but calulating interests and some other are no problem:
http://msdn.microsoft.com/en-us/library/daksysx3(VS.80).aspx
Calculate depreciation. DDB, SLN, SYD
Calculate future value.
FV
Calculate interest rate.
Rate
Calculate internal rate of return.
IRR, MIRR
Calculate number of periods.
NPer
Calculate payments.
IPmt, Pmt, PPmt
Calculate present value.
NPV, PV
In this question someone asked for ways to display disk usage in Linux. I'd like to take this one step further down the cli-path... how about a shell script that takes the output from something like a reasonable answer to the previous question and generates a graph/chart from it (output in a png file or something)? This may be a bit too much code to ask for in a regular question, but my guess is that someone already has a oneliner laying around somewhere...
If some ASCII chars are "graphical" enough for you, I can recommend ncdu. It is a very nice interactive CLI tool, which helps me a lot to step down large directories without doing cd bigdir ; du -hs over and over again.
I would recommend munin. It is designed for exactly this sort of thing - graphing CPU usage, memory usage, disc-usage and such. sort of like MRTG (but MRTG is primarily aimed at graphing router's traffic, graphing anything but bandwidth with it is very hackish)
Writing Munin plugins is very easy (it was one of the projects goals). They can be written in almost anything (shell script, perl/python/ruby/etc, C, anything that can be execute and produce an output). The plugin output format is basically disc1usage.value 1234. And debugging the plugins is very easy (compared to MRTG)
I've set it up on my laptop to monitor disc-usage, bandwidth usage (by pulling data from my ISP's control panel, it graphs my two download "bins", uploads and newsgroup usage), load average and number of processes. Once I got it installed (currently slightly difficult on OS X, but it's trivial on Linux/FreeBSD), I had written a plugin in a few minutes, and it worked, first time!
I would describe how it's setup, but the munin site will do that far better than I could!
There's an example installation here
Some alternatives are nagios and cacti. You could also write something similar using rrdtool. Munin, MRTG and Cacti are basically all far-nicer-to-use systems based around this graphing tool.
If you want something really, really simple, you could do..
import os
import time
while True:
disc_usage = os.system("df -h / | awk '{print $3}'")
log = open("mylog.txt")
log.write(disc_usage + "\n")
log.close()
time.sleep(60*5)
Then..
f = open("mylog.txt")
lines = f.readlines()
# Convert each line to a float number
lines = [float(cur_line) for cur_line in lines]
# Get the biggest and smallest
biggest = max(lines)
smallest = min(lines)
for cur_line in lines:
base = (cur_line - smallest) + 1 # make lowest value 1
normalised = base / (biggest - smallest) # normalise value between 0 and 1
line_length = int(round(normalised * 28)) # make a graph between 0 and 28 characters wide
print "#" * line_length
That'll make a simple ascii graph of the disc usage. I really really don't recommend you use something like this. Why? The log file will get bigger, and bigger, and bigger. The graph will get progressively slower to graph. RRDTool uses a rolling-database system to store it's data, so the file will never get bigger than about 50-100KB, and it's consistently quick to graph as the file is a fixed length.
In short. If you want something to easily graph almost anything, use munin. If you want something smaller and self-contained, write something with RRDTool.
We rolled our own at work using RRDtool (the data storage back end to tools like MRTG). We run a perl script every 5 minutes that takes a du per partition and stuffs it into an RRD database and then uses RRD's graph function to build graphs. It takes a while to igure out how to set up the .rrd files (for instance, I had to re-learn RPN to do some of the calculations I wanted to do) but if you have some data you want to graph over time, RRD tool's a good bet.
I guess there are a couple of options:
For a pure CLI solution, use something like gnuplot. See here for example usage. I haven't used gnuplot since my student days :-)
Not really a pure CLI solution, but download something like JFreeChart and write a simple Java app that reads stdin and creates your chart.
Hope this helps.