Python/Pandas aggregating by date - python-3.x

I am trying to count and plot the number of data points I have for each area by day, so far I have:
But I would like to show the number of instances of each county per day, with the end goal of plotting them on a line graph, like:
Only I would want to plot each county on its own line, rather than the total which I have plotted above.
Update:
I have managed to get this from the answers provided:
Which is great and exactly what I was looking for. However, in hindsight, this looks a little messy and not very descriptive even for the short period plotted let alone if I were to plot this for a couple of years worth of data.
So I'm thinking to plot this indivually on an 8 grid plot. But when I try to plot this for one county I am getting the boolean values. As below:
What would be the best way to plot only the True values?

You can try
df.county.groupby([df.date_stamp, df.county]).count().unstack().plot();
df.county...count() is the numerical series you want to plot.
groupby([df.date_stamp, df.county]) groups first by date_stamp, then by country (the order matters).
unstack will create a Dataframe whose index is the time stamp, and columns are counties.
plot(); will plot it (and the ; suppresses the unnecessary output).
Edit
To plot it on separate plots, you could do something like
for county in df.county.unique():
this_county = df[df.county == county]
this_county.county.groupby(df.date_stamp).count().plot();
title(county);
show();

pd.crosstab(df['date_stamp'],df['county']).plot()
EDIT: question changed, if you want them in subplots instead of lines:
pd.crosstab(df['date_stamp'],df['county']).plot(subplots=True)
The key in drawing each county as a separate line is that each county needs to be in a different column. If you just want to count them, crosstab is then probably the shortest way to achieve that result. For example:
Then the result is:
When subplots=True:

Related

Excel scatterplot graph update automatically

My problem is as follows:
The user inputs two numbers between 2 and 25, these numbers are used to create a grid. Every point on the grid has (x,y) coordinates. Based on the amount of points the user chose, my excel sheet is filled up with up to 25x25 (x,y) coordinates.
Example: A 6x7 grid is chosen by the user, the table is filled with 42 (x,y) coordinates and all other values in the table are set to "".
Now I want to use a scatterplot with lines connecting each array to plot the data.
Problem 1: If I only select the 6x7 part of the table that has values in it and create the scatterplot the result is correct. Until the user specifies a different grid, for example 8x9, then the graph is obviously missing two rows and two columns of input data.
Problem 2: If I select the entire 25x25 part of the table, including all the "" values, the graph axes get messed up. The y-axis works properly, but the x-axis shows sequential values (0-7) instead of the x-coordinates.
Problem 3: If I replace all the "" values in the table to 0 or NaN and plot the entire table the axes are correct, but the lines between the scatter data get messed up.
Question:
Is there a way to automatically change the input data for the plot, or is there a way to correctly display the values on the x-axis if I select all the data?
Not sure this will work in your case, but it's worth a try, especially since no one's addressed your post in 3+ hours. I've had success with this approach: 1) charting the largest data set, 2) copying the resulting chart, and 3) trimming the data it draws from to produce all smaller data sets.
To get this to work takes a lot of thought in laying out that largest data set so that all the other plots follow as needed. To illustrate, I've somewhat mimicked your data and in the animated gif I show largest data set, plus 2 others produced by copying it. Then I demonstrate how to make the second one, including the rescaling required to make all plots scaled equally. Notice that I've arranged things so that only one set of x-values feeds all the series. If you can do this, it makes working with the Excel's interface much easier.
After wrestling with it all night I came to the following solution:
Instead of setting all the empty cells to "" or zero the cells should be be set to #N/A (not available). The graph properly ignores the #N/A cells exactly like I want it to and updates when values are entered into them.

Excel - Plot average of two plots with inconsistent time (X) axis

I have managed to plot two different data sets on the same axis however, I'm also looking to plotting another line showing their average.
The main problem is that both data sets have different X (time) values so it's not possible to add an average column at the end and plot that. (See the highlighted row 22 for example, corresponding Time values are different)
Is there any way I can plot an average of two plots on the same axis?
One idea that might work is to place the values of both series, one above the other in two new columns, sort this new data according to time, smooth it, then plot the smoothed combined data. Alternatively, you could do the smoothing by simply plotting the new sorted series, adding a moving average trendline to it, then change the formatting of the new series so that it is no longer visible (but the trendline is). Something like this:
In the above picture, series 3 is the plot of the sorted aggregate data of series 1 and 2. If you change the formatting of series 3 so that there is no line, you get something like this:
For my relatively small mock data sets, the results are admittedly poor (it was based on just 25 data points in each series), but if you have a large amount of closely spaced data, and you play around with the moving average window size, you might get something acceptable. If not, you should probably just interpolate both datasets to obtain two consistent time series.

Highlight points on an Excel Line Graph according to a list

I have 2 data series, which records how much a user is meditating/attentive (out of 100) plotted onto a graph. The x axis is the number of seconds since the start of the experiment, and the y axis shows the value for meditation/attention at that point of time.
I have a 3rd set of data that is a series of key timestamps during the experiment (not exactly matching the timestamps from attention/meditation values).
I want to create a graph where you can compare how the attention/meditation values change at the key points
Whether the key points are highlighted by a line or dots I don't care. I tried adding the 3rd data set as a secondary axis, but it still uses the original x-axis of the main graph and I don't know how to make excel do what I want.
Thanks in advance
You should use an XY Scatter chart, not a line chart. A line chart ignores any numerical value in the X values, treats each X value as a text label, and uses the X values from the first series as X values for all series.
You can format the first two series so that they use lines and not markers, and the third so it uses markers without lines.
You may find this link helpful: superuser.com/questions/825692 You don't need to use the secondary axis, just add another series with tag times and constant 45 value, then format vertical error bars to 100% and horizontal to 0%.

chart in excel to represent correlation of 3 parameters

Please dont eat me because of this question :)
I have some data in excel and I would like to make a graphical representation of those data. Structure of my data:
persons ID : from 1 to 485 to every person, there is one parameter like average jumping distance and another parameter (like height) and finally there is a class to which every person belongs to (1, 2 or 3).
To assign persons to classes I have used k-means algorithm.
Now I would like to make a graph of this result. How can I do it please in excel (or by using another tool)?
Thank you
I would use a scatter (XY chart with markers and no lines). Plot average jumping distance on one axis, height on the second axis. Then for the classes I would separate all the data into 3 series and use different colors for each series. I would adjust the marker size to see which one works best with the data.
Here is a fast example to give you an idea how to it would look like. Its not as easy as just clicking once to insert the chart from the data though:

Excel Chart doesn't keep format

I have a table (came from a pivot table) where I have formatted the column 4 cells to show 1 billion as 1. But when I select the table and insert a chart, I am getting my units in millions. So the 14.8 billion number for Mexico is showing up as 14,800 on the chart. Why might this be happening and how can I fix this? This is also making all my other bars negligibly small. Note that the first three columns are not in billions and are totally different things. Some are percentages, some are other small numbers.
Table:
Chart:
You need a secondary horizontal axis and some formatting on the Axes.
In Excel 2013
First change the Chart Type to Combo and select Clustered Bar for both sets of data, then Check
Secondary Axis for the Percentage Series.
Then set up the axis limits so they match, e.g.
Percentage: min -.5 max 2
Billions: min -5e9 max 20e9
Then set the percentage format on the source data to a custom Number format of "";(0)%;0%
Then set the Billions format as 0,,,;"";0
You will get something like this:
EDIT
Now that we have the general principles, we can apply them to your specific data.
I will also switch to Excel 2010 do show the different menus.
The data selection looks like this
Select the non-Billion series (plural!) and check the secondary axis
If the larger data is always positive then you can use custom formatting to clean up the axis
Align the primary and secondary axes so that the grid lines match on both
The end result is clean and readable.
Mixing percentages and numbers for the smaller numbers is not handled by this but I would suggest that that would be confusing anyway?
The simplest way to fix this might be to plot cells containing the billions values divided by 10^9 rather than to plot the billions themselves, though via a secondary axis may be possible.
Using Excel 2007. For the purple bars, the example on the left uses ColumnE values, on the right ColumnF values. E1 contains =F1/10^9 and F1 contains =14800000000:
It appears that there are 3 questions here: 1) "Why might this be happening", 2) "how can I fix this", and 3) something like "how can I plot data which lie on two widely differing ranges, and make them all reasonably visible anyway", even if there was no explicit question on this.
There are several ways to solve issue #2 about the units (e.g., billions) and numbers (e.g., 14.8 vs. 14,800.0) shown in the axis, each one with its own pros and cons:
Use Format Axis -> Axis Options -> Display units.
This might be the answer to your issue #1 as well, you might have the following selection: Display units -> Millions, and unchecked Show display units... Otherwise, I wouldn't know why you chart shows what it shows.
Use faked tick marks, as indicated in the (excellent) site of Jon Peltier
http://peltiertech.com/Excel/Charts/ArbitraryAxis.html
It gives detailed instructions on how to create tick marks on an axis with arbitrary labels (which may be text, numbers, etc.), which is more generic than what the OP wants here. In this particular case, the labels will be the desired numbers.
Create new cells containing data that would be plotted exactly the way you want.
As for your issue #3, I guess the only option is to have a Secondary Axis (see the answer by pnuts).
Thus, to come up with the best final chart for you might use a combination of one of the options I gave here and a secondary axis.

Resources