I have a dataframe for which I am plotting the sorted values from the columns as a line and then plotting and labeling various percentiles along that line.
I would like to have 12 subplots per figure and as many figures as I need depending on the number of columns (which will vary on my real datasets).
Here is a version of my script with a simple dataframe. This example produces 2 figures, each with 12 subplots. There are only 17 entries in the dataframe, so one figure has 7 emply subplots. This is the result that I need, however, the script is not elegant and not effiecient.
I am learning python and tend to revert back to old scripting habits when I can't figure out the efficient python way. Could someone show me how to produce the same figures with more elegant python? If I don't have 7 empty subplots, that fine too. I've tried various combinations. I think I'm getting tripped up on not understanding the inner workings of pandas versus numpy. I have a pandas Dataframe, but am manipulating the columns with numpy functions. In my trials I had many errors when attempting to apply the np.sort to the dataframe columns, which is why I reverted back to taking a single column at a time out of the dataframe to manipulate and then plot.
df = pd.DataFrame(np.random.randint(0,100,size=(15, 17)), columns=list('ABCDEFGHIJKLMNOPQ'))
p = np.array([0.0, 25.0, 50.0, 75.0, 90.0, 95.0, 99.0, 100.0])
j=2
i=1
fig, axs = plt.subplots(4,3, figsize=(8.5, 11), facecolor='w', edgecolor='k')
fig.subplots_adjust(hspace=0.4, wspace=0.4)
for f in df.columns:
d=df[f]
d=np.sort(d)
perc = np.nanpercentile(d, p)
if i >12:
fig, axs = plt.subplots(4,3, figsize=(8.5, 11), facecolor='w', edgecolor='k')
i=1
j=j+1
plt.subplot(4, 3, i)
plt.plot(d)
plt.plot((len(df[f])-1) * p/100., perc, 'ro')
plt.xticks((len(df[f])-1)* p/100., map(str, p))
plt.show()
i=i+1
A slightly different approach
create all figures and axes upfront and flatten the array of axes
then use more unto date Matplotlib API to plot against axis
have used pandas instead of numpy as I found it simpler to define x-co-ordinates from index of series rather than an array
the x-axis is still somewhat ugly even after rotation as you want ticks that are close together for 90+ percentiles
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import math
df = pd.DataFrame(np.random.randint(0,100,size=(15, 17)), columns=list('ABCDEFGHIJKLMNOPQ'))
p = np.array([0.0, 25.0, 50.0, 75.0, 90.0, 95.0, 99.0, 100.0])
# create all the figures and axis that are going to be used...
axs = np.array([])
for _ in range(math.ceil(len(df.columns)/12)):
fig, ax_t = plt.subplots(4,3, figsize=(8.5, 11), facecolor='w', edgecolor='k')
fig.subplots_adjust(hspace=0.4, wspace=0.4)
axs = np.concatenate((axs, np.array(ax_t).flatten()))
for i, ax in enumerate(np.array(axs)):
# NB will have created empty axis where there is no column
if i==len(df.columns): break
d = df.loc[:,df.columns[i]].sort_values().reset_index(drop=True)
ax.plot(d.index/(len(df)-1), d)
q = d.quantile(p/100)
ax.plot(q, "ro")
ax.set_xticks(p/100)
ax.tick_params(axis='x', labelrotation = 90)
# annotate 0th, 50th and 100th percentiles
for a in q.loc[[0,.5,1]].index:
ax.annotate(round(q[a],1), xy=(a, q[a]), xycoords='data', xytext=(3, 3), textcoords='offset points',)
Related
I have many datasets taken from multiple excel files that I would like to plot on the same graph each with a different color.
I have created 4 spreadsheets with random data for testing.
The first column defines the measurement, the code should select one of this containing 5 rows of data (X, Y), and add them to a dataframe. The results should be 1 dataset for every file to be plot all together on the same graph and having each plot of a different color.
Spreadsheets
I have been using modified pieces of codes taken on here from people which were trying to do the same thing. The problem is that I cannot color each plot differently because the program counts them as one, because due to the pd.concat() it merge these into 1 line. Do you know how I could overcome this?
Other questions asking to plot multiple datasets in single graph are almost all about a small number of dataset, while in my case I have like 50, thus cannot create a subplot for each one of them, unless there is a way to do this automatically
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import glob
import os
from os import path
import sys
import openpyxl
# create a list of all excel files in the directory
xlsx_files=glob.glob(r'C:\Users\exx762\Desktop\*.xlsx')
files=[]
n=len(xlsx_files)
index=0
# select chunk of data needed from each file and add to dataframe
for file in xlsx_files:
index+=1
files.append(pd.read_excel(file))
df_files=pd.concat(files)
ph_loops=df_files[df_files['Measurement']==2]
x = ph_loops['X']
y = ph_loops['Y']
# plot elements in the dataframe
ax=plt.subplot()
colors=plt.cm.jet(np.linspace(0, 1, n))
ax.set_prop_cycle('color', list(colors))
ax.plot(x, y, marker='.', c=colors[index-1], linewidth=0.5, markersize=2)
print(colors[index-1])
ax.tick_params(axis='y', color='k')
ax.set_xlabel('X', fontsize=12, weight='bold')
ax.set_ylabel('Y', fontsize=12, weight='bold')
ax.set_title(file+'\n')
ax.tick_params(width=2)
plt.plot()
plt.show()
> Actual result
You can add an id field (I used name below) to the dataframes as you concatenate them, then you can plot in a loop. Example:
# Create example dataframes
dfs = []
for i in range(1, 4):
df = pd.DataFrame(np.random.randn(10, 2), columns=['x', 'y'])
df.insert(0, 'name', i)
dfs.append(df)
result = pd.concat(dfs, ignore_index=True)
# Plot
fig, ax = plt.subplots()
for name, group in result.groupby('name'):
group.plot(x='x', y='y', ax=ax, label=name)
plt.show()
I'm trying to combine seaborn's heatmap and kdeplot in one figure, but so far the result is not very promising since I cannot find a way to make them overlap. As a result, the heatmap is just squeezed to the left side of the figure.
I think the reason is that seaborn doesn't seem to recognize the x-axis as the same one in two charts (see picture below), although the data points are exactly the same. The only difference is that for heatmap I needed to pivot them, while for the kdeplot pivoting is not needed.
Therefore, data for the axis are coming from the same dataset, but in the different forms as it can be seen in the code below.
The dataset sample looks something like this:
X Y Z
7,75 280 52,73
3,25 340 54,19
5,75 340 53,61
2,5 180 54,67
3 340 53,66
1,75 340 54,81
4,5 380 55,18
4 240 56,49
4,75 380 55,17
4,25 180 55,40
2 420 56,42
2,25 380 54,90
My code:
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
f, ax = plt.subplots(figsize=(11, 9), dpi=300)
plt.tick_params(bottom='on')
# dataset is just a pandas frame with data
X1 = dataset.iloc[:, :3].pivot("X", "Y", "Z")
X2 = dataset.iloc[:, :2]
ax = sns.heatmap(X1, cmap="Spectral")
ax.invert_yaxis()
ax2 = plt.twinx()
sns.kdeplot(X2.iloc[:, 1], X2.iloc[:, 0], ax=ax2, zorder=2)
ax.axis('tight')
plt.show()
Please help me with placing kdeplot on top of the heatmap. Ideally, I would like my final plot to look something like this:
Any tips or hints will be greatly appreciated!
The question can be a bit hard to understand, because the dataset can't be "just some data". The X and Y values need to lie on a very regular grid. No X,Y combination can be repeated, but not all values appear. The kdeplot will then show where the used values of X,Y are concentrated.
Such a dataset can be simulated by first generating dummy data for a full grid, and then take a subset.
Now, a seaborn heatmap uses categorical X and Y axes. Such axes are very hard to align with the kdeplot. To obtain a similar heatmap with numerical axes, ax.pcolor() can be used.
from matplotlib import pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
xs = np.arange(2, 10, 0.25)
ys = np.arange(150, 400, 10)
# first create a dummy dataset over a full grid
dataset = pd.DataFrame({'X': np.repeat(xs, len(ys)),
'Y': np.tile(ys, len(xs)),
'Z': np.random.uniform(50, 60, len(xs) * len(ys))})
# take a random subset of the rows
dataset = dataset.sample(200)
fig, ax = plt.subplots(figsize=(11, 9), dpi=300)
X1 = dataset.pivot("X", "Y", "Z")
collection = ax.pcolor(X1.columns, X1.index, X1, shading='nearest', cmap="Spectral")
plt.colorbar(collection, ax=ax, pad=0.02)
# default, cut=3, which causes a lot of surrounding whitespace
sns.kdeplot(x=dataset["Y"], y=dataset["X"], cut=1.5, ax=ax)
fig.tight_layout()
plt.show()
I have a data frame with 36 columns. I want to plot histograms for each feature in one go (6x6) using seaborn. Basically reproducing df.hist() but with seaborn. My code below shows the plot for only the first feature and all other come empty.
Test dataframe:
df = pd.DataFrame(np.random.randint(0,100,size=(100, 36)), columns=range(0,36))
My code:
import seaborn as sns
# plot
f, axes = plt.subplots(6, 6, figsize=(20, 20), sharex=True)
for feature in df.columns:
sns.distplot(df[feature] , color="skyblue", ax=axes[0, 0])
I guess it would make sense to loop over the axes and features simultaneously.
f, axes = plt.subplots(6, 6, figsize=(20, 20), sharex=True)
for ax, feature in zip(axes.flat, df.columns):
sns.distplot(df[feature] , color="skyblue", ax=ax)
Numpy arrays are flattened by row-wise, i.e. you would get the first 6 features in the first row, the features 6 to 11 in the second row etc.
If this is not what you want, you can define the index for the axes array manually,
f, axes = plt.subplots(6, 6, figsize=(20, 20), sharex=True)
for i, feature in enumerate(df.columns):
sns.distplot(df[feature] , color="skyblue", ax=axes[i%6, i//6])
e.g. the above will fill the subplots column by column.
I can create a simple columnar diagram in a matplotlib according to the 'simple' dictionary:
import matplotlib.pyplot as plt
D = {u'Label1':26, u'Label2': 17, u'Label3':30}
plt.bar(range(len(D)), D.values(), align='center')
plt.xticks(range(len(D)), D.keys())
plt.show()
But, how do I create curved line on the text and numeric data of this dictionarie, I do not know?
ΠΆ_OLD = {'10': 'need1', '11': 'need2', '12': 'need1', '13': 'need2', '14': 'need1'}
Like the picture below
You may use numpy to convert the dictionary to an array with two columns, which can be plotted.
import matplotlib.pyplot as plt
import numpy as np
T_OLD = {'10' : 'need1', '11':'need2', '12':'need1', '13':'need2','14':'need1'}
x = list(zip(*T_OLD.items()))
# sort array, since dictionary is unsorted
x = np.array(x)[:,np.argsort(x[0])].T
# let second column be "True" if "need2", else be "False
x[:,1] = (x[:,1] == "need2").astype(int)
# plot the two columns of the array
plt.plot(x[:,0], x[:,1])
#set the labels accordinly
plt.gca().set_yticks([0,1])
plt.gca().set_yticklabels(['need1', 'need2'])
plt.show()
The following would be a version, which is independent on the actual content of the dictionary; only assumption is that the keys can be converted to floats.
import matplotlib.pyplot as plt
import numpy as np
T_OLD = {'10': 'run', '11': 'tea', '12': 'mathematics', '13': 'run', '14' :'chemistry'}
x = np.array(list(zip(*T_OLD.items())))
u, ind = np.unique(x[1,:], return_inverse=True)
x[1,:] = ind
x = x.astype(float)[:,np.argsort(x[0])].T
# plot the two columns of the array
plt.plot(x[:,0], x[:,1])
#set the labels accordinly
plt.gca().set_yticks(range(len(u)))
plt.gca().set_yticklabels(u)
plt.show()
Use numeric values for your y-axis ticks, and then map them to desired strings with plt.yticks():
import matplotlib.pyplot as plt
import pandas as pd
# example data
times = pd.date_range(start='2017-10-17 00:00', end='2017-10-17 5:00', freq='H')
data = np.random.choice([0,1], size=len(times))
data_labels = ['need1','need2']
fig, ax = plt.subplots()
ax.plot(times, data, marker='o', linestyle="None")
plt.yticks(data, data_labels)
plt.xlabel("time")
Note: It's generally not a good idea to use a line graph to represent categorical changes in time (e.g. from need1 to need2). Doing that gives the visual impression of a continuum between time points, which may not be accurate. Here, I changed the plotting style to points instead of lines. If for some reason you need the lines, just remove linestyle="None" from the call to plt.plot().
UPDATE
(per comments)
To make this work with a y-axis category set of arbitrary length, use ax.set_yticks() and ax.set_yticklabels() to map to y-axis values.
For example, given a set of potential y-axis values labels, let N be the size of a subset of labels (here we'll set it to 4, but it could be any size).
Then draw a random sample data of y values and plot against time, labeling the y-axis ticks based on the full set labels. Note that we still use set_yticks() first with numerical markers, and then replace with our category labels with set_yticklabels().
labels = np.array(['A','B','C','D','E','F','G'])
N = 4
# example data
times = pd.date_range(start='2017-10-17 00:00', end='2017-10-17 5:00', freq='H')
data = np.random.choice(np.arange(len(labels)), size=len(times))
fig, ax = plt.subplots(figsize=(15,10))
ax.plot(times, data, marker='o', linestyle="None")
ax.set_yticks(np.arange(len(labels)))
ax.set_yticklabels(labels)
plt.xlabel("time")
This gives the exact desired plot:
import matplotlib.pyplot as plt
from collections import OrderedDict
T_OLD = {'10' : 'need1', '11':'need2', '12':'need1', '13':'need2','14':'need1'}
T_SRT = OrderedDict(sorted(T_OLD.items(), key=lambda t: t[0]))
plt.plot(map(int, T_SRT.keys()), map(lambda x: int(x[-1]), T_SRT.values()),'r')
plt.ylim([0.9,2.1])
ax = plt.gca()
ax.set_yticks([1,2])
ax.set_yticklabels(['need1', 'need2'])
plt.title('T_OLD')
plt.xlabel('time')
plt.ylabel('need')
plt.show()
For Python 3.X the plotting lines needs to explicitly convert the map() output to lists:
plt.plot(list(map(int, T_SRT.keys())), list(map(lambda x: int(x[-1]), T_SRT.values())),'r')
as in Python 3.X map() returns an iterator as opposed to a list in Python 2.7.
The plot uses the dictionary keys converted to ints and last elements of need1 or need2, also converted to ints. This relies on the particular structure of your data, if the values where need1 and need3 it would need a couple more operations.
After plotting and changing the axes limits, the program simply modifies the tick labels at y positions 1 and 2. It then also adds the title and the x and y axis labels.
Important part is that the dictionary/input data has to be sorted. One way to do it is to use OrderedDict. Here T_SRT is an OrderedDict object sorted by keys in T_OLD.
The output is:
This is a more general case for more values/labels in T_OLD. It assumes that the label is always 'needX' where X is any number. This can readily be done for a general case of any string preceding the number though it would require more processing,
import matplotlib.pyplot as plt
from collections import OrderedDict
import re
T_OLD = {'10' : 'need1', '11':'need8', '12':'need11', '13':'need1','14':'need3'}
T_SRT = OrderedDict(sorted(T_OLD.items(), key=lambda t: t[0]))
x_val = list(map(int, T_SRT.keys()))
y_val = list(map(lambda x: int(re.findall(r'\d+', x)[-1]), T_SRT.values()))
plt.plot(x_val, y_val,'r')
plt.ylim([0.9*min(y_val),1.1*max(y_val)])
ax = plt.gca()
y_axis = list(set(y_val))
ax.set_yticks(y_axis)
ax.set_yticklabels(['need' + str(i) for i in y_axis])
plt.title('T_OLD')
plt.xlabel('time')
plt.ylabel('need')
plt.show()
This solution finds the number at the end of the label using re.findall to accommodate for the possibility of multi-digit numbers. Previous solution just took the last component of the string because numbers were single digit. It still assumes that the number for plotting position is the last number in the string, hence the [-1]. Again for Python 3.X map output is explicitly converted to list, step not necessary in Python 2.7.
The labels are now generated by first selecting unique y-values using set and then renaming their labels through concatenation of the strings 'need' with its corresponding integer.
The limits of y-axis are set as 0.9 of the minimum value and 1.1 of the maximum value. Rest of the formatting is as before.
The result for this test case is:
I have a code that runs a rolling window (30) average over a range (i.e. 300)
So I have 10 averages but they plot against ticks 1-10 rather than spaced over every window of 30.
The only way I can get it to look right is to plot it over (len(windowlength)) but the x-axis isnt right.
Is there any way to manually space the results?
windows30 = (sliding_window(sequence, 30))
Overall_Mean = mean(sequence)
fig, (ax) = plt.subplots()
plt.subplots_adjust(left=0.07, bottom=0.08, right=0.96, top=0.92, wspace=0.20, hspace=0.23)
ax.set_ylabel('mean (%)')
ax.set_xlabel(' Length') # axis titles
ax.yaxis.grid(True, linestyle='-', which='major', color='lightgrey', alpha=0.5)
ax.plot(windows30, color='r', marker='o', markersize=3)
ax.plot([0, len(sequence)], [Overall_Mean, Overall_Mean], lw=0.75)
plt.show()
From what I have understood you have a list of length 300 but only holds 10 values inside. If that is the case, you can remove the other values that are None from your windows30 list using the following solution.
Code Demonstration:
import numpy as np
import random
import matplotlib.pyplot as plt
# Generating the list of Nones and numbers
listofzeroes = [None] * 290
numbers = random.sample(range(50), 10)
numbers.extend(listofzeroes)
# Removing Nones from the list
numbers = [value for value in numbers if value is not None]
step = len(numbers)
x_values = np.linspace(0,300,step) # Generate x-values
plt.plot(x_values,numbers, color='red', marker='o')
This is a working example, the relevant code for you is after the second comment.
Output:
The above code will work independently of where the Nones are located in your list. I hope this solves your problem.