Gantt Chart for USGS Hydrology Data with Python? - python-3.x

I have a compiled a dataframe that contains USGS streamflow data at several different streamgages. Now I want to create a Gantt chart similar to this. Currently, my data has columns as site names and a date index as rows.
Here is a sample of my data.
The problem with the Gantt chart example I linked is that my data has gaps between the start and end dates that would normally define the horizontal time-lines. Many of the examples I found only account for the start and end date, but not missing values that may be in between. How do I account for the gaps where there is no data (blanks or nan in those slots for values) for some of the sites?
First, I have a plot that shows where the missing data is.
import missingno as msno
msno.bar(dfp)
Now, I want time on the x-axis and a horizontal line on the y-axis that tracks when the sites contain data at those times. I know how to do this the brute force way, which would mean manually picking out the start and end dates where there is valid data (which I made up below).
from datetime import datetime
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as dt
df=[('RIO GRANDE AT EMBUDO, NM','2015-7-22','2015-12-7'),
('RIO GRANDE AT EMBUDO, NM','2016-1-22','2016-8-5'),
('RIO GRANDE DEL RANCHO NEAR TALPA, NM','2014-12-10','2015-12-14'),
('RIO GRANDE DEL RANCHO NEAR TALPA, NM','2017-1-10','2017-11-25'),
('RIO GRANDE AT OTOWI BRIDGE, NM','2015-8-17','2017-8-21'),
('RIO GRANDE BLW TAOS JUNCTION BRIDGE NEAR TAOS, NM','2015-9-1','2016-6-1'),
('RIO GRANDE NEAR CERRO, NM','2016-1-2','2016-3-15'),
]
df=pd.DataFrame(data=df)
df.columns = ['A', 'Beg', 'End']
df['Beg'] = pd.to_datetime(df['Beg'])
df['End'] = pd.to_datetime(df['End'])
fig = plt.figure(figsize=(10,8))
ax = fig.add_subplot(111)
ax = ax.xaxis_date()
ax = plt.hlines(df['A'], dt.date2num(df['Beg']), dt.date2num(df['End']))
How do I make a figure (like the one shown above) with the dataframe I provided as an example? Ideally I want to avoid the brute force method.
Please note: values of zero are considered valid data points.
Thank you in advance for your feedback!

Find date ranges of non-null data
2020-02-12 Edit to clarify logic in loop
df = pd.read_excel('Downloads/output.xlsx', index_col='date')
Make sure the dates are in order:
df.sort_index(inplace=True)
Loop thru the data and find the edges of the good data ranges. Get the corresponding index values and the name of the gauge and collect them all in a list:
# Looping feels like defeat. However, I'm not clever enough to avoid it
good_ranges = []
for i in df:
col = df[i]
gauge_name = col.name
# Start of good data block defined by a number preceeded by a NaN
start_mark = (col.notnull() & col.shift().isnull())
start = col[start_mark].index
# End of good data block defined by a number followed by a Nan
end_mark = (col.notnull() & col.shift(-1).isnull())
end = col[end_mark].index
for s, e in zip(start, end):
good_ranges.append((gauge_name, s, e))
good_ranges = pd.DataFrame(good_ranges, columns=['gauge', 'start', 'end'])
Plotting
Nothing new here. Copied pretty much straight from your question:
fig = plt.figure(figsize=(10,8))
ax = fig.add_subplot(111)
ax = ax.xaxis_date()
ax = plt.hlines(good_ranges['gauge'],
dt.date2num(good_ranges['start']),
dt.date2num(good_ranges['end']))
fig.tight_layout()

Here's an approach that you could use, it's a bit hacky so perhaps some else will produce a better solution but it should produce your desired output. First use pd.where to replace non NaN values with an integer which will later determine the position of the lines on y-axis later, I do this row by row so that all data which belongs together will be at the same height. If you want to increase the spacing between the lines of the gantt chart you can add a number to i, I've provided an example in the comments in the code block below.
The y-labels and their positions are produced in the data munging steps, so this method will work regardless of the number of columns and will position the labels correctly when you change the spacing described above.
This approach returns matplotlib.pyplot.axes and matplotlib.pyplot.Figure object, so you can adjust the asthetics of the chart to suit your purposes (i.e. change the thickness of the lines, colours etc.). Link to docs.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = pd.read_excel('output.xlsx')
dates = pd.to_datetime(df.date)
df.index = dates
df = df.drop('date', axis=1)
new_rows = [df[s].where(df[s].isna(), i) for i, s in enumerate(df, 1)]
# To increase spacing between lines add a number to i, eg. below:
# [df[s].where(df[s].isna(), i+3) for i, s in enumerate(df, 1)]
new_df = pd.DataFrame(new_rows)
### Plotting ###
fig, ax = plt.subplots() # Create axes object to pass to pandas df.plot()
ax = new_df.transpose().plot(figsize=(40,10), ax=ax, legend=False, fontsize=20)
list_of_sites = new_df.transpose().columns.to_list() # For y tick labels
x_tick_location = new_df.iloc[:, 0].values # For y tick positions
ax.set_yticks(x_tick_location) # Place ticks in correct positions
ax.set_yticklabels(list_of_sites) # Update labels to site names

Related

How to plot subplots from a condition applied on a single column and the data available on another single column of the same dataframe?

I have a dataframe which is much like the one following:
data = {'A':[21,22,23,24,25,26,27,28,29,30,11,12,13,14,15,16,17,18,19,20,1,2,3,4,5,6,7,8,9,10],
'B':[8,8,8,8,8,8,8,8,8,8,5,5,5,5,5,5,5,5,5,5,3,3,3,3,3,3,3,3,3,3],
'C':[10,15,23,17,18,26,24,30,35,42,44,42,38,36,34,30,27,25,27,24,1,0,2,3,5,26,30,40,42,50]}
data_df = pd.DataFrame(data)
data_df
I would like to have the subplots, the number of subplots should be equal to number of unique values of column 'B'. X axis = Values in column 'A' and Y axis = values in Column 'C'.
The code that I tried:
fig = px.line(data_df,
x='A',
y='C',
color='B',
facet_col = 'B',
)
fig.show()
gives output like
However, I would like to have the graphs in a single column, each graph autoscaled to the relevant area and resolution on the axes.
Possibility: Can I somehow make use of groupby command to do it?
Since I may have other number of unique values in column 'B' (for example 5 unique values) based on other data, I would like to have this piece of code to work dynamic. Kindly help me.
PS: plotly express module is used to plot the graph.
In order to stack all subplot in one column, and make sure that each xaxis is independent, just add the following in your px.line() call:
facet_col_wrap=1
And then follow up with:
fig.update_xaxes(matches=None)
Plot 1: Default setup with px.line(facet_col = 'B')
If you'd like to display all x-axis labels just include this:
fig.update_xaxes(showticklabels = True)
Plot 2: Show x-axes for all subplots
Complete code:
import plotly.express as px
import pandas as pd
data = {'A':[21,22,23,24,25,26,27,28,29,30,11,12,13,14,15,16,17,18,19,20,1,2,3,4,5,6,7,8,9,10],
'B':[8,8,8,8,8,8,8,8,8,8,5,5,5,5,5,5,5,5,5,5,3,3,3,3,3,3,3,3,3,3],
'C':[10,15,23,17,18,26,24,30,35,42,44,42,38,36,34,30,27,25,27,24,1,0,2,3,5,26,30,40,42,50]}
data_df = pd.DataFrame(data)
data_df
fig = px.line(data_df,
x='A',
y='C',
color='B',
facet_col = 'B',
facet_col_wrap=1
)
fig.update_xaxes(matches=None, showticklabels = True)
fig.show()
You can instead use the argument facet_row = 'B' which will automatically stack the subplots by rows. Then to automatically rescale, you'll want to set all of the x data to the same array of values, which can be done by looping through fig.data and modifying fig.data[i]['x'] for each i.
import pandas as pd
import plotly.express as px
data = {'A':[21,22,23,24,25,26,27,28,29,30,11,12,13,14,15,16,17,18,19,20,1,2,3,4,5,6,7,8,9,10],
'B':[8,8,8,8,8,8,8,8,8,8,5,5,5,5,5,5,5,5,5,5,3,3,3,3,3,3,3,3,3,3],
'C':[10,15,23,17,18,26,24,30,35,42,44,42,38,36,34,30,27,25,27,24,1,0,2,3,5,26,30,40,42,50]}
data_df = pd.DataFrame(data)
fig = px.line(data_df,
x='A',
y='C',
color='B',
facet_row = 'B',
)
for fig_data in fig.data:
fig_data['x'] = list(range(len(fig_data['y'])))
fig.show()

Not able to set the row order in seaborn.relplot

i am new to python and seaborn, i am using the dataset from kaggle, now i want to visualize the data, this is my code
def add_if_zero(x):
if (x['stays_in_weekend_nights']+x['stays_in_week_nights'])==0:
return 1
else:
return (x['stays_in_weekend_nights']+x['stays_in_week_nights'])
_data['total_days_of_stay']=_data.apply(lambda x:add_if_zero(x),axis=1)
x_order=['January','February','March','April','May','May','May','August','September','October','November','December']
sns.relplot(x='arrival_date_month',
y='total_days_of_stay',hue='hotel',
kind='line',data=_data,height=10,aspect=.9,row_order=x_order)
i am getting the plot and i want the order to be from jan-dec but i am getting a random variable in the x-axis
i tried passing the order to the row_order attribute in array as well as in string format but nothing seems to work
as per the documentation
row_order, col_orderlists of strings, optional Order to organize the
rows and/or columns of the grid in, otherwise the orders are inferred
from the data objects.
UPDATE 1:
You may wanna use lineplot() that returns a matplotlib Axes object, which is easy to configure. The basic idea here is to plot the numerical order int first and then add labels. Please consider the following example that uses a sample dataset from seaborn.
import calendar
import seaborn as sns
import matplotlib.pyplot as plt
# sample dataset
df = sns.load_dataset('flights')
# map month name with numerical order
d = dict((v,k) for k,v in enumerate(calendar.month_name[1:], start=1))
df['month_num'] = df.month.map(d)
# plot
fig, ax = plt.subplots(figsize=(10, 5))
sns.lineplot(x='month_num', y='passengers', data=df, hue='year', ax=ax)
# set xticks position and labels
ax.set_xticks(range(1, len(d)+1))
ax.set_xticklabels(d.keys(), rotation=30)
Use sns.catplot , pass the desired order as a list of strings to the function and set the kind='point', as follows:
sns.catplot(... ,
kind='point',
order=['Jan', 'Feb' , ...., 'Dec'])

Curve fitting for large datasets in Python

I have a very large set of data, ( around 100k points) and I want to fit a curve to this plot.
I tried the filters suggested by answers to another question, but that lead to overfitting.
I am using numpy and matplotlib as of now.
This is the type of scatter plot I am trying to fit.
Edit 1:
Please ignore the data points to the side of the central main set of data points(Thus only a single curve can fit this)
Here is the dataset, download the file as a text file to separate the columns, consider the columns 3 and 9 ( 1-based indexing), the y-axis has column 3 while the x-axis plots the difference of column 3 and column 9.
Edit 2: Ignore the negative values
Edit 3: As there appears to be a lot of noise, consider the column 33 which accounts for probability and consider stars only which have >90% probability
Here is are comparison scatterplots using the data in your link, along with the python code I used to read, parse, and plot the data. Note that my plot also has an inverted y axis for direct comparison. This shows me that the data in the posted link, parsed per your directions, cannot be fit as it is per your question. My hope is that you can find some error in my work, and a model can in fact be made.
import matplotlib.pyplot as plt
dataFileName = 'temp.dat'
dataCount = 0
xlist = []
ylist = []
with open(dataFileName) as f:
for line in f:
if line[0] == '#': # comments
continue
spl = line.split()
col3 = float(spl[2])
col9 = float(spl[8])
if col3 < 0.0 or col9 < 0.0:
continue
x = abs(col3 - col9)
y = col3
xlist.append(x)
ylist.append(y)
f = plt.figure()
axes = f.add_subplot(111)
axes.invert_yaxis()
axes.scatter(xlist, ylist,color='black', marker='o', lw=0, s=1)
plt.show()

MatPlotLib Plot last few items differently

I'm exploring MatPlotLib and would like to know if it is possible to show last few items in a dataset differently.
Example: If my dataset contains 100 numbers, I want to display last 5 items in different color.
So far I could do it with one last record using annotate, but want to show last few items dotted with 'red' color as against the blue line.
I could finally achieve this by changing few things in my code.
Below is what I have done.
Let me know in case there is a better way. :)
series_df = pd.read_csv('my_data.csv')
series_df = series_df.fillna(0)
series_df = series_df.sort_values(['Date'], ascending=True)
# Created a new DataFrame for last 5 items series_df2
plt.plot(series_df["Date"],series_df["Values"],color="red", marker='+')
plt.plot(series_df2["Date"],series_df2["Values"],color="blue", marker='+')
You should add some minimal code example or a figure with the desired output to make your question clear. It seems you want to highlight some of the last few points with a marker. You can achieve this by calling plot() twice:
import numpy as np
import matplotlib.pyplot as plt
N = 50
x = np.arange(N)
y = np.random.rand(N)
plt.figure()
plt.plot(x, y)
plt.plot(x[-5:], y[-5:], ls='', c='tab:red', marker='.', ms=10)

Plotting a chart a plot in which the Y text data and X numeric data from dictionary. Matplotlib & Python 3 [duplicate]

I can create a simple columnar diagram in a matplotlib according to the 'simple' dictionary:
import matplotlib.pyplot as plt
D = {u'Label1':26, u'Label2': 17, u'Label3':30}
plt.bar(range(len(D)), D.values(), align='center')
plt.xticks(range(len(D)), D.keys())
plt.show()
But, how do I create curved line on the text and numeric data of this dictionarie, I do not know?
ΠΆ_OLD = {'10': 'need1', '11': 'need2', '12': 'need1', '13': 'need2', '14': 'need1'}
Like the picture below
You may use numpy to convert the dictionary to an array with two columns, which can be plotted.
import matplotlib.pyplot as plt
import numpy as np
T_OLD = {'10' : 'need1', '11':'need2', '12':'need1', '13':'need2','14':'need1'}
x = list(zip(*T_OLD.items()))
# sort array, since dictionary is unsorted
x = np.array(x)[:,np.argsort(x[0])].T
# let second column be "True" if "need2", else be "False
x[:,1] = (x[:,1] == "need2").astype(int)
# plot the two columns of the array
plt.plot(x[:,0], x[:,1])
#set the labels accordinly
plt.gca().set_yticks([0,1])
plt.gca().set_yticklabels(['need1', 'need2'])
plt.show()
The following would be a version, which is independent on the actual content of the dictionary; only assumption is that the keys can be converted to floats.
import matplotlib.pyplot as plt
import numpy as np
T_OLD = {'10': 'run', '11': 'tea', '12': 'mathematics', '13': 'run', '14' :'chemistry'}
x = np.array(list(zip(*T_OLD.items())))
u, ind = np.unique(x[1,:], return_inverse=True)
x[1,:] = ind
x = x.astype(float)[:,np.argsort(x[0])].T
# plot the two columns of the array
plt.plot(x[:,0], x[:,1])
#set the labels accordinly
plt.gca().set_yticks(range(len(u)))
plt.gca().set_yticklabels(u)
plt.show()
Use numeric values for your y-axis ticks, and then map them to desired strings with plt.yticks():
import matplotlib.pyplot as plt
import pandas as pd
# example data
times = pd.date_range(start='2017-10-17 00:00', end='2017-10-17 5:00', freq='H')
data = np.random.choice([0,1], size=len(times))
data_labels = ['need1','need2']
fig, ax = plt.subplots()
ax.plot(times, data, marker='o', linestyle="None")
plt.yticks(data, data_labels)
plt.xlabel("time")
Note: It's generally not a good idea to use a line graph to represent categorical changes in time (e.g. from need1 to need2). Doing that gives the visual impression of a continuum between time points, which may not be accurate. Here, I changed the plotting style to points instead of lines. If for some reason you need the lines, just remove linestyle="None" from the call to plt.plot().
UPDATE
(per comments)
To make this work with a y-axis category set of arbitrary length, use ax.set_yticks() and ax.set_yticklabels() to map to y-axis values.
For example, given a set of potential y-axis values labels, let N be the size of a subset of labels (here we'll set it to 4, but it could be any size).
Then draw a random sample data of y values and plot against time, labeling the y-axis ticks based on the full set labels. Note that we still use set_yticks() first with numerical markers, and then replace with our category labels with set_yticklabels().
labels = np.array(['A','B','C','D','E','F','G'])
N = 4
# example data
times = pd.date_range(start='2017-10-17 00:00', end='2017-10-17 5:00', freq='H')
data = np.random.choice(np.arange(len(labels)), size=len(times))
fig, ax = plt.subplots(figsize=(15,10))
ax.plot(times, data, marker='o', linestyle="None")
ax.set_yticks(np.arange(len(labels)))
ax.set_yticklabels(labels)
plt.xlabel("time")
This gives the exact desired plot:
import matplotlib.pyplot as plt
from collections import OrderedDict
T_OLD = {'10' : 'need1', '11':'need2', '12':'need1', '13':'need2','14':'need1'}
T_SRT = OrderedDict(sorted(T_OLD.items(), key=lambda t: t[0]))
plt.plot(map(int, T_SRT.keys()), map(lambda x: int(x[-1]), T_SRT.values()),'r')
plt.ylim([0.9,2.1])
ax = plt.gca()
ax.set_yticks([1,2])
ax.set_yticklabels(['need1', 'need2'])
plt.title('T_OLD')
plt.xlabel('time')
plt.ylabel('need')
plt.show()
For Python 3.X the plotting lines needs to explicitly convert the map() output to lists:
plt.plot(list(map(int, T_SRT.keys())), list(map(lambda x: int(x[-1]), T_SRT.values())),'r')
as in Python 3.X map() returns an iterator as opposed to a list in Python 2.7.
The plot uses the dictionary keys converted to ints and last elements of need1 or need2, also converted to ints. This relies on the particular structure of your data, if the values where need1 and need3 it would need a couple more operations.
After plotting and changing the axes limits, the program simply modifies the tick labels at y positions 1 and 2. It then also adds the title and the x and y axis labels.
Important part is that the dictionary/input data has to be sorted. One way to do it is to use OrderedDict. Here T_SRT is an OrderedDict object sorted by keys in T_OLD.
The output is:
This is a more general case for more values/labels in T_OLD. It assumes that the label is always 'needX' where X is any number. This can readily be done for a general case of any string preceding the number though it would require more processing,
import matplotlib.pyplot as plt
from collections import OrderedDict
import re
T_OLD = {'10' : 'need1', '11':'need8', '12':'need11', '13':'need1','14':'need3'}
T_SRT = OrderedDict(sorted(T_OLD.items(), key=lambda t: t[0]))
x_val = list(map(int, T_SRT.keys()))
y_val = list(map(lambda x: int(re.findall(r'\d+', x)[-1]), T_SRT.values()))
plt.plot(x_val, y_val,'r')
plt.ylim([0.9*min(y_val),1.1*max(y_val)])
ax = plt.gca()
y_axis = list(set(y_val))
ax.set_yticks(y_axis)
ax.set_yticklabels(['need' + str(i) for i in y_axis])
plt.title('T_OLD')
plt.xlabel('time')
plt.ylabel('need')
plt.show()
This solution finds the number at the end of the label using re.findall to accommodate for the possibility of multi-digit numbers. Previous solution just took the last component of the string because numbers were single digit. It still assumes that the number for plotting position is the last number in the string, hence the [-1]. Again for Python 3.X map output is explicitly converted to list, step not necessary in Python 2.7.
The labels are now generated by first selecting unique y-values using set and then renaming their labels through concatenation of the strings 'need' with its corresponding integer.
The limits of y-axis are set as 0.9 of the minimum value and 1.1 of the maximum value. Rest of the formatting is as before.
The result for this test case is:

Resources