Matplotlib plot displays date which is not in the pandas dateframe - python-3.x

I have a simple data-frame which has a list of 10 days. Out of those 10 days 1 is a holiday. I have assigned a value (1) to each date in the data frame. I am trying to plot the data. The 5th of January is supposed to be a holiday, but it gets printed on the plot even though I have excluded it during the creation of the dictionary using which the data frame was created. My code is as follows.
from datetime import date, timedelta
import matplotlib.pyplot as plt
from pandas import DataFrame, to_datetime
START_DATE = date(2019,1,1)
DATE_LIST = []
for i in range(10):
DATE = START_DATE + timedelta(i)
if DATE != date(2019,1,5):
DATE_LIST.append(DATE)
DATE_DICTIONARY = {}.fromkeys(DATE_LIST, 1)
DATAFRAME = DataFrame({"Value":DATE_DICTIONARY})
DATAFRAME.reindex(to_datetime(DATAFRAME.index)).plot(legend=False)
plt.show()
The output when the
if DATE != date(2019,1,5):
is removed and the 5th of January is also added to the DATE_LIST looks
something like this
Figure when the january 5th is added
I want the exact same format of x axis but, without 5th of januray but upon execution of my code this is what I get.
Figure when january 5th is excluded

It's not surprising that the number 5 is still present between 4 and 6, independent on if you actually have that number in your data.
If you want to treat dates as categories you can use matplotlib.
from datetime import date, timedelta
import matplotlib.pyplot as plt
from pandas import DataFrame, to_datetime
START_DATE = date(2019,1,1)
DATE_LIST = []
for i in range(10):
DATE = START_DATE + timedelta(i)
if DATE != date(2019,1,5):
DATE_LIST.append(DATE)
DATE_DICTIONARY = {}.fromkeys(DATE_LIST, 1)
df = DataFrame({"Value":DATE_DICTIONARY})
df = df.reindex(to_datetime(df.index))
plt.plot(df.index.astype(str), df.values)
plt.setp(plt.gca().get_xticklabels(), rotation=45, ha="right")
plt.tight_layout()
plt.show()

Related

Keyerror in time/Date Components of datetime - what to do?

I am using a pandas DataFrame with datetime indexing. I know from the
Xarray documentation, that datetime indexing can be done as ds['date.year'] with ds being the DataArray of xarray, date the date index and years of the dates. Xarray points to datetime components which again leads to DateTimeIndex, the latter being panda documentation. So I thought of doing the same with pandas, as I really like this feature.
However, it is not working for me. Here is what I did so far:
# Import required modules
import pandas as pd
import numpy as np
# Create DataFrame (name: df)
df=pd.DataFrame({'Date': ['2017-04-01','2017-04-01',
'2017-04-02','2017-04-02'],
'Time': ['06:00:00','18:00:00',
'06:00:00','18:00:00'],
'Active': [True,False,False,True],
'Value': np.random.rand(4)})
# Combine str() information of Date and Time and format to datetime
df['Date']=pd.to_datetime(df['Date'] + ' ' + df['Time'],format = '%Y-%m-%d %H:%M:%S')
# Make the combined data the index
df = df.set_index(df['Date'])
# Erase the rest, as it is not required anymore
df = df.drop(['Time','Date'], axis=1)
# Show me the first day
df['2017-04-01']
Ok, so this shows me only the first entries. So far, so good.
However
df['Date.year']
results in KeyError: 'Date.year'
I would expect an output like
array([2017,2017,2017,2017])
What am I doing wrong?
EDIT:
I have a workaround, which I am able to go on with, but I am still not satisfied, as this doesn't explain my question. I did not use a pandas DataFrame, but an xarray Dataset and now this works:
# Load modules
import pandas as pd
import numpy as np
import xarray as xr
# Prepare time array
Date = ['2017-04-01','2017-04-01', '2017-04-02','2017-04-02']
Time = ['06:00:00','18:00:00', '06:00:00','18:00:00']
time = [Date[i] + ' ' + Time[i] for i in range(len(Date))]
time = pd.to_datetime(time,format = '%Y-%m-%d %H:%M:%S')
# Create Dataset (name: ds)
ds=xr.Dataset({'time': time,
'Active': [True,False,False,True],
'Value': np.random.rand(4)})
ds['time.year']
which gives:
<xarray.DataArray 'year' (time: 4)>
array([2017, 2017, 2017, 2017])
Coordinates:
* time (time) datetime64[ns] 2017-04-01T06:00:00 ... 2017-04-02T18:00:00
Just in terms of what you're doing wrong, your are
a) trying to call an index as a series
b) chaning commands within a string df['Date'] is a single column df['Date.year'] is a column called 'Date.year'
if you're datetime is the index, then use the .year or dt.year if it's a series.
df.index.year
#or assuming your dtype is a proper datetime (your code indicates it is)
df.Date.dt.year
hope that helps bud.

Convert csv time format

I was trying to convert the time format from CSV file like "21-03-2019 00:10:00" to "2019-03-21 00:10:00" I spent hours on this finally still doesn't work hope you all guys can point out my wrong place.
This time I am using Python 3
import pandas as pd
import datetime
data = pd.read_csv('/Users/dongmintian994410/Downloads/Data/FM02.csv', header=0)
for i in range(0, len(data)):
row = data.iloc[i]['Date Time']
now I can print out the first row which including the time array however I don't know how to continue .
I would like to convert the time format like "21-03-2019 00:10:00" to "2019-03-21 00:10:00"
from datetime import datetime
import pandas as pd
data = pd.read_csv('/Users/dongmintian994410/Downloads/Data_Capiatalwater/FM002 2.csv', header=0)
for i in range(0, len(data)):
row = data.iloc[i]['Date Time']
datetime_str = row
datetime_object = datetime.strptime(datetime_str, '%d/%m/%Y %H:%M:%S')
print(datetime_object)
finally, I figured this out

Empty plot on Bokeh Tutorial Exercise

I'm following the bokeh tutorial and in the basic plotting section, I can't manage to show a plot. I only get the axis. What am I missing?
Here is the code:
df = pd.DataFrame.from_dict(AAPL)
weekapple = df.loc["2000-03-01":"2000-04-01"]
p = figure(x_axis_type="datetime", title="AAPL", plot_height=350, plot_width=800)
p.xgrid.grid_line_color=None
p.ygrid.grid_line_alpha=0.5
p.xaxis.axis_label = 'Time'
p.yaxis.axis_label = 'Value'
p.line(weekapple.date, weekapple.close)
show(p)
I get this:
My result
I'm trying to complete the exercise here (10th Code cell - Exercise with AAPL data) I was able to follow all previous code up to that point correctly.
Thanks in advance!
In case this is still relevant, this is how you should do you selection:
df = pd.DataFrame.from_dict(AAPL)
# Convert date column in df from strings to the proper datetime format
date_format="%Y-%m-%d"
df["date"] = pd.to_datetime(df['date'], format=date_format)
# Use the same conversion for selected dates
weekapple = df[(df.date>=dt.strptime("2000-03-01", date_format)) &
(df.date<=dt.strptime("2000-04-01", date_format))]
p = figure(x_axis_type="datetime", title="AAPL", plot_height=350, plot_width=800)
p.xgrid.grid_line_color=None
p.ygrid.grid_line_alpha=0.5
p.xaxis.axis_label = 'Time'
p.yaxis.axis_label = 'Value'
p.line(weekapple.date, weekapple.close)
show(p)
To make this work, before this code, I have (in my Jupyter notebook):
import numpy as np
from bokeh.io import output_notebook, show
from bokeh.plotting import figure
import bokeh
import pandas as pd
from datetime import datetime as dt
bokeh.sampledata.download()
from bokeh.sampledata.stocks import AAPL
output_notebook()
As descried at, https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html, .loc is used in operations with the index (or boolean lists); date is not in the index in your dataframe (it is a regular column).
I hope this helps.
You dataframe sub-view is empty:
In [3]: import pandas as pd
...: from bokeh.sampledata.stocks import AAPL
...: df = pd.DataFrame.from_dict(AAPL)
...: weekapple = df.loc["2000-03-01":"2000-04-01"]
In [4]: weekapple
Out[4]:
Empty DataFrame
Columns: [date, open, high, low, close, volume, adj_close]
Index: []

Reindexing A Pandas Data Frame gives only Nans

I wanted to do the tutorial Python for Finance, Part I: Yahoo & Google Finance API, pandas, and matplotlib. As the Google API is not working properly, I used IEX data. When I try to reindex the pandas data Frame, to have consecutive dates, all values that were available before are replaced with NaN. The new index I think has something to do with the two different types of the indexes. I am quite new to programming and have no clue how to solve this problem after reading the pandas documentation.
The code looks like this.
Modules used
matplotlib==2.2.3
pandas_datareader==0.6.0
googlefinance.client==1.3.0
pandas==0.23.4
googlefinance==0.7
import pandas as pd
pd.core.common.is_list_like = pd.api.types.is_list_like
import pandas_datareader as pdr
import matplotlib.pyplot as plt
from datetime import datetime
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
#tickers = ["AAPL","MSFT","SPY"]
# We would like all available data from 01/01/2017 until 9/1/2018.
start_date = datetime(2017,1,1)
end_date = datetime(2018,9,1)
# User pandas_reader.data.DataReader to load the desired data.
aapl_data = pdr.DataReader("SPY",'iex',start_date,end_date, pause = 5)
#ms_data = pdr.DataReader('SPY','iex',start_date,end_date,pause = 5)
print(aapl_data.tail(9))
#print(ms_data.tail(9))
print(aapl_data.index.dtype)
close = aapl_data
# Getting all weekdays between start_date and end_date
all_weekdays = pd.date_range(start=start_date, end=end_date, freq='B')
print(all_weekdays.dtype)
# How do we align the existing prices in adj_close with our new set of dates?
# All we need to do is reindex close using all_weekdays as the new index
close = close.reindex(all_weekdays)
# Reindexing will insert missing values (NaN) for the dates that were not present
print(close)
You need convert both the index to datetime dtype.
Use pd.to_datetime : Docs
aapl_data = pdr.DataReader("SPY",'iex',start_date,end_date, pause = 5)
aapl_data.index = pd.to_datetime(aapl_data.index) # converts index to datetime dtype
close = aapl_data
all_weekdays = pd.date_range(start=start_date, end=end_date, freq='B')
close = close.reindex(all_weekdays)
print(aapl_data.index.dtype)
# Output:
datetime64[ns]
print(all_weekdays.dtype)
# Output:
datetime64[ns]

Parse Years in Python 3.4 Pandas and Bokeh from counter dictionary

I'm struggling with creating a Bokeh time series graph from the output of the counter function from collections.
import pandas as pd
from bokeh.plotting import figure, output_file, show
import collections
plotyears = []
counter = collections.Counter(plotyears)
output_file("years.html")
p = figure(width=800, height=250, x_axis_type="datetime")
for number in sorted(counter):
yearvalue = number, counter[number]
p.line(yearvalue, color='navy', alpha=0.5)
show(p)
The output of yearvalue when printed is:
(2013, 132)
(2014, 188)
(2015, 233)
How can I make bokeh make the years as x-axis and numbers as y-axis. I have tried to follow the Time series tutorial, but I can't use the pd.read_csv and parse_dates=['Date'] functionalities since I'm not reading a csv file.
The simple way is to convert your data into a pandas DataFrame (with pd.DataFrame) and after create a datetime column with your year column.
simple example :
import pandas as pd
from bokeh.plotting import figure, output_notebook, show
output_notebook()
years = [2012,2013,2014,2015]
val = [230,120,200,340]
# Convert your data into a panda DataFrame format
data=pd.DataFrame({'year':years, 'value':val})
# Create a new column (yearDate) equal to the year Column but with a datetime format
data['yearDate']=pd.to_datetime(data['year'],format='%Y')
# Create a line graph with datetime x axis and use datetime column(yearDate) for this axis
p = figure(width=800, height=250, x_axis_type="datetime")
p.line(x=data['yearDate'],y=data['value'])
show(p)

Resources