What I am trying to figure out is how to add "Cases" and "Deaths" for each day, so that it starts with: "1/19/2020 Cases" and "1/19/2020 Deaths" then "1/20/2020 Cases" etc. It seems the append function does not work for this, and I don't know how else to add this. It doesn't seem like python has a way to do this task. My eventual goal is to make this a pandas dataframe.
import pandas as pd
dates = pd.date_range(start = '1/19/2020', end = '12/31/2021')
lst = dates.repeat(repeats = 2)
print(lst)
Thanks
If I am not mistaken, I don't think there's a way to do it with purely pandas. However with python and datetime, you can do so:
import pandas as pd
from datetime import timedelta, date
def daterange(start_date, end_date):
# Credit: https://stackoverflow.com/a/1060330/10640517
for n in range(int((end_date - start_date).days)):
yield start_date + timedelta(n)
dates = []
start_date = date(2020, 1, 19) # Start date here
end_date = date(2021, 12, 31) # End date here
for single_date in daterange(start_date, end_date):
dates.append(single_date.strftime("%m/%d/%Y") + " Cases")
dates.append(single_date.strftime("%m/%d/%Y") + " Deaths")
pdates = pd.DataFrame(dates)
print (pdates)
Is this what you want? If not, I can delete it.
Related
I am new to Python and I have trouble getting data into one dataframe.
I have the following code.
from pandas_datareader import data as pdr
from datetime import date
from datetime import timedelta
import yfinance as yf
yf.pdr_override()
import pandas as pd
# tickers list
ticker_list = ['0P0001A532.CO','0P00018Q4V.CO','0P00017UBI.CO','0P00000YYT.CO','PFIBAA.CO','PFIBAB.CO','PFIBAC.CO','PFIDKA.CO','PFIGLA.CO','PFIMLO.CO','PFIKRB.CO','0P00019SMI.F','WEKAFKI.CO','0P0001CICW.CO','WEISTA.CO','WEISTS.CO','WEISA.CO','WEITISOP.CO']
today = date.today()
# We can get data by our choice by days bracket
if date.today().weekday()==0:
start_date = (today + timedelta((4 + today.weekday()) % 7)) - timedelta(days=7) # Friday. If it is monday we do not have a price since it is based on the previous day close.
else:
start_date = today - timedelta(days=1)
files=[]
allData = []
dafr_All = []
def getData(ticker):
print(ticker)
data = pdr.get_data_yahoo(ticker, start= start_date, end=(today + timedelta(days=2)))['Adj Close']
dataname = ticker+'_'+str(today)
files.append(dataname)
allData.append(data)
SaveData(data, dataname)
# Create a data folder in your current dir.
def SaveData(df, filename):
df.to_csv('./data/'+filename+'.csv')
#This loop will iterate over ticker list, will pass one ticker to get data, and save that data as file.
for tik in ticker_list:
getData(tik)
for i in range(0,11):
df1= pd.read_csv('./data/'+ str(files[i])+'.csv')
print (df1.head())
I get several csv files containing the adjusted close values (if there exists an adjusted close).
I want to save all the data to a dataframe where the first column consist of tickers, while the second column consist of adjusted close values. The dataframe then needs to be exported into a csv-file.
I am trying to format the column 'Data' to make a pattern with dates.
The formats I have are:
1/30/20 16:00
1/31/2020 23:59
2020-02-02T23:43:02
Here is the code for the dataframe.
import requests
import pandas as pd
import numpy as np
url = "https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports"
csv_only = [i.split("=")[1][1:-1] for i in requests.get(url).text.split(" ") if '.csv' in i and 'title' in i]
combo = [pd.read_csv(url.replace("github","raw.githubusercontent").replace("/tree/","/")+"/"+f) for f in csv_only]
one_df = pd.concat(combo,ignore_index=True)
one_df["País"] = one_df["Country/Region"].fillna(one_df["Country_Region"])
one_df["Data"] = one_df["Last Update"].fillna(one_df["Last_Update"])
I tried adding the code bellow but it doesn't bring the result I wanted
pd.to_datetime(one_df['Data'])
one_df.style.format({"Data": lambda t: t.strftime("%m/%d/%Y")})
Any help?
UPDATE
This is the complete code, but it doesn't work. Many exceptions printed with different date formats.
import requests
import pandas as pd
import numpy as np
from datetime import datetime
url = "https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports"
csv_only = [i.split("=")[1][1:-1] for i in requests.get(url).text.split(" ") if '.csv' in i and 'title' in i]
combo = [pd.read_csv(url.replace("github","raw.githubusercontent").replace("/tree/","/")+"/"+f) for f in csv_only]
one_df = pd.concat(combo,ignore_index=True)
df = pd.DataFrame()
DATE_FORMATS = ["%m/%d/%y %H:%M", "%m/%d/%Y %H:%M", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M:%S"]
df["Região"] = one_df["Province/State"].fillna(one_df["Admin2"])
df["País"] = one_df["Country/Region"].fillna(one_df["Country_Region"])
df["Data"] = one_df["Last Update"].fillna(one_df["Last_Update"])
df["Confirmados"] = one_df["Confirmed"]
df["Mortes"] = one_df["Deaths"]
df["Recuperados"] = one_df["Recovered"]
def parse(x_):
for fmt in DATE_FORMATS :
try:
tmp = datetime.strptime(x_, fmt).strftime("%m/%d/%Y")
return tmp
except ValueError:
print(x_)
pd.to_datetime(df['Data'])
df['Data'] = df['Data'].apply(lambda x: parse(x))
#df['Data'].strftime('%m/%d/%Y')
#df['Data'] = df['Data'].map(lambda x: x.strftime('%m/%d/%Y') if x else '')
df.to_excel(r'C:\Users\guilh\Downloads\Covid2\Covid-19.xlsx', index=False, encoding="utf8")
print(df)
from datetime import datetime
import pandas as pd
You could save all possible formats in a list as -
DATE_FORMATS = ["%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%m/%d/%y %H:%M", "%m/%d/%Y %H:%M"]
Define a function that loops through the formats and tries to parse it.
(Fixed a bug, where the print statement should have been outside the for loop)
issues = set()
def parse(x_):
for fmt in DATE_FORMATS:
try:
return datetime.strptime(x_, fmt).strftime("%m/%d/%Y")
except ValueError:
pass
issues.add(x_)
sample = ["1/30/20 16:00", "1/31/2020 23:59", "2020-02-02T23:43:02"]
df = pd.DataFrame({'data': sample})
df['data'] = df['data'].apply(lambda x: parse(x))
assert df['Data'].isna().sum() == len(issues) == 0, "Issues observed, nulls observed in dataframe"
print("Done")
Output
data
0 01/30/2020
1 01/31/2020
2 02/02/2020
If df.apply() comes across a particular date format that hasn't been defined in the list, it would simply print None since nothing would be returned by the function parse()
also here, letting pd.to_datetime infer the format does the trick:
import pandas as pd
s = pd.to_datetime(["1/30/20 16:00", "1/31/2020 23:59", "2020-02-02T23:43:02"])
print(s)
# DatetimeIndex(['2020-01-30 16:00:00', '2020-01-31 23:59:00',
# '2020-02-02 23:43:02'],
# dtype='datetime64[ns]', freq=None)
Note that if your date/time format generally provides the day first (e.g. 30.1.2021 for Jan 30th 2021), set keyword dayfirst=True.
Say for the year of 2020, how do I iterate through the days in the months so that my outcome would be in the following format:
Jan1
Jan2
Jan3
....
Jan31
Feb1
I've tried so many things online but I couldnt find an answer. Please help :(
Both of these methods will handle leap years correctly out of the box.
Using a simple while loop:
from datetime import datetime, timedelta
def iter_days(year):
dt = datetime(year, 1, 1)
while dt.year == year:
yield dt
dt += timedelta(days=1)
Using date rules:
from datetime import datetime
from dateutil.rrule import rrule, DAILY
def iter_days(year):
first_date = datetime(year, 1, 1)
last_date = datetime(year, 12, 31)
return rrule(DAILY, dtstart=first_date, until=last_date)
Both would be used the same:
for dt in iter_days(2020):
print(dt.strftime('%b%-d'))
The format string '%b%-d' will give you the format you specified in your question. I don't know if that was a requirement or not.
This is crude but gets what you want for 2020. You'll need to change 366 to 365 for non-leap-years.
#!/usr/bin/python3
import datetime
startDate = '2020-01-01'
start = datetime.datetime.strptime(startDate, '%Y-%m-%d')
for dayNum in range(0,366):
dayOfYear = start + datetime.timedelta(days=dayNum)
print(dayOfYear.strftime('%b %d, %Y'))
The calendar module offers quite a bit of functionality.
Here is a solution that works for any given year
import calendar as cal
for mi in range(1,13):
_, days = cal.monthrange(2020, mi)
for d in range(1, days+1):
print(cal.month_name[mi], d)
I have two tables:
-event dates
-return dates
Some event dates are not at a trading day.
How can I change the event date to the next trading day?
So if event date is not in return dates, take the next day in return dates.
The approach to change weekend days to working days does not work because of days like Christmas.
The best would be to look up the next day in the return table.
for i in event['date']:
if i is not in return ['date'].values:
event ['date']=i+datetime.timedelta(days=1)
but this doenst work
I am working with dataframes and dates have the format datetime64[ns]. If the event date does not exist in return date than event date plus one day
Edit
After the clarifications concerning the desired logic, here is the new solution
from datetime import datetime, timedelta
import numpy as np
import pandas as pd
# Create two df
event_date = datetime.now()
event_dates = pd.DataFrame([datetime(2020, 2, _) for _ in range(1, 29)], columns=['date'])
print(event_dates.date[0])
# 2020-02-01 00:00:00
return_dates = pd.DataFrame([datetime(2020, 1, _) for _ in range(1, 32)], columns=['date'])
# Apply logic
event_dates.date = [_ if _ in return_dates.date else _ + timedelta(days=1) for _ in event_dates.date]
print(event_dates.date[0])
# 2020-02-02 00:00:00
Base Python
Here is a solution using the standard datetime library
from datetime import datetime
from typing import List
def get_next_trade_date(date: datetime, date_list: List[datetime]) -> datetime: # The annotations here are just to specify the types of the objects
if date in date_list: # Check if the date is contained in the list
return date
delta, res = None, None # Initialize both to None
for _ in date_list:
tmp = abs((date - _).days) # Time difference in current iteration
if not delta or tmp < delta: # See bullet point 1.
delta, res = tmp, _
return res
if __name__ == '__main__':
event_date = datetime.now()
return_dates = [datetime(2020, 1, _) for _ in range(1, 32)]
print(get_next_trade_date(event_date, return_dates))
# 2020-01-01 00:00:00
Notice that
The condition not delta or tmp < delta is twofold: in the first iteration delta, res are both None so we will overwrite them with tmp, _. We catch this by using not delta. The other part (tmp < delta) is more obvious: if we have a new minimal delta then we overwrite delta, res.
I only considered days intervals ((date - _).days), you could go further into details (see datetime.timedelta for more info)
coming from R I believe there must be a simpler solution using numpy - see below
Numpy
This solution uses numpy. (date_list - date) is an array of timedeltas, (date_list - date).argmin() returns the index of the minimal value.
from datetime import datetime
import numpy as np
def get_next_trade_date(date: datetime, date_list: np.ndarray) -> datetime:
return date_list[(date_list - date).argmin()]
if __name__ == '__main__':
event_date = datetime.now()
return_dates = np.array([datetime(2020, 1, _) for _ in range(1, 32)])
print(get_next_trade_date(event_date, return_dates))
# 2020-01-01 00:00:00
Need help in Pulling stock value of last business day of a month in a time series/dataframe
I am executing the fol code:
import pandas as pd
import datetime
import matplotlib
import warnings
warnings.filterwarnings('ignore')
start = datetime.datetime(2014, 3, 31)
end = datetime.datetime(2017, 9, 30)
stocks = ['AAPL', 'GOOG']
col = 'Adj Close'
df = web.get_data_yahoo(stocks,start,end)
data = df.ix[col]
dataframe = pd.DataFrame(data)
This gives me a dataframe with all the close values.
I want to get the values only from the last business day of the month
Please ignore the question, I've managed to find the answer