how to Get week number from specified year date in python? - python-3.x
I have a time-series data and i want to get the week number from the initial date
date
20180401
20180402
20180902
20190130
20190401
Things Tried
Code
df["date"]= pd.to_datetime(df.date,format='%Y%m%d')
df["week_no"]= df.date.dt.week
But the week getting reset in 2019 results in getting a common week number of 2018.
is there anything we can do in it ??
You can use this function that will calculate the difference between two days in weeks:
def Wdiff(fromdate, todate):
d = pd.to_datetime(todate) - pd.to_datetime(fromdate)
return int(d / np.timedelta64(1, 'W'))
You can create a datetime object with the specified date, then retrieve the week number using the isocalendar method:
import datetime
myDate = datetime.date(2018, 4, 1)
week = myDate.isocalendar()[1]
print(week)
You could then calculate the total number of remaining weeks in 2018, then add the total number of weeks in each year in between, and finally add the week number of the current date.
For example, this code would print the number of weeks from the 1st of April 2018 to the 6th May 2020:
import datetime
myDate = datetime.date(2018, 4, 1)
currentDate = datetime.date(2020, 5, 6)
weeks = datetime.date(myDate.year, 12, 28).isocalendar()[1] -
myDate.isocalendar()[1]
for i in range(myDate.year, currentDate.year):
weeks += datetime.date(i, 12, 28).isocalendar()[1]
weeks += currentDate.isocalendar()[1]
print(weeks)
Note that because of the way isocalendar works, the 28th of December will always be in the last week of the given year.
The ISO year consists of 52 or 53 full weeks, and where a week starts on a Monday and ends on a Sunday. The first week of an ISO year is the first (Gregorian) calendar week of a year containing a Thursday. This is called week number 1, and the ISO year of that Thursday is the same as its Gregorian year.
You can get more information about isocalendar here: https://docs.python.org/3/library/datetime.html
To get the week number, but as a 2-digit string (with leading zero),
you can run:
df['week_no'] = df.date.dt.strftime('%W')
The result, for slightly extended source data is:
date week_no
0 2018-04-01 13
1 2018-04-02 14
2 2018-09-02 35
3 2018-12-30 52
4 2018-12-31 53
5 2019-01-01 00
6 2019-01-02 00
7 2019-01-03 00
8 2019-01-04 00
9 2019-01-05 00
10 2019-01-06 00
11 2019-01-07 01
12 2019-01-30 04
13 2019-04-01 13
Note that the last day of 2018 (monday) has week No == 53 and "initial" days
in 2019 (up to 2019-01-06 - Sunday) have week No == 00.
If you want this column as int, append .astype(int) to the above code.
Related
Handle ValueError while creating date in pd
I'm reading a csv file with p, day, month, and put it in a df. The goal is to create a date from day, month, current year, and I run into this error for 29th of Feb: ValueError: cannot assemble the datetimes: day is out of range for month I would like when this error occurs, to replace the day by the day before. How can we do that? Below are few lines of my pd and datex at the end is what I would like to get p day month year datex 0 p1 29 02 2021 28Feb-2021 1 p2 18 07 2021 18Jul-2021 2 p3 12 09 2021 12Sep-2021 Right now, my code for the date is only the below, so I have nan where the date doesn't exist. df['datex'] = pd.to_datetime(df[['year', 'month', 'day']], errors='coerce')
You could try something like this : df['datex'] = pd.to_datetime(df[['year', 'month', 'day']], errors='coerce') Indeed, you get NA : p day year month datex 0 p1 29 2021 2 NaT 1 p2 18 2021 7 2021-07-18 2 p3 12 2021 9 2021-09-12 You could then make a particular case for these NA : df.loc[df.datex.isnull(), 'previous_day'] = df.day -1 p day year month datex previous_day 0 p1 29 2021 2 NaT 28.0 1 p2 18 2021 7 2021-07-18 NaN 2 p3 12 2021 9 2021-09-12 NaN df.loc[df.datex.isnull(), 'datex'] = pd.to_datetime(df[['previous_day', 'year', 'month']].rename(columns={'previous_day': 'day'})) p day year month datex previous_day 0 p1 29 2021 2 2021-02-28 28.0 1 p2 18 2021 7 2021-07-18 NaN 2 p3 12 2021 9 2021-09-12 NaN You have to create a new day column if you want to keep day = 29 in the day column.
Find earliest date within daterange
I have the following market data: data = pd.DataFrame({'year': [2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020], 'month': [10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11], 'day': [1,2,5,6,7,8,9,12,13,14,15,16,19,20,21,22,23,26,27,28,29,30,2,3,5,6,9,10,11,12,13,16,17,18,19,20,23,24,25,26,27,30]}) data['date'] = pd.to_datetime(data) data['spot'] = [77.3438,78.192,78.1044,78.4357,78.0285,77.3507,76.78,77.13,77.0417,77.6525,78.0906,77.91,77.6602,77.3568,76.7243,76.5872,76.1374,76.4435,77.2906,79.2239,78.8993,79.5305,80.5313,79.3615,77.0156,77.4226,76.288,76.5648,77.1171,77.3568,77.374,76.1758,76.2325,76.0401,76.0529,76.1992,76.1648,75.474,75.551,75.7018,75.8639,76.3944] data = data.set_index('date') I'm trying to find the spot value for the first day of the month in the date column. I can find the first business day with below: def get_month_beg(d): month_beg = (d.index + pd.offsets.BMonthEnd(0) - pd.offsets.MonthBegin(normalize=True)) return month_beg data['month_beg'] = get_month_beg(data) However, due to data issues, sometimes the earliest date from my data does not match up with the first business day of the month. We'll call the earliest spot value of each month the "strike", which is what I'm trying to find. So for October, the spot value would be 77.3438 (10/1/21) and in Nov it would be 80.5313 (which is on 11/2/21 NOT 11/1/21). I tried below, which only works if my data's earliest date matches up with the first business date of the month (eg it works in Oct, but not in Nov) data['strike'] = data.month_beg.map(data.spot) As you can see, I get NaN in Nov because the first business day in my data is 11/2 (spot rate 80.5313) not 11/1. Does anyone know how to find the earliest date within a date range (in this case the earliest date of each month)? I was hoping the final df would like like below: data = pd.DataFrame({'year': [2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020], 'month': [10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11], 'day': [1,2,5,6,7,8,9,12,13,14,15,16,19,20,21,22,23,26,27,28,29,30,2,3,5,6,9,10,11,12,13,16,17,18,19,20,23,24,25,26,27,30]}) data['date'] = pd.to_datetime(data) data['spot'] = [77.3438,78.192,78.1044,78.4357,78.0285,77.3507,76.78,77.13,77.0417,77.6525,78.0906,77.91,77.6602,77.3568,76.7243,76.5872,76.1374,76.4435,77.2906,79.2239,78.8993,79.5305,80.5313,79.3615,77.0156,77.4226,76.288,76.5648,77.1171,77.3568,77.374,76.1758,76.2325,76.0401,76.0529,76.1992,76.1648,75.474,75.551,75.7018,75.8639,76.3944] data['strike'] = [77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,77.3438,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313,80.5313] data = data.set_index('date')
I Believe, We can get the first() for every year and month combination and later on join that with main data. data2=data.groupby(['year','month']).first().reset_index() #join data 2 with data based on month and year later on year month day spot 0 2020 10 1 77.3438 1 2020 11 2 80.5313 Based on the question, What i have understood is that we need to take every month's first day and respective 'SPOT' column value. Correct me if i have understood it wrong.
Strike = Spot value from first day of each month To do this, we need to do the following: Step 1. Get the Year/Month value from the Date column. Alternate, we can use Year and Month columns you already have in the DataFrame. Step 2: We need to groupby Year and Month. That will give all the records by Year+Month. From this, we need to get the first record (which will be the earliest date of the month). The earliest date can either be 1st or 2nd or 3rd of the month depending on the data in the column. Step 3: By using transform in Groupby, pandas will send back the results to match the dataframe length. So for each record, it will send the same result. In this example, we have only 2 months (Oct & Nov). However, we have 42 rows. Transform will send us back 42 rows. The code: groupby('[year','month'])['date'].transform('first') will give first day of month. Use This: data['dy'] = data.groupby(['year','month'])['date'].transform('first') or: data['dx'] = data.date.dt.to_period('M') #to get yyyy-mm value Step 4: Using transform, we can also get the Spot value. This can be assigned to Strike giving us the desired result. Instead of getting first day of the month, we can change it to return Spot value. The code will be: groupby('date')['spot'].transform('first') Use this: data['strike'] = data.groupby(['year','month'])['spot'].transform('first') or data['strike'] = data.groupby('dx')['spot'].transform('first') Putting all this together The full code to get Strike Price using Spot Price from first day of month import pandas as pd import numpy as np data = pd.DataFrame({'year': [2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020], 'month': [10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11], 'day': [1,2,5,6,7,8,9,12,13,14,15,16,19,20,21,22,23,26,27,28,29,30,2,3,5,6,9,10,11,12,13,16,17,18,19,20,23,24,25,26,27,30]}) data['date'] = pd.to_datetime(data) data['spot'] = [77.3438,78.192,78.1044,78.4357,78.0285,77.3507,76.78,77.13,77.0417,77.6525,78.0906,77.91,77.6602,77.3568,76.7243,76.5872,76.1374,76.4435,77.2906,79.2239,78.8993,79.5305,80.5313,79.3615,77.0156,77.4226,76.288,76.5648,77.1171,77.3568,77.374,76.1758,76.2325,76.0401,76.0529,76.1992,76.1648,75.474,75.551,75.7018,75.8639,76.3944] #Pick the first day of month Spot price as the Strike price data['strike'] = data.groupby(['year','month'])['spot'].transform('first') #This will give you the first row of each month print (data) The output of this will be: year month day date spot strike 0 2020 10 1 2020-10-01 77.3438 77.3438 1 2020 10 2 2020-10-02 78.1920 77.3438 2 2020 10 5 2020-10-05 78.1044 77.3438 3 2020 10 6 2020-10-06 78.4357 77.3438 4 2020 10 7 2020-10-07 78.0285 77.3438 5 2020 10 8 2020-10-08 77.3507 77.3438 6 2020 10 9 2020-10-09 76.7800 77.3438 7 2020 10 12 2020-10-12 77.1300 77.3438 8 2020 10 13 2020-10-13 77.0417 77.3438 9 2020 10 14 2020-10-14 77.6525 77.3438 10 2020 10 15 2020-10-15 78.0906 77.3438 11 2020 10 16 2020-10-16 77.9100 77.3438 12 2020 10 19 2020-10-19 77.6602 77.3438 13 2020 10 20 2020-10-20 77.3568 77.3438 14 2020 10 21 2020-10-21 76.7243 77.3438 15 2020 10 22 2020-10-22 76.5872 77.3438 16 2020 10 23 2020-10-23 76.1374 77.3438 17 2020 10 26 2020-10-26 76.4435 77.3438 18 2020 10 27 2020-10-27 77.2906 77.3438 19 2020 10 28 2020-10-28 79.2239 77.3438 20 2020 10 29 2020-10-29 78.8993 77.3438 21 2020 10 30 2020-10-30 79.5305 77.3438 22 2020 11 2 2020-11-02 80.5313 80.5313 23 2020 11 3 2020-11-03 79.3615 80.5313 24 2020 11 5 2020-11-05 77.0156 80.5313 25 2020 11 6 2020-11-06 77.4226 80.5313 26 2020 11 9 2020-11-09 76.2880 80.5313 27 2020 11 10 2020-11-10 76.5648 80.5313 28 2020 11 11 2020-11-11 77.1171 80.5313 29 2020 11 12 2020-11-12 77.3568 80.5313 30 2020 11 13 2020-11-13 77.3740 80.5313 31 2020 11 16 2020-11-16 76.1758 80.5313 32 2020 11 17 2020-11-17 76.2325 80.5313 33 2020 11 18 2020-11-18 76.0401 80.5313 34 2020 11 19 2020-11-19 76.0529 80.5313 35 2020 11 20 2020-11-20 76.1992 80.5313 36 2020 11 23 2020-11-23 76.1648 80.5313 37 2020 11 24 2020-11-24 75.4740 80.5313 38 2020 11 25 2020-11-25 75.5510 80.5313 39 2020 11 26 2020-11-26 75.7018 80.5313 40 2020 11 27 2020-11-27 75.8639 80.5313 41 2020 11 30 2020-11-30 76.3944 80.5313 Previous Answer to get the first day of each month (within the column data) One way to do it is to create a dummy column to store the first day of each month. Then use drop_duplicates() and retain only the first row. Key assumption: The assumption with this logic is that we have at least 2 rows for each month. If there is only one row for a month, then it will not be part of the duplicates and you will NOT get that month's data. That will give you the first day of each month. import pandas as pd import numpy as np data = pd.DataFrame({'year': [2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020,2020], 'month': [10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11,11], 'day': [1,2,5,6,7,8,9,12,13,14,15,16,19,20,21,22,23,26,27,28,29,30,2,3,5,6,9,10,11,12,13,16,17,18,19,20,23,24,25,26,27,30]}) data['date'] = pd.to_datetime(data) data['spot'] = [77.3438,78.192,78.1044,78.4357,78.0285,77.3507,76.78,77.13,77.0417,77.6525,78.0906,77.91,77.6602,77.3568,76.7243,76.5872,76.1374,76.4435,77.2906,79.2239,78.8993,79.5305,80.5313,79.3615,77.0156,77.4226,76.288,76.5648,77.1171,77.3568,77.374,76.1758,76.2325,76.0401,76.0529,76.1992,76.1648,75.474,75.551,75.7018,75.8639,76.3944] #create a dummy column to store the first day of the month data['dx'] = data.date.dt.to_period('M') #drop duplicates while retaining only the first row of each month dx = data.drop_duplicates('dx',keep='first') #This will give you the first row of each month print (dx) The output of this will be: year month day date spot dx 0 2020 10 1 2020-10-01 77.3438 2020-10 22 2020 11 2 2020-11-02 80.5313 2020-11 If there is only one row for a given month, then you can use groupby the month and take the first record. data.groupby(['dx']).first() This will give you: year month day date spot dx 2020-10 2020 10 1 2020-10-01 77.3438 2020-11 2020 11 2 2020-11-02 80.5313
data['strike']=data.groupby(['year','month'])['spot'].transform('first') I guess this can be achieved by this without creating any other dataframe.
Python3 How to convert date into monthly periods where the first period is September
Working with a group that has a Fiscal Year that starts in September. I have a dataframe with a bunch of dates that I want to calculate a monthly period that = 1 in September. What works: # Convert date column to datetime format df['Hours_Date'] = pd.to_datetime(df['Hours_Date']) # First quarter starts in September - Yes! df['Quarter'] = pd.PeriodIndex(df['Hours_Date'], freq='Q-Aug').strftime('Q%q') What doesn't work: # Gives me monthly periods starting in January. Don't want. df['Period'] = pd.PeriodIndex(df['Hours_Date'], freq='M').strftime('%m') # Gives me an error df['Period'] = pd.PeriodIndex(df['Hours_Date'], freq='M-Aug').strftime('%m') Is there a way to adjust the monthly frequency?
I think it is not implemented, check anchored offsets. Possible solution is subtract or Index.shift 8 for shift by 8 months: rng = pd.date_range('2017-04-03', periods=10, freq='m') df = pd.DataFrame({'Hours_Date': rng}) df['Period'] = (pd.PeriodIndex(df['Hours_Date'], freq='M') - 8).strftime('%m') Or: df['Period'] = pd.PeriodIndex(df['Hours_Date'], freq='M').shift(-8).strftime('%m') print (df) Hours_Date Period 0 2017-04-30 08 1 2017-05-31 09 2 2017-06-30 10 3 2017-07-31 11 4 2017-08-31 12 5 2017-09-30 01 6 2017-10-31 02 7 2017-11-30 03 8 2017-12-31 04 9 2018-01-31 05
I think 'M-Aug' is not applicable for month , so you can do little bit adjust by using np.where, Data From Jez np.where(df['Hours_Date'].dt.month-8<=0,df['Hours_Date'].dt.month+4,df['Hours_Date'].dt.month-8) Out[271]: array([ 8, 9, 10, 11, 12, 1, 2, 3, 4, 5], dtype=int64)
Excel indexmatch, vlookup
I have a holiday calendar for several years in one table. Can anyone help – How to arrange this data by week and show holiday against week? I want to reference this data in other worksheets and hence arranging this way will help me to use formulae on other sheets. I want the data to be: col A having week numbers and column B showing holiday for year 1, col. C showing holiday for year 2, etc. Fiscal Week 2015 2014 2013 2012 Valentine's Day 2 2 2 3 President's Day 3 3 3 4 St. Patrick's Day 7 7 7 7 Easter 10 12 9 11 Mother's Day 15 15 15 16 Memorial Day 17 17 17 18 Flag Day 20 19 19 20 Father's Day 21 20 20 21 Independence Day 22 22 22 23 Labor Day 32 31 31 32 Columbus Day 37 37 37 37 Thanksgiving 43 43 43 43 Christmas 47 47 47 48 New Year's Day 48 48 48 49 ML King Day 51 51 51 52
It's not too clear what year 1 is, so I'm going to assume that's 2015, and year 2 is 2014, etc. Here's how you could set it up, if I understand correctly. Use this index/match formula (psuedo-formula): =Iferror(Index([holiday names range],match([week number],[2015's week numbers in your table],0)),"") It looks like this: (=IFERROR(INDEX($A$3:$A$17,MATCH($H3,B$3:B$17,0)),""), in the cell next to the week numbers) You can then drag the formula over, and the matching group (in above picture, B3:B17) will "slide over" as you drag the formula over.
Spotfire how to calculate end of month
I have a formula in spotfire Sum([sales])/10000*28 But this 28 is nothing but (EOMONTH(X,-1)-EOMONTH(X,-2)) x is nothing but 31Mar2015 so it is like when current month is march it is like end of month of feb - end of the month of jan and I have to populate this for all the months like when the end of period is April then we calculate no of days between end of month of march - end of month of feb . Can anyone pls help
We can simplify this if all you need is the delta value. Correct me if I'm wrong but taking the end of the month of one month minus the EOM of another is really just the number of days in the first value's month. E.g. Input of March then I'm going to look into February and see that it is 28 days (unless leap year then 29). No need for EOMONTH-EOMONTH calculations since these calendar values are static outside of leap years. Below is a spotfire expression that should provide the value you need. I also included some modulo logic to handle leap years (if current month is march then check if leap year. If so then result = 29, otherwise 28.) Case when Month(DateTimeNow()) = 1 then 31 when Month(DateTimeNow()) = 2 then 31 when Month(DateTimeNow()) = 3 and Mod(Year(DateTimeNow()),4)=0 and (Mod(Year(DateTimeNow()),100)!=0 or Mod(Year(DateTimeNow()),400)=0) then 29 when Month(DateTimeNow()) = 3 then 28 when Month(DateTimeNow()) = 4 then 31 when Month(DateTimeNow()) = 5 then 30 when Month(DateTimeNow()) = 6 then 31 when Month(DateTimeNow()) = 7 then 30 when Month(DateTimeNow()) = 8 then 31 when Month(DateTimeNow()) = 9 then 31 when Month(DateTimeNow()) = 10 then 30 when Month(DateTimeNow()) = 11 then 31 when Month(DateTimeNow()) = 12 then 30 end You can replace Month(DateTimeNow()) if you're not actually needing the current month and have an input column. Also, you could add your result calculation Sum([sales])/10000*28 into the above or just insert the above as a calculated column which your formula references: Sum([sales])/10000*[calc_col] Let me know if this works for you and if you have any questions. For reference: Leap Year logic without the messy Spotfire code: if( 0 == year % 4 and (0 != year % 100 or 0 == year % 400) ) { # Then year is a leap year. } Modified from this perl code. Days of Month reference Edit: As per #Niko if we don't use the leap year logic we can clean up our code a little bit. Really just depends on the necessity of leap year logic for this solution. Case Month(DateTimeNow()) when 1 then 31 when 2 then 31 when 3 then 28 when 4 then 31 when 5 then 30 when 6 then 31 when 7 then 30 when 8 then 31 when 9 then 31 when 10 then 30 when 11 then 31 when 12 then 30 end
I take the first of the month,add one month and take one day off. DateAdd('day',-1,date(Year(DateTimeNow()),Month(DateAdd('month',1,DateTimeNow())),1)) DateTimeNow() can be replaced by your date column. Once you have the last day of month for a given date, you can extract the Day(/above expression here/)
came back to this after running into a similar problem. I made #clesiemo3 's answer a little smaller: CASE WHEN Month([Date]) IN (1, 3, 5, 7, 8, 10, 12) THEN 31 WHEN Month([Date]) IN (4, 6, 9, 11) THEN 30 WHEN (Month([Date])=2) AND (Mod(Year([Date]),4)=0) AND ((Mod(Year([Date]),100)!=0) or (Mod(Year([Date]),400)=0)) THEN 29 ELSE 28 END