How to use the condition with other rows (previous moments in time series data), in pandas, python3 - python-3.x

Question

I have a pandas.dataframe df. It is a time series data, with 1000 rows and 3 columns. What I want is given in the pseudo-code below.
for each row
if the value in column 'colA' at [this_row-1] is higher than
the value in column 'B' at [this_row-2] for more than 3%
then set the value in 'colCheck' at [this_row] as True.
Finally, pickout all the rows in the df where 'colCheck' are True.
I will use the following example to further demonstrate my purpose.
df =
'colA', 'colB', 'colCheck'
Dates
2017-01-01, 20, 30, NAN
2017-01-02, 10, 40, NAN
2017-01-03, 50, 20, False
2017-01-04, 40, 10, True
First, when this_row = 2 (the 3rd row, where the date is 2017-01-03), the value in colA at [this_row-1] is 10, the value in colB at [this_row-2] is 30. So (10-30)/30 = -67% < 3%, so the value in colCheck at [this_row] is False.
Likewise, when this_row = 3, (50-40)/40 = 25% > 3%, so the value in colCheck at [this_row] is True.
Last but not least, the first two rows in colCheck should be NAN, since the calculation needs to access [this_row-2] in colB. But the first two rows do not have [this_row-2].
Besides, the criteria of 3% and [row-1] in colA, [row-2] in colB are just examples. In my real project, they are situational, e.g. 4% and [row-3].
I am looking for concise and elegant approach. I am using Python3.
Thanks.

Answer 1

You can rearrange the maths and use pd.Series.shift
df.colA.shift(1).div(df.colB.shift(2)).gt(1.03)
Dates
2017-01-01 False
2017-01-02 False
2017-01-03 False
2017-01-04 True
dtype: bool
Using pd.DataFrame.assign we can create a copy with the new column
df.assign(colCheck=df.colA.shift(1).div(df.colB.shift(2)).gt(1.03))
colA colB colCheck
Dates
2017-01-01 20 30 False
2017-01-02 10 40 False
2017-01-03 50 20 False
2017-01-04 40 10 True
If you insisted on leaving the first two as NaN, you could use iloc
df.assign(colCheck=df.colA.shift(1).div(df.colB.shift(2)).gt(1.03).iloc[2:])
colA colB colCheck
Dates
2017-01-01 20 30 NaN
2017-01-02 10 40 NaN
2017-01-03 50 20 False
2017-01-04 40 10 True
And for maximum clarity:
# This creates a boolean array of when your conditions are met
colCheck = (df.colA.shift(1) / df.colB.shift(2)) > 1.03
# This chops off the first two `False` values and creates a new
# column named `colCheck` and assigns to it the boolean values
# calculate just above.
df.assign(colCheck=colCheck.iloc[2:])
colA colB colCheck
Dates
2017-01-01 20 30 NaN
2017-01-02 10 40 NaN
2017-01-03 50 20 False
2017-01-04 40 10 True