Sorting dictionary by keys that are dates - sorteddictionary

When developing a telegram bot, you need to sort the dictionary by date, to output news in chronological order. The problem is the difference in key formats (dates). There is the format %d.%m at %H:M%, and there is %d.%m.%Y at %H:M%.
for k,v in sorted(Dnews_dict.items(), key=lambda x: DT.strptime(x[1].get("time"),'%d.%m.%Y at %H:%M')):
news = f"<b>{v['time']}</b>\n"\
f"{hlink(v['title'],v['url'])}"
await message.answer(news)
This code works fine, but only with 1 date type. As an option, i tried to add a string length condition (length is constant).
if len(round_data) == 18:
for k,v in sorted(Dnews_dict.items(), key=lambda x:DT.strptime(x[1].get("time"),'%d.%m.%Y в %H:%M')):
news = f"<b>{v['time']}</b>\n"\
f"{hlink(v['title'],v['url'])}"
await message.answer(news)
else:
for k,v in sorted(Dnews_dict.items(), key=lambda x:DT.strptime(x[1].get("time"),'%d.%m at %H:%M')):
news = f"<b>{v['time']}</b>\n"\
f"{hlink(v['title'],v['url'])}"
await message.answer(news)
But the condition doesn’t work. But that condition doesn’t work. How can this dilemma be resolved?
enter image description here

While I am unfamiliar with the telegraph bot, the following is a way to deal with datetime data having mixed formats e you have isolated the specific text containing the date time data:
So given a list of mixed datetime string data of the following formats:
datelist = ['08.02.2022 at 23:53',
'13.07 at 18:13',
'23.11.2022 at 19:55',
'15.02 at 01:06',
'09.07.2022 at 14:57',
'09.08 at 04:06',
'19.04.2022 at 07:19',
'28.10 at 21:56',
'19.10.2022 at 02:18',
'23.04 at 18:15']
from dateutil import parser #Utility for handling all the messy dates you encounter along the way
from dateutil.relativedelta import *
for dt in datelist:
print(parser.parse(dt))
Yields:
2022-08-02 23:53:00
2023-01-13 18:13:00
2022-11-23 19:55:00
2023-01-15 01:06:00
2022-09-07 14:57:00
2023-01-09 04:06:00
2022-04-19 07:19:00
2023-01-28 21:56:00
2022-10-19 02:18:00
2023-01-23 18:15:00
If the year 2023 is of concern, you can do the following:
for dt in datelist:
dtx = parser.parse(dt)
if dtx.year == 2023:
dtx = dtx + relativedelta(year=2022)
print(dtx)
Which produces a result keyed to the year 2022:
2022-08-02 23:53:00
2022-01-13 18:13:00
2022-11-23 19:55:00
2022-01-15 01:06:00
2022-09-07 14:57:00
2022-01-09 04:06:00
2022-04-19 07:19:00
2022-01-28 21:56:00
2022-10-19 02:18:00
2022-01-23 18:15:00

Related

To check if the continuity of dates are missing in a column

I want to check in my dataframe's column that if there is a missing date for a certain month then the code should output the following month in the format MMM- YYYY
The data set looks like this :
date_start_balance date_end_balance start_balance
22.02.16 22.03.16 3590838
22.04.16 22.05.16 69788
15.06.16 21.07.16 452165
Both date cols are in datetime format. Now in the above data set the dates are missing for March and May in the start col and this should be returned as MMM-YYYYY
I have tried the following code :
import datetime
dates = df1['date_start_balance'].tolist()
missing = []
for i in range(0,len(dates)-1):
if dates[i+1].month - dates[i+1].month != 1:
for j in range(dates[i].month+1,dates[i+1].month):
missing.append(datetime(dates[i].year, j,1))
print(missing)
You can first create a date range with pd.date_range
march = pd.date_range(start='2016-05-01', end='2016-05-31')
And then you will have the list with the dates that you already have, in the example there is only one date: 2016-05-15:
your_list = [datetime.datetime.strptime('15052016', "%d%m%Y").date()]
And then you can calculate the difference between the range and your list and get the dates that you are missing:
march.difference(your_list)
DatetimeIndex(['2016-05-01', '2016-05-02', '2016-05-03', '2016-05-04',
'2016-05-05', '2016-05-06', '2016-05-07', '2016-05-08',
'2016-05-09', '2016-05-10', '2016-05-11', '2016-05-12',
'2016-05-13', '2016-05-14', '2016-05-16', '2016-05-17',
'2016-05-18', '2016-05-19', '2016-05-20', '2016-05-21',
'2016-05-22', '2016-05-23', '2016-05-24', '2016-05-25',
'2016-05-26', '2016-05-27', '2016-05-28', '2016-05-29',
'2016-05-30', '2016-05-31'],
dtype='datetime64[ns]', freq=None)

Replace values in observations (i.e., multiple columns within multiple rows) based on multiple conditionals

I am trying to replace the values of 3 columns within multiple observations based on two conditionals ( e.g., specific ID after a particular date).
I have seen similar questions.
Pandas Multiple Conditions Function based on Column
Pandas replace, multi column criteria
Pandas: How do I assign values based on multiple conditions for existing columns?
Replacing values in a pandas dataframe based on multiple conditions
However, they did not quite address my problem or I can't quite manipulate them to solve my problem.
This code will generate a dataframe similar to mine:
df = pd.DataFrame({'SUR_ID': {0:'SUR1', 1:'SUR1', 2:'SUR1', 3:'SUR1', 4:'SUR2', 5:'SUR2'}, 'DATE': {0:'05-01-2019', 1:'05-11-2019', 2:'06-15-2019', 3:'06-20-2019', 4: '05-15-2019', 5:'06-20-2019'}, 'ACTIVE_DATE': {0:'05-01-2019', 1:'05-01-2019', 2:'05-01-2019', 3:'05-01-2019', 4: '05-01-2019', 5:'05-01-2019'}, 'UTM_X': {0:'444895', 1:'444895', 2:'444895', 3:'444895', 4: '445050', 5:'445050'}, 'UTM_Y': {0:'4077528', 1:'4077528', 2:'4077528', 3:'4077528', 4: '4077762', 5:'4077762'}})
Output Dataframe:
What I am trying to do:
I am trying to replace UTM_X,UTM_Y, AND ACTIVE_DATE with
[444917, 4077830, '06-04-2019']
when
SUR_ID is "SUR1" and DATE >= "2019-06-04 12:00:00"
This is a poorly adapted version of the solution for question 1 in attempts to fix my problem- throws error:
df.loc[[df['SUR_ID'] == 'SUR1' and df['DATE'] >='2019-06-04 12:00:00'], ['UTM_X', 'UTM_Y', 'Active_Date']] = [444917, 4077830, '06-04-2019']
First ensure that the column Date is of type datetime, and then when using 2 conditions, they need to be between parenthesis individually. so you can do:
df.DATE = pd.to_datetime(df.DATE)
df.loc[ (df['SUR_ID'] == 'SUR1') & (df['DATE'] >= pd.to_datetime('2019-06-04 12:00:00')),
['UTM_X', 'UTM_Y', 'ACTIVE_DATE']] = [444917, 4077830, '06-04-2019']
See the difference between what you wrote for the boolean mask:
[df['SUR_ID'] == 'SUR1' and df['DATE'] >='2019-06-04 12:00:00']
and what is here with parenthesis
(df['SUR_ID'] == 'SUR1') & (df['DATE'] >= pd.to_datetime('2019-06-04 12:00:00'))
Use:
df['UTM_X']=df['UTM_X'].mask(df['SUR_ID'].eq('SUR1') & (pd.to_datetime(df['DATE'])>= pd.to_datetime("2019-06-04 12:00:00")),444917)
df['UTM_Y']=df['UTM_Y'].mask(df['SUR_ID'].eq('SUR1') & (pd.to_datetime(df['DATE'])>= pd.to_datetime("2019-06-04 12:00:00")),4077830)
df['ACTIVE_DATE']=df['ACTIVE_DATE'].mask(df['SUR_ID'].eq('SUR1') & (pd.to_datetime(df['DATE'])>= pd.to_datetime("2019-06-04 12:00:00")),'06-04-2019')
Output:
SUR_ID DATE ACTIVE_DATE UTM_X UTM_Y
0 SUR1 05-01-2019 05-01-2019 444895 4077528
1 SUR1 05-11-2019 05-01-2019 444895 4077528
2 SUR1 06-15-2019 06-04-2019 444917 4077830
3 SUR1 06-20-2019 06-04-2019 444917 4077830
4 SUR2 05-15-2019 05-01-2019 445050 4077762
5 SUR2 06-20-2019 05-01-2019 445050 4077762

Simple datetime conversion from integer or string

Is there a simple way to convert a start and end time input into a list of evenly separated times? the input can be string or integer with format 1000,"1000",or "10:00" in 2400hr format. I've managed to accomplish this in a messy looking way, is there a tighter more efficient way to create this list? As you'll notice I created an array first and then called .tolist() to make the time transformation iteration easier. The problem is that an input of 1030 or 1015 would need to be translated into 1050 or 1025 to create the right spacing but if there were a way I could call a datetime.timedelta or something and cleanly make the array?
start="1000"
end="1600"
total_minutes=(int(end[:2])*60)+int(end[2:])-(int(start[:2])*60)-
int(start[2:])
dog=list(range(0,int(total_minutes),25))
walk=dog_df["Walk Length"][dog_df.index[dog_df["Name"]==self.name][0]]
if walk=='half':
self.dogarr=np.array([(x-25,x,x+25,x+50) for x in dog])
elif walk=='full':
self.dogarr=np.array([(x-25,x,x+25,x+50,x+75,x+100) for x in dog])
else:
self.dogarr=np.array([(x,x+25,x+50) for x in dog])
if int(start[2])!=0:
start=start[:2]+str(int(int(start[2:])*1.667))
self.dogarr+=(int(start))
self.dogarr=self.dogarr.tolist()
z=0
while z<len(self.dogarr):
for timespot in self.dogarr[z].copy():
self.dogarr[z][self.dogarr[z].index(timespot)]=time.strftime('%H%M', time.gmtime(self.dogarr[z][self.dogarr[z].index(timespot)]*36))
z+=1
self.dogarr=np.array(self.dogarr)```
array([['1115', '1130', '1145', '1200'],
['1130', '1145', '1200', '1215'],
['1145', '1200', '1215', '1230'],
['1200', '1215', '1230', '1245'],
['1215', '1230', '1245', '1300']], dtype='<U4')
I'm sure you can figure out to parse times from any number of existing questions. The crux of your question seems to be how to create evenly separated times within a range. Here's a simple way:
start = datetime.datetime(2018,12,20,10) # or use strptime etc.
end = datetime.datetime(2018,12,24,18)
count = 10
interval = (end - start) / count
dt = start
while dt <= end:
print(dt)
dt += interval
The output is:
2018-12-20 10:00:00
2018-12-20 20:24:00
2018-12-21 06:48:00
2018-12-21 17:12:00
2018-12-22 03:36:00
2018-12-22 14:00:00
2018-12-23 00:24:00
2018-12-23 10:48:00
2018-12-23 21:12:00
2018-12-24 07:36:00
2018-12-24 18:00:00

How to convert a milliseconds into the normal time inside the list?

I'm parcing a web-site on Python 3.6.2
I got a long list (it has about 17000 symbols):
[[1194570000000, 1806.22], [1194829200000, 1792.02], [1194915600000, 1777.35], [1195002000000, 1783.27], [1195088400000, 1782.55], [1195174800000, 1760.89], [1195434000000, 1751.49], [1195520400000, 1747.99], [1195606800000, 1741.47], [1195693200000, 1726.78], [1195779600000, 1730.69], [1196038800000, 1750.07], [1196125200000, 1738.24], [1196211600000, 1730.1], [1196298000000, 1744.72], [1196384400000, 1759.95], [1196643600000, 1759.74], [1196730000000, 1754.45], [1196816400000, 1772.8], [1196902800000, 1786.09], [1196989200000, 1790.25], [1197248400000, 1796.63], [1197334800000, 1815.07], [1197421200000, 1818.52], [1197507600000, 1816.8], [1197594000000, 1796.78], [1197853200000, 1772.76], [1197939600000, 1789.9], [1198026000000, 1781.98], [1198112400000, 1794.38], [1198198800000, 1797.66], [1198458000000, 1807.15], [1198544400000, 1801.53], [1198630800000, 1799.07], [1198717200000, 1800.78], [1198803600000, 1800.3], [1198890000000, 1800.23], [1199926800000, 1825.09], [1200013200000, 1827.88], [1200272400000, 1828.87], [1200358800000, 1832.82], [1200445200000, 1788.15], [1200531600000, 1746.4], [1200618000000, 1729.26], [1200877200000, 1657.58], [1200963600000, 1606.25], [1201050000000, 1603.64], [1201136400000, 1621.03], [1201222800000, 1657.67], [1201482000000, 1628.03], [1201568400000, 1629.02], [1201654800000, 1617.45], [1201741200000, 1585.1], [1201827600000, 1615.29], [1202086800000, 1666.69], [1202173200000, 1673.41], [1202259600000, 1633.71], [1202346000000, 1625.94], [1202432400000, 1624.1], [1202691600000, 1638.08], [1202778000000, 1701.16], [1202864400000, 1721.58], [1202950800000, 1756.86], [1203037200000, 1732.13], [1203296400000, 1751.02], [1203382800000, 1760.49], [1203469200000, 1755.08], [1203555600000, 1780.1], [1203642000000, 1778.15], [1203987600000, 1786.42], [1204074000000, 1777.88], [1204160400000, 1767.45], [1204246800000, 1764.72], [1204506000000, 1734.83], [1204592400000, 1739.02], [1204678800000, 1730.82], [1204765200000, 1738.13], [1204851600000, 1710.22], [1205197200000, 1714.67], [1205283600000, 1748.51], [1205370000000, 1727.69], [1205456400000, 1735.35], [1205715600000, 1681.17], [1205802000000, 1680.49], [1205888400000, 1687.95], [1205974800000, 1658.55], [1206061200000, 1669.67], [1206320400000, 1666.83], [1206406800000, 1682.01], [1206493200000, 1673.85], [1206579600000, 1692.85], [1206666000000, 1711.2], [1206921600000, 1732.77], [1207008000000, 1747.96], [1207094400000, 1757.32], [1207180800000, 1754.46], [1207267200000, 1745.45], [1207526400000, 1764.22], [1207612800000, 1758.4], [1207699200000, 1770.59], [1207785600000, 1776.8], [1207872000000, 1775.61], [1208131200000, 1759.18], [1208217600000, 1782.33], [1208304000000, 1803.16], [1208390400000, 1819.44], [1208476800000, 1825.26], [1208736000000, 1834.73], [1208822400000, 1819.78], [1208908800000, 1818.49], [1208995200000, 1804.03], [1209081600000, 1797.37], [1209340800000, 1819.62], [1209427200000, 1816.22], [1209513600000, 1788.91], [1209859200000, 1818.08], [1209945600000, 1825.41], [1210032000000, 1835.86], [1210118400000, 1860.42], [1210204800000, 1890.98], [1210550400000, 1912.79], [1210636800000, 1921.4], [1210723200000, 1930.14], [1210809600000, 1961.11], [1210896000000, 1976.18], [1211155200000, 1982.32], [1211241600000, 1962.19], [1211328000000, 1971.47], [1211414400000, 1958.48], [1211500800000, 1937.04], [1211760000000, 1924.79], [1211846400000, 1920.34], [1211932800000, 1910.95], [1212019200000, 1937.63], [1212105600000, 1939.91], [1212364800000, 1954.14], [1212451200000, 1943.88], [1212537600000, 1917.49], [1212624000000, 1908.67], [1212710400000, 1920.71], [1212796800000, 1888.57], [1212969600000, 1902.74], [1213056000000, 1893.78], [1213142400000, 1910.03], [1213574400000, 1934.58], [1213660800000, 1954.97], [1213747200000, 1971.5], [1213833600000, 1967.82], [1213920000000, 1958.94], [1214179200000, 1925.44], [1214265600000, 1897.82], [1214352000000, 1898.14], [1214438400000, 1888.13], [1214524800000, 1853.59], [1214784000000, 1874.87], [1214870400000, 1843.88], [1214956800000, 1836.45], [1215043200000, 1801.29], [1215129600000, 1799.02], [1215388800000, 1801.73], [1215475200000, 1778.26], [1215561600000, 1782.45], [1215648000000, 1781.25], [1215734400000, 1768.2], [1215993600000, 1764.23], [1216080000000, 1737.92], [1216166400000, 1725.59], [1216252800000, 1752.41], [1216339200000, 1738.93], [1216598400000, 1718.19], [1216684800000, 1712.45], [1216771200000, 1718.95], [1216857600000, 1686.71], [1216944000000, 1587.53], [1217203200000, 1581.52], [1217289600000, 1524.75], [1217376000000, 1557.89], [1217462400000, 1581.82], [1217548800000, 1580.59], [1217808000000, 1559.64], [1217894400000, 1513.67], [1217980800000, 1503.23], [1218067200000, 1527.85], [1218153600000, 1482.64], [1218412800000, 1445.45], [1218499200000, 1508.53], [1218585600000, 1514.26], [1218672000000, 1539.56], [1218758400000, 1535.05], [1219017600000, 1535.95], [1219104000000, 1477.17], [1219276800000, 1483.31], [1219363200000, 1476.09], [1219622400000, 1442.8], [1219708800000, 1364.55], [1219795200000, 1367.89], [1219881600000, 1391.75], [1219968000000, 1404.44], [1220227200000, 1426.43], [1220313600000, 1418.14], [1220400000000, 1389.37], [1220486400000, 1360.89], [1220572800000, 1255.55], [1220832000000, 1293.47], [1220918400000, 1239.64], [1221004800000, 1132.71], [1221091200000, 1138.64], [1221177600000, 1157.26], [1221436800000, 1125.8], [1221523200000, 1023.4], [1221609600000, 964.38], [1221696000000, 963.58], [1221782400000, 1096.9], [1222041600000, 1158.72], [1222128000000, 1126.96], [1222214400000, 1138.86], [1222300800000, 1107.07], [1222387200000, 1093.35], [1222646400000, 1056.35], [1222732800000, 994.01], [1222819200000, 1025.51], [1222905600000, 997.97], [1222992000000, 916.35], [1223251200000, 808.1], [1223337600000, 757.5], [1223424000000, 682.82], [1223510400000, 694.25], [1223596800000, 694.19], [1223856000000, 672.24], [1223942400000, 718.12], [1224028800000, 682.73], [1224115200000, 614.08], [1224201600000, 573.74], [1224460800000, 578.35], [1224547200000, 602.25], [1224633600000, 584.62], [1224720000000, 563.86], [1224806400000, 514.53], [1225069200000, 514.39], [1225155600000, 488.76], [1225242000000, 529.27], [1225328400000, 611.05], [1225414800000, 625.17], [1225501200000, 689.32], [1225846800000, 732.17], [1225933200000, 650.25], [1226019600000, 640.83], [1226278800000, 679.3], [1226365200000, 589.93], [1226451600000, 589.87], [1226538000000, 517.05], [1226624400000, 542.3], [1226883600000, 501.9], [1226970000000, 479.59], [1227056400000, 478.47], [1227142800000, 450.07], [1227229200000, 457.15], [1227488400000, 478.68], [1227574800000, 517.51], [1227661200000, 505.31], [1227747600000, 528.05], [1227834000000, 534.75], [1228093200000, 536.86], [1228179600000, 510.96], [1228266000000, 516.41], [1228352400000, 519.82], [1228438800000, 509.61], [1228698000000, 542.01], [1228784400000, 559.07], [1228870800000, 571.89], [1228957200000, 575.07], [1229043600000, 547.36], [1229302800000, 575.82], [1229389200000, 571.53], [1229475600000, 574.57], [1229562000000, 552.97], [1229648400000, 534.69], [1229907600000, 568.61], [1229994000000, 580.45], [1230080400000, 586.15], [1230166800000, 566.55], [1230253200000, 557.38], [1230512400000, 555.76], [1230598800000, 547.45], [1230685200000, 548.16], [1231635600000, 563.58], [1231722000000, 575.88], [1231808400000, 572.75], [1231894800000, 576.31], [1231981200000, 549.98], [1232067600000, 553.97], [1232326800000, 530.95], [1232413200000, 512.12], [1232499600000, 508.34], [1232586000000, 514.01], [1232672400000, 489.04], [1232931600000, 509.06], [1233018000000, 528.42], [1233104400000, 539.64], [1233190800000, 546.1], [1233277200000, 556.09], [1233536400000, 555.23], [1233622800000, 565.5], [1233709200000, 576.94], [1233795600000, 583.89], [1233882000000, 606.37], [1234141200000, 650.84], [1234227600000, 670.48], [1234314000000, 665.39], [1234400400000, 657.63], [1234486800000, 660.24], [1234746000000, 663.05], [1234832400000, 637.9], [1234918800000, 600.87], [1235005200000, 621.45], [1235091600000, 607.74], [1235437200000, 597.27], [1235523600000, 620.61], [1235610000000, 631.12], [1235696400000, 632.53], [1235955600000, 616.06], [1236042000000, 614.71], [1236128400000, 634.31], [1236214800000, 633.7], [1236301200000, 636.42], [1236646800000, 676.8], [1236733200000, 687.33], [1236819600000, 664.34], [1236906000000, 677.91], [1237165200000, 673.76], [1237251600000, 680.96], [1237338000000, 678.9], [1237424400000, 690.34], [1237510800000, 713.94], [1237770000000, 765.98], [1237856400000, 761.21], [1237942800000, 792.88], [1238029200000, 815.54], [1238115600000, 817.43], [1238371200000, 757.88], [1238457600000, 768.05], [1238544000000, 771.56], [1238630400000, 799.54], [1238716800000, 806.71], [1238976000000, 826.06], [1239062400000, 805.96], [1239148800000, 816.81], [1239235200000, 867.51], [1239321600000, 897.57], [1239580800000, 905.51], [1239667200000, 943.68], [1239753600000, 947.12], [1239840000000, 952.44], [1239926400000, 974.06], [1240185600000, 950.03], [1240272000000, 921.33], [1240358400000, 938.02], [1240444800000, 972.55], [1240531200000, 987.71], [1240790400000, 969.15], [1240876800000, 929.82], [1240963200000, 975.75], [1241049600000, 1009.64], [1241395200000, 1042.23], [1241481600000, 1069.43], [1241568000000, 1084.97], [1241654400000, 1163.66], [1241740800000, 1161.41], [1242086400000, 1181.97], [1242172800000, 1176.05], [1242259200000, 1108.11], [1242345600000, 1137.12], [1242604800000, 1121.17], [1242691200000, 1151.36], [1242777600000, 1171.26], [1242864000000, 1164.11], [1242950400000, 1162.57], [1243209600000, 1165.35], [1243296000000, 1142.74], [1243382400000, 1171.37], [1243468800000, 1172.33], [1243555200000, 1203.54], [1243814400000, 1262.5], [1243900800000, 1289.68], [1243987200000, 1268.4], [1244073600000, 1250.81], [1244160000000, 1285.42], [1244419200000, 1262.42], [1244505600000, 1277.5], [1244592000000, 1304.35], [1244678400000, 1300.95], [1245024000000, 1264.32], [1245110400000, 1255.02], [1245196800000, 1233], [1245283200000, 1183.33], [1245369600000, 1192.41], [1245628800000, 1142.32], [1245715200000, 1069.01], [1245801600000, 1118.55], [1245888000000, 1127.34], [1245974400000, 1141.99], [1246233600000, 1124.32], [1246320000000, 1144.29], [1246406400000, 1128.87], [1246492800000, 1122.6], [1246579200000, 1106.46], [1246838400000, 1076.98], [1246924800000, 1076.09], [1247011200000, 1042.85], [1247097600000, 1041.93], [1247184000000, 1010.26], [1247443200000, 997.8], [1247529600000, 1037.49], [1247616000000, 1044.34], [1247702400000, 1050.26], [1247788800000, 1063.75], [1248048000000, 1089.15], [1248134400000, 1101.04], [1248220800000, 1093.33], [1248307200000, 1107.34], [1248393600000, 1145.76], [1248652800000, 1183.04], [1248739200000, 1172.51], [1248825600000, 1152.45], [1248912000000, 1185.01], [1248998400000, 1203.98], [1249257600000, 1249.48], [1249344000000, 1235.59], [1249430400000, 1251.26], [1249516800000, 1252.76], [1249603200000, 1245.49], [1249862400000, 1263.41], [1249948800000, 1250.72], [1250035200000, 1227.87], [1250121600000, 1271.28], [1250208000000, 1269.32], [1250467200000, 1216.47], [1250553600000, 1234.36], [1250640000000, 1222.69], [1250726400000, 1239.34], [1250812800000, 1270.14], [1251072000000, 1309.02], [1251158400000, 1311.1], [1251244800000, 1319.67], [1251331200000, 1322.77], [1251417600000, 1346.3], [1251676800000, 1344.57], [1251763200000, 1372.85], [1251849600000, 1367.47], [1251936000000, 1384.82], [1252022400000, 1381], [1252281600000, 1399.53], [1252368000000, 1451.52], [1252454400000, 1486.98], [1252540800000, 1516.02], [1252627200000, 1535.25], [1252886400000, 1518.48], [1252972800000, 1550.23], [1253059200000, 1577.56], [1253145600000, 1616.07], [1253232000000, 1643.75], [1253491200000, 1620.48], [1253577600000, 1653.28], [1253664000000, 1652.49], [1253750400000, 1632.9], [1253836800000, 1643.01], [1254096000000, 1657.87], [1254182400000, 1690.57], [1254268800000, 1711.31], [1254355200000, 1726.62], [1254441600000, 1668.86], [1254700800000, 1698.15], [1254787200000, 1723.63], [1254873600000, 1796.48], [1254960000000, 1855.47], [1255046400000, 1866.12], [1255305600000, 1909.47], [1255392000000, 1909.81], [1255478400000, 1947.76], [1255564800000, 1947.59], [1255651200000, 1929.36], [1255910400000, 1947.13], [1255996800000, 1977.64], [1256083200000, 1975.57], [1256169600000, 1962.69], [1256256000000, 2009.06], [1256518800000, 2038.76], [1256605200000, 1973.44], [1256691600000, 1900.16], [1256778000000, 1865.5], [1256864400000, 1901.15], [1257123600000, 1867.59], [1257210000000, 1840.98], [1257382800000, 1864.77], [1257469200000, 1880.08], [1257728400000, 1920.11], [1257814800000, 1948.4], [1257901200000, 1962.72], [1257987600000, 1922.58], [1258074000000, 1902.03], [1258333200000, 1949.41], [1258419600000, 1943.86], [1258506000000, 1972.27], [1258592400000, 1945.69], [1258678800000, 1930.92], [1258938000000, 1952.74], [1259024400000, 1928.06], [1259110800000, 1933.98], [1259197200000, 1864.53], [1259283600000, 1820.72], [1259542800000, 1869.23], [1259629200000, 1909.74], [1259715600000, 1914.24], [1259802000000, 1928.28], [1259888400000, 1907.94], [1260147600000, 1908.85], [1260234000000, 1893.61], [1260320400000, 1881.92], [1260406800000, 1876.59], [1260493200000, 1888.93], [1260752400000, 1894.9], [1260838800000, 1915.16], [1260925200000, 1938.58], [1261011600000, 1945.07], [1261098000000, 1962.33], [1261357200000, 1965.04], [1261443600000, 1961.31], [1261530000000, 1953.76], [1261616400000, 1937.68], [1261702800000, 1927.8], [1261962000000, 1940.8], [1262048400000, 1939.3], [1262134800000, 1944.21], [1262221200000, 1956.6], [1263171600000, 2065.07], [1263258000000, 2064.64], [1263344400000, 2090.27], [1263430800000, 2128.52], [1263517200000, 2176.48], [1263776400000, 2194.19], [1263862800000, 2205.26], [1263949200000, 2268.81], [1264035600000, 2294.71], [1264122000000, 2199.22], [1264381200000, 2193.58], [1264467600000, 2144.89], [1264554000000, 2149.68], [1264640400000, 2218.12], [1264726800000, 2237.17], [1264986000000, 2229.98], [1265072400000, 2270.82], [1265158800000, 2334.71], [1265245200000, 2293.67], [1265331600000, 2170.48], [1265590800000, 2154.39], [1265677200000, 2140.96], [1265763600000, 2160.78], [1265850000000, 2157.81], [1265936400000, 2152.04], [1266195600000, 2166.07], [1266282000000, 2208.66], [1266368400000, 2241.27], [1266454800000, 2204.89], [1266541200000, 2194.01], [1266973200000, 2205.58], [1267059600000, 2195.12], [1267146000000, 2206.47], [1267232400000, 2241.65], [1267405200000, 2254.96], [1267491600000, 2259.17], [1267578000000, 2269.09], [1267664400000, 2290.47], [1267750800000, 2316.19], [1268096400000, 2312.49], [1268182800000, 2316.02], [1268269200000, 2300.43], [1268355600000, 2323.75], [1268614800000, 2333.29], [1268701200000, 2339.15], [1268787600000, 2366.15], [1268874000000, 2373.97], [1268960400000, 2367.54], [1269219600000, 2352.68], [1269306000000, 2371.24], [1269392400000, 2359.14], [1269478800000, 2365.72], [1269565200000, 2416.75], [1269820800000, 2421.75], [1269907200000, 2453.62], [1269993600000, 2485.89], [1270080000000, 2520.2], [1270166400000, 2509.26], [1270425600000, 2527.51], [1270512000000, 2549.75], [1270598400000, 2510.39], [1270684800000, 2495.96], [1270771200000, 2537.35], [1271030400000, 2556.07], [1271116800000, 2599.57], [1271203200000, 2646.47], [1271289600000, 2646.61], [1271376000000, 2592.88], [1271635200000, 2500.75], [1271721600000, 2540.25], [1271808000000, 2538.07], [1271894400000, 2509.43], [1271980800000, 2500.58], [1272240000000, 2562.76], [1272326400000, 2519.97], [1272412800000, 2457.82], [1272499200000, 2491.67], [1272585600000, 2501.05], [1272931200000, 2429.29], [1273017600000, 2362.58], [1273104000000, 2322.6], [1273190400000, 2204.22], [1273536000000, 2208.73], [1273622400000, 2243.5], [1273708800000, 2275.73], [1273795200000, 2209.73], [1274054400000, 2196.03], [1274140800000, 2239.5], [1274227200000, 2172.53], [1274313600000, 2113.25], [1274400000000, 2041.37], [1274659200000, 2096.79], [1274745600000, 2019.49], [1274832000000, 2045.77], [1274918400000, 2103.37], [1275004800000, 2117.9], [1275264000000, 2138.24], [1275350400000, 2120.55], [1275436800000, 2137.98], [1275523200000, 2200.41], [1275609600000, 2183.65], [1275868800000, 2120.63], [1275955200000, 2132.57], [1276041600000, 2135.47], [1276128000000, 2138.14], [1276214400000, 2139.16], [1276560000000, 2154.37], [1276646400000, 2166.77], [1276732800000, 2170.14], [1276819200000, 2157], [1277078400000, 2203.37], [1277164800000, 2188.95], [1277251200000, 2177.6], [1277337600000, 2152.36], [1277424000000, 2127.99], [1277683200000, 2136.44], [1277769600000, 2079.88], [1277856000000, 2041.62], [1277942400000, 1996.26], [1278028800000, 1983.89], [1278288000000, 1996.61], [1278374400000, 2029.39], [1278460800000, 2033.7], [1278547200000, 2090.78], [1278633600000, 2071.53], [1278892800000, 2088.88], [1278979200000, 2116.48], [1279065600000, 2115.64], [1279152000000, 2114.95], [1279238400000, 2091.12], [1279497600000, 2087.68], [1279584000000, 2079.61], [1279670400000, 2105.69], [1279756800000, 2113.61], [1279843200000, 2146.42], [1280102400000, 2167.61], [1280188800000, 2175.95], [1280275200000, 2181.87], [1280361600000, 2206.88], [1280448000000, 2199.06], [1280707200000, 2249.88], [1280793600000, 2238.43], [1280880000000, 2244.51], [1280966400000, 2247.07], [1281052800000, 2238.67], [1281312000000, 2260.88], [1281398400000, 2235.64], [1281484800000, 2206.03], [1281571200000, 2183.63], [1281657600000, 2182.92], [1281916800000, 2174.21], [1282003200000, 2209.7], [1282089600000, 2213.49], [1282176000000, 2219.71], [1282262400000, 2203.89], [1282521600000, 2211.16], [1282608000000, 2183.58], [1282694400000, 2185.76], [1282780800000, 2200.34], [1282867200000, 2201.8], [1283126400000, 2229.02], [1283212800000, 2205.47], [1283299200000, 2235.49], [1283385600000, 2265.73], [1283472000000, 2272.88], [1283731200000, 2307.1], [1283817600000, 2289.12], [1283904000000, 2316], [1283990400000, 2336.63], [1284076800000, 2353.65], [1284336000000, 2364.86], [1284422400000, 2345.28], [1284508800000, 2330.23], [1284595200000, 2332.73], [1284681600000, 2329.07], [1284940800000, 2311.08], [1285027200000, 2340.75], [1285113600000, 2322.33], [1285200000000, 2318.23], [1285286400000, 2314.83], [1285545600000, 2322.81], [1285632000000, 2273.99], [1285718400000, 2291.21], [1285804800000, 2299.94], [1285891200000, 2324.88], [1286150400000, 2345.1], [1286236800000, 2349.47], [1286323200000, 2362.85], [1286409600000, 2343.36], [1286496000000, 2324.29], [1286755200000, 2350.54], [1286841600000, 2356.69], [1286928000000, 2379.54], [1287014400000, 2385.66], [1287100800000, 2391.12], [1287360000000, 2391.33], [1287446400000, 2388.31], [1287532800000, 2383.03], [1287619200000, 2368.92], [1287705600000, 2365.76], [1287964800000, 2377.83], [1288051200000, 2374.57], [1288137600000, 2371.17], [1288224000000, 2368.49], [1288310400000, 2362.03], [1288573200000, 2394.35], [1288659600000, 2388.84], [1288746000000, 2391.23], [1289178000000, 2419.32], [1289264400000, 2434.15], [1289350800000, 2421.7], [1289437200000, 2436.95], [1289523600000, 2420.74], [1289610000000, 2420.22], [1289782800000, 2430.49], [1289869200000, 2436.24], [1289955600000, 2452.63], [1290042000000, 2502.14], [1290128400000, 2509.4], [1290387600000, 2521.59], [1290474000000, 2492.79], [1290560400000, 2499.77], [1290646800000, 2539.31], [1290733200000, 2541.65], [1290992400000, 2535.83], [1291078800000, 2548.12], [1291165200000, 2579.55], [1291251600000, 2630.43], [1291338000000, 2684.82], [1291597200000, 2706.76], [1291683600000, 2722.66], [1291770000000, 2699.41], [1291856400000, 2704.61], [1291942800000, 2720.08], [1292202000000, 2755.7], [1292288400000, 2775.37], [1292374800000, 2787.9], [1292461200000, 2782.11], [1292547600000, 2785.73], [1292806800000, 2822.49], [1292893200000, 2824.16], [1292979600000, 2831.9], [1293066000000, 2842.81], [1293152400000, 2842.59], [1293411600000, 2809.56], [1293498000000, 2811.79], [1293584400000, 2807.98], [1293670800000, 2810.86], [1293757200000, 2810.44], [1294707600000, 2916.55], [1294794000000, 2970.84], [1294880400000, 2985.02], [1294966800000, 2970.21], [1295226000000, 3013.12], [1295312400000, 3023.03], [1295398800000, 3034.26], [1295485200000, 2994.76], [1295571600000, 3030.34], [1295830800000, 3004.56], [1295917200000, 3007.85], [1296003600000, 3033.62], [1296090000000, 3054.33], [1296176400000, 3034.96], [1296435600000, 3001.76], [1296522000000, 3047.94], [1296608400000, 3085.91], [1296694800000, 3079.5], [1296781200000, 3081.59], [1297040400000, 3065.24], [1297126800000, 3057.8], [1297213200000, 3037.93], [1297299600000, 2994.59], [1297386000000, 3019.97], [1297645200000, 3066.61], [1297731600000, 3077.68], [1297818000000, 3075.74], [1297904400000, 3098.59], [1297990800000, 3110.37], [1298250000000, 3145.66], [1298336400000, 3114.48], [1298509200000, 3101.39], [1298595600000, 3140.65], [1298854800000, 3181.54], [1298941200000, 3187.78], [1299027600000, 3160.24], [1299114000000, 3185.79], [1299200400000, 3174.66], [1299286800000, 3178.22], [1299632400000, 3205.61], [1299718800000, 3142.39], [1299805200000, 3093.05], [1300064400000, 3095.52], [1300150800000, 2986.98], [1300237200000, 3010.72], [1300323600000, 3017.64], [1300410000000, 3040.04], [1300669200000, 3047.59], [1300755600000, 3023.41], [1300842000000, 3038.04], [1300928400000, 3068.87], [1301014800000, 3083.17], [1301270400000, 3087.51], [1301356800000, 3067.37], [1301443200000, 3098.34], [1301529600000, 3113.9], [1301616000000, 3141.69], [1301875200000, 3174.5], [1301961600000, 3172.26], [1302048000000, 3188.25], [1302134400000, 3176.8], [1302220800000, 3191.89], [1302480000000, 3185.84], [1302566400000, 3127.59], [1302652800000, 3108.23], [1302739200000, 3095.78], [1302825600000, 3089.41], [1303084800000, 3015.84], [1303171200000, 2981.92], [1303257600000, 3033.67], [1303344000000, 3053.12], [1303430400000, 3054.46], [1303689600000, 3063.89], [1303776000000, 3014.28], [1303862400000, 3014.37], [1303948800000, 3007.57], [1304035200000, 2967.66], [1304380800000, 2945.87], [1304467200000, 2901.66], [1304553600000, 2855.25], [1304640000000, 2850.45], [1304985600000, 2878.71], [1305072000000, 2866.19], [1305158400000, 2802.41], [1305244800000, 2802.41], [1305417600000, 2790.78], [1305504000000, 2790.78], [1305590400000, 2802.58], [1305676800000, 2805.54], [1305763200000, 2819.33], [1305849600000, 2795.36], [1306022400000, 2720.91], [1306108800000, 2720.91], [1306195200000, 2738.45], [1306281600000, 2742.91], [1306368000000, 2759.38], [1306454400000, 2774], [1306713600000, 2801.78], [1306800000000, 2829.22], [1306886400000, 2818.64], [1306972800000, 2799.42], [1307059200000, 2806.33], [1307318400000, 2790.76], [1307404800000, 2818.34], [1307491200000, 2836.65], [1307577600000, 2857.04], [1307664000000, 2874.5], [1308009600000, 2879.79], [1308096000000, 2905.71], [1308182400000, 2858.4], [1308268800000, 2849.95], [1308528000000, 2822.09], [1308614400000, 2846.21], [1308700800000, 2845.19], [1308787200000, 2815.21], [1308873600000, 2831.43], [1309132800000, 2814.01], [1309219200000, 2825.49], [1309305600000, 2852.74], [1309392000000, 2869.25], [1309478400000, 2909.79], [1309737600000, 2924.97], [1309824000000, 2926.85], [1309910400000, 2918.67], [1309996800000, 2961.3], [1310083200000, 2967.96], [1310342400000, 2930.53], [1310428800000, 2910.43], [1310515200000, 2927.7], [1310601600000, 2943.07], [1310688000000, 2947.03], [1310947200000, 2943.98], [1311033600000, 2925.17], [1311120000000, 2938.49], [1311206400000, 2941.49], [1311292800000, 2969.48], [1311552000000, 2955.88], [1311638400000, 2970.93], [1311724800000, 2974.8], [1311811200000, 2964.83], [1311897600000, 2959.33], [1312156800000, 2984.21], [1312243200000, 2951.08], [1312329600000, 2908.34], [1312416000000, 2840.94], [1312502400000, 2689.22], [1312761600000, 2599.58], [1312848000000, 2449.5], [1312934400000, 2534.46], [1313020800000, 2435.75], [1313107200000, 2509.19], [1313366400000, 2617.46], [1313452800000, 2607.63], [1313539200000, 2648.05], [1313625600000, 2615.57], [1313712000000, 2531.74], [1313971200000, 2553.75], [1314057600000, 2548.08], [1314144000000, 2538.26], [1314230400000, 2543.95], [1314316800000, 2515.13], [1314576000000, 2603.59], [1314662400000, 2605.23], [1314748800000, 2637.56], [1314835200000, 2630.24], [1314921600000, 2610.72], [1315180800000, 2579.43], [1315267200000, 2579.33], [1315353600000, 2621.89], [1315440000000, 2626.82], [1315526400000, 2598.39], [1315785600000, 2513.64], [1315872000000, 2525.82], [1315958400000, 2527.06], [1316044800000, 2546.5], [1316131200000, 2517.3], [1316390400000, 2486.41], [1316476800000, 2518.97], [1316563200000, 2531.8], [1316649600000, 2408.69], [1316736000000, 2247.31], [1316995200000, 2223.62], [1317081600000, 2291.87], [1317168000000, 2272.35], [1317254400000, 2279.69], [1317340800000, 2257.6], [1317600000000, 2204.21], [1317686400000, 2127.62], [1317772800000, 2097.34], [1317859200000, 2132.02]]
It has following structure:
[Date, price]
How can I convert all milliseconds to normal date (DD/MM/YYYY) inside the list?
Those look like milliseconds since the epoch. You can use the Python datetime library.
import datetime
def timestamp(ms):
return datetime.datetime.fromtimestamp(ms / 1000.0).strftime('%d-%m-%Y %H:%M')
def use_timestamps(lst):
return [[timestamp(ms), price] for ms, price in lst]
Here is the output using the list you provided:
https://pastebin.com/zJx94EpQ

Remove certain dates in list. Python 3.4

I have a list that has several days in it. Each day have several timestamps. What I want to do is to make a new list that only takes the start time and the end time in the list for each date.
I also want to delete the Character between the date and the time on each one, the char is always the same type of letter.
the time stamps can vary in how many they are on each date.
Since I'm new to python it would be preferred to use a lot of simple to understand codes. I've been using a lot of regex so pleas if there is a way with this one.
the list has been sorted with the command list.sort() so it's in the correct order.
code used to extract the information was the following.
file1 = open("test.txt", "r")
for f in file1:
list1 += re.findall('20\d\d-\d\d-\d\dA\d\d\:\d\d', f)
listX = (len(list1))
list2 = list1[0:listX - 2]
list2.sort()
here is a list of how it looks:
2015-12-28A09:30
2015-12-28A09:30
2015-12-28A09:35
2015-12-28A09:35
2015-12-28A12:00
2015-12-28A12:00
2015-12-28A12:15
2015-12-28A12:15
2015-12-28A14:30
2015-12-28A14:30
2015-12-28A15:15
2015-12-28A15:15
2015-12-28A16:45
2015-12-28A16:45
2015-12-28A17:00
2015-12-28A17:00
2015-12-28A18:15
2015-12-28A18:15
2015-12-29A08:30
2015-12-29A08:30
2015-12-29A08:35
2015-12-29A08:35
2015-12-29A10:45
2015-12-29A10:45
2015-12-29A11:00
2015-12-29A11:00
2015-12-29A13:15
2015-12-29A13:15
2015-12-29A14:00
2015-12-29A14:00
2015-12-29A15:30
2015-12-29A15:30
2015-12-29A15:45
2015-12-29A15:45
2015-12-29A17:15
2015-12-29A17:15
2015-12-30A08:30
2015-12-30A08:30
2015-12-30A08:35
2015-12-30A08:35
2015-12-30A10:45
2015-12-30A10:45
2015-12-30A11:00
2015-12-30A11:00
2015-12-30A13:00
2015-12-30A13:00
2015-12-30A13:45
2015-12-30A13:45
2015-12-30A15:15
2015-12-30A15:15
2015-12-30A15:30
2015-12-30A15:30
2015-12-30A17:15
2015-12-30A17:15
And this is how I want it to look like:
2015-12-28 09:30
2015-12-28 18:15
2015-12-29 08:30
2015-12-29 17:15
2015-12-30 08:30
2015-12-30 17:15
First of all, you should convert all your strings into proper dates, Python can work with. That way, you have a lot more control on it, also to change the formatting later. So let’s parse your dates using datetime.strptime in list2:
from datetime import datetime
dates = [datetime.strptime(item, '%Y-%m-%dA%H:%M') for item in list2]
This creates a new list dates that contains all your dates from list2 but as parsed datetime object.
Now, since you want to get the first and the last date of each day, we somehow have to group your dates by the date component. There are various ways to do that. I’ll be using itertools.groupby for it, with a key function that just looks at the date component of each entry:
from itertools import groupby
for day, times in groupby(dates, lambda x: x.date()):
first, *mid, last = times
print(first)
print(last)
If we run this, we already get your output (without date formatting):
2015-12-28 09:30:00
2015-12-28 18:15:00
2015-12-29 08:30:00
2015-12-29 17:15:00
2015-12-30 08:30:00
2015-12-30 17:15:00
Of course, you can also collect that first and last date in a list first to process the dates later:
filteredDates = []
for day, times in groupby(dates, lambda x: x.date()):
first, *mid, last = times
filteredDates.append(first)
filteredDates.append(last)
And you can also output your dates with a different format using datetime.strftime:
for date in filteredDates:
print(date.strftime('%Y-%m-%d %H:%M'))
That would give us the following output:
2015-12-28 09:30
2015-12-28 18:15
2015-12-29 08:30
2015-12-29 17:15
2015-12-30 08:30
2015-12-30 17:15
If you don’t want to go the route through parsing those dates, of course you could also do this simply by working on the strings. Since they are nicely formatted (i.e. they can be easily compared), you can do that as well. It would look like this then:
for day, times in groupby(list2, lambda x: x[:10]):
first, *mid, last = times
print(first)
print(last)
Producing the following output:
2015-12-28A09:30
2015-12-28A18:15
2015-12-29A08:30
2015-12-29A17:15
2015-12-30A08:30
2015-12-30A17:15
Because your data is ordered you just need to pull the first and last value from each group, you can use re.sub to remove the single letter replacing it with a space then split each date string just comparing the dates:
from re import sub
def grp(l):
it = iter(l)
prev = start = next(it).replace("A"," ")
for dte in it:
dte = dte.replace("A"," ")
# if we have a new date, yield that start and end
if dte.split(None, 1)[0] != prev.split(None,1)[0]:
yield start
yield prev
start = dte
prev = dte
yield start, prev
l=["2015-12-28A09:30", "2015-12-28A09:30", .....................
l[:] = grp(l)
This could also certainly be done as your process the file without sorting by using a dict to group:
from re import findall
from collections import OrderedDict
with open("dates.txt") as f:
od = defaultdict(lambda: {"min": "null", "max": ""})
for line in f:
for dte in findall('20\d\d-\d\d-\d\dA\d\d\:\d\d', line):
dte, tme = dte.split("A")
_dte = "{} {}".format(dte, tme)
if od[dte]["min"] > _dte:
od[dte]["min"] = _dte
if od[dte]["max"] < _dte:
od[dte]["max"] = _dt
print(list(od.values()))
Which will give you the start and end time for each date.
[{'min': '2016-01-03 23:59', 'max': '2016-01-03 23:59'},
{'min': '2015-12-28 00:00', 'max': '2015-12-28 18:15'},
{'min': '2015-12-30 08:30', 'max': '2015-12-30 17:15'},
{'min': '2015-12-29 08:30', 'max': '2015-12-29 17:15'},
{'min': '2015-12-15 08:41', 'max': '2015-12-15 08:41'}]
The start for 2015-12-28 is also 00:00 not 9:30.
if you dates are actually as posted one per line you don't need a regex either:
from collections import defaultdict
with open("dates.txt") as f:
od = defaultdict(lambda: {"min": "null", "max": ""})
for line in f:
dte, tme = line.rstrip().split("A")
_dte = "{} {}".format(dte, tme)
if od[dte]["min"] > _dte:
od[dte]["min"] = _dte
if od[dte]["max"] < _dte:
od[dte]["max"] = _dte
print(list(od.values()
Which would give you the same output.

Resources