DataFrame merging with ordered indices and different columns - python-3.x

Question

I have two pandas data-frames, which I wanted to merge. The data-frames have different columns and overlapping indices. I want to merge them, keeping the order of indices intact.
Dataframe (d1)
Dec 16 Dec 15
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN <-- 'all Nan' row
Other Long Term Liabilities 8.37 9.30
Long Term Provisions 13.53 12.74 <-- Not present in d2
Total Non-Current Liabilities 21.90 22.04
CURRENT LIABILITIES NaN NaN <-- 'all Nan' row
Trade Payables 32.49 24.26
Dataframe (d2)
Dec 11 Dec 10
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN
Deferred Tax Liabilities [Net] 0.00 7.40 <-- Not present in d1
Other Long Term Liabilities 14.13 0.00
Total Non-Current Liabilities 14.13 7.40
CURRENT LIABILITIES NaN NaN
Trade Payables 77.35 60.40
I tried the following ways to merge these data-frames, but none of them worked.
d1.merge(d2, how='left', left_index=True,right_index=True)
d1.merge(d2, how='outer', left_index=True,right_index=True)
pd.merge_ordered(d1,d2,left_on=['Dec 16'],right_on=['Dec 11'])
pd.concat([d1.merge(d2, how='left', left_index=True,right_index=True),d1.merge(d2, how='right', left_index=True,right_index=True)]).drop_duplicates(subset='Dec 16',keep='last')
I am expecting the resulting dataframe to look like this
Dec 16 Dec 15 Dec 11 Dec 10
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN NaN NaN
Deferred Tax Liabilities [Net] NaN NaN 0.00 7.40 <-- from d2
Other Long Term Liabilities 8.37 9.30 14.13 0.00 <-- d1+d2 merged
Long Term Provisions 13.53 12.74 NaN NaN <-- from d1
Total Non-Current Liabilities 21.90 22.04 14.13 7.40 <-- d1+d2 merged
CURRENT LIABILITIES NaN NaN NaN NaN
Trade Payables 32.49 24.26 77.35 60.40
Note that the overall order matters (e.g all NaN rows need to be in same order), but not the order of merged indices between the 'all NaN' rows. Also the columns of d1 should come prior to d2 columns.

Answer 1

Use how=outer with merge and reindex with custom order
In [1424]: order_index = ['NON-CURRENT LIABILITIES', 'Deferred Tax Liabilities [Net]',
'Other Long Term Liabilities', 'Long Term Provisions',
'Total Non-Current Liabilities', 'CURRENT LIABILITIES',
'Trade Payables']
In [1425]: df1.merge(df2,how='outer',left_index=True,right_index=True).reindex(order_index)
Out[1425]:
Dec 16 Dec 15 Dec 11 Dec 10
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN NaN NaN
Deferred Tax Liabilities [Net] NaN NaN 0.00 7.4
Other Long Term Liabilities 8.37 9.30 14.13 0.0
Long Term Provisions 13.53 12.74 NaN NaN
Total Non-Current Liabilities 21.90 22.04 14.13 7.4
CURRENT LIABILITIES NaN NaN NaN NaN
Trade Payables 32.49 24.26 77.35 60.4
Also, join works
In [1426]: df1.join(df2, how='outer').reindex(order_index)
Out[1426]:
Dec 16 Dec 15 Dec 11 Dec 10
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN NaN NaN
Deferred Tax Liabilities [Net] NaN NaN 0.00 7.4
Other Long Term Liabilities 8.37 9.30 14.13 0.0
Long Term Provisions 13.53 12.74 NaN NaN
Total Non-Current Liabilities 21.90 22.04 14.13 7.4
CURRENT LIABILITIES NaN NaN NaN NaN
Trade Payables 32.49 24.26 77.35 60.4
Details
In [1417]: df1
Out[1417]:
Dec 16 Dec 15
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN
Other Long Term Liabilities 8.37 9.30
Long Term Provisions 13.53 12.74
Total Non-Current Liabilities 21.90 22.04
CURRENT LIABILITIES NaN NaN
Trade Payables 32.49 24.26
In [1418]: df2
Out[1418]:
Dec 11 Dec 10
Balance Sheet
NON-CURRENT LIABILITIES NaN NaN
Deferred Tax Liabilities [Net] 0.00 7.4
Other Long Term Liabilities 14.13 0.0
Total Non-Current Liabilities 14.13 7.4
CURRENT LIABILITIES NaN NaN
Trade Payables 77.35 60.4