With pandas why does `DataFrame(foo) is foo` = False? - python-3.x

Question

Starting with the example...
In [1]: import pandas as pd
In [2]: from sklearn.datasets import load_iris
In [3]: iris = load_iris()
In [4]: X = pd.DataFrame(data=iris.data, columns=iris.feature_names)
In [5]: output_df = pd.DataFrame(X)
In [6]: X is output_df
Out[6]: False
In [7]: list(X.columns)
Out[7]:
['sepal length (cm)',
'sepal width (cm)',
'petal length (cm)',
'petal width (cm)']
In [8]: output_df['y'] = iris.target
In [9]: list(X.columns)
Out[9]:
['sepal length (cm)',
'sepal width (cm)',
'petal length (cm)',
'petal width (cm)',
'y']
[6] says that X is output_df is False, meaning they are not the same object. If they are not the same object, then adding a column to one of them should not affect the other one.
However [9] tells us that adding a column to output_df definitely did add the same column to X, which implies they actually are the same object.
Why is there a disconnect here?
(pd.__version__ == 0.24.1 and python --version = Python 3.7.1, in case it matters)

Answer 1

There's some decoupling between a DataFrame and its underlying data, which is stored in its BlockManager. In your example the underlying BlockManager data is the same, so changing it on one DataFrame will impact the other:
In [1]: import pandas as pd; pd.__version__
Out[1]: '0.24.1'
In [2]: df = pd.DataFrame({'A': list('abc'), 'B': [10, 20, 30]})
In [3]: df2 = pd.DataFrame(df)
In [4]: df is df2
Out[4]: False
In [5]: df._data is df2._data
Out[5]: True
In [6]: df._data
Out[6]:
BlockManager
Items: Index(['A', 'B'], dtype='object')
Axis 1: RangeIndex(start=0, stop=3, step=1)
IntBlock: slice(1, 2, 1), 1 x 3, dtype: int64
ObjectBlock: slice(0, 1, 1), 1 x 3, dtype: object
Essentially DataFrame serves as a wrapper around the underlying data, so these actually are different objects, it's just that certain components of them happen to be shared. As a basic example, you can add dummy attributes to one without impacting the other:
In [7]: df.foo = 'bar'
In [8]: df.foo
Out[8]: 'bar'
In [9]: df2.foo
---------------------------------------------------------------------------
AttributeError: 'DataFrame' object has no attribute 'foo'
To get around the issue of shared underlying data you'll need to explicitly tell the DataFrame constructor to copy the input data via the copy parameter:
In [10]: df2 = pd.DataFrame(df, copy=True)
In [11]: df._data is df2._data
Out[11]: False
In [12]: df['C'] = [1.1, 2.2, 3.3]
In [13]: df
Out[13]:
A B C
0 a 10 1.1
1 b 20 2.2
2 c 30 3.3
In [14]: df2
Out[14]:
A B
0 a 10
1 b 20
2 c 30