Memory consumption in Python - lists, subscripting, and pointers - python-3.x

Question

I'm trying to understand how much memory Python objects use.
In the following code, I check the memory of a numpy array vs list as well as a subscripted numpy array:
import sys, os, psutil, numpy as np
def size_of(obj):
return f'{sys.getsizeof(obj) / 1000000:,.0f} MB'
def get_memory_usage():
process = psutil.Process(os.getpid())
return f'{process.memory_info().rss / 1000000:,.0f} MB'
# Numpy vs List
print(f'(1) Mem usage: {get_memory_usage()}')
ONE_HUNDRED_MIL_NP = np.random.randint(-128,127,int(10**8),dtype='int8')
print(f'(2) Mem usage: {get_memory_usage()}, ONE_HUNDRED_MIL_NP: {size_of(ONE_HUNDRED_MIL_NP)}')
ONE_HUNDRED_MIL_LIST = list(np.random.choice(127, int(10**8), replace=True).astype('int8'))
print(f'(3) Mem usage: {get_memory_usage()}, ONE_HUNDRED_MIL_LIST: {size_of(ONE_HUNDRED_MIL_LIST)}')
# Now try subscriping
FOURCOLS = np.random.randint(-128,127,size=(int(10**8),4),dtype='int8')
print(f'(4) Mem usage: {get_memory_usage()}, FOURCOLS: {size_of(FOURCOLS)}')
FOURCOLS_PERMUTED = FOURCOLS[np.random.randint(0,len(FOURCOLS),size=len(FOURCOLS),dtype='int32')]
print(f'(5) Mem usage: {get_memory_usage()}, FOURCOLS_PERMUTED: {size_of(FOURCOLS_PERMUTED)}')
This returns:
(1) Mem usage: 187 MB
(2) Mem usage: 287 MB, ONE_HUNDRED_MIL_NP: 100 MB
(3) Mem usage: 3,526 MB, ONE_HUNDRED_MIL_LIST: 900 MB
(4) Mem usage: 3,926 MB, FOURCOLS: 400 MB
(5) Mem usage: 4,326 MB, FOURCOLS_PERMUTED: 400 MB
Notes:
Output (2) makes sense. One int8 is 8 bits (one byte) and 100 million bytes is 100 MB
Output (3) I don't understand:
The first issue is that sys.getsizeof() shows the objects takes up 900 MB, but psutil shows that the process now takes up 3,239 MB more memory (3526-287=3239). Where is this phantom memory usage coming from?
Where does the 900 MB come from? (From Python: Size of Reference?, I'm assuming that there's 100 MB of the numpy object plus 100 million pointers, which are 8 bytes each, so 100 MB + 800 MB = 900 MB?)
Output (4) Makes sense. 400 million int8s is 400 MB.
Output (5) I don't understand. Is a copy being made or references? If references, we're only referencing 100 million rows, right? How does this make 400 MB?
Thanks

Answer 1

ONE_HUNDRED_MIL_NP = np.random.randint(-128,127,int(10**8),dtype='int8')
This makes an array. ONE_HUNDRED_MIL_NP.nbytes is a good measure of the array size. An array has some basic info like shape, strides, dtype, but the bulk of the space is a 1d data buffer that contains bytes, in this case on byte per element.
ONE_HUNDRED_MIL_LIST = list(np.random.choice(127, int(10**8), replace=True).astype('int8'))
This produces a list from that array. A list has a databuffer that contains references to objects else where in memory. getsizeof just measures that size of that buffer, and says nothing about the objects. Here those objects are numpy.int8 objects, extracted from the array. They don't actually reference elements of the array, but rather are copies of those values.
A better way to get a list from an array is with arr.tolist().
FOURCOLS = np.random.randint(-128,127,size=(int(10**8),4),dtype='int8')
This is just another array. the 2d shape doesn't change how much memory it takes up.
FOURCOLS_PERMUTED = FOURCOLS[np.random.randint(0,len(FOURCOLS),size=len(FOURCOLS),dtype='int32')]
This is an example of advanced indexing. It creates a new array with its own data buffer (not a view with a shared buffer). Yes, you are just indexing the first dimension, but the data buffer stores all values, not references to rows of FOURCOLS.
A list of lists does store references or points to the nested lists. So shuffling the outer list would just shuffle the references. Multidimensional c arrays also store references or pointers. But multidimensional numpy arrays use a different model. The data is a flat c array. Multidimensionality is produced by the shape/strides iteration code.
So looking at your numbers:
(1) Mem usage: 187 MB
base usage.
(2) Mem usage: 287 MB, ONE_HUNDRED_MIL_NP: 100 MB
adds 100mb to base.
(3) Mem usage: 3,526 MB, ONE_HUNDRED_MIL_LIST: 900 MB
The 900 is roughly the memory used by the list's data buffer. The rest of total usage increase is storage for those 10**8 np.int8 objects.
(4) Mem usage: 3,926 MB, FOURCOLS: 400 MB
This shows another 400 MB memory usage.
(5) Mem usage: 4,326 MB, FOURCOLS_PERMUTED: 400 MB
And yet another 400.
Without the (3) list creating mem usage should show an orderly increase by the array size.