How to append chunks of a pickled pandas DataFrame - python-3.x

Question

I have pickled a pandas data frame on my server and I'm sending it via a socket connection and I can receive the data but I can't seem to append the chunks of data back together to the original Dataframe format that's all I'm trying to achieve! I have a feeling its the way I'm appending as its turning into a list because of data = [] but I tried an empty pd Dataframe and that didn't work so I'm kinda a bit lost as to how ill append these values
data = []
FATPACKET = 0
bytelength = self.s.recv(BUFFERSIZE)
length = int(pickle.loads(bytelength))
print(length)
ammo = 0
while True:
print("Getting Data....")
packet = self.s.recv(1100)
FATPACKET = int(sys.getsizeof(packet))
ammo += FATPACKET
print(str(FATPACKET) + ' Got this much of data out of ' +str(length))
print("Getting Data.....")
data.append(packet)
print(ammo)
if not ammo > length:
break
print(data)
unpickled = pickle.loads(data)
self.s.close()
print("Closing Connection!")
print(unpickled)
when I try this code Im constantly running into this
TypeError: a bytes-like object is required, not 'list'
or I run into this
_pickle.UnpicklingError: invalid load key, '\x00'
which is the first couple digits of my pickled Dataframe sorry this is my first time messing around with the pickle module so I'm not very knowledgeable

Answer 1

It would help if we could also see exactly what you're doing on the sending end. However, it's apparent that you have several problems.
First, in the initial recv, it's obvious you intended that to only obtain the initial pickle object you used to encode the length of the remaining bytes. However, that recv might also receive an initial segment of the remaining bytes (or even all of the remaining bytes, depending on how large that is). So how much of it should you give to the initial pickle.loads?
You would be better off creating a fixed length field to contain the size of the remaining data. That is often done with the struct module. On the sending side:
import struct
# Pickle the data to be sent
data = pickle.dumps(python_obj)
data_len = len(data)
# Pack data length into 4 bytes, encoded in network byte order
data_len_to_send = struct.pack('!I', data_len)
# Send exactly 4 bytes (due to 'I'), then send data
conn.sendall(data_len_to_send)
conn.sendall(data)
On the receiving side, as the exception said, pickle.loads takes a byte string not a list. So part of solving this will be to concatenate all the list elements into a single byte string before calling loads:
unpickled = pickle.loads(b''.join(data))
Other issues on receiving side: use len(packet) to get the buffer size. sys.getsizeof provides the internal memory used by the bytes object which includes unspecified interpreter overhead and isn't what you need here.
After recv, the first thing you should do is check for an empty buffer which indicates the end-of-stream (len(packet) == 0 or packet == '' or not packet even). That would happen for example if the sender got killed before completing the send (or the network link goes down, or some bug on the sender side, etc.). Otherwise, if the connection ends prematurely, your program will never reach the break and hence it will be in a very tight infinite loop.
So, altogether you could do something like this:
# First, obtain fixed-size content length
buf = b''
while len(buf) < 4:
tbuf = recv(4 - len(buf))
if tbuf == '':
raise RuntimeError("Lost connection with peer")
buf += tbuf
# Decode (unpack) length (note that unpack returns an array)
len_to_recv = struct.unpack('!I', buf)[0]
data = []
len_recved = 0
while len_recvd < len_to_recv:
buf = self.s.recv(min(len_to_recv - len_recvd, BUFFERSIZE))
if buf == '':
raise RuntimeError("Lost connection with peer")
data.append(buf)
len_recvd += len(buf)
unpickled_obj = pickle.loads(b''.join(data))
EDIT: moved parenthesis