I'm facing a strange issue. I cannot decode the data received through a socket connection while it's working with the same code in python 2.7. I know that the data type received in python 2 is a string an bytes in python 3. But I don't understand why I'm receiving an error when I try to decode.
I'm sending exactly the same datas(copy/paste to be sure) except that I need to perform .encode() for python 3 to avoid to received "TypeError, a bytes-like object is required, not 'str' "
Python2:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(15)
s.connect((SERVERIP, SERVERPORT))
s.send(message)
data = ''
while True:
new_data = s.recv(4096)
if not new_data:
break
data += new_data
s.close()
Python 3
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(15)
s.connect((SERVERIP, SERVERPORT))
s.send(message)
data = ''
while True:
new_data = s.recv(4096)
if not new_data:
break
data += new_data.decode('utf-8') #same result with new_data.decode()
s.close()
Python 2 new_data content:
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\x05\xc1\xdd\x12B#\x18\x00\xd0\x07r\xa3\xb6\xfdv]t\xa1T&\xb5d\x91\xd1tA\x98]F\xfeB\x1a\x0f\xdf9yu\x10s\xa3\xa29:\xdbl\xae\xe9\xe8\xd9H\xc8v\xa8\xd0K\x8c\xde\xd7\xef\xf9\xc4uf\xca\xfd \xdd\xb7\x0c\x9a\x84\xe9\xec\xb7\xf1\xf3\x97o\\k\xd5E\xc3\r\x11(\x9d{\xf7!\xdc*\x8c\xd5\x1c\x0b\xadG\xa5\x1e(\x97dO\x9b\x8f\x14\xaa\xddf\xd7I\x1e\xbb\xd4\xe7a\xe4\xe6a\x88\x8b\xf5\xa0\x08\xab\x11\xda\xea\xb8S\xf0\x98\x94\x1c\x9d\xa24>9\xbai\xd3\x1f\xe6\xcc`^\x91\xca\x02j\x1aLy\xccj\x0fdVn\x17#\xb0\xc1#\x80hX#\xb0\x06\n\x0b\xc0\xf2x\xfe\x01?\x05\x1f\xc1\xc5\x00\x00\x00'
Python3 new_data content:
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\x05\xc1\xdb\x12B#\x00\x00\xd0\x0f\xf2\xc0\xda\xb5\xcbC\x0f"-\xb9gPM\x0f\x85&\x8b)\xb7\x1d\x1a\x1f\xdf9\xe3\xbc\xbe\xfd\x9e\xd9A\xe3:\x851,\xcf\xc4\xe5\x865|\xa5\xcb\xbb\xcbs\xa8\x8f\xcc\x1b\xf7\x06\xc5\x8f\xfa\xba\x84\xd8>\xea\xc0\xa5b\xe6\xceC\xea\xd0\x88\xebM\t\xd7\xf8\xc1*#hI\xd6F\x80\xb3B[\xa7\x99\x91\xbe\x16%Q\xf5\x1d(\xa0\x93\x87\n\x13\xbe\x92\x91\xcc\xbfT\x98b\xd3\x0b=\xc0\xd5\xb3\xdf}\xcc\xc9\xb1\xe4\'\xb1\xe25\xcc{tl\xe5\x92\xf34x\xd5\xa1\xf9K\xa4\xa8k\xa8 dU\xd7\x1e\xce\xb4\x02\xean\xc3\x10#\x05\x13L\x14\xa0(H\xd2d\xb8a\xbc\xdd\xee\x7f\x1b\xe5\xf1\xd2\xc5\x00\x00\x00'
And so when in python3 I'm receiving this error when I try to decode:
'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
The data received are not the same. The difference start after 'x12B#'. Someone has an explanation?
I'm not managing the server side so don't ask me to check this side!
Thanks,
Matthieu
For Python 3 you need to work with bytes, the data you have is not a text string so don't try and interpret it as one.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(15)
s.connect((SERVERIP, SERVERPORT))
s.send(message)
data = b''
while True:
new_data = s.recv(4096)
if not new_data:
break
data += new_data
s.close()
That should be all you need to receive the data: start with an empty bytes object created using b'' or just bytes(), but you will also have to be aware you are working with bytes when you come to process the data so that code will probably need changing as well.
You next step in processing this is probably:
import gzip
text = gzip.decompress(data)
and at this point it may be appropriate to change that to:
text = gzip.decompress(data).decode('ascii')
using whatever encoding is appropriate here (the sample data you posted above only contains ascii when decompressed so that might be all you need, or you might want utf-8 or some other encoding but you'll have to find out what was used to encode the data as you shouldn't attempt to guess). However it looks like it contains some pipe-separated fields so you might want to split the fields first and decode or otherwise process them individually:
fields = gzip.decompress(b).split(b'|')
Related
In an ambitious attempt making a VNC with Python, I am trying to continuously send screenshots of one user (Server) to another (Client). After hours of trying and hitting Ballmer's peak, I managed to do it. However, now my problem is sending multiple images, a continuous stream of them. I first tried to write all the binary data to one file, which didn't work. When the second image was opened, it crashed. I thought this might be because the binary data somehow got corrupted, so instead I tried making a new file for every image, yet I have the same problem. I know that Tcp is a constant stream of data so that it would be hard to know the end of the first image and start of the next, but by creating another file, I thought I would be all good.
Any help in fixing this and/or increasing the efficiency of this is greatly appreciated :)
Server side:
import socket
from PIL import Image, ImageGrab
PORT = 10007
HOST = '127.0.0.1'
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT))
s.listen()
conn, addr = s.accept()
with conn:
counter = 3
while counter > 0:
image = ImageGrab.grab(bbox=None)
image.save('test.png')
f = open('test.png', 'rb')
l = f.read(1024)
while (l):
conn.send(l)
l = f.read(1024)
f.close()
print('Done sending curr image')
counter -= 1
conn.close()
Client side:
import socket
from PIL import Image
HOST = '127.0.0.1'
PORT = 10007
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
counter = 1
while counter != 3:
fname = 'image' + str(counter) + '.png'
with open(fname, 'wb') as file:
print('file opened')
while True:
data = s.recv(1024)
if not data:
break
else:
file.write(data)
file.close()
currImg = Image.open(fname)
currImg.show()
counter += 1
s.close()
Your receiver does not know when one file finishes and the next begins. The easiest way to fix that is to send the length of the file (perhaps as a 4-byte unsigned value) to the receiver before sending each file. Then the receiver can read the length, read the file, read the length, read the file, ...
To improve efficiency you can stop saving the file data into an actual file at both ends and instead save it into (and, obviously, read it from) an in-memory buffer. See this answer for explanations of how to do that. In Python 3 it looks like you would use the BytesIO module.
Another improvement would be to only send image data for the parts of the screen that have changed since the previous send. For that you'll need to figure out how to compare the current capture against the previous one. For a first pass you could use PIL.ImageChops.difference followed by PIL.Image.getbbox and then encode and send only that region of the current capture. For that to work, the sender will have to tell the receiver not only the size of the PNG but also the location in the output screen image where the new image patch should be painted. So you'll want to send a position in addition to the size and the encoded image data.
I'm playing around with sockets and I've got this block of code:
while True:
client_socket.send(bytes('<BHP> ', 'utf-8'))
cmd_buffer = ''
while '\n' not in cmd_buffer:
cmd_buffer += client_socket.recv(1024)
response = run_command(cmd_buffer)
client_socket.send(response)
I know in Python 3 socket.send() method accepts bytes-like arguments.
When I use the syntax bytes(string, [encoding]) it throws an error saying:
TypeError: must be str, not bytes. When I edit the code to take in string
like this:
client_socket.send('<BHP> ')
It throws: TypeError: a bytes-like object is required, not 'str'
So what does it want me to pass in ? Bytes or a string ???
try
client_socket.sendall()
instead of
client_socket.send()
example
import socket
import os
def main():
file_ori = 'Location of file to send'
address = 'Destination IP'
port = 'Destintion Port'
clients_socket = socket.socket(socet.AF_INET, socet.SOCK_STREAM)
client_socet.connect((adrees,int(port))
file = open(file_ori, "rb")
file_size = os.path.getsize(file_ori)
data = file.read(file_size)
file.close()
client_socket.sendall(data)
client_socket.close()
if __name__ == '__main__':
main()
if you want to send a value,
BHP = 'hello world'
#BHP = str(5.3482) if you want to send a number, change it's type to string first.
data = BHP.encode()
client_socet.sendall(data)
you can do also
data = BHP.encode('utf-16')
if you want.
'utf-8' is default anyway.
We have a socket in python3 that receive x bytes from client, our problem lies when the client sends more bytes than x, when this happens our buffer gets overwritten and we lose previous bytes. We need a way to avoid lose the first bytes. We'll appreciate any help. Thanks!
class Connection(object):
def __init__(self, socket, directory):
self.sock = socket
self.directory = directory
def handle(self):
while(True):
data = self.sock.recv(4096)
if len(data) > 0:
...
we expect to stop the socket from receving or some way to avoid losing the bytes that we already have in the buffer
You could do the following:
def receivallData(sock, buffer_size=4096):
buf = sock.recv(buffer_size)
while buf:
yield buf
if len(buf) < buffer_size: break
buf = sock.recv(buffer_size)
You can read more on this here:
Python Socket Receive Large Amount of Data
you can follow this logic:
create a buffer in which you store all the receive data
append the data you receive at each iteration of the loop to this buffer so you won't lose it
check if you receive the full data in order to process it
The example below explains how to create the buffer and append the data to it (in the example I exit the loop if no more data is available on the socket or socket closed)
total_data = []
while True:
data = self.sock.recv(4096)
if not data:
break
total_data.append(data)
# TODO: add processing on total_data
print "".join(total_data)
when I write the bytes into serial, I get the data which I want, but there's a None appearing that my serial doesn't have this one.How can I delete the "None"?
Thanks!
#encoding = utf-8
import serial
def getValues():
ser = serial.Serial('COM3', baudrate=9600, timeout=1)
str = 'ZREADMACS'
ser.write(bytes(str, encoding='utf8'))
data = ser.readline()
print(data)
while 1:
# data = ser.readline()
# if data:
# print(str(data)[2:63])
# else:
# print(getValues())
userInput = input('Are you kidding me?')
if userInput == "1":
print(getValues())
output like this, but I don't want "None".
Are you kidding me?1
b'client:TESTR~~address:R0003~~radiation:01000~~voltage:21000~~current:21000~~temprature:01000~~Li_power:02000~~time_delay:00010~~acs_712_zero:00000~~'
None
Are you kidding me?1
b'client:TESTR~~address:R0003~~radiation:01000~~voltage:21000~~current:21000~~temprature:01000~~Li_power:02000~~time_delay:00010~~acs_712_zero:00000~~'
None
On your code you call print twice.
On the first time you call it with data and data is assigned to the readline result.
On the second time, you call it directly on the getValues() result.
The problem here is that getValues() returns nothing so python translates it as None and then you print it.
If you remove the last print it will be good.
You can also add return data to your getValues() function.
This question already has an answer here:
TypeError: str does not support buffer interface [duplicate]
(1 answer)
Closed 6 years ago.
`import socket
def Main():
host = "127.0.0.1"
port = 2000
s = socket.socket()
s.connect((host, port))
message = input("-> ")
while message != "q":
s.send(message)
data = s.recv(1024)
print("Received from server: " + str(data))
message = input("-> ")
s.close()
if __name__ == '__main__':
Main() #a problem with this code`
errors are given here(http://pastebin.com/AvaGSJzB)
please, help me
i don't know what to do
The error means that you are trying to pass a Unicode string to a interface that expects binary data such as bytes, bytearray:
encoding = 'utf-8'
s.send(message.encode(encoding))
data = s.recv(1024).decode(encoding) # assume the whole answer arrived
Or you could wrap the socket object in a text file using s.makefile(), to avoid encoding the text manually.