I have a weird issue. After reading about websocket-client, it seems easy enough, send data via websocket.send() and receive via recv().
I have setup my websocket to read and send a binary file. The codes are below
ws = websocket.WebSocket()
ws.connect(uri)
text_list = list()
# setup the receiving portion
receiving_tread = threading.Thread(target=thread_receiving, args=(ws, text_list))
receiving_tread.start()
ws.send_binary([48]) # telling server file has started
f = open(binary_file, 'rb')
chunk = f.read(8000)
while chunk != b'':
ws.send(chunk)
time.sleep(0.5)
chunk = f.read(8000)
ws.send_binary([49]) # tell server file has ended
The process to receive the data:
def thread_receiving(ws, text_list):
data = json.loads(ws.recv())
text_list.append(data)
I know the data gets sent because I get the first part of the data returned, then it stops. So I get only 1 return and somehow the recv() stops listening. Note: I have no access to the server or the server websocket so I have to troubleshoot from the client side.
Can anyone tell me, what it is I'm doing wrong?
Okay, I've found the answer, it seems that the receiving function exits after it runs. So we need to keep it running.
Therefore,
def thread_receiving(ws, text_list):
data = json.loads(ws.recv())
while len(data) > 0
text_list.append(data)
data = json.loads(ws.recv())
Related
I have created a global tweepy stream and I have one function where I want to disconnect it if it's active and after to start to track the word received as argument (with async=True).
When I call the function for the first time it's working properly but after I receive errors.
Here is the sequence where I m creating the stream:
myStreamListener = MyStreamListener()
myStream = tweepy.Stream(auth = api.auth, listener = myStreamListener)
And here I use it:
def cauta_dupa_cuvant(word):
myStream.disconnect()
myStream.filter(languages=["en"], track = [word],is_async=True)
Any idea?
In Python, I am trying to build a simple server/client application that sends video bytes to the client.
I have been trying to figure this out for months.
The server is listening to a connection, and is receiving this much data every second, length of 1316
for example:
data = sock_reciever.recv(16000000)
If I print data, I get this:
b'G\x05\x14\x1e\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x14\x1f\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x14\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x14\x11\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x14\x12\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x14\x13\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffG\x05\x17\x12\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\r\xff\xe0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
As this comes in, I can send it back out via sock_sender.sendto(data, (SENDMCAST_GRP, SENDMCAST_PORT)), and watch the video in VLC player.
Ultimately, I want to take this data and send it to the client through the socket via conn.send(data).
However, on the client side, I believe data is coming in in the wrong order.
I've compared the data between server and client and it appears to be the same, but it builds up on the client side, which might be causing some sort of overflow.
On the client side, I take the data coming in with data = sock_reciever.recv(16000000) and send it back out via sock_sender.sendto(data, (SENDMCAST_GRP, SENDMCAST_PORT)).
But I do not get the same results on the client side, as on the server side.
How do I add headers to this data to ensure it arrives to the client in sequential order?
If possible, please explain this to me like you would explain what oxygen is to a fish.
Which Sockets I Am Using
I am trying to use TCP sockets between server and client. I thought using a stream socket would be more reliable, but I might be wrong, the only reason being that I would like to do a reverse connection, so that the client connects to the server and then receives data.
I am also using sock_sender.sendto(data, (SENDMCAST_GRP, SENDMCAST_PORT)) just to relay the incoming data on the client side, it is not the actual connection between server and client.
The socket used for the communication between server/client
is called sock_client.
This is the server-side code:
sock_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# recieves connection from client
conn, address = sock_client.accept()
# sends data to client
conn.send(data)
Client Side is set up like so:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
data = s.recv(2400000)
Then the data is just relayed back out for use with other applications via sock_sender.sendto(data, (SENDMCAST_GRP, SENDMCAST_PORT)).
I need to ensure the data between the client/server socket connection is accurate and complete.
Let me explain it a little bit. The video can be sent in the form of bytes with the aid of struct. It is part of the python standard library string services.
It interprets strings as packed binary data and performs conversions between Python values and C structs represented as Python strings. This is helpful handling binary data in files, socket network connections and other sources. In server.py, the frame matrix is pickled dump as a string in variable a and then used struct.pack to add a reference payloadsize (for the string in a). The payload_size is attached using + to the a and then .sendall sends the whole frame. To simply explain it, lets suppose our frame is just 3x3 matrix:
frame=np.array([[1,2,3],[4,5,6],[7,8,9]])
a = pickle.dumps(frame)
The pickled form of frame is shown below as a string:
b'\x80\x03cnumpy.core.multiarray\n_reconstruct\nq\x00cnumpy\nndarray\nq\
x01K\x00\x85q\x02C\x01bq\x03\x87q\x04Rq\x05(K\x01K\x03K\x03\x86q\x06cnumpy
\ndtype\nq\x07X\x02\x00\x00\x00i4q\x08K\x00K\x01\x87q\tRq\n(K\x03X\x01\x00
\x00\x00
Here payload_size: len(a) is 193 and its string representation in a 4 byte 'L' format is b'\xc1\x00\x00\x00'.
Then, message = struct.pack("L", len(a))+a
b'\xc1\x00\x00\x00\x80\x03cnumpy.core.multiarray\n_reconstruct\nq\x00cnumpy\nndarray\nq\
x01K\x00\x85q\x02C\x01bq\x03\x87q\x04Rq\x05(K\x01K\x03K\x03\x86q\x06cnumpy
\ndtype\nq\x07X\x02\x00\x00\x00i4q\x08K\x00K\x01\x87q\tRq\n(K\x03X\x01\x00
\x00\x00
And now the message also contains the payload_size infront of the data. The len(message) = 197.
In client.py the s.recv(4096) means at the most 4096 bytes will be received through a blocking call.
The data is appended to the string in 4096 bytes per packet. The rest of the part is quite obvious.
For reference working versions of server.py and client.py are below. First run the server.py and then in another command window run client.py. The transmitted and received frames will be displayed. You can increase the number of maximum bytes. I hope this solves your problem!
server.py
import socket,cv2,pickle
import struct
serversocket = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 9999
serversocket.bind((host, port))
serversocket.listen(5)
while True:
clientsocket,addr = serversocket.accept()
print("Got a connection from %s" % str(addr))
if clientsocket:
vid = cv2.VideoCapture(0)
while(vid.isOpened()):
img, frame = vid.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
print(frame)
a = pickle.dumps(frame)
message =struct.pack("L", len(a))+a
clientsocket.sendall(message)
cv2.imshow('Transmitted',frame)
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
clientsocket.close()
client.py
import socket,cv2,pickle
import struct
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 9999
s.connect((host, port))
data = b""
payload_size = struct.calcsize("L")
while True:
while len(data) < payload_size:
packet= s.recv(4096)
if not packet: break
data += packet
packed_msg_size = data[:payload_size]
data = data[payload_size:]
msg_size = struct.unpack("L", packed_msg_size)[0]
while len(data) < msg_size:
data += s.recv(4096)
frame_data = data[:msg_size]
data = data[msg_size:]
frame=pickle.loads(frame_data)
cv2.imshow('Received',frame)
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
s.close()
I'm trying to create a simple echo server that can handle multiple connections, I am using select and need to use non blocking sockets but I get an error saying 'BlockingIOError: [Errno 35] Resource temporarily unavailable' on the line where I receive the data.
Here's the server and client code
inputs = [server]
while True:
inready, outready, excready = select.select(inputs, [], [])
for s in inready:
if s == server:
client, address = server.accept()
client.setblocking(0)
print(address)
inready.append(client)
else:
data = s.recv(1024);
if data:
s.send(data)
else:
inputs.remove(s)
s.close()
This little error took me also a while to find:
inready.append(client)
Correct is, of course:
inputs.append(client)
Since inready was accidentally modified, the else block was executed when it shouldn't have been.
I have a little problem with my code, I'm doing a HTTP Proxy Server and I send it a random number of HTTP Request and I want that my program close when I stop of send.
I think the problem is in the accept because the program still working always
I tried to put a recv after the accept for checking if there if empty but the program does't arrive there
My code is the following
from socket import *
from _thread import *
MAX_DATA_RECV = 4096 # max number of bytes we receive at once
def start(port_5, my_port):
s=socket(AF_INET, SOCK_STREAM)
s.bind(('', my_port))
s.listen(1)
while 1:
try:
conn, client_addr = s.accept()
except KeyboardInterrupt:
print('\nProgram closed. Interrupted by the user')
exit()
proxy_thread(conn, client_addr)
s.close()
def proxy_thread(conn, client_addr):
# get the request from browser
request = conn.recv(MAX_DATA_RECV).decode('utf-8')
# parse the first line
first_line = request.split('n')[0]
# get url
url = first_line.split(' ')[1]
# find the webserver and port
http_pos = url.find("://") # find pos of ://
if (http_pos==-1):
temp = url
else:
temp = url[(http_pos+3):] # get the rest of url
port_pos = temp.find(":") # find the port pos (if any)
# find end of web server
webserver_pos = temp.find("/")
if webserver_pos == -1:
webserver_pos = len(temp)
webserver = ""
port = -1
if (port_pos==-1 or webserver_pos < port_pos): # default port
port = 80
webserver = temp[:webserver_pos]
else: # specific port
port = int((temp[(port_pos+1):])[:webserver_pos-port_pos-1])
webserver = temp[:port_pos]
print("Connect to:", webserver, port)
# create a socket to connect to the web server
s = socket(AF_INET, SOCK_STREAM)
s.connect((webserver, port))
s.send(request.encode()) # send request to webserver
print(temp)
while 1:
# receive data from web server
data = s.recv(MAX_DATA_RECV)
if (len(data) > 0):
# send to browser
conn.send(data)
else:
break
s.close()
conn.close()
If someone is able to help me, thanks in advance
I want that my program close when I stop of send
At the network level there is no implicit indicator that the other side will not send anymore. If all requests would be done through a single TCP connection then the end of the connection might be treated as such an indicator. But you are using a new TCP connection for every request so you need to define your own condition(s) how the server should determine that the client will not send anymore.
This could for example a timeout, i.e. if the client has not sent any more request for 20 seconds then the client is treated as dead. Or it might be a special message by the client to signal the end - in which case your code needs to explicitly look for this message.
So I'm connected to an IRC chat using socket.connec
I log in by passing my variables via socket.send
The log in is successful and then I sit in a while true loop using
Socket.recv(1024)
If I just continually print the response everything looks fine, but let's say I want to add to the end of the string... I noticed that socket.recv doesn't always get the full message (only grabs up to 1024 as expected) and the remainder of the message is in the next iteration of the loop.
This makes its it impossible to process the feedback line by line.
Is there a better way to constantly read the data without it getting trunked? Is it possible to figure out the size of the response before receiving it so the buffer can be set dynamically?
TCP is a stream-based protocol. Buffer the bytes received and only extract complete messages from the stream.
For complete lines, look for newline characters in the buffer.
Example server:
import socket
class Client:
def __init__(self,socket):
self.socket = socket
self.buffer = b''
def getline(self):
# if there is no complete line in buffer,
# add to buffer until there is one.
while b'\n' not in self.buffer:
data = self.socket.recv(1024)
if not data:
# socket was closed
return ''
self.buffer += data
# break the buffer on the first newline.
# note: partition(n) return "left of n","n","right of n"
line,newline,self.buffer = self.buffer.partition(b'\n')
return line + newline
srv = socket.socket()
srv.bind(('',5000))
srv.listen(1)
conn,where = srv.accept()
client = Client(conn)
print(f'Client connected on {where}')
while True:
line = client.getline()
if not line:
break
print(line)
Example client:
s=socket()
s.connect(('127.0.0.1',5000))
s.sendall(b'line one\nline two\nline three\nincomplete')
s.close()
Output on server:
Client connected on ('127.0.0.1', 2667)
b'line one\n'
b'line two\n'
b'line three\n'