After a lot of trials with httpio, requests, socket I'm seeing that no matter how I approach the solution it still behaves like a synchronous program even with an async function. the code is -
import socket
import time
import asyncio
t1 = time.time()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
target = '72.221.171.130'
#async socket to connect to the target with a given port
async def connect(port):
try:
s.connect((target, port))
print(port)
except Exception as e:
print(e)
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait([connect(port) for port in range(4100, 4150)]))
#printing time taken to scan ports two decimal places
print('Time taken to scan ports:', round((time.time() - t1),3), 'seconds')
I looking for the open port which is 4145. but this takes around 20 seconds to execute. I'm a beginner in async programming so I would really appreciate your help.
Related
I'm almost always receiving on a socket in blocking mode and this works fine. Very occasionally I don't want to wait - if there is data on the socket I want it now, otherwise I will try again later.
I thought I could do this using the flags argument to socket.recv(), but it seems not to work. I can achieve the effect I want using the socket.setblocking() and socket.settimeout() calls, but this seems clumsy.
From the python socket documentation the flags argument takes the same meanings as for Unix recv:
MSG_DONTWAIT (since Linux 2.2)
Enables nonblocking operation; if the operation would block, the
call fails with the error EAGAIN or EWOULDBLOCK. This provides
similar behavior to setting the O_NONBLOCK flag (via the fcntl(2)
F_SETFL operation), but differs in that MSG_DONTWAIT is a per-call
option, whereas O_NONBLOCK is a setting on the open file description
(see open(2)), which will affect all threads in the calling process
and as well as other processes that hold file descriptors referring
to the same open file description.
I read this to mean I could pass socket.MSG_DONTWAIT to get non-blocking operation on that call only. Possibly this isn't correct - I could also read this as it would always return an error as the call in principle would be blocking. In which case, this is all irrelevant.
Some example code:
A simple blocking call. As expected, this takes about 0.5s.
MSG_DONTWAIT: I hoped would be a non-blocking call, returning very quickly. Actually this also took about 0.5s
Non-blocking by re-configuring the port. This actually takes about 50us, so it clearly isn't blocking in the way the first two calls were.
import socket
import time
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.settimeout(0.5)
starttime = time.time()
try:
m = sock.recv(100)
except socket.timeout as e:
pass
endtime = time.time()
print(f'sock.recv(100) took {endtime-starttime}s') # 0.5s
starttime = time.time()
try:
m = sock.recv(100, socket.MSG_DONTWAIT)
except socket.timeout as e:
pass
endtime = time.time()
print(f'sock.recv(100, socket.MSG_DONTWAIT) took {endtime-starttime}s') # 0.5s
starttime = time.time()
timeout = sock.gettimeout()
sock.setblocking(0)
try:
m = sock.recv(100)
except BlockingIOError as e:
pass
sock.settimeout(timeout)
endtime = time.time()
print(f'sock.recv(100) with non-blocking set took {endtime-starttime}s') # 4.96e-5s
Questions:
Am I just wrong about the use of MSG_DONTWAIT? Should it work in the way I am trying to use it?
Is there a better way to toggle blocking and non-blocking calls to recv()
Regarding "1. Am I just wrong about the use of MSG_DONTWAIT? Should it work in the way I am trying to use it?":
No, you are not wrong, but there is a small issue with the way you test. Specifically, your
MSG_DONTWAIT test is for a blocking socket with a timeout of 0.5s. This is because you have sock.settimeout(0.5) before your first test (perhaps you overlooked that this affects your second test).
If I update the exception type in your MSG_DONTWAIT test (which is a another indication that the socket is blocking) and try in a "clean" session, I get what you expect you would get:
>>> import socket
>>> import time
>>>
>>> sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
>>> starttime = time.time()
>>> try:
... m = sock.recv(100, socket.MSG_DONTWAIT)
... except BlockingIOError as e:
... pass
...
>>> endtime = time.time()
>>> print(f'sock.recv(100, socket.MSG_DONTWAIT) took {endtime-starttime}s')
sock.recv(100, socket.MSG_DONTWAIT) took 0.0007114410400390625s
If I "forget" to exclude sock.settimeout(0.5), I get a socket.timeout exception after 0.5s:
>>> import socket
>>> import time
>>>
>>> sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
>>> sock.settimeout(0.5) # <= see this
>>>
>>> starttime = time.time()
>>> try:
... m = sock.recv(100, socket.MSG_DONTWAIT)
... except socket.timeout as e:
... pass
...
>>> endtime = time.time()
>>>
>>> print(f'sock.recv(100, socket.MSG_DONTWAIT) took {endtime-starttime}s')
sock.recv(100, socket.MSG_DONTWAIT) took 0.501746416091919s
Regarding "2. Is there a better way to toggle blocking and non-blocking calls to recv()": depending on the needs of your application, you may want to take a look at select (and the "Non-blocking Sockets" section in Socket Programming HOWTO and this)
What I’m Trying to Do:
I’m trying to create a client script that can listen for potential messages from the server and receive input from a user at any point that tells the script to send a message to the server all on a single socket. The server I’m working with will only connect to one client at a time, so everything is on one socket.
The script will facilitate interaction between a maya ui and a file with prewritten functions that determine what to send to the server.
How it works:
I have the script with two threads. The parent thread is the adding messages thread and there is a child thread that runs the listening in the background. The child thread has a constant background loop, listening for any messages from the server (for example an error message) and reads a message queue to see if anything has been added. If something is added to the queue, the listening loop stops, sends a message, then starts the listening loop again. The parent thread allows the user to add a command into the message queue using the add_message() attribute. The maya ui will have buttons that call functions to add commands to the message queue. I made this with python 2.7
Here is an simplified example of the client. I added a single message before starting the listening loop so you can see what it is supposed to look like.
import socket
import struct
import threading
import Queue
import time
class ThreadedClient(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self)
#set up queues
self.send_q = Queue.Queue(maxsize = 10)
#declare instance variables
self.host = host
self.port = port
#connect to socket
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.connect((self.host, self.port))
#LISTEN
def listen(self):
while True: #loop forever
try:
print 'checking for message...'
# stops listening if there's a message to send
if self.send_q.empty() == False:
self.send_message()
else:
print 'no message'
print 'listening...'
message = self.s.recv(4096)
print 'RECEIVED: ' + message
except socket.timeout:
pass
def start_listen(self):
t = threading.Thread(target = self.listen())
t.start()
#t.join()
print 'started listen'
#ADD MESSAGE
def add_message(self, msg):
#put message into the send queue
self.send_q.put(msg)
print 'ADDED MSG: ' + msg
#self.send_message()
#SEND MESSAGE
def send_message(self):
#send message
msg = self.get_send_q()
if msg == "empty!":
print "nothing to send"
else:
self.s.sendall(msg)
print 'SENDING: ' + msg
#restart the listening
#self.start_listen()
#SAFE QUEUE READING
#if nothing in q, prints "empty" instead of stalling program
def get_send_q(self):
if self.send_q.empty():
print "empty!"
return "empty!"
else:
msg = self.send_q.get()
return msg
if __name__ == '__main__':
port = 8001
address = 'localhost'
s = ThreadedClient(address, port)
s.start()
print('Server started, port: ', port)
s.add_message('hello world')
s.start_listen()
s.add_message('hello world')
And here is an example server for the client:
import socket
import sys
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 8001)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(1)
while True:
# Wait for a connection
print >>sys.stderr, 'waiting for a connection'
connection, client_address = sock.accept()
try:
print >>sys.stderr, 'connection from', client_address
# Receive the data in small chunks and retransmit it
while True:
data = connection.recv(16)
print >>sys.stderr, 'received "%s"' % data
if data:
print >>sys.stderr, 'sending data back to the client'
connection.sendall(data)
else:
print >>sys.stderr, 'no more data from', client_address
break
finally:
# Clean up the connection
connection.close()
Problem:
Once I start the listening loop thread, the script will no longer take any more input. Just as a test I added a line after starting the listening thread that would add a message to the queue and nothing happens.
Previous Approaches:
This question is essentially exactly what I’m trying to do as well, although it was never solved: How to handle chat client using threading and queues?
I have looked at Grab user input asynchronously and pass to an Event loop in python about creating a main loop that accepts user input and have tried implementing the queue system but am getting stuck.
I’ve also tried out How to use threading to get user input realtime while main still running in python approach but I can’t use raw_input() for my final usage of the code. I have also tried socket.timeout, but also loops without taking input.
One approach I was considering was making the script asynchronous, but from what I’ve read I don’t think it will resolve the issue.
TL/DR:
Is there a way to create a script that has a client listening to the server loop running in the background while at the same time being able to accept real time commands from the user?
I would greatly appreciate any help or nudge in the right direction, I've been stuck on this a while now.
The script works. There were just some errors throwing it off. In the start_listen() function
t = threading.Thread(target = self.listen())
should be
t = threading.Thread(target = self.listen)
and then in the __init__() after
self.s.connect((self.host, self.port))
add
self.s.settimeout(.1)
so that the script will cycle through the while loop.
I'm having difficulty understanding the behaviour of my altered echo server, which attempts to take advantage of python 3's asyncio module.
Essentially I have an infinite loop (lets say I want to stream some data from the server to the client indefinitely whilst the connection has been made) e.g. MyServer.py:
#! /usr/bin/python3
import asyncio
import os
import time
class MyProtocol(asyncio.Protocol):
def connection_made(self, transport):
peername = transport.get_extra_info('peername')
print('Connection from {}'.format(peername))
self.transport = transport
def connection_lost(self, exc):
asyncio.get_event_loop().stop()
def data_received(self, data):
i = 0
while True:
self.transport.write(b'>> %i' %i)
time.sleep(2)
i+=1
loop = asyncio.get_event_loop()
coro = loop.create_server(MyProtocol,
os.environ.get('MY_SERVICE_ADDRESS', 'localhost'),
os.environ.get('MY_SERVICE_PORT', 8100))
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except:
loop.run_until_complete(server.wait_closed())
finally:
loop.close()
Next when I connect with nc ::1 8100 and send some text (e.g. "testing") I get the following:
user#machine$ nc ::1 8100
*** Connection from('::1', 58503, 0, 0) ***
testing
>> 1
>> 2
>> 3
^C
Now when I attempt to connect using nc again, I do not get any welcome message and after I attempt to send some new text to the server I get an endless stream of the following error:
user#machine$ nc ::1 8100
Is there anybody out there?
socket.send() raised exception
socket.send() raised exception
...
^C
Just to add salt to the wound the socket.send() raised exception message continues to spam my terminal until I kill the python server process...
As I'm new to web technologies (been a desktop dinosaur for far too long!), I'm not sure why I am getting the above behaviour and I haven't got a clue on how to produce the intended behaviour, which loosely looks like this:
server starts
client 1 connects to server
server sends welcome message to client
4 client 1 sends an arbitrary message
server sends messages back to client 1 for as long as the client is connected
client 1 disconnects (lets say the cable is pulled out)
client 2 connects to server
Repeat steps 3-6 for client 2
Any enlightenment would be extremely welcome!
There are multiple problems with the code.
First and foremost, data_received never returns. At the transport/protocol level, asyncio programming is single-threaded and callback-based. Application code is scattered across callbacks like data_received, and the event loop runs the show, monitoring file descriptors and invoking the callbacks as needed. Each callback is only allowed to perform a short calculation, invoke methods on transport, and arrange for further callbacks to be executed. What the callback cannot do is take a lot of time to complete or block waiting for something. A while loop that never exits is especially bad because it doesn't allow the event loop to run at all.
This is why the code only spits out exceptions once the client disconnects: connection_lost is never called. It's supposed to be called by the event loop, and the never-returning data_received is not giving the event loop a chance to resume. With the event loop blocked, the program is unable to respond to other clients, and data_received keeps trying to send data to the disconnected client, and logs its failure to do so.
The correct way to express the idea can look like this:
def data_received(self, data):
self.i = 0
loop.call_soon(self.write_to_client)
def write_to_client(self):
self.transport.write(b'>> %i' % self.i)
self.i += 1
loop.call_later(2, self.write_to_client)
Note how both data_received and write_to_client do very little work and quickly return. No calls to time.sleep(), and definitely no infinite loops - the "loop" is hidden inside the kind-of-recursive call to write_to_client.
This change reveals the second problem in the code. Its MyProtocol.connection_lost stops the whole event loop and exits the program. This renders the program unable to respond to the second client. The fix could be to replace loop.stop() with setting a flag in connection_lost:
def data_received(self, data):
self._done = False
self.i = 0
loop.call_soon(self.write_to_client)
def write_to_client(self):
if self._done:
return
self.transport.write(b'>> %i' % self.i)
self.i += 1
loop.call_later(2, self.write_to_client)
def connection_lost(self, exc):
self._done = True
This allows multiple clients to connect.
Unrelated to the above issues, the callback-based code is a bit tiresome to write, especially when taking into account complicated code paths and exception handling. (Imagine trying to express nested loops with callbacks, or propagating an exception occurring inside a deeply embedded callback.) asyncio supports coroutines-based streams as alternative to callback-based transports and protocols.
Coroutines allow writing natural-looking code that contains loops and looks like it contains blocking calls, which under the hood are converted into suspension points that enable the event loop to resume. Using streams the code from the question would look like this:
async def talk_to_client(reader, writer):
peername = writer.get_extra_info('peername')
print('Connection from {}'.format(peername))
data = await reader.read(1024)
i = 0
while True:
writer.write(b'>> %i' % i)
await writer.drain()
await asyncio.sleep(2)
i += 1
loop = asyncio.get_event_loop()
coro = asyncio.start_server(talk_to_client,
os.environ.get('MY_SERVICE_ADDRESS', 'localhost'),
os.environ.get('MY_SERVICE_PORT', 8100))
server = loop.run_until_complete(coro)
loop.run_forever()
talk_to_client looks very much like the original implementation of data_received, but without the drawbacks. At each point where it uses await the event loop is resumed if the data is not available. time.sleep(n) is replaced with await asyncio.sleep(n) which does the equivalent of loop.call_later(n, <resume current coroutine>). Awaiting writer.drain() ensures that the coroutine pauses when the peer cannot process the output it gets, and that it raises an exception when the peer has disconnected.
I'm trying to create a code that is non-blocking and that allows me to create multiple clients to do some request on a server. However, I can't create more than 1 client simultaneously!
CLIENT.PY
import asyncio
PYTHONASYNCIODEBUG = 1
#ECHO CLIENT PROTOCOL
async def tcp_echo_client(message, loop):
# Send request to server
reader, writer = await asyncio.open_connection('127.0.0.1', 8888, loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
# Receive the information
if message == '1':
await asyncio.Task(read_position(reader))
else:
await asyncio.ensure_future(read_server(reader))
# Close the connection
print('Close the socket')
writer.close()
#ASYNCIO COROUTINES TO REQUEST INFORMATION
async def read_server(reader):
server_message = await reader.read()
print(type(server_message))
print('Received: %r' % server_message.decode())
async def read_position(reader):
while True:
print("I'm Here")
server_message = await reader.read(50)
position = server_message.split()
print(position)
print(type(position))
print('Received: %r' % server_message.decode())
#FUNCTION THAT CREATES THE CLIENT
def main(message):
'''This function creates the client'''
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(tcp_echo_client(message, loop))
finally:
pass
# This is how I create a new client
if __name__ == '__main__':
message = '2'
main(message)
message = '3'
main(message)
I want to create multiples clients, however, the code is blocking in the first main when I send the message('1'). I don't know why the code is blocking if I'm using asyncio. My server accepts multiples connections, because if I run this code seperatly I can do everything. The propose of this is to create clients every time I click a button at my Kivy app to send a request to the server.
This problems exists because I want to control a Robot and do a lot of things simultaneously, however with a blocking code I can't do it because I'm get stuck
Maybe it's a stupid question but I've only started coded 2 months ago and I haven't any help.
Your main function doesn't "create the client", as its docstring claims. It creates the client and runs it to completion. This is why multiple invocations of main() result in serial execution. main() being a regular function, that's exactly what you'd expect, asyncio doesn't change that. It's useful to remember that asyncio is single-threaded, so it can't do some "run in the background" magic, unless you cooperate.
To cooperate, you need to tell aysncio to start both clients, and then await them in parallel:
async def main(messages):
loop = asyncio.get_event_loop()
# launch the coroutines in parallel
tasks = [loop.create_task(tcp_echo_client(msg, loop)) for msg in messages]
# don't exit until all of them are done
await asyncio.gather(*tasks)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main(['2', '3']))
Note that when awaiting your coroutines, you don't need to wrap them in asyncio.ensure_future() or asyncio.Task() - asyncio will handle that automatically. await read_position(reader) and await read_server(reader) would work just fine and have the same meaning as the longer versions.
I trying to do a simple server. The server has to load data while client provides it. I want to use non-blocking 'recv' command, but whatever I do it still blocks. My code is above.
import socket
import fcntl
import os
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("localhost", 8888))
server_socket.listen()
client_socket, addr = server_socket.accept()
#fcntl.fcntl(client_socket, fcntl.F_SETFL, os.O_NONBLOCK) #mark1
client_socket.setblocking(False) #mark2
#client_socket.settimeout(0.0) #mark3
while True:
data = client_socket.recv(10)
if len(data) != 0:
print(data.decode("UTF8"))
else:
break
print("Exit")
I tried to use lines '#mark1', '#mark2', '#mark3'. These lines seem to be pretty good for me, but my program is still stuck on the second iteration with data = client_socket.recv(10) line.
Why do client_socket.setblocking(False) and others not affect?