I was reading << Black Hat Python >> and trying the first bhnet.py program.
On one terminal, I run script
./bhnet.py -l -p 9999 -c
And on another terminal, run script
./bhnet.py -t localhost -p 9999
then type
<ctrl-D>
or
ls -alt
<ctrl-D>
The first terminal would return
File "bhnet.py", line 186, in client_handler
cmd_buffer += client_socket.recv(1024).decode('utf-8')
ConnectionResetError: [Errno 54] Connection reset by peer
Below are the codes for the program
def client_handler(client_socket):
global upload
global execute
global command
# check for upload
if len(upload_destination):
# read all the buffer and write to destination
file_buffer = ""
# keep reading til none is available
while True:
data = client_socket.recv(1024)
if not data:
break
else:
file_buffer += data
# take the bytes and write them out
try:
file_descriptor = open(upload_destination,'wb')
file_descriptor.write(file_buffer)
file_descriptor.close()
# acknowledge that file being wrote out
client_socket.send(f"Successfully save file to {upload_destination}.\r \n")
except:
client_socket.send(f"Failed to save file to {upload_destination}.\r \n")
# check for command execution
if command:
while True:
#pop up a window
client_socket.send(b"<BHP:#> ")
# keep receiving data until \n
cmd_buffer = ""
while "\n" not in cmd_buffer:
cmd_buffer += client_socket.recv(1024).decode('utf-8')
response = run_command(cmd_buffer)
client_socket.send(response)
I googled and even tried to upgrade openssl and none of these work...
Thanks in advance!
It's hard to be certain since you don't provide the client side code. However, I'm fairly confident this is happening:
When you type Ctrl-D, you're giving an end of file to the client's input. That is causing the client to close the socket it had previously connected to the server. Doing this causes the client's operating system to send a TCP FIN packet to the server. The FIN only tells the server that the client is finished sending data; there is no way in the normal TCP session termination of telling a peer that the peer may not send any more data.
But then the server is trying to send to the client, after the client has closed its socket. When you try to send further data on a closed socket, then the destination peer's operating system sends a TCP RST packet. That isn't actually reported to the server on the send because the send function call is complete when the data is copied into the kernel -- whereas the RST is probably received by the kernel a few milliseconds later after it actually sent a data packet to the peer.
Hence that condition will be reported on the next operation on the socket which, here, is recv. Hence your program gets back an ECONNRESET error, which python translates into ConnectionResetError exception.
In other words:
Client Server
------ ------
close()
FIN => <OS receives FIN>
send(data)
<= "data"
RST =>
recv
<ECONNRESET>
One more thing: Depending on the exact timing, it's possible that your first call to recv in that loop is actually getting an end-of-file indicator (i.e. zero bytes). However, you're not checking for that, you just keep calling recv as long as there is no newline in the buffer. You really should be checking whether you got a zero byte string back from recv and terminate your loop in that case.
Once you've gotten the end-of-file indicator on the socket, you'll never get a newline added to the buffer. If the client had actually managed to receive the data you sent it before it closed the socket, then no RST would have been sent. In that case, your recv loop would run potentially forever, getting back zero bytes continuously but never finding the newline.
Related
First, thank for fixing my post. I'm still not sure how to include a sketch. I've been reading posts here for many months, but never posted one before.
My headless RasPi is running two sketches of mine, one reads data from a pm2.5 sensor (PMS7003) and the other is the program listed above that sends information to another Pi, the client, that turns on a pm2.5 capable air filter. (I live in California) The program that reads the PMS7003 sorts the data, called max_index, into one of six categories, 0 thru 5 and saves the current category to a text file. I'm using the 'w' mode during the write operation, so there is only one character in the text file at any time. The server program listed above reads the text file and sends it to a client that turns on the air filter for categories above 2. The client sends the word "done" back to the server to end the transaction.
Until you mentioned it, I didn't realize my mistake, clientsocket.recv(2). I'll fix that and try again.
So, the listener socket should go outside the while loop, leaving the send and receive inside???
Troubleshooting: I start the two programs using nice nohup python3 xxx.py & nice nohup python3 yyy.py. The program that reads the PMS7003 continues running and updating the text file with current category, but the server program falls out of existence after a few days. top -c -u pi reveals only the PMS7003 program running, while the server program is missing. Also, there's nothing in nohup.out or in socketexceptions.txt and I tried looking through system logs in /var/log but was overwhelmed by information and found nothing that made any sense to me.
Since writing to the socketexceptions.txt file is not in a try/except block, the crash might be happening there.
import socket
import time
index = " "
clientsocket = ""
def getmaxindex():
try:
with open('/home/pi/pm25/fan.txt','r')as f:
stat = f.read() #gets max_index from pm25b.py
return(stat)
except:
with open("/home/pi/pm25/socketexceptions.txt",'a')as f:
f.write("Failed to read max index")
def setup(index):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind(("192.168.1.70", 5050))
except:
with open("/home/pi/pm25/socketexceptions.txt",'a')as f:
f.write("Failed to bind")
try:
s.listen(1)
clientsocket, address = s.accept()
clientsocket.send(index)
rx = clientsocket.recv(2)
if rx == "done":
clientsocket.close()
except:
with open("/home/pi/pm25/socketexceptions.txt",'a')as f:
f.write("Failed to communicate with flient")
while True:
index = getmaxindex().encode('utf-8')
setup(index)
time.sleep(5)
enter code here
It is unknown what program is supposed to do and where exactly you run into problems, since there is only a code dump and no useful error description (what does "stop" mean - hang or exit, where exactly does it stop). But the following condition can never be met:
rx = clientsocket.recv(2)
if rx == "done":
The will receive at most 2 bytes (recv(2)) which is definitely not enough to store the value "done".
Apart from that it makes not real sense to recreate the same listener socket again and again, just to accept a single client and exchange some data. Instead the listener should only be created once and multiple accept should be called on the same listener socket, where each will result in a new client connection.
I have a python script that "streams" a very large gzip file using urllib3 and feeds it into a zlib.decompressobj. This zlib decompression object is configured to read gzip compression. If this initial http connection is interrupted then the zlib.decompressobj begins to throw errors after the connection is "resumed". See my source code below if you want to cut to the chase.
These errors occur despite the fact that the script initiates a new http connection using the Range header to specify the number of bytes previously read. It resumes from the completed read point present when the connection was broken. I believe this arbitrary resume point is the source of my problem.
If I don't try to decompress the chunks of data being read in by urllib3, but instead just write them to a file, everything works just fine. Without trying to decompress the stream everything works even when there is an interruption. The completed archive is valid, it is the same size as one downloaded by a browser and the MD5 hash of the .gz file is the same as if I had downloaded the file directly with Chrome.
On the other hand, if I try to decompress the chunks of data coming in after the interruption, even with the Range header specified, the zlib library throws all kinds of errors. The most recent was Error -3 while decompressing data: invalid block type
Additional Notes:
The site that I am using has the Accept-Range flag set to bytes meaning that I am able to submit modified Range headers to the server.
I am not using the requests library in this script as it ultimately manages urllib3. I am instead using urllib3 directly in an attempt to cut out the middle man.
This script it an oversimplification of my ultimate goal, which is to stream the compressed data directly from where it is hosted, enrich it and store it in a MySQL database on the local network.
I am heavily resource constrained inside of the docker container where this processing will occur.
The genesis of this question is present in a question I asked almost 3 weeks ago: requests.iter_content() thinks file is complete but it's not
The most common problem I am encountering with the urllib3 (and requests) library is the IncompleteRead(self._fp_bytes_read, self.length_remaining) error.
This error only appears if the urllib3 library has been patched to raise an exception when an incomplete read occurs.
My best guess:
I am guessing that the break in the data stream being fed to zlib.decompressobj is causing zlib to somehow lose context and start attempting to decompress the data again in an odd location. Sometimes it will resume, however the data stream is garbled, making me believe the byte location used as the new Range header fell at the front of some bytes which are then incorrectly interpreted as headers. I do not know how to counteract this and I have been trying to solve it for several weeks. The fact that the data are still valid when downloaded whole (without being decompressed prior to completion) even with an interruption occurs, makes me believe that some "loss of context" within zlib is the cause.
Source Code: (Has been updated to include a "buffer")
This code is a little bit slapped together so forgive me. Also, this target gzip file is quite a lot smaller than the actual file I will be using. Additionally, the target file in this example will no longer be available from Rapid7 in about a month's time. You may choose to substitute a different .gz file if that suits you.
import urllib3
import certifi
import inspect
import os
import time
import zlib
def patch_urllib3():
"""Set urllib3's enforce_content_length to True by default."""
previous_init = urllib3.HTTPResponse.__init__
def new_init(self, *args, **kwargs):
previous_init(self, *args, enforce_content_length = True, **kwargs)
urllib3.HTTPResponse.__init__ = new_init
#Path the urllib3 module to throw an exception for IncompleteRead
patch_urllib3()
#Set the target URL
url = "https://opendata.rapid7.com/sonar.http/2021-11-27-1638020044-http_get_8899.json.gz"
#Set the local filename
local_filename = '2021-11-27-1638020044-http_get_8899_script.json.gz'
#Configure the PoolManager to handle https (I think...)
http = urllib3.PoolManager(ca_certs=certifi.where())
#Initiate start bytes at 0 then update as download occurs
sum_bytes_read=0
session_bytes_read=0
total_bytes_read=0
#Dummy variable to silence console output from file write
writer=0
#Set zlib window bits to 16 bits for gzip decompression
decompressor = zlib.decompressobj(zlib.MAX_WBITS|16)
#Build a buffer list
buf_list=[]
i=0
while True:
print("Building request. Bytes read:",total_bytes_read)
resp = http.request(
'GET',
url,
timeout=urllib3.Timeout(connect=15, read=40),
preload_content=False)
print("Setting headers.")
#This header should cause the request to resume at "total_bytes_read"
resp.headers['Range'] = 'bytes=%s' % (total_bytes_read)
print("Local filename:",local_filename)
#If file already exists then append to it
if os.path.exists(local_filename):
print("File already exists.")
try:
print("Starting appended download.")
with open(local_filename, 'ab') as f:
for chunk in resp.stream(2048):
buf_list.append(chunk)
#Use i to offset the chunk being read from the "buffer"
#I.E. load 3 chunks (0,1,2) in the buffer list before starting to read from it
if i >2:
buffered_chunk=buf_list.pop(0)
writer=f.write(buffered_chunk)
#Comment out the below line to stop the error from occurring.
#File download should complete successfully even if interrupted when the following line is commented out.
decompressed_chunk=decompressor.decompress(buffered_chunk)
#Increment i so that the buffer list will fill before reading from it
i=i+1
session_bytes_read = resp._fp_bytes_read
#Sum bytes read is an updated value that isn't stored. It is only used for console print
sum_bytes_read = total_bytes_read + session_bytes_read
print("[+] Bytes read:",str(format(sum_bytes_read, ",")), end='\r')
print("\nAppended download complete.")
break
except Exception as e:
print(e)
#Add to total bytes read to current session bytes each time the loop needs to repeat
total_bytes_read=total_bytes_read+session_bytes_read
print("Bytes Read:",total_bytes_read)
#Mod the total_bytes back to the nearest chunk size so it can be - re-requested
total_bytes_read=total_bytes_read-(total_bytes_read%2048)-2048
print("Rounded bytes Read:",total_bytes_read)
#Pop the last entry off of the buffer since it may be incomplete
buf_list.pop()
#reset i so that the buffer has to rebuilt
i=0
print("Sleeping for 30 seconds before re-attempt...")
time.sleep(30)
#If file doesn't already exist then write to it directly
else:
print("File does not exist.")
try:
print("Starting initial download.")
with open(local_filename, 'wb') as f:
for chunk in resp.stream(2048):
buf_list.append(chunk)
#Use i to offset the chunk being read from the "buffer"
#I.E. load 3 chunks (0,1,2) in the buffer list before starting to read from it
if i > 2:
buffered_chunk=buf_list.pop(0)
#print("Buffered Chunk",str(i-2),"-",buffered_chunk)
writer=f.write(buffered_chunk)
decompressed_chunk=decompressor.decompress(buffered_chunk)
#Increment i so that the buffer list will fill before reading from it
i=i+1
session_bytes_read = resp._fp_bytes_read
print("[+] Bytes read:",str(format(session_bytes_read, ",")), end='\r')
print("\nInitial download complete.")
break
except Exception as e:
print(e)
#Set the total bytes read equal to the session bytes since this is the first failure
total_bytes_read=session_bytes_read
print("Bytes Read:",total_bytes_read)
#Mod the total_bytes back to the nearest chunk size so it can be - re-requested
total_bytes_read=total_bytes_read-(total_bytes_read%2048)-2048
print("Rounded bytes Read:",total_bytes_read)
#Pop the last entry off of the buffer since it may be incomplete
buf_list.pop()
#reset i so that the buffer has to rebuilt
i=0
print("Sleeping for 30 seconds before re-attempt...")
time.sleep(30)
print("Looping...")
#Finish writing from buffer into file
#BE SURE TO SET TO "APPEND" with "ab" or you will overwrite the start of the file
f = open(local_filename, 'ab')
print("[+] Finishing write from buffer.")
while not len(buf_list) == 0:
buffered_chunk=buf_list.pop(0)
writer=f.write(buffered_chunk)
decompressed_chunk=decompressor.decompress(buffered_chunk)
#Flush and close the file
f.flush()
f.close()
resp.release_conn()
Reproducing the error
To reproduce the error perform the following actions:
Run the script and let the download start
Be sure that line 65 decompressed_chunk=decompressor.decompress(chunk) is not commented out
Turn off your network connection until an exception is raised
Turn your network connection back on immediately.
If the decompressor.decompress(chunk) line is removed from the script then it will download the file and the data can be successfully decompressed from the file itself. However, if line 65 is present and an interruption occurs, the zlib library will not be able to continue decompressing the data stream. I need to decompress the data stream as I cannot store the actual file I am trying to use.
Is there some way to prevent this from occurring? I have now attempted to add a "buffer" list that stores the chunks; the script discards the last chunk after a failure and moves back to a point in the file that preceded the "failed" chunk. I am able to re-establish the connection and even pull back all the data correctly but even with a "buffer" my ability to decompress the stream is interrupted. I must not be smoothly recovering the data back to the buffer somehow.
Visualization:
I put this together very quickly in an attempt to better describe what I am trying to do...
I bet Mark Adler is hiding out there somewhere...
r+b doesn't append. You would need to use ab for that. It appears that on the re-try, you are reading the entire gzip file again from the start. With r+b, that file is written correctly to your output file, by overwriting what was read before.
However, you are feeding the initial read to the decompressor, and then the start of the file again. Not surprisingly, the decompressor then soon detects invalid compressed data.
I want to connect to 2018shell.picoctf.com by netcat using python. I need to read output(which I accomplished) and later write "210.205.230.140" and send it to the server.
import subprocess
proces = subprocess.Popen(["nc", "2018shell.picoctf.com", "14079"],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE)
data = proces.communicate(input=b'210.205.230.140')
I don't know why program gets stuck in proces.communicate(input=b'210.205.230.140') and I can't send it to subproces and get next messages sent by server or execute next commands.
I am trying to update the firmware of a controller through a serial interface. To do this, I must send a reboot message to the controller (no problem there) and then send another message (the character 'w') THE MOMENT it starts up so that it may start up in write mode. This is easily done with the minicom utility by pressing w continuously while the device restarts.
I want to achieve this functionality using python code instead, but I can't figure out how to send a message until the device is up without throwing exceptions (since the device is not connected).
This is what I have tried, but it does not work (with pyserial):
def send_w(serial_port, baud_rate):
msgw = "w_"
ans = ""
ser = serial.Serial(port=serial_port, baudrate=baud_rate,timeout = 10)
ser.write(msgw)
ans = ser.read(24)
ser.close()
print(ans)
return ans
def set_firmware_version(serial_port, baud_rate):
s = ""
try:
with serial.Serial(serial_port,baud_rate,timeout=1) as ser:
msgr = "%reset "+sk+"_"
ser.write(msgr)
ser.close()
print("reset")
except (IOError) as e:
print("Error in: set_firmware_version")
print(e)
return s
time.sleep(1)
send_w(serial_port, baud_rate)
set_firmware_version(sp,br)
This gives the following error:
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
I also tried sending the messages in a loop with a short timeout, but had the same problem. Is there any way to send a message continuously and disregard exceptions if the device is not found?
Any help will be greatly appreciated.
Full Traceback:
Traceback (most recent call last):
File "mc_config.py", line 69, in <module>
set_firmware_version(sp,br)
File "mc_config.py", line 64, in set_firmware_version
send_w(serial_port, baud_rate)
File "mc_config.py", line 46, in send_w
ans = ser.read(24)
File "/home/avidbots/.local/lib/python2.7/site-packages/serial/serialposix.py", line 501, in read
'device reports readiness to read but returned no data '
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
(I am using ubuntu 16.04 and python 3)
What if you put the excepting code into a try and then catch the exception with an except serial.serialutil.SerialException {...} block?
Clearly there's a significant window of time to submit the w ( otherwise the "press w" method wouldn't often work.) Your requirement, then, would be to retry only the part of the code that's absolutely necessary to send the w, so that you send it quickly enough to "catch" the system in its bootup state. Since the backtrace shows that the exceptions occurs in send_w, then you can add try/except blocks and a while loop around what is now one line at the end of set_firmware_version.
Instead of just this:
send_w(serial_port, baud_rate)
Something like this might solve the problem:
while True:
try:
send_w(serial_port, baud_rate)
break
except serial.serialutil.SerialException:
pass # retry
You may need to change your imports to expose that exception, fyi. And you may need to consider whether you're catching too many possible exceptions - it's possible that exception might also represent other errors that shouldn't be retried. You might also need to add a small sleep time there - this is essentially a busy wait loop (https://en.wikipedia.org/wiki/Busy_waiting).
I am working with a high refresh rate IMU (x-IO technologies NGIMU) which outputs all data in osc format. The manufacturer provides the following python script to serve the data on linux platforms ( I am running Ubuntu 16.04)
'''
NGIMU Demo python v2.7 script written by Tom Mitchell (teamxe.co.uk) 2016
Requires pyOSC https://trac.v2.nl/wiki/pyOSC
'''
import socket, OSC, threading, time
# Change this to the NGIMU IP address
send_address = '192.168.1.1', 9000
# Set the NGIMU to send to this machine's IP address
c = OSC.OSCClient()
c.connect(send_address)
msg = OSC.OSCMessage()
msg.setAddress('/wifi/send/ip')
msg.append(str(socket.gethostbyname(socket.gethostname())))
c.send(msg)
c.close()
# Set up receiver
receive_address = '192.168.1.2', 8000
s = OSC.OSCServer(receive_address)
s.addDefaultHandlers()
def sensorsHandler(add, tags, args, source):
print add + str(args)
def quaternionHandler(add, tags, args, source):
print add + str(args)
def batteryHandler(add, tags, args, source):
print add + str(args)
# Add OSC handlers
s.addMsgHandler("/sensors", sensorsHandler)
s.addMsgHandler("/quaternion", quaternionHandler)
s.addMsgHandler("/battery", batteryHandler)
# Start OSCServer
print "\nUse ctrl-C to quit."
st = threading.Thread(target = s.serve_forever)
st.start()
# Loop while threads are running
try :
while 1 :
time.sleep(10)
except KeyboardInterrupt :
print "\nClosing OSCServer."
s.close()
print "Waiting for Server-thread to finish"
st.join()
print "Done"
The IMU hosts its own network which I connect to with the computer that is to receieve the data.
I have installed pyOSC from the location referenced in the script.
When I run the script, no data is delivered, only the message "Use ctrl-C to quit".
All connections seem to take place properly. When the script is running, I can see the udp connection at the correct ip and port using the Ubuntu firewall configuration gui. I have tried disabling the firewall but that had no effect.
Separately, I have used another computer to send udp packets to that ip and port and confirmed their receipt.
To say that I am a coding novice is far too generous. Nonetheless, I need to get this script running. Any help you can offer is greatly appreciated.
The problem is that
socket.gethostbyname(socket.gethostname())
is not setting the correct IP. You should change to
msg.setAddress('/wifi/send/ip')
msg.append('192.168.1.2')