I have an micropython client on esp32 board, and Python on linux server. I am trying send 5.5kb text file from Python Server to MicroPython client. It sends successfully but MicroPython client does not receive all data. Codes as follows;
Python Server:
with open('downloads/%s' % (request_path), 'rb') as f:
data = f.read()
self.wfile.write(data) #data is 5.5kb
MicroPython Client
recvData = sock.read(4096).decode('utf-8').split("\r\n")
print("Response_Received:: %s" % recvData)
sock.close()
Response_Received:: ['HTTP/1.0 200 OK', 'Server: SimpleHTTP/0.6 Python/3.5.3', 'Date: Sat, 09 Jun 2018 09:29:41 GMT', '', '# Ity: asdasd\n# ksduygfkhsgdkjfksjdhfg\n kjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy98\n 47y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhs\n gdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349rio\n t34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r3\n 49riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkv\n nvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\n oiruy9847y397r349riot34jt;o\n giji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbs\n djkvbjcxbvhweioufhoiweuoiruy9847y397\n r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufh\n oiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhwei\n oufhoiweuoiruy9847y397r349riot\n 34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhw\n eioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjd\n hfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiwe\n uoiruy9847y397r349riot34jt;ogiji4vuij\n o4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhwe\n ioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfk\n hsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;o\n giji4vuijo4vjlkvnvl;kksduygfkhsgdk\n jfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvh\n weioufhoiweuoiruy9847y397r349riot34jt;ogiji\n 4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiru\n y9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;k4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcx\n bvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhs\n gdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdj\n nvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9bfkjcbsdjkvbjcxbvhweioufhoi847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\nnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\nnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufnvl;k']
Client receives only 4140 bytes of the array data in due to buffer size(4096), 4th element of the recvData is lost. MicroPython does not accept over this Buffer size. How can i receive all my data (5.5kb) in 4th element of recvData array without any loss?
I have tried to fragment the received data, but it was not successful.
while True:
chunck = s.recv(4096)
if not chunck:
break
fragments.append(chunck)
Since your goal is to write the file to the filesystem, the simplest solution is to stop trying to hold the entire file in memory. Instead of building up your fragments array, just write the received chunks to a file:
with open('datafile', 'w') as fd:
while True:
chunk = s.recv(4096)
if not chunk:
break
fd.write(chunk)
This requires a constant amount of memory and can be used to receive
files of arbitrary size.
Related
I am transmitting data with Python-can using isotp.stack().
Successfully able to do the same for 8 bytes of frame size but not able to transmit for 64 bytes of frame.
rxid and txid are enabled for 64 bytes of frame size as tested using capl script.
I even tried to add param 'tx_data_min_length': 64 in the stack, but did not work.
Below is my code that I am using.
bus = VirtualBus(net_channel=channel)
print(bus.bus.channel_info)
address = isotp.Address(addressing_mode=isotp.AddressingMode.Normal_29bits, rxid=rxid, txid=txid)
stack = isotp.CanStack(bus=bus.bus,
address=address,
error_handler=print,
params={'tx_padding': 0xCC,
'can_fd': True,
'tx_data_length': 64,
'rx_consecutive_frame_timeout': 100000})
stack.send(data)
while stack.transmitting():
stack.process()
time.sleep(stack.sleep_time())
ERROR:
E RuntimeError: (c_ubyte_Array_8) <class 'IndexError'>: invalid index
Is there any other params I am missing out?
Document that I have read https://can-isotp.readthedocs.io/en/latest/isotp/implementation.html
I'm using nRF Connect for Android to test a BLE peripheral. The peripheral is a BSX Insight residual muscle oxygen monitor whose software application is no longer functional or supported by the manufacturer. Thus, my only option to use my device (BSX) is to write my own control software. I've written a Python 3.7 script that I run within a tkinter routine on my 64-bit Win 10 laptop. Also, I'm using the Pygatt library and a BLED112 BT dongle.
I can connect to the peripheral, read and write values just fine to characteristics, but I'm sure that the "conversion" from the process used in nRF Connect and to my script is incomplete and inefficient. So the first thing I'd like to confirm is that the correct respective functions from Pygatt are used. Once I'm using the correct functions from Pygatt, then I can compare respective outputs for the two data (characteristic values) streams that I want to capture and store.
The basic process in nRF Connect:
1. scan
2. select/connect the BSX Insight
3. expose the service and characteristics of interest
4. enable CCCDs
5. write the "start data" values (04-02)
These are the process command results from the nRF Connect log file. Starting with number four:
4.
D 09:04:54.491 gatt.setCharacteristicNotification(00002a37-0000-1000-8000-00805f9b34fb, true) 11
D 09:04:54.496 gatt.setCharacteristicNotification(2e4ee00b-d9f0-5490-ff4b-d17374c433ef, true) 20x
D 09:04:54.499 gatt.setCharacteristicNotification(2e4ee00d-d9f0-5490-ff4b-d17374c433ef, true) 25x
D 09:04:54.516 gatt.setCharacteristicNotification(2e4ee00e-d9f0-5490-ff4b-d17374c433ef, true) 32x
D 09:04:54.519 gatt.setCharacteristicNotification(00002a63-0000-1000-8000-00805f9b34fb, true) 36
D 09:04:54.523 gatt.setCharacteristicNotification(00002a53-0000-1000-8000-00805f9b34fb, true) 40
The above resulted from using the nRF command "Enable CCCDs." Basically every characteristic that could be enabled was enabled which is fine. The 'x' are the three that I need enabled. The others are extra. Note, I've annotated the respective handles for these UUIDs on the end of the line.
V 09:05:39.211 Writing command to characteristic 2e4ee00a-d9f0-5490-ff4b-d17374c433ef
D 09:05:39.211 gatt.writeCharacteristic(2e4ee00a-d9f0-5490-ff4b-d17374c433ef, value=0x0402)
I 09:05:39.214 Data written to 2e4ee00a-d9f0-5490-ff4b-d17374c433ef, value: (0x) 04-02
A 09:05:39.214 "(0x) 04-02" sent
Number five is where I write 0402 to the UUID above. This action sends the data/value streams from:
2e4ee00d-d9f0-5490-ff4b-d17374c433ef, with a descriptor handle 26
2e4ee00e-d9f0-5490-ff4b-d17374c433ef, with a descriptor handle 33
Once I've done the basic steps above in nRF Connect, the two characteristic value streams become active, and I can immediately see the converted values in my Garmin Edge 810 head unit.
So attempting to duplicate the same process within my tkinter snippet:
# this function fires from the 'On' button click event
def powerON():
powerON_buttonevent = 1
print(f"\tpowerON_buttonevent OK {powerON_buttonevent}")
# Connect to the BSX Insight
try:
adapter = pygatt.BGAPIBackend() # serial_port='COM3'
adapter.start()
device = adapter.connect('0C:EF:AF:81:0B:76', address_type=pygatt.BLEAddressType.public)
print(f"\tConnected: {device}")
except:
print(f"BSX Insight connection failure")
finally:
# adapter.stop()
pass
# Enable only these CCCDs
try:
device.char_write_handle(21, bytearray([0x01, 0x00]), wait_for_response=True)
device.char_write_handle(26, bytearray([0x01, 0x00]), wait_for_response=True)
device.char_write_handle(33, bytearray([0x01, 0x00]), wait_for_response=True)
print(f"\te00b DESC: {device.char_read_long_handle(21)}") # notifiy e00b
print(f"\te00d DESC: {device.char_read_long_handle(26)}") # notify e00d SmO2
print(f"\te00e DESC: {device.char_read_long_handle(33)}") # notify e00e tHb
# Here's where I tested functions from Pygatt...
# print(f"\t{device.get_handle('UUID_here')}") # function works
# print(f"\tvalue_handle/characteristic_config_handle: {device._notification_handles('UUID_here')}") # function works
# print(f"{device.char_read('UUID_here')}")
# print(f"{device.char_read_long_handle(handle_here)}") # function works
except:
print(f"CCCD write value failure")
finally:
# adapter.stop()
pass
# Enable the data streams
try:
device.char_write('2e4ee00a-d9f0-5490-ff4b-d17374c433ef', bytearray([0x04, 0x02]), wait_for_response=True) # function works
print(f"\te00a Power ON: {device.char_read('2e4ee00e-d9f0-5490-ff4b-d17374c433ef')}")
except:
print(f"e00a Power ON write failure")
finally:
# adapter.stop()
pass
# Subscribe to SmO2 and tHb UUIDs
try:
def data_handler(handle, value):
"""
Indication and notification come asynchronously, we use this function to
handle them either one at the time as they come.
:param handle:
:param value:
:return:
"""
if handle == 25:
print(f"\tSmO2: {value} Handle: {handle}")
elif handle == 32:
print(f"\ttHb: {value} Handle: {handle}")
else:
print(f"\tvalue: {value}, handle: {handle}")
device.subscribe("2e4ee00d-d9f0-5490-ff4b-d17374c433ef", callback=data_handler, indication=False, wait_for_response=True)
device.subscribe("2e4ee00e-d9f0-5490-ff4b-d17374c433ef", callback=data_handler, indication=False, wait_for_response=True)
print(f"\tSuccess 2e4ee00d: {device.char_read('2e4ee00d-d9f0-5490-ff4b-d17374c433ef')}")
print(f"\tSuccess 2e4ee00e: {device.char_read('2e4ee00e-d9f0-5490-ff4b-d17374c433ef')}")
# this statement causes a run-on continuity when enabled
# while True:
# sleep(1)
except:
print("e00d/e00e subscribe failure")
finally:
adapter.stop()
# pass
Problem: in the output window of my Atom editor, the two data streams start as expected. For example:
I 09:05:39.983 Notification received from 2e4ee00d-d9f0-5490-ff4b-d17374c433ef, value: (0x) 00- 00-00-00-C0-FF-00-00-C0-FF-84-65-B4-3B-9E-AB-83-3C-FF-03
and...
I 09:05:39.984 Notification received from 2e4ee00e-d9f0-5490-ff4b-d17374c433ef, value: (0x) 1C-00-00-FF-03-FF-0F-63-00-00-00-00-00-00-16-32-00-00-00-00
I'll see about seven to ten lines of data before the "stream" stops. There'll be a gap of about 20 seconds, and then a big dump of values. This is different from the output from nRF Connect, which is immediate and continous.
I have the logs from nRF Connect and Python...but I'm not sure which log entry points to the cause of the stop. Might this issue be related to the Peripheral Preferred Connection Parameters? The nRF Connect property read shows:
ConnectionInterval = 50ms~100ms
SlaveLatency = 1
SuperTimeoutMonitor = 200
The Python log entry shows this:
INFO:pygatt.backends.bgapi.bgapi:Connection status: handle=0x0, flags=5, address=0xb'760b81afef0c', connection interval=75.000000ms, timeout=1000, latency=0 intervals, bonding=0xff
Thoughts anyone? (And truly, thanks in advance.)
I've answered my questions. I now have to solve the new problem of why my tKinter dialog is "not responding" as a separate issue.
Thanks All
Edit 3/31/2020: I re-wrote the script using pyQt and now have a functional app.
I'm reading binary file http://snap.stanford.edu/data/amazon/productGraph/image_features/image_features.b
using pyspark.
import array
from io import StringIO
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 4106)
def mapper(features):
a = array.array('f')
a.frombytes(features)
return a.tolist()
def byte_mapper(bytes):
return str(bytes)
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
When just product_id is selected from the rdd using
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
The output for product_id is
["b'1582480311'", "b'\\x00\\x00\\x00\\x00\\x88c-?\\xeb\\xe2'", "b'7#\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\xec/\\x0b?\\x00\\x00\\x00\\x00K\\xea'", "b'\\x00\\x00c\\x7f\\xd9?\\x00\\x00\\x00\\x00'", "b'L\\xa6\\n>\\x00\\x00\\x00\\x00\\xfe\\xd4'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\xe5\\xd0\\xa2='", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"]
The file is hosted on s3.
The file in each row has first 10 bytes for product_id next 4096 bytes as image_features
I'm able to extract all the 4096 image features but facing issue when reading the first 10 bytes and converting it into proper readable format.
EDIT:
Finally, the problem comes from the recordLength. It's not 4096 + 10 but 4096*4 + 10. Chaging to :
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 16394)
Should work.
Actually you can find this in the provided code from the web site you downloaded the binary file:
for i in range(4096):
feature.append(struct.unpack('f', f.read(4))) # <-- so 4096 * 4
Old answer:
I think the issue comes from your byte_mapper function.
That's not the correct way to convert bytes to string. You should be using decode:
bytes = b'1582480311'
print(str(bytes))
# output: "b'1582480311'"
print(bytes.decode("utf-8"))
# output: '1582480311'
If you're getting the error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 4: invalid start byte
That means product_id string contains non-utf8 characters. If you don't know the input encoding, it's difficult to convert into strings.
However, you may want to ignore those characters by adding option ignore to decode function:
bytes.decode("utf-8", "ignore")
I'm practicing a little bit with constructing a socket server on python. I'm currently running it locally on my linux mint system, and testing it with telnet localhost 20000.
The idea is quite simple. For now, I want the client(myself) to send a "hello" message to the server on which the server responses back with another "hello" message. Then the client can send any message on which the server does not respond, until the client says "bye". Then the server responds with another "bye", and the socket is closed.
I implemented this in the following way (in python 3):
1 from socket import *
2 serverPort = 20000
3 serverSocket = socket(AF_INET, SOCK_STREAM)
4 serverSocket.bind(('', serverPort))
5 serverSocket.listen(1)
6
7 print('The echo server is ready to receive')
8 while 1:
9 connectionSocket, addr = serverSocket.accept()
10 print('Processing client ', addr)
11 sentence = ""
12
13 try:
14 sentence = connectionSocket.recv(1024)
15 while sentence != "hello":
16 sentence = connectionSocket.recv(1024)
17 connectionSocket.send("hello")
18
19 sentence = connectionSocket.recv(1024)
20 while sentence != "bye":
21 sentence = connectionSocket.recv(1024)
22 connectionSocket.send("bye")
23 except error:
24 pass
25
26 print('Client closed ', addr)
27 connectionSocket.close()
It's a very simple program, in which I can't find any bug, so I was very surprised to see it not work. When I type in "hello" after running the server with python3 server.py and establishing a connection trough telnet localhost 20000 I simply get no response. I first thought that the problem lies in the equality test in line number 15, so I tested a few things with prints, and instead of "hello" The message received by the server was: b'hello\r\n. I understand the \n, because I type Enter after the "hello" message, but it's not clear to me why the b' and the \r appear.
Do you know what the problem is here and how I can fix it? On other examples on stack overflow the string comparision just works fine, so I can't figure out why it is not working for me.
Thanks in advance!
Update:
I use nc now instead of telnet, and changed every occurence of connectionSocket.recv(1024) into str(connectionSocket.recv(1024), 'utf-8'). The server gets the correct string now, but it still does not echo "hello" back to me, nor does it close the socket when I type "bye".
Use netcat instead of telnet to avoid the telnet protocol bytes.
Like this: nc localhost 20000 and type your input then.
Also, socket I/O is always bytes in Python 3, for instance see https://docs.python.org/3/library/socket.html#socket.socket.recv So you have to compare with b'hello' (or decode the bytes to a string first)
I am using requests stream for preforming a 'GET' download of a remote very large CSVs, then chunking response using response.iter_content(). This has been working for multiple data providers.
However, for one remote data provider, when using response.iter_content(), occasionally I am getting a ChunkedEncodingError, specifically:
ChunkedEncodingError: (
ProtocolError(
'Connection broken: IncompleteRead(921 bytes read, 103 more expected)',
IncompleteRead(921 bytes read, 103 more expected)),
)
Here is the Python 3 code, and I would like to know of an alternative to resolving this chunking exception problem:
tmp_csv_chunk_sum = 0
with open(
file=tmp_csv_file_path,
mode='wb',
encoding=encoding_write
) as csv_file_wb:
try:
for chunk in response.iter_content(chunk_size=8192):
if not chunk:
break
tmp_csv_chunk_sum += 8192
csv_file_wb.write(chunk)
csv_file_wb.flush()
os.fsync(csv_file_wb.fileno())
except Exception as ex:
self.logger.error(
"Request CSV Download: Exception",
extra={
'error_details': str(ex),
'chunk_total_sum': tmp_csv_chunk_sum
}
)
raise
I truly appreciate assistance, Thank you