Received message differs from sent message over serial connection - python-3.x

I am trying to connect two raspberries over a serial connection with pyserial. I wrote two little scripts, for transmitting and receiving. While there is a communication happening between those two, the received message is completely different, than the sent message.
For example if I do
serial1.write(b'hello')
on raspi1, then raspi2 receives:
print(serial2.readline().hex())
fa9b9fff
which is the hex representation of ú›Ÿÿ.
EDIT: Here is the receive and send methods:
sender:
def send_msg(_ser, _msg):
if _ser.isOpen(): # isOpen() is deprecated since version 3.0
try:
_ser.flushInput() # flush input buffer, discarding all its contents
_ser.flushOutput() # flush output buffer, aborting current output 
# and discard all that is in buffer
_ser.write(_msg)
_ser.flush()
except IOError:
print('error communicating...')
else:
print('cannot open serial port')
return
receiver:
def read_line(_ser, _eol_character=b'\n', _timeout=1, _encoding='utf-8'):
buffer = ""
timer = time.time()
while (time.time()-timer)<_timeout:
one_byte = _ser.read(1)
print(one_byte.hex())
if one_byte == _eol_character:
return buffer.encode(_encoding)
else:
buffer += str(one_byte, _encoding, errors='replace')
raise TimeoutError("Timed out while read_line(), make sure there is an EOF!")

The kind of gibberish you're getting indicates three possible causes:
-A mismatch of settings, most likely different baud rates
-Noise on the bus: frequently originating from lacking a common ground reference. This happens when you have not connected only connect the TX and RX signals but leave the GND floating (on desktop computers you'll likely have a common ground even if you don't connect the GND signals because of the safety ground on your house's wiring but on laptops or other battery-powered devices this is a problem and you have to use a third cable to connect GND on both sides of the bus).
-You're trying to communicate using different logic levels. This happens if one side of the bus is working on TTL levels (5V) and the other on 3.3V or when you have a real RS232 level (differential) combined with any of the others. Not to mention if you try to mix RS485 with RS232 or TTL, which, of course, won't work either.

Related

Slow BLE response times after second BLE device is connected

Screen grab from WireShark showing traffic when problem occurs
Short question - Referring to WireShark image, What is causing Master to send LL_CHANNEL_MAP_IND and why is it taking so long?
We are working on a hardware/software project that is utilizing the TI WL18xx as a Bluetooth controller. One of the main functions of this product is to communicate with our sensor hardware over a Bluetooth Low Energy connection. We have encountered an issue that we have had difficulty pinpointing, but feel may reside in the TI WL18xx hardware/firmware. Intermittently, after a second Bluetooth Low Energy device is connected, the response times from writing and notification of one of the characteristics on one of the connected devices becomes very long.
Details
The host product device is running our own embedded Linux image on a TI AM4376x processor. The kernel is 4.14.79 and our communication stack sits on top of Bluez5. The wifi/bluetooth chip is the Jorjin WG7831-BO, running TIInit_11.8.32.bts firmware version 4.5. It is based on the TI WL1831. The sensor devices that we connect to are our own and we use a custom command protocol which uses two characteristics to perform command handshakes. These devices work very well on a number of other platforms, including Mac, Windows, Linux, Chrome, etc.
The workflow that is causing problems is this;
A user space application allows the user to discover and connect to our sensor devices over BLE, one device at a time.
The initial connection requires a flurry of command/response type communication over the aforementioned BLE characteristics.
Once connected, the traffic is reduced significantly to occasional notifications of new measurements, and occasional command/response exchanges triggered by the user.
A single device always seems stable and performant.
When the user connects to a second device, the initial connection proceeds as expected.
However, once the second device's connection process completes, we start to see that the command/response response times become hundreds of times longer on the initially connected device.
The second device communication continues at expected speeds.
This problem only occurs with the first device about 30% of the time we follow this workflow.
Traces
Here is a short snippet of the problem that is formed from a trace log that is a mix of our library debug and btmon traces.
Everything seems fine until line 4102, at which we see the following:
ACL Data TX: Handle 1025 flags 0x00 dlen 22 #1081 [hci0] 00:12:48.654867
ATT: Write Command (0x52) len 17
Handle: 0x0014
Data: 580fd8c71bff00204e000000000000
D2PIO_SDK: GMBLNGIBlobSource.cpp(1532) : Blob cmd sent: 1bh to GDX-FOR 07100117; length = 15; rolling counter = 216; timestamp = 258104ms .
HCI Event: Number of Completed Packets (0x13) plen 5 #1082 [hci0] 00:12:49.387892
Num handles: 1
Handle: 1025
Count: 1
ACL Data RX: Handle 1025 flags 0x02 dlen 23 #1083 [hci0] 00:12:51.801225
ATT: Handle Value Notification (0x1b) len 18
Handle: 0x0016
Data: 9810272f1bd8ff00204e000000000000
D2PIO_SDK: GMBLNGIBlobSource.cpp(1745) : GetNextResponse(GDX-FOR 07100117) returns 1bh cmd blob after 3139=(261263-258124) milliseconds.
The elapsed time reported by GetNextResponse() for most cmds should be < 30 milliseconds. Response times were short when we opened and sent a bunch of cmds to device A.
The response times remained short when we opened and sent a bunch of cmds to device B. But on the first subsequent cmd sent to device A, the response time is more than 3 seconds!
Note that a Bluetooth radio can only do one thing at a time. Receive or transmit. On one single frequency. If you have two connections and two connection events happen at the same time, the firmware must decide which one to prioritize, and which one to skip. Maybe the firmware isn't smart enough to handle your specific case. Try with other connection parameters to see if something gets better. You can also try another Bluetooth dongle from a different manufacturer.

BlueZ remote device presence

Using BlueZ, which
is the official Linux Bluetooth stack
I'd like to know which of the below two methods are better suited for detecting a device's presence in the nearby.
To be more exact, I want to periodically scan for a Bluetooth device (not BLE => no advertisement packets are sent).
I found two ways of detecting it:
1.) Using l2ping
# l2ping BTMAC
2.) Using hcitool
# hcitool name BTMAC
Both approaches working.
I'd like to know, which approach would drain more battery of the scanned device?
Looking at solution #1 (l2ping's source):
It uses a standard socket connect call to connect to the remote device, then uses the send command to send data to it:
send(sk, send_buf, L2CAP_CMD_HDR_SIZE + size, 0)
Now, L2CAP_CMD_HDR_SIZE is 4, and default size is 44, so altogether 48 bytes are sent, and received back with L2CAP_ECHO_REQ.
For hcitool I just have found the entrypoint:
int hci_read_remote_name(int dd, const bdaddr_t *bdaddr, int len, char *name, int to);
My questions:
which of these approaches are better (less power-consuming) for the remote device? If there is any difference at all.
shall I reduce the l2ping's size? What shall be the minimum?
is my assumption correct that hci_read_remote_name also connects to the remote device and sends some kind of request to it for getting back its name?
To answer your questions:-
which of these approaches are better (less power-consuming) for the remote device? If there is any difference at all.
l2ping BTMAC is the more suitable command purely because this is what it is meant to do. While "hcitool name BTMAC" is used to get the remote device's name, "l2ping" is used to detect its presence which is what you want to achieve. The difference in power consumption is really minimal, but if there is any then l2ping should be less power consuming.
shall I reduce the l2ping's size? What shall be the minimum?
If changing the l2ping size requires modifying the source code then I recommend leaving it the same. By leaving it the same you are using the same command that has been used countless times and the same command that was used to qualify the BlueZ stack. This way there's less chance for error and any change would not result in noticeable performance or power improvements.
is my assumption correct that hci_read_remote_name also connects to the remote device and sends some kind of request to it for getting back its name?
Yes your assumption is correct. According the Bluetooth Specification v5.2, Vol 4, Part E, Section 7.1.19 Remote Name Request Command:
If no connection exists between the local device and the device
corresponding to the BD_ADDR, a temporary Link Layer connection will
be established to obtain the LMP features and name of the remote
device.
I hope this helps.

Arduino Random Characters sent to Bluetooth TX during Code Upload

When I upload code to my Arduino while the TX and RX pins are connected to my HC-05 module, a bunch of random characters are sent to the TX buffer, and when I connect to a device, those characters are sent and mess up communication. Is there a way that I can clear this buffer after uploading the code? I've just been disconnecting the wires whenever I upload, but I'd like to find an easier way. Thanks!
Well, if you use a serial port to both send data and the program of course you will see it on the other side of the BT... Possible solutions:
disconnect the BT module every time you want to program the Arduino
shut down the other BT device (or just disconnect it) when you have to program the Arduino
shut down the HC-05 (or keep it in reset state) until the arduino says that it is communicating (so use a GPIO to control the reset pin or a transistor to power the BT on at the beginning of the program)
use a 3-state driver between the HC-05 and the Arduino serial ports (one driver for TX and one for RX) and activate its outputs at the beginning of the arduino program.
I don't like djUniversal's solution because you cannot control what the PC transmits; if, for instance, you decide to use the byte 0xAA to signal the start of the transmission then if the PC sends 0xAA the other device thinks that the Arduino is transmitting. Choosing longer bytes sequences helps, because the sequence becomes less probable, but.....
Moreover you have to send it at EVERY command, not just at the beginning, because you have to reset the arduino to program it (and so the other device is not aware of WHEN to stop considering the data).
The only other way around it is to send a header of maybe a couple of bytes each time to send a message. The other program can wait for these characters before it starts to take commands. Until those characters are read from the buffer you would just do a Serial.read() loop to get rid of the garbage.
Also, if garbage characters are going to screw up your program really badly you might want to think about creating some kind of crude checksum also to confirm the correct transmission.
Need help coding? Let me know.

Tell when Bluetooth module connects on Arduino

I'm working on a project with Android and Arduino and am trying to figure out how on the Arduino side to tell if the Bluetooth is connected or not.
I'm using one of these Bluetooth Modules to connect. I know I can send a command through Android, but I'm trying to have an action happen automatically when they connect and not have to run a background application on the Android if possible.
Using the module supplied and nothing else you cannot: notice the module has four connectors:
Power (Vcc)
Ground
Tx (send)
Rx (receive)
Given this interface the only way to determine whether the bluetooth module is paired is to send something to the paired device and have it respond in such as way that your Arduino knows that it is connected. For instance, if your Android program always responds with "Hi there!" when it receives a string "Hello?", then by seingin "Hello?" your Arduino will know that it is paired with your Android phone/tablet. Your Arduino could poll (send the interrogation string) every minute (or every five seconds) to see if it is paired with your device.
There is a better way, but it will require some soldering on your part. If your module is HC-03/HC-05-based, then the PIO9 pin is the "paired indicator LED" (see datasheet here). You could connect that pin to an Arduino input pin and read the level: reading digital 1 will indicate that the device is paired, while reading digital 0 will indicate that it is not. It is possible, though not certain, that the pin on your module labeled STATE is exactly this kind of a pin, i.e. it indicates the paired status. Unfortunately. this pin it isn't connected to the header, so you'll have to solder a wire to the correctponding pad to connect it to your Arduino. You should test it first by connecting a multimeter in voltage mode to that pad and measure the potential between that pad and ground in paired and non-paired state. If this is the pin that responds to the paired state then you are golden. It might be that it indicates power (like the HC-03/05 PIO8 whilc blinks when on). If it turns out that the STATE pin is not the pairing status, then you should request a datasheet from your supplier, and use that to find the status LED connection: one is likely to exist. Once you found the correct pad, verify its function using the voltmeter again. Then solder a wire to that connection and read it from your Arduino.
IMPORTANT: Make sure that your Arduino never puts out a digital 1 on the Arduino pin connected to the bluetooth module status pin: these bluetooth modules run on 3.3V, and connecting any unprotected pins to 5V will be damaging. The Vcc and Txd pins are voltage shifted in the module you bought, but the LED/Status lines are likely not to be. So if the Arduino pin connected to "status" on your Bluetooth module is configured as output and you digitalWrite(HIGH) to it, you will likely damage the Bluetooth module.
Unfortuntaely, the HC-05 will switch states when paired, but won't output a 1 until it's actually connected to something.
For instance, I can unpair my phone from the HC-05, pair again, and then the LED will change state, but the output of the STATE pin is still 0. If I open up an app, and connect to the device manually then the LED, and STATE pin will change state. The LED will periodically blink twice, and the STATE pin outputs a 1 to the Arduino.
If you would like to read the the value of the STATE pin, connect a wire to any of the inputs to the arduino, and code Serial.println(digitalRead(inputPin)); inputPin being the wire to the input of the Arduino.
I've been fighting this thing for months, and have yet to find a way to make this thing automatically connect to my phone. It won't even allow for me to connect to it from my phone to the HC-05 unless I download an app onto my Android. It's possible to bind the HC-05 to a certain address, but even this did not work for me. I want to mess with the "AT+CLASS" command, but the documentation behind the instruction has hindered me thus far.
From the HC-05 datasheet we see that the connection status depends on the output from PI09. Apparently sending "AT+BIND?" to the module will return the status of PI08 & PI09 in the form,
"+ POLAR=PI08,PI09" however this makes no sense to me because in order to get this you must enter AT mode and entering AT mode will disrupt the paired connection, hence it will always send PI09 marked as "not connected".
THUS in order to see if the connection is still live from the arduinos POV I can only see 2 feasible ways:
Program arduino to, every so often, send a "hello?" and if it doesn't receive the expected "Hi back" response, then it is to assume that it isn't connected.
Connect PI09 to an arduino input pin and read it's value whenever you want to check if the connection is live or not
AT+STATE? will return the current state of the connection. Yes you will need to enter at mode, that is done by bringing up pin 11 HIGH on the HC05 module. It does require soldering but it's kinda worth it. It then allows full AT control of the device, then set it LOW to return it to normal working mode.
Another option, which I don't fully understand, is the AT+MPIO? command, which returns the state of all the pins in some strange masked format I don't understand yet.
I use the first option above so that I can terminal (Bluetooth) from my phone to the HC05 and switch on a led/relay etc (ie bring up pin 2 to HIGH) on the HC05. This required entering AT mode (pin 11 HIGH), sending the command AT+PIO=2,1 and then setting pin 11 to LOW to return to normal working mode.
Note: I noticed I had to put a 200ms delay in between high and AT and LOW commands. Angela's solution is nice - I use a cheap XBEE Bluetooth module (HC-05 Bluetooth Bee Master & Slave Module with Bluetooth XBee for Arduino uk2015) 2 units(HC05/6) for 5Stg which are laid out in XBEE format - handy for the 3.3v.

Serial port programming - Recognize end of received data

I am writing a serial port application using VC++, in which I can open a port on a switch device, send some commands and display their output. I am running a thread which always read open port for output of given command. My main thread waits until read completes, but problem is how do I recognize that command output ends, and I should signal main thread.
Almost any serial port communication requires a protocol. Some way for the receiver to discover that a response has been received in full. A very simple one is using a unique byte or character that can never appear in the rest of the data. A linefeed is standard, used by any modem for example.
This needs to get more elaborate when you need to transfer arbitrary binary data. A common solution for that is to send the length of the response first. The receiver can then count down the received bytes to know when it is complete. This often needs to be embellished with a specific start byte value so that the receiver has some chance to re-synchronize with the transmitter. And often includes a checksum or CRC so that the receiver can detect transmission errors. Further embellishments then is to make errors recoverable with ACK/NAK responses from the receiver. You'd be then well on your way in re-inventing TCP. The RATP protocol in RFC-916 is a good example, albeit widely ignored.

Resources