DRONEKIT - WARNING:dronekit:Link timeout, no heartbeat in last 5 seconds - linux

I started to using the Dronekit, Dronekit-STIL and Mavlink to simulate my python scripts. Afters some days using it without problem I started to receive the error: WARNING:dronekit:Link timeout, no heartbeat in last 5 seconds.
I had tried to reinstall all the things but nothings works.
I had install the PIP pachages on Linux Ubutun 18. I had try the same packages on Ubutun 20 but I receive the same error.
I had install this packages:
pymavlink>=2.3.3
MAVProxy-1.8.39
dronekit-2.9.2
dronekit-sitl-3.3.0
Python 2.7.17
Follow my steps to receive the error:
1 - dronekit-sitl copter --home=-25.56731,-42.61554,0,180
os: linux, apm: copter, release: stable
SITL already Downloaded and Extracted.
Ready to boot.
Execute: /home/cesar/.dronekit/sitl/copter-3.3/apm --home=-23.56731,-46.61554,0,180 --model=quad -I 0
Started model quad at -23.56731,-46.61554,0,180 at speed 1.0
bind port 5760 for 0
Starting sketch 'ArduCopter'
Serial port 0 on TCP port 5760
Starting SITL input
Waiting for connection ....
bind port 5762 for 2
Serial port 2 on TCP port 5762
bind port 5763 for 3
Serial port 3 on TCP port 5763
2 - mavproxy.py --master tcp:127.0.0.1:5760 --out udp:127.0.0.1:14551 --out udp:10.0.2.15:14550
Connect tcp:127.0.0.1:5760 source_system=255
Log Directory:
Telemetry log: mav.tlog
MAV> Waiting for heartbeat from tcp:127.0.0.1:5760
online system 1
STABILIZE> Mode STABILIZE
AP: Calibrating barometer
AP: Initialising APM...
AP: barometer calibration complete
AP: GROUND START
Init Gyro**
INS
G_off: 0.00, 0.00, 0.00
A_off: 0.00, 0.00, 0.00
A_scale: 1.00, 1.00, 1.00
3 - python hello.py
Start simulator (SITL)
Starting copter simulator (SITL)
SITL already Downloaded and Extracted.
Ready to boot.
Connecting to vehicle on: tcp:127.0.0.1:5760
WARNING:dronekit:Link timeout, no heartbeat in last 5 seconds
after 30s
ERROR:dronekit.mavlink:Exception in MAVLink input loop
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dronekit/mavlink.py", line 211, in mavlink_thread_in
fn(self)
File "/usr/local/lib/python2.7/dist-packages/dronekit/init.py", line 1371, in listener
self._heartbeat_error)
APIException: No heartbeat in 30 seconds, aborting.
Traceback (most recent call last):
File "hello.py", line 11, in
vehicle = connect(connection_string, wait_ready=True)
File "/usr/local/lib/python2.7/dist-packages/dronekit/init.py", line 3166, in connect
vehicle.initialize(rate=rate, heartbeat_timeout=heartbeat_timeout)
File "/usr/local/lib/python2.7/dist-packages/dronekit/init.py", line 2275, in initialize
raise APIException('Timeout in initializing connection.')
dronekit.APIException: Timeout in initializing connection.

Hard to say without knowing the content of hello.py.
Try connecting through udp:127.0.0.1:14551 in the hello.py script rather than tcp:127.0.0.1:5760.
Also, it looks like you're starting another SITL instance from your script, but again, hard to know without seeing the code.

Related

Glusterfs geo-replication issue

I have been using georep for the last two months and posted this on their GitHub but no answers so far.
Description of problem: after copying ~8TB without any issue, some nodes are flipping between Active and Faulty with the following error message in gsync log:
ssh> failed with UnicodeDecodeError: 'ascii' codec can't decode byte 0xf2 in position 60: ordinal not in range(128).
Default encoding in all machines is utf-8
Command to reproduce the issue:
gluster volume georeplication master_vol user#slave_machine::slave_vol start
The full output of the command that failed:
The command itself it's fine but you need to start it to fail, hence the command it's not the issue on it's own
Expected results:
No such failures, copy should go as planned
Mandatory info:
The output of the gluster volume info command:
Volume Name: volname
Type: Distributed-Replicate
Volume ID: d5a46398-9638-4b50-9db0-4cd7019fa526
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x 2 = 24
Transport-type: tcp
Bricks: 24 bricks (omited the names cause not relevant and too large)
Options Reconfigured:
features.ctime: off
cluster.min-free-disk: 15%
performance.readdir-ahead: on
server.event-threads: 8
cluster.consistent-metadata: on
performance.cache-refresh-timeout: 1
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: WARNING
performance.flush-behind: off
performance.cache-size: 5GB
performance.cache-max-file-size: 1GB
performance.io-thread-count: 32
performance.write-behind-window-size: 8MB
client.event-threads: 8
network.inode-lru-limit: 1000000
performance.md-cache-timeout: 1
performance.cache-invalidation: false
performance.stat-prefetch: on
features.cache-invalidation-timeout: 30
features.cache-invalidation: off
cluster.lookup-optimize: on
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
storage.owner-uid: 33
storage.owner-gid: 33
features.bitrot: on
features.scrub: Active
features.scrub-freq: weekly
cluster.rebal-throttle: lazy
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
The output of the gluster volume status command:
Don't really think this is relevant as everything seems fine, if needed I'll post it
The output of the gluster volume heal command:
Same as before
**- Provide logs present on following locations of client and server nodes -
/var/log/glusterfs/
Not the relevant ones as is georep, posting the exact issue: (this log is from master volume node)
[2022-09-23 09:53:32.565196] I [master(worker /bricks/brick1/data):1439:process] _GMaster: Entry Time Taken [{MKD=0}, {MKN=0}, {LIN=0}, {SYM=0}, {REN=0}, {RMD=0}, {CRE=0}, {duration=0.0000}, {UNL=0}]
[2022-09-23 09:53:32.565651] I [master(worker /bricks/brick1/data):1449:process] _GMaster: Data/Metadata Time Taken [{SETA=0}, {SETX=0}, {meta_duration=0.0000}, {data_duration=1663926812.5656}, {DATA=0}, {XATT=0}]
[2022-09-23 09:53:32.566270] I [master(worker /bricks/brick1/data):1459:process] _GMaster: Batch Completed [{changelog_end=1663925895}, {entry_stime=None}, {changelog_start=1663925895}, {stime=(0, 0)}, {duration=673.9491}, {num_changelogs=1}, {mode=xsync}]
[2022-09-23 09:53:32.668133] I [master(worker /bricks/brick1/data):1703:crawl] _GMaster: processing xsync changelog [{path=/var/lib/misc/gluster/gsyncd/georepsession/bricks-brick1-data/xsync/XSYNC-CHANGELOG.1663926139}]
[2022-09-23 09:53:33.358545] E [syncdutils(worker /bricks/brick1/data):325:log_raise_exception] : connection to peer is broken
[2022-09-23 09:53:33.358802] E [syncdutils(worker /bricks/brick1/data):847:errlog] Popen: command returned error [{cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-GcBeU5/38c083bada86a45a28e6710377e456f6.sock geoaccount#slavenode6 /usr/libexec/glusterfs/gsyncd slave mastervol geoaccount#slavenode1::slavevol --master-node masternode21 --master-node-id 08c7423e-c2b6-4d40-adc8-d2ded4f66608 --master-brick /bricks/brick1/data --local-node slavenode6 --local-node-id bc1b3971-50a7-4b32-a863-aaaa02419de6 --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin --master-dist-count 12}, {error=1}]
[2022-09-23 09:53:33.358927] E [syncdutils(worker /bricks/brick1/data):851:logerr] Popen: ssh> failed with UnicodeDecodeError: 'ascii' codec can't decode byte 0xf2 in position 60: ordinal not in range(128).
[2022-09-23 09:53:33.672739] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2022-09-23 09:53:45.477905] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
**- Is there any crash ? Provide the backtrace and coredump
Provided log up
Additional info:
Master volume: 12x2 Distributed-replicated setup, been working for a couple years no, no big issues as of today. 160TB of Data
Slave volume: 2x(5+1) Distributed-disperse setup, created exclusively to be a slave georep node. Managed to copy 11TB of data from master node, but it's failing.
The operating system / glusterfs version:
On ALL nodes: Glusterfs version= 9.6
Master nodes OS: CentOS 7
Slave nodes OS: Debian11
Extra questions
Don't really know if it's the place to ask this, but while we're at it, any guidance as of how to improve sync performance? Tried changing the parameter sync_jobs up to 9 (from 3) but as we've seen (while it was working) it'd only copy from 3 nodes max, at a "low" speed (about 40% of our bandwidth). It could go as high as 1Gbps but the max we got was 370Mbps.
Also, is there any in-depth documentation for georep? The basics we found were too basic and we did miss more doc to read and dig up into.

Connect TCP socket failed with: 111 (No connection between python in linux and twincat in windows)

I have been trying to get a connection between TwinCAT 3 on Windows and Python on Ubuntu. I already have the connection between Twincat 3 Windows and Python Windows working, but not to Ubuntu. I have a virtual machine set up through Oracle VM Virtualbox. I tried many things but so far had no success in creating the connection.
I have a bridged adapter network and tried to open the port of the IP address of the virtual machine in linux through sudo ufw allow
I have the following code:
pyads.open_port()
pyads.add_route('10.11.104.206.1.1','127.0.0.1')
pyads.close_port()
plc = pyads.Connection('10.11.104.206.1.1', 851)
plc.open()
try:
# try to connect to PLC
plc.read_state()
print('Connection succeeded')
except Exception:
print('Connection failed')
And this is the error I get:
2020-11-22T22:45:46+0100 Error: Connect TCP socket failed with: 111
Traceback (most recent call last):
File "/home/laurence/ws_moveit/devel/lib/moveit_tutorials/move_panda_LKO.py", line 15, in <module>
exec(compile(fh.read(), python_script, 'exec'), context)
File "/home/laurence/ws_moveit/src/moveit_tutorials/doc/move_panda_LKO/scripts/move_panda_LKO.py", line 64, in <module>
pyads.add_route('10.11.104.206.1.1','127.0.0.1')
File "/usr/local/lib/python3.8/dist-packages/pyads/ads.py", line 188, in add_route
return adsAddRoute(adr.netIdStruct(), ip_address)
File "/usr/local/lib/python3.8/dist-packages/pyads/pyads_ex.py", line 155, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pyads/pyads_ex.py", line 177, in adsAddRoute
raise ADSError(error_code)
pyads.pyads_ex.ADSError: ADSError: target port not found ADS Server not started (6).
These are the netid/IPadresses.
-
-
-
CX-52EE70
169.254.64.202
5.82.238.112.1.1
TCP_IP
-
LEENLAPTOP19
127.0.0.1
10.11.104.206.1.1
TCP_IP
I have tried combinations with other netid/IP addresses so sometimes I get other errors (110,113) but usually 111 which means connection refused, but I do not know what I am doing wrong. Any ideas?
Please make sure the plc runtime is running (a plc program) when you connect. If the plc is in config or exception mode the plc runtime ads port (851 (or 801 for TC2)) is not present. That is what the ADS error 6 target port not found is trying to tell us.

Azure-IoT - Raspberry pi3 Forever script exited with code: 7

I'm running a forever script sending data from raspberry pi 3 to azure-iot-hub, using the following,
root#raspberrypi3:~# forever start /home/pi/azure/iam/ble_azure.js
After working for about 1.5 days, I stopped getting messages at auzre-iot-hub, when I checked forever list I got the following:
root#raspberrypi3:~# forever list
info: Forever processes running
data: uid command script forever pid id logfile uptime
data: [0] NWgI /usr/bin/nodejs /home/pi/azure/iam/ble_azure.js 8990 3784 /root/.forever/NWgI.log 0:21:17:38.742
When I checked the log file I get this error message:
/home/pi/azure/iam/node_modules/applicationinsights/AutoCollection/Exceptions.js:27
throw error;
^
NotConnectedError: mqtt.js returned client disconnecting error
at translateError (/home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-translate-error.js:25:11)
at MqttTwinReceiver._handleError (/home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:201:42)
at /home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:64:18
at MqttClient._checkDisconnecting (/home/pi/azure/iam/node_modules/mqtt/lib/client.js:314:7)
at MqttClient.subscribe (/home/pi/azure/iam/node_modules/mqtt/lib/client.js:423:12)
at /home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:62:22
at _combinedTickCallback (internal/process/next_tick.js:73:7)
at process._tickCallback (internal/process/next_tick.js:104:9)
error: Forever detected script exited with code: 7
error: Script restart attempt #34
state has changed poweredOn
started scanning
[IoT hub Client] Connect error: mqtt.js returned premature close error
Through the log file, mqtt error keeps happening multiple times and forever handled it successfully, what I can't understand why after 1.5 days, I get this error:
**error: Forever detected script exited with code: 7
error: Script restart attempt #34**
Also why I keep getting such mqtt error, why it keeps disconnecting?
**NotConnectedError: mqtt.js returned client disconnecting error**
Forever --version
v0.15.3
root#raspberrypi3:~# uname -a
Linux raspberrypi3 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
Thanks
If you're using Client.fromConnectionString to instantiate the client object the SDK disconnects and reconnects every 45 minutes to renew the shared access signature token. (it doesn't happen with AMQP that uses a different authentication mechanism). It might be that when re-establishing the connection the client hits this "premature close" error that we've been tracking in this issue.
There are 2 things that could help limit potential errors linked to disconnecting/reconnecting:
Use X509 certificates instead of connection strings to authenticate.
Create a client object using Client.fromSharedAccessSignature and build a long-lived signature that does not require disconnecting and reconnecting as often.
Last but not least, the next release of the SDK (1.2.0) will include a retry/reconnect logic that is way more robust than what was there before. I'll update the issue to point to it when it is released.

Why are my COM ports busy?

I have a python program I am using to talk to a microcontroler. It opens the com ports like this:
def STM32_connect():
ports = list(serial.tools.list_ports.comports())
for p in ports:
if "STM32" in p.description:
connection = serial.Serial(p.device, timeout = .01)
return(connection)
print("ERROR: No STM32 Device Found")
sys.exit()
serial_connection = STM32_connect()
And then does a bunch of things, sending and receiving data until I close the program like this with a keyboard interput:
except:
print("\n Program Interrupted...")
finally:
print("\n Closing Serial Port\n")
serial_connection.close()
This all works fine. My problem is when the python script is improperly killed by disconnecting the USB cable or powering off the board I can no longer connect to the micro on any COM port. I receive this error:
Traceback (most recent call last):
File "/home/---/.local/lib/python3.5/site-packages/serial/serialposix.py", line 265, in open
self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
OSError: [Errno 16] Device or resource busy: '/dev/ttyACM2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "flextester.py", line 1, in <module>
from flex_usb_class import *
File "/home/---/Code/RobotFlexTester/flex_usb_class.py", line 30, in <module>
serial_connection = STM32_connect()
File "/home/---/Code/RobotFlexTester/flex_usb_class.py", line 25, in STM32_connect
connection = serial.Serial(p.device, timeout = .01)
File "/home/---/.local/lib/python3.5/site-packages/serial/serialutil.py", line 240, in __init__
self.open()
File "/home/janey/.local/lib/python3.5/site-packages/serial/serialposix.py", line 268, in open
raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 16] could not open port /dev/ttyACM2: [Errno 16] Device or resource busy: '/dev/ttyACM2'
I can change COM ports and I receive the exact same error at a different ttyACM port. But I can connect to the same device on the same port if I try to run a different python script. The problem seems to somehow be locked to the initial test script but ps -a does not show it to still be running. The problem goes away without me doing anything after about 30 seconds - 1 min.
the 30s to 1min delay are due to the internals of RS232 protocol. see the closing_wait option ( default 30 sec ) in setserial command ( http://manpages.ubuntu.com/manpages/zesty/man8/setserial.8.html ) when disconnecting USB cable and connection is killed the protocol waits for the time specified in the closing_wait option until it closes the port / session. session_lockout forbids attaching a second process to an open port...

Failure running Overtone and SuperCollider

I can't get overtone to work with supercollider server, I'm following the getting started guide at https://github.com/overtone/overtone/wiki/Getting-Started, I've got Jack audio server running through qjackctl, then I ran SuperCollider with scsynth -u 8888 which produced the following output:
Found 12 LADSPA plugins
JackDriver: client name is 'SuperCollider'
SC_AudioDriver: sample rate = 48000.000000, driver's block size = 1024
SuperCollider 3 server ready.
Zeroconf: registered service 'SuperCollider'
then in the clojure repl I connect to SC server:
(connect-external-server 8888)
then when I run (definst foo [] (saw 220))
I get the following error:
CompilerException java.util.concurrent.TimeoutException: deref! timeout
error. Dereference took longer than 5000 ms whilst blocking until the
following node has completed loading: #<synth-group[loading]: Inst foo
Container 41>, compiling:(form-init1483192646581877285.clj:131:7)
and scsynth outputs FAILURE IN SERVER /g_new Group 31 not found
also if I try (demo (sin-osc)) I get the error FAILURE IN SERVER /s_new Group 7 not found
although if I run using sclang:
s.boot;
{ SinOsc.ar(440, 0, 0.2) }.play;
it does produce a sound.
I'm running Manjaro Linux using the Linux 4.9.27 real time Manjaro kernel
and an HDA Intel PCH sound card.

Resources