I am using latest Kurento Media Server (6.4) and node.js app for one-2-one calls. However, Kurento process crashes from time to time inside libnice: (multiple crashes point to the same lib entries)
Segmentation fault (thread 139888166897408, pid 1093)
Stack trace:
[nice_output_stream_new]
/usr/lib/x86_64-linux-gnu/libnice.so.10:0x28C74
[nice_output_stream_new]
/usr/lib/x86_64-linux-gnu/libnice.so.10:0x2B00A
[nice_output_stream_new]
/usr/lib/x86_64-linux-gnu/libnice.so.10:0x2BD09
[nice_agent_gather_candidates]
/usr/lib/x86_64-linux-gnu/libnice.so.10:0x13E8B
[nice_agent_gather_candidates]
/usr/lib/x86_64-linux-gnu/libnice.so.10:0x1431E
[g_simple_permission_new]
/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0:0x75B21
[g_main_context_dispatch]
/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x49EAA
[g_main_context_dispatch]
/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x4A250
[g_main_loop_run]
/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x4A572
[gst_nice_src_get_type]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so:0x2F99
[g_test_get_filename]
/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x70965
[start_thread]
/lib/x86_64-linux-gnu/libpthread.so.0:0x8182
[clone]
/lib/x86_64-linux-gnu/libc.so.6:0xFA47D
At the same time, the most recent debug events recorded to log files are ice candidates pairings:
2016-03-17 08:43:12,095520 1093 [0x00007f3a407f0700] debug KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:214 newSelectedPairFull() <kmswebrtcendpoint43> New pair selected stream_id: 1, component_id: 1, local candidate: candidate:4 1 UDP 2013266431 222.250.42.158 55844 typ host, remot
e candidate: candidate:candidate:2003496507 1 UDP 25042687 222.250.42.151 51817 typ relay raddr 83.21.212.134 rport 51817
Application log shows that error happens at the moment after both SDP answers were generated, and just before startCommunication command is going to be fired:
AppServer-0 Thu, 17 Mar 2016 08:43:11 GMT kms:CallMediaPipeline generateSdpOffer 16
AppServer-0 Thu, 17 Mar 2016 08:43:11 GMT kms:server pipeline:sdpAnswer:caller sdpReady
AppServer-0 Thu, 17 Mar 2016 08:43:11 GMT kms:CallMediaPipeline generateSdpOffer 15
AppServer-0 Thu, 17 Mar 2016 08:43:11 GMT kms:server pipeline:sdpAnswer:callee sdpReady
AppServer-0 reconnect to server 0 100 ff19ebc8-b114-495e-bf31-31f188f6ea8e
Full stack trace and log in can be seen in this gist
$ kurento-media-server -v
Version: 6.4.1~1.g3ffe480
Found modules:
Module: 'core' version '6.4.1~2.g4ed0cfc'
Module: 'elements' version '6.4.1~3.g8e842ad'
Module: 'filters' version '6.4.1~3.g06e2b4f'
# dpkg -l |grep -i libnice
ii libnice10:amd64 0.1.13.1~20160224182402.77.g7bbb87a.trusty amd64 ICE library (shared library)
I can reproduce this by using two Chrome browsers (Mac + Win) and by making around 10-15 calls (call out - hang - call out - hang ..)
If anyone could give any hints, suggestions or directions, I would really appreciate that.
Thanks!
UPD: Problem solved after switching TURN server from reciprocate-turn-server 1.9.7 to coturn 4.4.2.1
Related
I've been trying to troubleshoot this problem for some days now.
A couple of minutes after starting an SSH connection to my Namecheap server (on Mac/windows/cPanel's "Terminal"), it crashes and give the following error message :
Error: The connection to the server ended in failure at {TIME} PM. (SIGKILL)
and :
Exit Code: 137
I've tried to create some kind of log file for any SIGKILL signal, but, it seems like none can be made on a Namecheap server :
auditctl doesn't exist,
We can't get systemtap because no package managers are available.
Precision :
uname -a : Linux [-n] 2.6.32-954.3.5.lve1.4.78.el6.x86_64 #1 SMP Thu Mar 26 08:20:27 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
I calculated the time between each crash : around 6min.
I don't have a very good knowledge of Linux servers, and maybe didn't include needed information. So please ask for any specificities!
linux：
[root#localhost bin]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
[root#localhost bin]# cat /proc/version
Linux version 4.14.0-115.5.1.el7a.06.aarch64 (mockbuild#arm-buildhost1) (gcc version 4.8.5 20150623 (NeoKylin 4.8.5-36) (GCC)) #1 SMP Tue Jun 18 10:34:55 CST 2019
[root#localhost bin]# file /bin/bash
/bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.7.0, BuildID[sha1]=8a346ec01d611062313a5a4ed2b0201ecc9d9fa1, stripped
JxBrower7.7：
i used this demo,the line 55 is:Browser browser = engine.newBrowser();
enter code here
public static void main(String[] args) {
Engine engine = Engine.newInstance(
EngineOptions.newBuilder(OFF_SCREEN).build());
Browser browser = engine.newBrowser();
enter code here
[root#localhost bin]# java -jar test.jar
Exception in thread "main" com.teamdev.jxbrowser.navigation.TimeoutException: Failed to execute task withing 45 seconds.
at com.teamdev.jxbrowser.navigation.internal.NavigationImpl.loadAndWait(NavigationImpl.java:248)
at com.teamdev.jxbrowser.navigation.internal.NavigationImpl.loadUrlAndWait(NavigationImpl.java:105)
at com.teamdev.jxbrowser.navigation.internal.NavigationImpl.loadUrlAndWait(NavigationImpl.java:82)
at com.teamdev.jxbrowser.navigation.internal.NavigationImpl.loadUrlAndWait(NavigationImpl.java:74)
at com.teamdev.jxbrowser.engine.internal.EngineImpl.newBrowser(EngineImpl.java:458)
at com.pinnet.HelloWorld.main(HelloWorld.java:55)
linux logs at /var/logs/messages:
22 09:48:53 localhost dbus[8661]: [system] Activating via systemd: service name='org.bluez' unit='dbus-org.bluez.service'
May 22 09:48:54 localhost abrt-hook-ccpp: Process 90562 (chromium) of user 0 killed by SIGABRT - dumping core
May 22 09:48:54 localhost abrt-hook-ccpp: Process 90566 (chromium) of user 0 killed by SIGABRT - ignoring (repeated crash)
May 22 09:48:54 localhost abrt-hook-ccpp: Process 90561 (chromium) of user 0 killed by SIGABRT - ignoring (repeated crash)
May 22 09:48:54 localhost abrt-hook-ccpp: Process 90593 (chromium) of user 0 killed by SIGABRT - ignoring (repeated crash)
May 22 09:48:55 localhost abrt-hook-ccpp: Process 90624 (chromium) of user 0 killed by SIGABRT - ignoring (repeated crash)
May 22 09:48:55 localhost abrt-hook-ccpp: Process 90623 (chromium) of user 0 killed by SIGABRT - ignoring (repeated crash)
May 22 09:48:56 localhost abrt-server: Duplicate: core backtrace
May 22 09:48:56 localhost abrt-server: DUP_OF_DIR: /var/spool/abrt/ccpp-2020-05-21-16:55:06-33694
May 22 09:48:56 localhost abrt-server: Deleting problem directory ccpp-2020-05-22-09:48:54-90562 (dup of ccpp-2020-05-21-16:55:06-33694)
May 22 09:48:56 localhost abrt-server: /bin/sh: reporter-mailx: 未找到命令
May 22 09:49:18 localhost dbus[8661]: [system] Failed to activate service 'org.bluez': timed out
I have 3 servers, 2 on AWS and one on Digital Ocean, and the timezone for all is set to CDT. But when I check the current time on all 3 by using the date command via command line, none of them matches.
Server1: Wed Jun 12 23:36:01 CDT 2019
Server2: Wed Jun 12 23:45:51 CDT 2019
Server3: Wed Jun 12 23:38:39 CDT 2019
Could anyone please suggest what needs to be done here? Thanks.
Since you have not explicitly said that you have ntp running on them, you'll need to install that. Once that is installed and set up properly, you should show the same exact time on all of them.
My background is mostly Windows programming in C and C++. Recently I've had the chance to work with some embedded Linux systems also, but I'm still new at this.
Right now I'm working on a utility for Openwrt that needs to react to network and system events that occur during normal operation.
I've been able to use Hotplug for some events, but others still elude me. I can parse the output of the system log using logread, but that seems primitive and hackish.
In particular I'd like to get a callback similar to what hotplug does for some of the 'kern.info kernel' and 'daemon.info' events. For example:
Mar 31 19:42:32 OpenWrt kern.info kernel: [ 369.540000] device wlan0 left promiscuous mode
Mar 31 19:42:32 OpenWrt kern.info kernel: [ 369.540000] br-lan: port 2(wlan0) entered disabled state
Mar 31 19:42:32 OpenWrt kern.info kernel: [ 369.730000] device wlan1 left promiscuous mode
Mar 31 19:42:32 OpenWrt kern.info kernel: [ 369.730000] br-lan: port 3(wlan1) entered disabled state
Mar 31 19:42:34 OpenWrt kern.info kernel: [ 371.360000] device wlan0 entered promiscuous mode
Mar 31 19:45:56 OpenWrt daemon.info hostapd: wlan0: STA 04:f7:e4:00:00:00 IEEE 802.11: authenticated
Mar 31 19:45:56 OpenWrt daemon.info hostapd: wlan0: STA 04:f7:e4:00:00:00 IEEE 802.11: associated (aid 1)
Mar 31 19:45:56 OpenWrt daemon.info hostapd: wlan0: STA 04:f7:e4:00:00:00 WPA: pairwise key handshake completed (WPA)
Mar 31 19:45:56 OpenWrt daemon.info hostapd: wlan0: STA 04:f7:e4:00:00:00 WPA: group key handshake completed (WPA)
Mar 31 19:45:56 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPREQUEST(br-lan) 10.1.1.51 04:f7:e4:00:00:00
Mar 31 19:45:56 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPNAK(br-lan) 10.1.1.51 04:f7:e4:00:00:00 wrong network
Mar 31 19:46:00 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPDISCOVER(br-lan) 04:f7:e4:00:00:00
Mar 31 19:46:00 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPOFFER(br-lan) 192.168.1.198 04:f7:e4:1c:09:00
Mar 31 19:46:00 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPDISCOVER(br-lan) 04:f7:e4:1c:09:00
Mar 31 19:46:00 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPOFFER(br-lan) 192.168.1.198 04:f7:e4:1c:09:00
Mar 31 19:46:01 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPREQUEST(br-lan) 192.168.1.198 04:f7:e4:1c:09:00
Mar 31 19:46:01 OpenWrt daemon.info dnsmasq-dhcp[5005]: DHCPACK(br-lan) 192.168.1.198 04:f7:e4:1c:09:00 My-iPhone
Log entries like the DHCPOFFER (as seen in your example) are generated individually by the corresponding process (for example, by udhcpc) using the Unix syslog mechanism (kind-of like the Windows Event Logging API)
By default on OpenWRT logging is handled by the syslogd process provided by the busybox package. This is fairly primitive and simply sends messages to the circular buffer you see using logread and/or to a UDP socket.
You can upgrade logging on OpenWRT to use the syslog-ng package. This has a much more advanced configuration and you should be able to use this to send filtered log events to a script that you can write to do what you need with them.
opkg install syslog-ng
syslog-ng is a GPL product but the documentation is now buried beneath a commercial web site, one would hope you can get it from the source code , via http://freecode.com/projects/syslog-ng. Note that OpenWRT seems to provide version 1.6.12 which I had trouble finding the documentation for when I implemented it on my OpenWRT devices, but eventually I found it via the wayback machine: https://web.archive.org/web/20070406054439/http://www.balabit.com/products/syslog_ng/reference-1.6/syslog-ng.html/x731.html (for example)
A configuration file fragment that would pull out those DHCP messages and send them to a standalone log file would look a bit like:
source src { unix-stream("/dev/log"); internal(); };
destination dhcp_messages { file("/var/log/dhcpmessages"); };
filter f_dhcp { match("dnsmasq-dhcp"); };
log {
source(src);
filter(f_dhcp);
destination(dhcp_messages);
};
You might probably find the pipe() or program() destination drivers the most useful for your application. For example, using a program() driver you could send selected messages to a shell script that parses them and saves them into a sqlite database.
I did a logger with programmatically configured SocketAppender for Chainsaw. I successfully receive messages in the SimpleReceiver of Chainsaw from code running on localhost, but from remote host i'm just getting like
63 Fri Sep 23 14:44:08 MSD
2011 INFO org.apache.log4j.chainsaw.messages.MessageCenter Connection
received from
my.host.com:50299 Chainsaw-WorkerThread log chainsaw
68 Fri Sep 23 15:12:22 MSD
2011 INFO org.apache.log4j.chainsaw.messages.MessageCenter Connection
lost! :: null Chainsaw-WorkerThread
log chainsaw 69 Fri Sep 23 15:12:22 MSD
2011 DEBUG org.apache.log4j.net.SocketReceiver accepted
socket Chainsaw-WorkerThread log chainsaw
70 Fri Sep 23 15:12:22 MSD
2011 DEBUG org.apache.log4j.net.SocketReceiver socket not null -
creating and starting
socketnode Chainsaw-WorkerThread log chainsaw
71 Fri Sep 23 15:12:22 MSD
2011 DEBUG org.apache.log4j.net.SocketReceiver w*aiting to accept
socket* Chainsaw-WorkerThread log chainsaw
but no tab with log messages from remote host. What's wrong? No firewall is running.
Tested with v2 and v2.1-trunk-today.
Solved yesterday. Log level in big app's plugin wasn't set correctly.