I have a Quectel EG25-G setup but whenever I connect (right after a reboot) it connects for 20 seconds and then disconnects.
It happens at the
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4569] device (ttyUSB2): state change: activated -> failed (reason 'modem-no-carrier', sys-iface-state: 'managed')
Jun 24 04:59:32 linux NetworkManager[5093]: <warn> [1656046772.4677] device (ttyUSB2): Activation: failed for connection 'telstra'
point but I can't read why this is happening.
Can someone point me in the right direction please ?
configuration:
nvidia#localhost:~$ sudo cat /etc/NetworkManager/system-connections/telstra
[sudo] password for nvidia:
[connection]
id=telstra
uuid=9ae31580-093b-45ec-8b8c-3c22e7eca76f
type=gsm
permissions=
[gsm]
apn=telstra.internet
number=*99#
password-flags=1
[ipv4]
dns-search=
method=auto
[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=auto
output of NetworkManager and ping google.com :
nvidia#localhost:~$ journalctl -u NetworkManager -f &
[1] 7916
nvidia#localhost:~$ ping google.com &-- Logs begin at Fri 2022-06-24 04:58:32 UTC. --
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0549] device (l4tbr0): Activation: starting connection 'l4tbr0' (73df77e3-1d83-4794-afcb-e6079b37272b)
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0575] device (l4tbr0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0602] device (l4tbr0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0609] device (l4tbr0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0612] device (l4tbr0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0628] device (l4tbr0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0636] device (l4tbr0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Jun 24 04:58:39 localhost.localdomain NetworkManager[5093]: <info> [1656046719.0681] device (l4tbr0): Activation: successful, device activated.
Jun 24 04:58:41 localhost.localdomain NetworkManager[5093]: <info> [1656046721.6864] manager: startup complete
Jun 24 04:58:43 localhost.localdomain NetworkManager[5093]: <info> [1656046723.1537] bluez: use BlueZ version 5
[2] 7917
nvidia#localhost:~$ ping: google.com: Name or service not known
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.7954] modem["ttyUSB2"]: modem state changed, 'disabled' --> 'enabling' (reason: user preference)
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.7970] manager: (ttyUSB2): new Broadband device (/org/freedesktop/NetworkManager/Devices/10)
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.7976] device (ttyUSB2): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.7983] device (ttyUSB2): modem state 'enabling'
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.7997] device (ttyUSB2): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.8004] policy: auto-activating connection 'telstra'
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.8025] device (ttyUSB2): Activation: starting connection 'telstra' (9ae31580-093b-45ec-8b8c-3c22e7eca76f)
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.8030] device (ttyUSB2): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:01 localhost.localdomain NetworkManager[5093]: <info> [1656046741.8032] manager: NetworkManager state is now CONNECTING
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.5602] modem["ttyUSB2"]: modem state changed, 'enabling' --> 'enabled' (reason: user-requested)
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.6982] modem["ttyUSB2"]: modem state changed, 'enabled' --> 'registered' (reason: unknown)
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.7381] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8055] modem["ttyUSB2"]: modem state changed, 'connecting' --> 'connected' (reason: user-requested)
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <warn> [1656046742.8173] device (ttyUSB2): ip-ifname: set ifname 'ttyUSB2', unknown ifindex
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8177] device (ttyUSB2): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8183] device (ttyUSB2): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <warn> [1656046742.8190] device (ttyUSB2): interface ttyUSB2 not up for IP configuration
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8191] modem["ttyUSB2"]: using modem-specified IP timeout: 20 seconds
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8253] loaded PPP plugin /usr/lib/aarch64-linux-gnu/NetworkManager/libnm-ppp-plugin.so
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8264] ppp-manager: starting PPP connection
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8317] ppp-manager: pppd started with pid 7929
Jun 24 04:59:02 localhost.localdomain pppd[7929]: Plugin /usr/lib/pppd/2.4.7/nm-pppd-plugin.so loaded.
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: Plugin /usr/lib/pppd/2.4.7/nm-pppd-plugin.so loaded.
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (plugin_init): initializing
Jun 24 04:59:02 localhost.localdomain pppd[7929]: pppd 2.4.7 started by root, uid 0
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_phasechange): status 3 / phase 'serial connection'
Jun 24 04:59:02 localhost.localdomain pppd[7929]: Using interface ppp0
Jun 24 04:59:02 localhost.localdomain pppd[7929]: Connect: ppp0 <--> /dev/ttyUSB2
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_phasechange): status 5 / phase 'establish'
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: Using interface ppp0
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: Connect: ppp0 <--> /dev/ttyUSB2
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8628] manager: (ppp0): new Ppp device (/org/freedesktop/NetworkManager/Devices/11)
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_phasechange): status 6 / phase 'authenticate'
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (get_credentials): passwd-hook, requesting credentials...
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (get_credentials): got credentials from NetworkManager
Jun 24 04:59:02 localhost.localdomain pppd[7929]: CHAP authentication succeeded
Jun 24 04:59:02 localhost.localdomain pppd[7929]: CHAP authentication succeeded
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_phasechange): status 8 / phase 'network'
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: CHAP authentication succeeded
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8875] devices added (path: /sys/devices/virtual/net/ppp0, iface: ppp0)
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.8875] device added (path: /sys/devices/virtual/net/ppp0, iface: ppp0): no ifupdown configuration found.
Jun 24 04:59:02 localhost.localdomain pppd[7929]: Could not determine remote IP address: defaulting to 10.64.64.64
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: Could not determine remote IP address: defaulting to 10.64.64.64
Jun 24 04:59:02 localhost.localdomain pppd[7929]: local IP address 10.247.106.201
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: local IP address 10.247.106.201
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: remote IP address 10.64.64.64
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: primary DNS address 101.168.244.101
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: secondary DNS address 101.168.244.103
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_phasechange): status 9 / phase 'running'
Jun 24 04:59:02 localhost.localdomain pppd[7929]: remote IP address 10.64.64.64
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_ip_up): ip-up event
Jun 24 04:59:02 localhost.localdomain pppd[7929]: primary DNS address 101.168.244.101
Jun 24 04:59:02 localhost.localdomain pppd[7929]: nm-ppp-plugin: (nm_ip_up): sending IPv4 config to NetworkManager...
Jun 24 04:59:02 localhost.localdomain pppd[7929]: secondary DNS address 101.168.244.103
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9212] ppp-manager: (IPv4 Config Get) reply received.
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9268] device (ppp0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9297] device (ttyUSB2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9341] device (ppp0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9350] device (ttyUSB2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9356] device (ttyUSB2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9359] manager: NetworkManager state is now CONNECTED_LOCAL
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9387] manager: NetworkManager state is now CONNECTED_SITE
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9391] policy: set 'telstra' (ppp0) as default for IPv4 routing and DNS
Jun 24 04:59:02 localhost.localdomain NetworkManager[5093]: <info> [1656046742.9437] device (ttyUSB2): Activation: successful, device activated.
Jun 24 04:59:03 localhost.localdomain NetworkManager[5093]: <info> [1656046743.1280] policy: set-hostname: set hostname to 'linux' (from address lookup)
Jun 24 04:59:03 linux NetworkManager[5093]: <info> [1656046743.9009] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 24 04:59:05 linux pppd[7929]: sif6addr: ioctl(SIOCSIFADDR): Permission denied (line 2613)
Jun 24 04:59:05 linux NetworkManager[5093]: sif6addr: ioctl(SIOCSIFADDR): Permission denied (line 2613)
Jun 24 04:59:05 linux NetworkManager[5093]: cif6addr: ioctl(SIOCDIFADDR): No such address
Jun 24 04:59:05 linux pppd[7929]: cif6addr: ioctl(SIOCDIFADDR): No such address
[2]+ Exit 2 ping google.com
nvidia#localhost:~$
nvidia#localhost:~$ ping google.com
PING google.com (172.217.24.46) 56(84) bytes of data.
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=1 ttl=115 time=34.7 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=2 ttl=115 time=74.3 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=3 ttl=115 time=71.4 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=4 ttl=115 time=55.9 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=5 ttl=115 time=54.4 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=6 ttl=115 time=52.7 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=7 ttl=115 time=51.7 ms
64 bytes from hkg07s23-in-f14.1e100.net (172.217.24.46): icmp_seq=8 ttl=115 time=52.0 ms
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4567] modem["ttyUSB2"]: modem state changed, 'connected' --> 'registered' (reason: user-requested)
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4569] device (ttyUSB2): state change: activated -> failed (reason 'modem-no-carrier', sys-iface-state: 'managed')
Jun 24 04:59:32 linux NetworkManager[5093]: <warn> [1656046772.4677] device (ttyUSB2): Activation: failed for connection 'telstra'
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4690] manager: NetworkManager state is now CONNECTED_LOCAL
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4711] device (ttyUSB2): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4955] policy: set-hostname: set hostname to 'localhost.localdomain' (no default device)
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4965] device (ppp0): state change: disconnected -> unmanaged (reason 'connection-assumed', sys-iface-state: 'external')
Jun 24 04:59:32 linux NetworkManager[5093]: <info> [1656046772.4985] policy: auto-activating connection 'telstra'
Jun 24 04:59:32 localhost.localdomain NetworkManager[5093]: <info> [1656046772.5016] device (ttyUSB2): Activation: starting connection 'telstra' (9ae31580-093b-45ec-8b8c-3c22e7eca76f)
Jun 24 04:59:32 localhost.localdomain NetworkManager[5093]: <info> [1656046772.5035] device (ttyUSB2): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:32 localhost.localdomain NetworkManager[5093]: <info> [1656046772.5038] manager: NetworkManager state is now CONNECTING
Jun 24 04:59:32 localhost.localdomain NetworkManager[5093]: <info> [1656046772.5361] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jun 24 04:59:39 localhost.localdomain NetworkManager[5093]: <info> [1656046779.6407] modem["ttyUSB2"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Jun 24 04:59:39 localhost.localdomain NetworkManager[5093]: <info> [1656046779.6622] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jun 24 04:59:46 localhost.localdomain NetworkManager[5093]: <info> [1656046786.7669] modem["ttyUSB2"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Jun 24 04:59:46 localhost.localdomain NetworkManager[5093]: <info> [1656046786.7880] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.8956] modem["ttyUSB2"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <warn> [1656046793.8960] modem-broadband[ttyUSB2]: failed to connect modem: Sending command failed: 'Resource temporarily unavailable'
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.8962] device (ttyUSB2): state change: prepare -> failed (reason 'unknown', sys-iface-state: 'managed')
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.8971] manager: NetworkManager state is now CONNECTED_LOCAL
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <warn> [1656046793.8988] device (ttyUSB2): Activation: failed for connection 'telstra'
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9007] device (ttyUSB2): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9028] policy: auto-activating connection 'telstra'
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9052] device (ttyUSB2): Activation: starting connection 'telstra' (9ae31580-093b-45ec-8b8c-3c22e7eca76f)
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9064] device (ttyUSB2): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9072] manager: NetworkManager state is now CONNECTING
Jun 24 04:59:53 localhost.localdomain NetworkManager[5093]: <info> [1656046793.9246] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jun 24 05:00:01 localhost.localdomain NetworkManager[5093]: <info> [1656046801.0289] modem["ttyUSB2"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Jun 24 05:00:01 localhost.localdomain NetworkManager[5093]: <info> [1656046801.0454] modem["ttyUSB2"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
... etc...
seeing i can't delete this and i've found the answer:
Someone here suggested to update to the latest ModemManager like:
Code:
sudo apt-get install software-properties-common # contains add-apt-repository
sudo add-apt-repository ppa:aleksander-m/modemmanager-bionic
sudo apt-get update
sudo apt-get install modemmanager libqmi-utils libmbim-utils policykit-1
sudo reboot
and it all started working. Must have been a a ModemHandler bug in the older verision of ModemManager that the base image of NVidia comes with.
Related
Recently installed Docker, ElasticSearch 7.17.6. docker-compose up -d worked fine
but when trying to bring up the ElasticSearch container, its status remains Exited(1) & can't start the container.
Command to start: sudo docker container start <container-ID>
See below Exception for command: sudo docker logs <Container-ID>
Exception in thread "main" java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/jvm.options
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.tools.launchers.JvmOptionsParser.readJvmOptionsFiles(JvmOptionsParser.java:168)
at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:124)
at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
var/log/messages file shows below error:
msg="error detaching from network es_elastic: could not find network attachment for container <Container-ID> to network es_elastic"
Nov 1 13:11:07 ES11 dockerd: time="2022-11-01T13:11:07.567246932-04:00" level=info msg="initialized VXLAN UDP port to 4789 "
Nov 1 13:11:07 ES11 kernel: br0: port 2(<Device_name2>) entered disabled state
Nov 1 13:11:07 ES11 kernel: br0: port 1(<Device_name1>) entered disabled state
Nov 1 13:11:07 ES11 kernel: ov-1-f: renamed from br0
Nov 1 13:11:07 ES11 kernel: device <Device_name2> left promiscuous mode
Nov 1 13:11:07 ES11 kernel: ov-1-f: port 2(<Device_name2>) entered disabled state
Nov 1 13:11:07 ES11 kernel: device <Device_name1> left promiscuous mode
Nov 1 13:11:07 ES11 kernel: ov-1-f: port 1(<Device_name1>) entered disabled state
Nov 1 13:11:07 ES11 kernel: vx-1-f: renamed from <Device_name1>
Nov 1 13:11:07 ES11 kernel: : renamed from <Device_name2>
Nov 1 13:11:07 ES11 avahi-daemon[891]: Withdrawing workstation service for vx-1-f.
Nov 1 13:11:07 ES11 NetworkManager[999]: <info> [ID.7289] manager: (): new Veth device (/org/freedesktop/NetworkManager/Devices/144)
Nov 1 13:11:07 ES11 kernel: : renamed from eth0
Nov 1 13:11:07 ES11 NetworkManager[999]: <info> [ID.7693] manager: (): new Veth device (/org/freedesktop/NetworkManager/Devices/145)
Nov 1 13:11:07 ES11 dockerd: time="2022-11-01T13:11:07.ID-04:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object
File jvm.options were missing from the config directory at /u01/es11/config but unsure why the error shows different location and how come it worked in other nodes es12 & es13. But containers started fine after placing these files.
I am unable to restart my apache server to successfully install the SSL certificates.
I get the following error
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
I have tried several articles and the root cause seems to be the following
Mar 29 13:05:09 localhost.localdomain httpd\[1234546\]: (98)Address already in use: AH00072: make_sock: could not bind to address \[::\]:80
Mar 29 13:05:09 localhost.localdomain httpd\[1234546\]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
I am able to diagnose the issue and get the following output and is also attached. I am unable to proceed further. Can you please help ?
Server - AlmaLinux 8
Host - IONOS
Server version: Apache/2.4.37 (AlmaLinux)
-- Unit session-62994.scope has finished starting up.
-
-- Unit session-62994.scope has finished starting up.
-
-- The unit session-62994.scope has successfully entered the 'dead' state.
Mar 31 06:07:10 localhost.localdomain dhclient\[1326\]: XMT: Solicit on ens192, interval 110600ms.
Mar 31 06:07:10 localhost.localdomain dhclient\[1326\]: RCV: Advertise message on ens192 from fe80::250:56ff:fe8c:84c6.
Mar 31 06:07:10 localhost.localdomain dhclient\[1326\]: RCV: Advertise message on ens192 from fe80::250:56ff:fe9a:f13a.
Mar 31 06:07:30 localhost.localdomain sshd\[1297516\]: Invalid user sui from 167.99.68.65 port 48488
Mar 31 06:07:30 localhost.localdomain sshd\[1297516\]: pam_unix(sshd:auth): check pass; user unknown
Mar 31 06:07:30 localhost.localdomain sshd\[1297516\]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=167.99.68.65
Mar 31 06:07:32 localhost.localdomain sshd\[1297516\]: Failed password for invalid user sui from 167.99.68.65 port 48488 ssh2
Mar 31 06:07:34 localhost.localdomain sshd\[1297516\]: Received disconnect from 167.99.68.65 port 48488:11: Bye Bye \[preauth\]
Mar 31 06:07:34 localhost.localdomain sshd\[1297516\]: Disconnected from invalid user sui 167.99.68.65 port 48488 \[preauth\]
Mar 31 06:07:44 localhost.localdomain unix_chkpwd\[1297520\]: password check failed for user (root)
Mar 31 06:07:44 localhost.localdomain sshd\[1297518\]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.173.27 user=root
Mar 31 06:07:46 localhost.localdomain sshd\[1297518\]: Failed password for root from 61.177.173.27 port 58626 ssh2
Mar 31 06:07:46 localhost.localdomain unix_chkpwd\[1297521\]: password check failed for user (root)
\[root#localhost \~\]# ss --listening --tcp --numeric --processes
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:80 0.0.0.0:\* users:(("nginx",pid=1087,fd=10),("nginx",pid=1086,fd=10),("nginx",pid=1084,fd=10))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:\* users:(("sshd",pid=1335,fd=5))
LISTEN 0 128 0.0.0.0:443 0.0.0.0:\* users:(("nginx",pid=1087,fd=11),("nginx",pid=1086,fd=11),("nginx",pid=1084,fd=11))
LISTEN 0 128 \[::\]:22 \[::\]:\* users:(("sshd",pid=1335,fd=7))
LISTEN 0 80 \*:3306 *:* users:(("mysqld",pid=1098,fd=19))
Tried -
apachectl configtest - Result: syntax ok
setenforce 0
every time I try to do:
$ docker exec
I get the error message:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:83: executing setns process caused \"exit status 16\""
Session 1 (works like expected):
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine latest baa5d63471ea 7 weeks ago 4.8 MB
hello-world latest c54a2cc56cbb 5 months ago 1.85 kB
$ docker run --rm --name alpine -it alpine sh
/ # pwd
/
Session 2:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7bd39b37aee2 alpine "sh" 22 seconds ago Up 21 seconds alpine
$ docker exec -it alpine sh
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:83: executing setns process caused \"exit status 16\""
$ docker exec -it 7bd39b37aee2 sh
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:83: executing setns process caused \"exit status 16\""
/var/log/syslog shows some warnings, but I was neither able to understand the root cause not finding matching answers.
Thanks for any hint.
= = = = = = = = = = = = = = = = = = = = = = = = =
$ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 1.13.0-rc3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 4
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-53-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.487 GiB
Name: pb7tt6ts
ID: YQ4G:ETTP:5VCM:PAJD:F3KB:O7JN:AZOF:VLTI:SKH4:BTSR:KP7D:NXIZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
= = =
/var/log/syslog docker restart and steps above
= = =
Dec 13 14:28:09 pb7tt6ts systemd[1]: Stopping Docker Socket for the API.
Dec 13 14:28:09 pb7tt6ts systemd[1]: Starting Docker Socket for the API.
Dec 13 14:28:09 pb7tt6ts systemd[1]: Listening on Docker Socket for the API.
Dec 13 14:28:09 pb7tt6ts systemd[1]: Starting Docker Application Container Engine...
Dec 13 14:28:09 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:09.291301057+01:00" level=info msg="libcontainerd: new containerd process, pid: 1448"
Dec 13 14:28:10 pb7tt6ts kernel: [25908.125394] audit: type=1400 audit(1481635690.357:28): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=1466 comm="apparmor_parser"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.370364923+01:00" level=info msg="[graphdriver] using prior storage driver: aufs"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.387915069+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.388367650+01:00" level=warning msg="Your kernel does not support swap memory limit."
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.388465142+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.388508739+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.389419384+01:00" level=info msg="Loading containers: start."
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.397339748+01:00" level=info msg="Firewalld running: false"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.628011070+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.743703578+01:00" level=info msg="Loading containers: done."
Dec 13 14:28:10 pb7tt6ts kernel: [25908.510718] aufs au_opts_verify:1597:dockerd[1462]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.808510166+01:00" level=info msg="Daemon has completed initialization"
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.808575966+01:00" level=info msg="Docker daemon" commit=4d92237 graphdriver=aufs version=1.13.0-rc3
Dec 13 14:28:10 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:10.820562161+01:00" level=info msg="API listen on /var/run/docker.sock"
Dec 13 14:28:10 pb7tt6ts systemd[1]: Started Docker Application Container Engine.
Dec 13 14:28:10 pb7tt6ts console-kit-daemon[3106]: console-kit-daemon[3106]: GLib-CRITICAL: Source ID 226 was not found when attempting to remove it
Dec 13 14:28:10 pb7tt6ts console-kit-daemon[3106]: GLib-CRITICAL: Source ID 226 was not found when attempting to remove it
Dec 13 14:28:16 pb7tt6ts kernel: [25914.206672] aufs au_opts_verify:1597:dockerd[1460]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 14:28:16 pb7tt6ts kernel: [25914.388393] aufs au_opts_verify:1597:dockerd[1460]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 14:28:16 pb7tt6ts kernel: [25914.492197] aufs au_opts_verify:1597:dockerd[1460]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <warn> [1481635696.7320] device (vethff6f844): failed to find device 35 'vethff6f844' with udev
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7340] manager: (vethff6f844): new Veth device (/org/freedesktop/NetworkManager/Devices/46)
Dec 13 14:28:16 pb7tt6ts systemd-udevd[1614]: Could not generate persistent MAC address for vethff6f844: No such file or directory
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <warn> [1481635696.7345] device (veth13c2a1d): failed to find device 36 'veth13c2a1d' with udev
Dec 13 14:28:16 pb7tt6ts systemd-udevd[1615]: Could not generate persistent MAC address for veth13c2a1d: No such file or directory
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7417] manager: (veth13c2a1d): new Veth device (/org/freedesktop/NetworkManager/Devices/47)
Dec 13 14:28:16 pb7tt6ts kernel: [25914.509027] device veth13c2a1d entered promiscuous mode
Dec 13 14:28:16 pb7tt6ts kernel: [25914.509240] IPv6: ADDRCONF(NETDEV_UP): veth13c2a1d: link is not ready
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7632] devices added (path: /sys/devices/virtual/net/vethff6f844, iface: vethff6f844)
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7632] device added (path: /sys/devices/virtual/net/vethff6f844, iface: vethff6f844): no ifupdown configuration found.
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7639] devices added (path: /sys/devices/virtual/net/veth13c2a1d, iface: veth13c2a1d)
Dec 13 14:28:16 pb7tt6ts NetworkManager[1343]: <info> [1481635696.7640] device added (path: /sys/devices/virtual/net/veth13c2a1d, iface: veth13c2a1d): no ifupdown configuration found.
Dec 13 14:28:16 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:16.965015836+01:00" level=warning msg="Your kernel does not support swap memory limit."
Dec 13 14:28:16 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:16.965090775+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Dec 13 14:28:16 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:16.965117179+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Dec 13 14:28:17 pb7tt6ts kernel: [25914.808163] eth0: renamed from vethff6f844
Dec 13 14:28:17 pb7tt6ts acvpnagent[2339]: Function: tableCallbackHandler File: RouteMgr.cpp Line: 1723 Invoked Function: recv Return Code: 11 (0x0000000B) Description: unknown
Dec 13 14:28:17 pb7tt6ts NetworkManager[1343]: <info> [1481635697.0599] devices removed (path: /sys/devices/virtual/net/vethff6f844, iface: vethff6f844)
Dec 13 14:28:17 pb7tt6ts acvpnagent[2339]: A new network interface has been detected.
Dec 13 14:28:17 pb7tt6ts NetworkManager[1343]: <info> [1481635697.0600] device (vethff6f844): driver 'veth' does not support carrier detection.
Dec 13 14:28:17 pb7tt6ts acvpnagent[2339]: Function: logInterfaces File: RouteMgr.cpp Line: 2105 Invoked Function: logInterfaces Return Code: 0 (0x00000000) Description: IP Address Interface List: 192.168.178.24 172.17.0.1 9.145.68.34 FE80:0:0:0:D8B4:C1E0:F8E4:DB77 FE80:0:0:0:42:44FF:FEC9:5D85 FE80:0:0:0:60A9:A1FF:FEED:F31C
Dec 13 14:28:17 pb7tt6ts NetworkManager[1343]: <info> [1481635697.0604] device (veth13c2a1d): link connected
Dec 13 14:28:17 pb7tt6ts NetworkManager[1343]: <info> [1481635697.0605] device (docker0): link connected
Dec 13 14:28:17 pb7tt6ts kernel: [25914.823988] IPv6: ADDRCONF(NETDEV_CHANGE): veth13c2a1d: link becomes ready
Dec 13 14:28:17 pb7tt6ts kernel: [25914.824039] docker0: port 1(veth13c2a1d) entered forwarding state
Dec 13 14:28:17 pb7tt6ts kernel: [25914.824061] docker0: port 1(veth13c2a1d) entered forwarding state
Dec 13 14:28:18 pb7tt6ts acvpnagent[2339]: Function: tableCallbackHandler File: RouteMgr.cpp Line: 1723 Invoked Function: recv Return Code: 11 (0x0000000B) Description: unknown
Dec 13 14:28:18 pb7tt6ts avahi-daemon[1217]: Joining mDNS multicast group on interface veth13c2a1d.IPv6 with address fe80::60a9:a1ff:feed:f31c.
Dec 13 14:28:18 pb7tt6ts avahi-daemon[1217]: New relevant interface veth13c2a1d.IPv6 for mDNS.
Dec 13 14:28:18 pb7tt6ts avahi-daemon[1217]: Registering new address record for fe80::60a9:a1ff:feed:f31c on veth13c2a1d.*.
Dec 13 14:28:32 pb7tt6ts kernel: [25929.850840] docker0: port 1(veth13c2a1d) entered forwarding state
Dec 13 14:28:36 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:36.704565159+01:00" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 16\\\"\"\n"
Dec 13 14:28:36 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:36.705362948+01:00" level=error msg="Handler for POST /v1.25/exec/8a78f29ef71d4c3ab982a8dd7a4a325e280766072dea7337860874a72c42f42c/resize returned error: rpc error: code = 2 desc = containerd: process not found for container"
Dec 13 14:28:46 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:46.921880770+01:00" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 16\\\"\"\n"
Dec 13 14:28:46 pb7tt6ts dockerd[1436]: time="2016-12-13T14:28:46.922576933+01:00" level=error msg="Handler for POST /v1.25/exec/5ad25668cac553118b8c702f02c69b427436eb67d1488d4170641bcacfdad50b/resize returned error: rpc error: code = 2 desc = containerd: process not found for container"
As recommended I reverted to a main version of docker and installed docker-engine 1.12.4
$ docker info
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 3
Server Version: 1.12.4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 11
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-53-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.487 GiB
Name: pb7tt6ts
ID: YQ4G:ETTP:5VCM:PAJD:F3KB:O7JN:AZOF:VLTI:SKH4:BTSR:KP7D:NXIZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Furthermore, no success but different error:
$ docker exec -it alpine sh
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 17\\\"\"\n"
Corresponding /var/log/syslog from service docker start (21:00), docker run ... (21:01), docker exec ... (21:01)
Dec 13 21:00:01 pb7tt6ts systemd[1]: Starting Docker Socket for the API.
Dec 13 21:00:01 pb7tt6ts systemd[1]: Listening on Docker Socket for the API.
Dec 13 21:00:01 pb7tt6ts systemd[1]: Starting Docker Application Container Engine...
Dec 13 21:00:01 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:01.468921183+01:00" level=info msg="libcontainerd: new containerd process, pid: 8686"
Dec 13 21:00:02 pb7tt6ts kernel: [49419.124965] audit: type=1400 audit(1481659202.536:37): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=8700 comm="apparmor_parser"
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.550070413+01:00" level=info msg="[graphdriver] using prior storage driver \"aufs\""
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.572067603+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.572336166+01:00" level=warning msg="Your kernel does not support swap memory limit."
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.572799562+01:00" level=info msg="Loading containers: start."
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.579465999+01:00" level=info msg="Firewalld running: false"
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.779165187+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.903085523+01:00" level=info msg="Loading containers: done."
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.903179108+01:00" level=info msg="Daemon has completed initialization"
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.903208197+01:00" level=info msg="Docker daemon" commit=1564f02 graphdriver=aufs version=1.12.4
Dec 13 21:00:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:00:02.923282443+01:00" level=info msg="API listen on /var/run/docker.sock"
Dec 13 21:00:02 pb7tt6ts systemd[1]: Started Docker Application Container Engine.
Dec 13 21:01:01 pb7tt6ts kernel: [49477.834789] aufs au_opts_verify:1597:dockerd[8692]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 21:01:01 pb7tt6ts kernel: [49477.896566] aufs au_opts_verify:1597:dockerd[8692]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 21:01:01 pb7tt6ts kernel: [49478.080340] aufs au_opts_verify:1597:dockerd[8692]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 21:01:01 pb7tt6ts kernel: [49478.192100] aufs au_opts_verify:1597:dockerd[8682]: dirperm1 breaks the protection by the permission bits on the lower branch
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <warn> [1481659261.6125] device (veth2b5b07c): failed to find device 47 'veth2b5b07c' with udev
Dec 13 21:01:01 pb7tt6ts systemd-udevd[8810]: Could not generate persistent MAC address for vethc2e4873: No such file or directory
Dec 13 21:01:01 pb7tt6ts kernel: [49478.196917] device vethc2e4873 entered promiscuous mode
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6215] manager: (veth2b5b07c): new Veth device (/org/freedesktop/NetworkManager/Devices/63)
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <warn> [1481659261.6222] device (vethc2e4873): failed to find device 48 'vethc2e4873' with udev
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6241] manager: (vethc2e4873): new Veth device (/org/freedesktop/NetworkManager/Devices/64)
Dec 13 21:01:01 pb7tt6ts systemd-udevd[8809]: Could not generate persistent MAC address for veth2b5b07c: No such file or directory
Dec 13 21:01:01 pb7tt6ts kernel: [49478.211913] IPv6: ADDRCONF(NETDEV_UP): vethc2e4873: link is not ready
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6454] devices added (path: /sys/devices/virtual/net/veth2b5b07c, iface: veth2b5b07c)
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6454] device added (path: /sys/devices/virtual/net/veth2b5b07c, iface: veth2b5b07c): no ifupdown configuration found.
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6507] devices added (path: /sys/devices/virtual/net/vethc2e4873, iface: vethc2e4873)
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.6507] device added (path: /sys/devices/virtual/net/vethc2e4873, iface: vethc2e4873): no ifupdown configuration found.
Dec 13 21:01:01 pb7tt6ts kernel: [49478.557310] eth0: renamed from veth2b5b07c
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.9915] devices removed (path: /sys/devices/virtual/net/veth2b5b07c, iface: veth2b5b07c)
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.9916] device (veth2b5b07c): driver 'veth' does not support carrier detection.
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.9919] device (vethc2e4873): link connected
Dec 13 21:01:01 pb7tt6ts NetworkManager[1343]: <info> [1481659261.9937] device (docker0): link connected
Dec 13 21:01:01 pb7tt6ts kernel: [49478.573434] IPv6: ADDRCONF(NETDEV_CHANGE): vethc2e4873: link becomes ready
Dec 13 21:01:01 pb7tt6ts kernel: [49478.573503] docker0: port 1(vethc2e4873) entered forwarding state
Dec 13 21:01:01 pb7tt6ts kernel: [49478.573527] docker0: port 1(vethc2e4873) entered forwarding state
Dec 13 21:01:03 pb7tt6ts avahi-daemon[1217]: Joining mDNS multicast group on interface vethc2e4873.IPv6 with address fe80::d02a:ecff:fea8:662c.
Dec 13 21:01:03 pb7tt6ts avahi-daemon[1217]: New relevant interface vethc2e4873.IPv6 for mDNS.
Dec 13 21:01:03 pb7tt6ts avahi-daemon[1217]: Registering new address record for fe80::d02a:ecff:fea8:662c on vethc2e4873.*.
Dec 13 21:01:17 pb7tt6ts kernel: [49493.628038] docker0: port 1(vethc2e4873) entered forwarding state
Dec 13 21:02:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:02:02.072027206+01:00" level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value \"oci runtime error: exec failed: container_linux.go:247: starting container process caused \\\"process_linux.go:83: executing setns process caused \\\\\\\"exit status 17\\\\\\\"\\\"\\n\""
Dec 13 21:02:02 pb7tt6ts dockerd[8675]: time="2016-12-13T21:02:02.072759152+01:00" level=error msg="Handler for POST /v1.24/exec/00c0dcac7a178129a17cd9eb833d154d428f2a6efbcd0f421ab3c5c54e52a236/resize returned error: rpc error: code = 2 desc = containerd: process not found for container"
From the linked issue is this comment which appears to be the root cause:
I think I found the root reason. It's nothing to do with Docker.
Actually docker exec always fail because of Symantec AutoProtect
running on my system. It loads a custom kernel module that add some
file operation hooks, which affects the result of setns.
$ lsmod | grep symev
symev_custom_dkms_x86_64 72166 2 symap_custom_dkms_x86_64
The workaround is to disable Symantec AutoProtect and reboot.
sudo update-rc.d autoprotect disable
I'm running weave with kubernetes/cni . I have a wordpress/mysql pod running on a kube minion. When I hit the url of the wordpress service via the browser, my node goes down (on azure). I upgraded to 8cores and 14gb ram and now when I do hit the wordpress url I find that I can't access the internet i.e. curl google.com which I could do before hitting the wordpress url.
I was curious so I tail -f /var/log/syslog and found the following which may be relevant; Please note I have an nginx pod on the same node and I can access it's url without incident, however the following happens when I hit the wordpress installation page:
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.723678] device vethwepl4fe8074 entered promiscuous mode
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.732271] eth0: renamed from vethwepg4fe8074
Jul 21 16:38:27 sc-minion-1 systemd-udevd[8388]: Could not generate persistent MAC address for vethwepl4fe8074: No such file or directory
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.744321] IPv6: ADDRCONF(NETDEV_UP): vethwepl4fe8074: link is not ready
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.744661] IPv6: ADDRCONF(NETDEV_CHANGE): vethwepl4fe8074: link becomes ready
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.744730] weave: port 3(vethwepl4fe8074) entered forwarding state
Jul 21 16:38:27 sc-minion-1 kernel: [ 6319.744736] weave: port 3(vethwepl4fe8074) entered forwarding state
Jul 21 16:38:27 sc-minion-1 docker[883]: time="2016-07-21T16:38:27.987193241Z" level=error msg="Handler for GET /images/nginx/json returned error: No such image: nginx"
Jul 21 16:38:27 sc-minion-1 kubelet[2088]: I0721 16:38:27.987696 2088 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 16:38:28 sc-minion-1 kernel: [ 6320.564156] IPv6: eth0: IPv6 duplicate address fe80::3c2b:b1ff:fe47:fe26 detected!
Jul 21 16:38:42 sc-minion-1 kernel: [ 6334.748036] weave: port 3(vethwepl4fe8074) entered forwarding state
Jul 21 16:38:42 sc-minion-1 docker[883]: time="2016-07-21T16:38:42.988502303Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 21 16:38:42 sc-minion-1 docker[883]: time="2016-07-21T16:38:42.988545804Z" level=error msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 21 16:39:02 sc-minion-1 docker[883]: time="2016-07-21T16:39:02.989940573Z" level=error msg="Not continuing with pull after error: Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/nginx/images. You may want to check your internet connection or if you are behind a proxy."
I am trying to boot Fedora 20 with serial output,so I modify the boot command line with:
menuentry 'Fedora (3.18.0) 20 (Heisenbug)' --class fedora --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-0-rescue-690525b7662a4bbca483ccdfdac3f6dc-advanced-d27ee4d5-522c-48e8-abc5-73b42bd81ae4' {
load_video
insmod gzio
insmod part_gpt
insmod ext2
set root='hd1,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt2 --hint-efi=hd1,gpt2 --hint-baremetal=ahci1,gpt2 86088439-feab-4ac8-9cca-792414d9fff0
else
search --no-floppy --fs-uuid --set=root 86088439-feab-4ac8-9cca-792414d9fff0
fi
linuxefi /vmlinuz-3.18.0 root=UUID=d27ee4d5-522c-48e8-abc5-73b42bd81ae4 ro text no_console_suspend hpet=disable console=ttyS0,115200 console=tty0
initrdefi /initramfs-3.18.0.img
}
And the serial output seemed to stop at:
'a start job is running for Show Plymouth Boot Screen',
and did not go on.
and here are the journalctl message:
Jan 06 19:02:13 localhost.localdomain systemd[1]: Mounted /boot.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Mounting /boot/efi...
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Activation of DM RAID sets.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Encrypted Volumes.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Reached target Encrypted Volumes.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Mounted /boot/efi.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Local File Systems.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Reached target Local File Systems.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Mark the need to relabel after reboot.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Reconfigure the system on administrator request.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Relabel all filesystems, if necessary.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Recreate Volatile Files and Directories...
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Security Auditing Service...
Jan 06 19:02:13 localhost.localdomain auditd[468]: Error - audit support not in kernel
Jan 06 19:02:13 localhost.localdomain auditd[468]: Cannot open netlink audit socket
Jan 06 19:02:13 localhost.localdomain auditd[468]: The audit daemon is exiting.
Jan 06 19:02:13 localhost.localdomain auditctl[469]: Error - audit support not in kernel
Jan 06 19:02:13 localhost.localdomain auditctl[469]: Error - audit support not in kernel
Jan 06 19:02:13 localhost.localdomain auditctl[469]: Cannot open netlink audit socket
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Recreate Volatile Files and Directories.
Jan 06 19:02:13 localhost.localdomain systemd[1]: auditd.service: main process exited, code=exited, status=1/FAILURE
Jan 06 19:02:13 localhost.localdomain systemd[1]: Failed to start Security Auditing Service.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Unit auditd.service entered failed state.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Starting Update UTMP about System Reboot/Shutdown...
Jan 06 19:02:13 localhost.localdomain systemd-journal[394]: Permanent journal is using 24.0M (max 601.3M, leaving 902.0M of free 2.1G, current limit 601.3M).
Jan 06 19:02:13 localhost.localdomain systemd-journal[394]: Time spent on flushing to /var is 172.987ms for 1168 entries.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
Jan 06 19:02:13 localhost.localdomain systemd[1]: Started Update UTMP about System Reboot/Shutdown.
Jan 06 19:02:17 localhost.localdomain kernel: random: nonblocking pool is initialized
finally I solved this problem by appending:
console=tty console=ttyS0,115200n8
rather than:
console=tty0 console=ttyS0,115200
in boot command line
I don't know why,but it works,thanks god.