when i run yum command：
> yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
/usr/lib64/python2.6/lib-dynload/arraymodule.so: cannot read file data: Input/output error
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)]
Current version of Python is 2.6.6，not other。
system logs:
Oct 16 09:56:50 localhost kernel: mptbase: ioc0: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000) cb_idx mptscsih_io_done
Oct 16 09:56:50 localhost kernel: LSI Debug log info 31080000 for channel 0 id 0
Oct 16 09:56:50 localhost kernel: mptbase: ioc0: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000) cb_idx mptscsih_io_done
Oct 16 09:56:50 localhost kernel: LSI Debug log info 31080000 for channel 0 id 0
Oct 16 09:56:50 localhost kernel: sd 6:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 16 09:56:50 localhost kernel: sd 6:0:0:0: [sda] Sense Key : Medium Error [current]
Oct 16 09:56:50 localhost kernel: Info fld=0x4d59fc8
Oct 16 09:56:50 localhost kernel: sd 6:0:0:0: [sda] Add. Sense: Unrecovered read error
Oct 16 09:56:50 localhost kernel: sd 6:0:0:0: [sda] CDB: Read(10): 28 00 04 d5 9f c8 00 00 08 00
Oct 16 09:56:50 localhost kernel: end_request: critical medium error, dev sda, sector 81108936
Who know how to fix? Thank you!
Input/output error indicates that you system cannot read the file. Your log indicates that the hard drive is failing. Reinstall yum through RPM if you must, but ultimately backup your critical data and salvage the storage array.
Related
I'm working on non-Linux OS and now trying to enable bluetooth on Raspberry Pi 4.
I have some necessary drivers such as: gpio, uart (pl011 and mini-uart), mailbox and expgpio through that mailbox.
To enable bluetooth I make some steps:
I configure GPIOs as described in Linux's dts to make UART0 connected
to BT/WiFi chip;
I set BT_ON expgpio to 1 through mailbox (it is made by default, just ensure);
I wrote some command to UART0 and nothing happened =( UART driver return success and reading command answer is always timeouted.
I think I could forget some step for initialization procedure, but as I can see in Linux log there is only firmware downloading and many commands, such as read device name, can be executed prior to it.
May be I forget to enable some clock source or a regulator, but I don't have any idea where start my research.
There is a part of Raspbian kernel log with additional debug info:
Jan 28 05:17:13 raspberrypi kernel: [ 15.321055] Bluetooth: Core ver 2.22
Jan 28 05:17:13 raspberrypi kernel: [ 15.321093] device class 'bluetooth': registering
Jan 28 05:17:13 raspberrypi kernel: [ 15.321149] NET: Registered PF_BLUETOOTH protocol family
Jan 28 05:17:13 raspberrypi kernel: [ 15.321158] Bluetooth: HCI device and connection manager initialized
Jan 28 05:17:13 raspberrypi kernel: [ 15.321176] Bluetooth: HCI socket layer initialized
Jan 28 05:17:13 raspberrypi kernel: [ 15.321189] Bluetooth: L2CAP socket layer initialized
Jan 28 05:17:13 raspberrypi kernel: [ 15.321208] Bluetooth: SCO socket layer initialized
Jan 28 05:17:13 raspberrypi kernel: [ 15.335356] Bluetooth: HCI UART driver ver 2.3
Jan 28 05:17:13 raspberrypi kernel: [ 15.335377] Bluetooth: HCI UART protocol H4 registered at id 0
Jan 28 05:17:13 raspberrypi kernel: [ 15.335387] bus: 'serial': add driver hci_uart_h5
Jan 28 05:17:13 raspberrypi kernel: [ 15.335456] Bluetooth: HCI UART protocol Three-wire (H5) registered at id 2
Jan 28 05:17:13 raspberrypi kernel: [ 15.335480] bus: 'platform': add driver hci_bcm
Jan 28 05:17:13 raspberrypi kernel: [ 15.335641] bus: 'serial': add driver hci_uart_bcm
Jan 28 05:17:13 raspberrypi kernel: [ 15.335679] Bluetooth: HCI UART protocol Broadcom registered at id 7
Jan 28 05:17:13 raspberrypi kernel: [ 15.337922] Bluetooth: TTY name ttyAMA0
Jan 28 05:17:13 raspberrypi kernel: [ 15.338543] Bluetooth: hci_uart_register_dev
Jan 28 05:17:13 raspberrypi kernel: [ 15.338599] device: 'hci0': device_add
Jan 28 05:17:13 raspberrypi kernel: [ 15.345358] device: 'rfkill1': device_add
Jan 28 05:17:13 raspberrypi kernel: [ 15.345497] Bluetooth: HCI UART protocol set. Proto H4; id 0
Jan 28 05:17:13 raspberrypi kernel: [ 15.345530] Bluetooth: hci_uart_open hci0 5d898f04
Jan 28 05:17:13 raspberrypi kernel: [ 15.345543] Bluetooth: hci_uart_setup: START
Jan 28 05:17:13 raspberrypi kernel: [ 15.345550] Bluetooth: hci_uart_setup: init speed = 0
Jan 28 05:17:13 raspberrypi kernel: [ 15.345557] Bluetooth: hci_uart_setup: oper speed = 0
Jan 28 05:17:13 raspberrypi kernel: [ 15.352975] Bluetooth: hci0: type 1 len 3
Jan 28 05:17:13 raspberrypi kernel: [ 15.353010] Bluetooth skb: 00000000: 01 03 10 00
Jan 28 05:17:13 raspberrypi kernel: [ 15.353026] Bluetooth: hci_uart_write_work written 4
Jan 28 05:17:13 raspberrypi kernel: [ 15.353760] Bluetooth: hci0: type 1 len 3
Jan 28 05:17:13 raspberrypi kernel: [ 15.353826] Bluetooth skb: 00000000: 01 01 10 00
....
a lot of lines
....
Jan 28 05:17:13 raspberrypi btuart[479]: bcm43xx_init
Jan 28 05:17:13 raspberrypi btuart[479]: Flash firmware /lib/firmware/brcm/BCM4345C0.hcd
Jan 28 05:17:13 raspberrypi btuart[479]: Set Controller UART speed to 3000000 bit/s
Jan 28 05:17:13 raspberrypi btuart[479]: Device setup complete
Jan 28 05:17:13 raspberrypi systemd[1]: Starting Load/Save RF Kill Switch Status...
Jan 28 05:17:13 raspberrypi systemd[1]: Started Configure Bluetooth Modems connected by UART.
Jan 28 05:17:13 raspberrypi systemd[1]: Reached target Multi-User System.
Jan 28 05:17:13 raspberrypi systemd[1]: Reached target Graphical Interface.
Jan 28 05:17:13 raspberrypi systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jan 28 05:17:13 raspberrypi systemd[625]: Reached target Bluetooth.
Jan 28 05:17:13 raspberrypi systemd[1]: Started Load/Save RF Kill Switch Status.
Jan 28 05:17:13 raspberrypi systemd[1]: Created slice system-bthelper.slice.
Jan 28 05:17:13 raspberrypi systemd[1]: Starting Raspberry Pi bluetooth helper...
Jan 28 05:17:13 raspberrypi systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Jan 28 05:17:13 raspberrypi systemd[1]: Finished Update UTMP about System Runlevel Changes.
Jan 28 05:17:13 raspberrypi bthelper[774]: Raspberry Pi BDADDR already set
Jan 28 05:17:13 raspberrypi systemd[1]: Finished Raspberry Pi bluetooth helper.
Jan 28 05:17:13 raspberrypi kernel: [ 15.490868] Bluetooth: hci0: type 1 len 8
Jan 28 05:17:13 raspberrypi kernel: [ 15.490909] Bluetooth skb: 00000000: 01 1c fc 05 01 02 00 01 01
Jan 28 05:17:13 raspberrypi kernel: [ 15.490930] Bluetooth: hci_uart_write_work written 9
Thank you in advance
For H4 protocol UART with Hardware Flow Control must be used. Adding HFC support to PL011 UART driver resolves the problem.
I been getting this error on my manjaro linux machine, here is some more info:
- Journal begins at Mon 2021-03-08 18:37:49 EET, ends at Tue 2021-03-09 16:21:19 EET. --
Mar 09 11:02:26 manjaro kernel: tpm_crb MSFT0101:00: can't request region for resource [mem 0xcfbb6000-0xcfbb9fff]
Mar 09 11:02:29 manjaro kernel: kfd kfd: STONEY not supported in kfd
Mar 09 11:02:32 manjaro systemd-backlight[1332]: Failed to get backlight or LED device 'backlight:acpi_video0': No such device
Mar 09 11:02:32 manjaro systemd[1]: Failed to start Load/Save Screen Backlight Brightness of backlight:acpi_video0.
Subject: A start job for unit systemd-backlight#backlight:acpi_video0.service has failed
Defined-By: systemd
Support: https://forum.manjaro.org/c/support
A start job for unit systemd-backlight#backlight:acpi_video0.service has finished with a failure.
The job identifier is 1354 and the job result is failed.
Mar 09 11:02:32 manjaro systemd-backlight[1333]: Failed to get backlight or LED device 'backlight:acpi_video1': No such device
Mar 09 11:02:32 manjaro systemd[1]: Failed to start Load/Save Screen Backlight Brightness of backlight:acpi_video1.
Subject: A start job for unit systemd-backlight#backlight:acpi_video1.service has failed
Defined-By: systemd
Support: https://forum.manjaro.org/c/support
A start job for unit systemd-backlight#backlight:acpi_video1.service has finished with a failure.
The job identifier is 1360 and the job result is failed.
I don't know if the kfd error it's happening because of the first error.
I would like to know what it actually means, where is it coming from, and how can I fix it?
And maybe a word on the systemd-backlight#backlight:acpi_video1.service error.
The setup i have:
Cpu:
AMD A9-9420 RADEON R5, 5 COMPUTE CORES 2C+3G, 2586 MHz
GPU:
ATI Stoney [Radeon R2/R3/R4/R5 Graphics]
4GB RAM, 250GB SSD
OS: Linux manjaro 5.9.16-1-MANJARO #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020 x86_64 GNU/Linux
The connection to a Bluetooth device via RFCOMM fails on Linux/Bluez with Connection refused at the call of
connect(s, (struct sockaddr *)&addr, sizeof(addr));.
The device was successfully paired. An RFCOMM connection to that device from Android or Windows can be successfully established, so the problem seems to be locaed with Bluez diver and/or blueotoothd.
With Linux/Bluez the bluetoothctl and Wireshark traces show that it fist connects and then after about 2 seconds a disconnection is done. The reason for the disconnection is not clear.
The same problem happens with different Linux releases, on PC with USB Bluetooth (Linux ubuntu 4.15.0-33-generic #36~16.04.1-Ubuntu SMP Wed Aug 15 17:21:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux) or Raspberry Pi 3 (Jessie, Stretch).
I have checked numerous other thread having the same/similar problem. Most have no or no clear answer.
The Wireshark trace screenshot shows the disconnect after 2.2 seconds.
The corresponding bluetoothd syslog output:
Aug 31 16:43:54 ubuntu bluetoothd[926]: src/adapter.c:connected_callback() hci0 device F6:65:0A:E5:DE:E1 connected eir_len 22
Aug 31 16:43:54 ubuntu bluetoothd[926]: src/device.c:device_create() dst F6:65:0A:E5:DE:E1
Aug 31 16:43:54 ubuntu bluetoothd[926]: src/device.c:device_new() address F6:65:0A:E5:DE:E1
Aug 31 16:43:55 ubuntu bluetoothd[926]: src/device.c:device_new() Creating device /org/bluez/hci0/dev_F6_65_0A_E5_DE_E1
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/adapter.c:dev_disconnected() Device F6:65:0A:E5:DE:E1 disconnected, reason 3
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/adapter.c:adapter_remove_connection()
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/adapter.c:adapter_remove_connection() Removing temporary device /org/bluez/hci0/dev_F6_65_0A_E5_DE_E1
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/device.c:device_remove() Removing device /org/bluez/hci0/dev_F6_65_0A_E5_DE_E1
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/device.c:btd_device_unref() Freeing device /org/bluez/hci0/dev_F6_65_0A_E5_DE_E1
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/device.c:device_free() 0x563aa2a270a0
Aug 31 16:43:57 ubuntu bluetoothd[926]: plugins/policy.c:disconnect_cb() reason 3
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/adapter.c:bonding_attempt_complete() hci0 bdaddr F6:65:0A:E5:DE:E1 type 0 status 0xe
Aug 31 16:43:57 ubuntu bluetoothd[926]: src/adapter.c:resume_discovery()
reason 3 points to MGMT_DEV_DISCONN_REMOTE in include/net/bluetooth/mgmt.h of the kernel sources. This would mean that it is the device who initiates the disconnect. But the highlighted line in the Wireshark trace shows that it is the host that initiates the disconnection.
Many thanks for any help in advance.
The incorrect RFCOMM channel was used. It instantly works when the correct RFCOMM channel is used.
sdptool records F6:65:0A:E5:DE:E1 shows on which channel the RFCOMM is:
Service Name: Serial Port
Service RecHandle: 0x10000
Service Class ID List:
"Serial Port" (0x1101)
Protocol Descriptor List:
"L2CAP" (0x0100)
"RFCOMM" (0x0003)
Channel: 5
I have a cluster of 3 kafka brokers Version 0.10.2.1. Each broker has it's own host 2 cpu / 16G RAM, In addition we are using docker to wrap the broker process.
The problems is as follows:
Almost every day at the same time we see all of our kafka clients failed for 10 minutes.
At the beginning I thought it is related to Kafka No broker in ISR for partition
But after a while I discover that the broker just crash due to OOM-killer.
I also played with the Xmx and Xms before I discover that it is the OOM-killer. I had:
-Xmx2048M -Xms2048M
-Xmx4096M -Xms2048M
Same behavior for both
In addition currently we don't have ulimit
>> ulimit
unlimited
less kern.log
LOGS:
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761019] run-parts invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761022] run-parts cpuset=/ mems_allowed=0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761026] CPU: 1 PID: 12266 Comm: run-parts Not tainted 4.4.0-59-generic #80-Ubuntu
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761027] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761029] 0000000000000286 000000004811d7da ffff880036967af0 ffffffff813f7583
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761031] ffff880036967cc8 ffff880439f2f000 ffff880036967b60 ffffffff8120ad5e
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761033] ffffffff81cd2dc7 0000000000000000 ffffffff81e67760 0000000000000206
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761036] Call Trace:
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761041] [<ffffffff813f7583>] dump_stack+0x63/0x90
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761044] [<ffffffff8120ad5e>] dump_header+0x5a/0x1c5
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761048] [<ffffffff81192722>] oom_kill_process+0x202/0x3c0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761049] [<ffffffff81192b49>] out_of_memory+0x219/0x460
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761052] [<ffffffff81198abd>] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761054] [<ffffffff81198eb6>] __alloc_pages_nodemask+0x286/0x2a0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761056] [<ffffffff81198f6b>] alloc_kmem_pages_node+0x4b/0xc0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761060] [<ffffffff8107ea5e>] copy_process+0x1be/0x1b70
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761063] [<ffffffff81391bcc>] ? apparmor_file_alloc_security+0x5c/0x220
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761066] [<ffffffff811ed05a>] ? kmem_cache_alloc+0x1ca/0x1f0
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761070] [<ffffffff81347bd3>] ? security_file_alloc+0x33/0x50
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761073] [<ffffffff810caf11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761074] [<ffffffff810805a0>] _do_fork+0x80/0x360
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761076] [<ffffffff81080929>] SyS_clone+0x19/0x20
Jan 23 06:25:16 kafka10-172-40-103-177 kernel: [16504862.761080] [<ffffffff818384f2>] entry_SYSCALL_64_fastpath+0x16/0x71
And ....
Jan 24 06:25:25 kafka10-172-40-103-177 kernel: [16591270.954463] Out of memory: Kill process 16123 (java) score 134 or sacrifice child
Jan 24 06:25:25 kafka10-172-40-103-177 kernel: [16591270.958609] Killed process 16123 (java) total-vm:11977548kB, anon-rss:2035780kB, file-rss:67848kB
Any suggestion of how to approach this ??
We found the problem.
First I will say that adding more RAM to the machine also solved the problem but it is "expensive solution".
The problem was as follows:
Since I was working with EC2 ubuntu distribution I got daily crontabs in all of my cluster exactly at the same time. One of the scripts was mlocate this script apparently took too many resources.
I assume that since all cluster of kafka has some issues with IO and Memory, brokers was trying to use more memory and then the OOM killer killed them.
When 2 of my 3 brokers were down some services were down.
So the solution was:
Change the crontab to work in different hours of the day in each
broker.
Disable mlocate
I also faced the same issue below mentioned blog helped me out :
https://docs.confluent.io/current/kafka/deployment.html
How to decide Kafka Cluster size
https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html
And please make sure that the swap is enabled on all the brokers.
I'm trying to understand the output from the backtrace I captured using kprobes. If you'd like to see the full stack trace, that's fine, but it's not necessary for the question. Below is an excerpt:
Jul 24 16:31:34 scilinx kernel: [<ffffffff813a2b2e>] ? ata_scsi_port_error_handler+0x4be/0x710
Jul 24 16:31:34 scilinx kernel: [<ffffffff813a2ea8>] ? ata_scsi_cmd_error_handler+0x128/0x180
Jul 24 16:31:34 scilinx kernel: [<ffffffff813a2f98>] ? ata_scsi_error+0x98/0xd0
Jul 24 16:31:34 scilinx kernel: [<ffffffff81386cfa>] ? scsi_error_handler+0x12a/0x810
Jul 24 16:31:34 scilinx kernel: [<ffffffff81386bd0>] ? scsi_error_handler+0x0/0x810
Jul 24 16:31:34 scilinx kernel: [<ffffffff8109aef6>] ? kthread+0x96/0xa0
You'll notice that scsi_error_handler is adjacent to itself in the call stack but I cannot find why this is. Here is the scsi_error_handler function for this kernel. As you can see, it does not call itself. So why does the stacktrace show it adjacent to itself like this?
Thanks.