We have developed an application using ipnc_rdk version 5.0 for TMS320DM365 .
We have memory leak in the application.
We cross compiled valgrind for arm using arm- arago.
But when we run valgrind on the device, it shows illegal instruction error.
We saw couple of posts telling that valgrind doesn't support armv5.
We got valgrind patches for armv5 at below link, but they fail to apply in the valgrind source.
https://bugs.kde.org/show_bug.cgi?id=248998
We tried applying patches for valgrind 3.9.0 3.8.1 & 3.2.1 versions
Is there a valgrind version for armv5 ?
If not how can we debug memory leak error in our application.
Also i ran top utility to check memory usage and there was increase in memory. Also kerenel calls oom killer. Please find the attached log which gives the top utility output and also the out of memory log
Top utility output just before oom killer was called:
shrd, 8K buff, 8352K cached
CPU: 86% usr 13% sys 0% nic 0% idle 0% io 0% irq 0% sirq
Load average: 2.86 1.31 0.51 4/54 690
PID PPID USER STAT VSZ %MEM %CPU COMMAND
663 655 root R 59184 132% 97% /opt/ipnc/stillImage.out
668 655 root R 3080 7% 3% top
1 0 root S 1624 4% 0% init [5]
269 2 root SW 0 0% 0% [kswapd0]
627 1 root S 6152 14% 0% ./boa -c /etc
655 1 root S 3080 7% 0% -sh
529 1 root S 2976 7% 0% /usr/sbin/inetd
651 1 root S 2964 7% 0% /sbin/syslogd -n -C64 -m 20
653 1 root S 2900 6% 0% /sbin/klogd -n
646 1 root S 2900 6% 0% /usr/sbin/telnetd
634 1 root S 2704 6% 0% avahi-daemon: running [10.local]
637 1 root S 1940 4% 0% /usr/sbin/avahi-dnsconfd -D
630 1 root S 1764 4% 0% avahi-autoipd: [eth0] bound 169.254.11
631 630 root S 1764 4% 0% avahi-autoipd: [eth0] callout dispatch
622 2 root SW 0 0% 0% [flush-1:0]
620 2 root SW 0 0% 0% [flush-ubifs_0_0]
5 2 root SW 0 0% 0% [kworker/u:0]
4 2 root SW 0 0% 0% [kworker/0:0]
3 2 root SW 0 0% 0% [ksoftirqd/0]
2 0 root SW 0 0% 0% [kthreadd]
stillImage.out invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
[ 96.003955] Backtrace:
[ 96.006595] Function entered at [<c0030504>] from [<c03330c8>]
[ 96.012605] r7:00000042 r6:00000000 r5:000200da r4:c1f06000
[ 96.018703] Function entered at [<c03330b0>] from [<c007cad0>]
[ 96.024712] Function entered at [<c007ca58>] from [<c007cf50>]
[ 96.030710] Function entered at [<c007cf00>] from [<c007d478>]
[ 96.036727] Function entered at [<c007d1b4>] from [<c00809bc>]
[ 96.042602] Function entered at [<c008056c>] from [<c007a5d0>]
[ 96.048656] Function entered at [<c007a574>] from [<c0132024>]
[ 96.054670] Function entered at [<c0131fb8>] from [<c00794bc>]
[ 96.060669] Function entered at [<c00793cc>] from [<c007b6c8>]
[ 96.066683] Function entered at [<c007b220>] from [<c007b78c>]
[ 96.072694] Function entered at [<c007b718>] from [<c01317a8>]
[ 96.078718] Function entered at [<c0131620>] from [<c00a2b34>]
[ 96.084813] Function entered at [<c00a2a90>] from [<c00a35e8>]
[ 96.090777] r8:400b4000 r7:c1f07f70 r6:400b4000 r5:00001000 r4:c1d85680
[ 96.098026] Function entered at [<c00a3534>] from [<c00a3738>]
[ 96.103955] r8:400b4000 r7:00001000 r6:c1d85680 r5:00000000 r4:0004e000
[ 96.111222] Function entered at [<c00a36f4>] from [<c002d020>]
[ 96.117225] r8:c002d1a4 r7:00000004 r6:00351d08 r5:400b4000 r4:00001000
[ 96.124460] Mem-info:
[ 96.126777] DMA per-cpu:
[ 96.129412] CPU 0: hi: 6, btch: 1 usd: 4
[ 96.134472] active_anon:2952 inactive_anon:179 isolated_anon:0
[ 96.134504] active_file:95 inactive_file:192 isolated_file:0
[ 96.134531] unevictable:0 dirty:191 writeback:0 unstable:0
[ 96.134557] free:220 slab_reclaimable:86 slab_unreclaimable:437
[ 96.134584] mapped:154 shmem:1911 pagetables:71 bounce:0
[ 96.163348] DMA free:880kB min:880kB low:1100kB high:1320kB active_anon:11808kBinactive_anon:716kB active_file:380kB inactive_file:768kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:48768kB mlocked:0kB dirty:764kB writeback:0kB mapped:616kB shmem:7644kB slab_reclaimable:344kB slab_unreclaimable:1748kB kernel_stack:432kB pagetables:284kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1760 all_unreclaimable? yes
[ 96.202321] lowmem_reserve[]: 0 0 0
[ 96.206059] DMA: 6*4kB 7*8kB 8*16kB 1*32kB 4*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 880kB
[ 96.216739] 2198 total pagecache pages
[ 96.224107] 12288 pages of RAM
[ 96.227313] 317 free pages
[ 96.230128] 1096 reserved pages
[ 96.233410] 389 slab pages
[ 96.236289] 187 pages shared
[ 96.239319] 0 pages swap cached
[ 96.242494] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 96.250121] [ 529] 0 529 744 20 0 0 0 inetd
[ 96.257877] [ 627] 0 627 1538 22 0 0 0 boa
[ 96.265408] [ 630] 0 630 441 20 0 0 0 avahi-autoipd
[ 96.273789] [ 631] 0 631 441 15 0 0 0 avahi-autoipd
[ 96.282186] [ 634] 0 634 676 54 0 0 0 avahi-daemon
[ 96.290549] [ 637] 0 637 485 22 0 0 0 avahi-dnsconfd
[ 96.299030] [ 646] 0 646 725 15 0 0 0 telnetd
[ 96.306996] [ 651] 0 651 741 34 0 0 0 syslogd
[ 96.314876] [ 653] 0 653 725 22 0 0 0 klogd
[ 96.322563] [ 655] 0 655 770 29 0 0 0 sh
[ 96.330018] [ 663] 0 663 14796 1080 0 0 0 stillImage.out
[ 96.338552] [ 668] 0 668 770 54 0 0 0 top
[ 96.346062] Out of memory: Kill process 663 (stillImage.out) score 66 or sacrifice child
[ 96.354411] Killed process 663 (stillImage.out) total-vm:59184kB, anon-rss:3824kB, file-rss:496kB
Thanks and Regards,
Arpitha
Related
I have a memory cgroup with 1 process in it.
And look for rss memory usage in that cgroup (in memory.stat) and it is much bigger than rss memory of the process (from /proc/[pid]/status).
The only process pid in cgroup:
$ cat /sys/fs/cgroup/memory/karim/master/cgroup.procs
3744924
The memory limit in cgroup:
$ cat /sys/fs/cgroup/memory/karim/master/memory.limit_in_bytes
7340032000
rss of the cgroup is 990 MB:
$ cat /sys/fs/cgroup/memory/karim/master/memory.stat
cache 5990449152
rss 990224384
rss_huge 0
shmem 0
mapped_file 13516800
dirty 1081344
writeback 270336
pgpgin 4195191
pgpgout 2490628
pgfault 5264589
pgmajfault 0
inactive_anon 0
active_anon 990240768
inactive_file 5862830080
active_file 127021056
unevictable 0
hierarchical_memory_limit 7340032000
total_cache 5990449152
total_rss 990224384
total_rss_huge 0
total_shmem 0
total_mapped_file 13516800
total_dirty 1081344
total_writeback 270336
total_pgpgin 4195191
total_pgpgout 2490628
total_pgfault 5264589
total_pgmajfault 0
total_inactive_anon 0
total_active_anon 990240768
total_inactive_file 5862830080
total_active_file 127021056
total_unevictable 0
rss of the process is 165 MB:
$ cat /proc/3744924/status
Name: [main] /h
Umask: 0002
State: S (sleeping)
Tgid: 3744924
Ngid: 0
Pid: 3744924
PPid: 3744912
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1001 1001 1001 1001
FDSize: 256
Groups: 1000 1001
NStgid: 3744924
NSpid: 3744924
NSpgid: 3744912
NSsid: 45028
VmPeak: 2149068 kB
VmSize: 2088876 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 245352 kB
VmRSS: 198964 kB
RssAnon: 165248 kB
RssFile: 33660 kB
RssShmem: 56 kB
VmData: 575400 kB
VmStk: 132 kB
VmExe: 3048 kB
VmLib: 19180 kB
VmPTE: 1152 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 17
SigQ: 0/241014
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180000002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: fff
Cpus_allowed_list: 0-11
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 94902
nonvoluntary_ctxt_switches: 1903
Why is there such a big difference?
My child process is trying to access an PCI address space. It works fine most of the times.
But, sometimes the child process is going to zombie state. dmesg logs shows the following bus error.
[ 501.134156] Caused by (from MCSR=10008): Bus - Read Data Bus Error
[ 501.134169] Oops: Machine check, sig: 7 [#1]
There is no core file generated in this case.
[Linux:/]$ ps -axl | grep tes1
4 0 6805 32495 20 0 0 0 exit Zl ? 0:05 [test1] <defunct>
[Linux:/]$
Core is generated for SIGSEGV error by the child process. So I assume it has nothing to do with permission/ulimit settings.
Can someone help me to understand why core is not getting generated in this case?
Child Process:
--------------
[Linux:/]$ cat /proc/6805/status
Name: test1
State: Z (zombie)
Tgid: 6805
Pid: 6805
PPid: 32495
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 0
Groups:
Threads: 2
SigQ: 18/13007
SigPnd: 0000000002000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001006
SigCgt: 0000000182000200
CapInh: 0000000000000000
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: 3
Cpus_allowed_list: 0-1
voluntary_ctxt_switches: 8998
nonvoluntary_ctxt_switches: 857
Stack:
-------
[Linux:/]$ cat /proc/6805/stack
[<00000000>] (nil)
[<c0008640>] __switch_to+0xc0/0x160
[<c004b4f4>] do_exit+0x5d4/0xa70
[<c000c694>] die+0x224/0x310
[<c000ce44>] machine_check_exception+0x124/0x1e0
[<c00123bc>] ret_from_mcheck_exc+0x0/0x14c
[Linux:/]$
Parent Process:
---------------
[Linux:/]$ cat /proc/32495/status
Name: test
State: S (sleeping)
Tgid: 32495
Pid: 32495
PPid: 21911
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups:
VmPeak: 4820 kB
VmSize: 4820 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 2548 kB
VmRSS: 2548 kB
VmData: 1284 kB
VmStk: 132 kB
VmExe: 900 kB
VmLib: 1976 kB
VmPTE: 24 kB
VmSwap: 0 kB
Threads: 1
SigQ: 19/13007
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000010000
SigIgn: 0000000000001006
SigCgt: 0000000043816ef9
CapInh: 0000000000000000
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: 3
Cpus_allowed_list: 0-1
voluntary_ctxt_switches: 274
nonvoluntary_ctxt_switches: 145
[Linux:/]$
I understand that the PCI hardware which is mmaped, is not responding. So, it is appropriate that only the kernel to deal with the error.
The error won't be propagated to user level, because this is not software fault. So, We do not get a core dump (either kernel or user space), since it is not a software failure.
The Machine check exception handler in the kernel tells what the hardware failure was, and what address/data is relevant (depending on the cause) - Need to be investigated from the hardware perspective further.
I use linux perf tool to collect branch info of programs, and the command and result is as follow:
$ sudo perf record -b /bin/ls
Error:
No hardware sampling interrupt available.
No APIC? If so then you can boot the kernel with the "lapic" boot parameter to force-enable it.
the content in /pro/cpuinfo is below:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5405 # 2.00GHz
stepping : 10
microcode : 0xa07
cpu MHz : 1994.921
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 **apic** sep mtrr pge mca cmov pat pse36 clflush dts acpi strong text mmx fxsr sse sse2 ss ht tm
pbe syscall nx lm constant_tsc arch_perfmon pebs **bts** rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx tm2 ssse3 cx16 xtpr pdcm
dca sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
bugs :
bogomips : 3989.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
apic and bts in flags entry is strengthened(I want but just encapsuled by "**") and I don't know what else is import for this case. And the other 7 processors are same to processor 0.
The boot parameter "lapic" is added by modifying /boot/grub/grub.cfg:
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-0ed8a872-4eb7-4339-a0bb-6c0033da582e' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 ced80bc6-08a9-4909-9717-97658cf0c4fd
else
search --no-floppy --fs-uuid --set=root ced80bc6-08a9-4909-9717-97658cf0c4fd
fi
linux /vmlinuz-4.2.0-42-generic root=/dev/mapper/fedora_hustyong-root ro **lapic** quiet splash $vt_handoff
initrd /initrd.img-4.2.0-42-generic
}
just add lapic in linux entry.
But no sense after rebooting.
My questions:
1) What does the error info means?
2) Does the perf tool branch sampling use Intel Branch Trace Store(BTS)? Or Last Branch Record(LBR)?
3) How can I look up the LBR support?
4) what is different of the LBR and BTS support between x86 32bit and 64bit?
My OS is Ubuntu 14.04 64bit:
$ uname -a
Linux user-S5000VSA 4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
The perf install instructions:
$ sudo apt-get install linux-tools-common
$ sudo apt-get install linux-tools-4.2.0-27-generic linux-cloud-tools-4.2.0-27-generic
update:
the content of /proc/interrupts:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 127780 52084 127784 126729 127706 128431 127785 126822 IO-APIC 2-edge timer
1: 52 42 3 2 62 49 5 2 IO-APIC 1-edge i8042
8: 0 0 0 0 0 0 0 1 IO-APIC 8-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
12: 1428 1307 52 47 1424 1324 53 58 IO-APIC 12-edge i8042
14: 0 0 0 0 0 0 0 0 IO-APIC 14-edge ata_piix
15: 0 0 0 0 0 0 0 0 IO-APIC 15-edge ata_piix
17: 47 276 1004 49 52 295 993 50 IO-APIC 17-fasteoi radeon
20: 31062 4201 7533 29935 31080 4297 7540 29824 IO-APIC 20-fasteoi ata_piix
22: 0 0 0 0 0 0 0 0 IO-APIC 22-fasteoi uhci_hcd:usb3, uhci_hcd:usb5
23: 0 0 0 0 0 0 0 0 IO-APIC 23-fasteoi ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb4
25: 2 755654 3 3 1 1 3 6 PCI-MSI 2621440-edge eth0
27: 0 0 0 0 0 0 0 1 PCI-MSI 131072-edge ioat-msi
NMI: 6756 678 6894 5867 861 2168 4994 3700 Non-maskable interrupts
LOC: 343554 578094 1736638 773135 219952 777567 1459249 689292 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 6756 678 6894 5867 861 2168 4994 3700 Performance monitoring interrupts
IWI: 6756 678 6894 5867 861 2168 4994 3700 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 82594 294601 142535 259797 77845 316210 84927 261455 Rescheduling interrupts
CAL: 4749 9296 7358 31330 7560 8564 5751 20364 Function call interrupts
TLB: 5933 2044 12867 11215 6563 4682 8669 8272 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 292 292 292 292 292 292 292 292 Machine check polls
HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
I install ubuntu 16.10 64bit in my PC and run perf record -b successfully. I think maybe it's wrong in kernel or linux-tools-4.2.0-27-generic or linux-cloud-tools-4.2.0-27-generic package.
I have a custom board based on am335x. I am building a buildroot for a minimal root filesystem (minimal by size) : boots off as initramfs. I am able to boot the image but unfortunately there are no /dev/mmcblk0 entry.
I would like to know how to make it create that entry.
The following is some log and data to help.
======================================================================
# ls /dev
console ram14 tty26 tty55
cpu_dma_latency ram15 tty27 tty56
full ram2 tty28 tty57
hwrng ram3 tty29 tty58
i2c-0 ram4 tty3 tty59
i2c-1 ram5 tty30 tty6
input ram6 tty31 tty60
kmem ram7 tty32 tty61
kmsg ram8 tty33 tty62
loop-control ram9 tty34 tty63
loop0 random tty35 tty7
loop1 shm tty36 tty8
loop2 tty tty37 tty9
loop3 tty0 tty38 ttyO0
loop4 tty1 tty39 ttyO1
loop5 tty10 tty4 ttyS0
loop6 tty11 tty40 ttyS1
loop7 tty12 tty41 ttyS2
mem tty13 tty42 ttyS3
network_latency tty14 tty43 ubi_ctrl
network_throughput tty15 tty44 urandom
null tty16 tty45 usbmon0
port tty17 tty46 vcs
psaux tty18 tty47 vcs1
ptmx tty19 tty48 vcsa
pts tty2 tty49 vcsa1
ram0 tty20 tty5 watchdog
ram1 tty21 tty50 watchdog0
ram10 tty22 tty51 zero
ram11 tty23 tty52
ram12 tty24 tty53
ram13 tty25 tty54
# cat /proc/partitions
major minor #blocks name# cat /proc/devices
Character devices:
1 mem
4 /dev/vc/0
4 tty
4 ttyS
5 /dev/tty
5 /dev/console
5 /dev/ptmx
7 vcs
10 misc
13 input
29 fb
89 i2c
90 mtd
128 ptm
136 pts
153 spi
166 ttyACM
180 usb
188 ttyUSB
189 usb_device
226 drm
248 ttySDIO
249 usbmon
250 ttyO
251 watchdog
252 ptp
253 pps
254 rtc
Block devices:
1 ramdisk
259 blkext
7 loop
8 sd
31 mtdblock
65 sd
66 sd
67 sd
68 sd
69 sd
70 sd
71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
179 mmc
# cat /etc/fstab
# <file system> <mount pt> <type> <options> <dump> <pass>
/dev/root / ext2 rw,noauto 0 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts defaults,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs mode=0777 0 0
tmpfs /tmp tmpfs mode=1777 0 0
tmpfs /run tmpfs mode=0755,nosuid,nodev 0 0
sysfs /sys sysfs defaults 0 0
==================================================================
It is booting busybox and using devtmpfs which are defaults for buildroot.
If I use TI's minimalfs, it does show /dev/mmcblk0 so the kernel and device tree are ok. How to solve this.
Regards,
Preetam
You can create this node manually by mknod
mknod [OPTION]... NAME TYPE [MAJOR MINOR]
In your case type would be b (block). Major number you may find in sources or use
cat /proc/devices the minor number is a partition number.
If this is not the case, try to enable udev in buildroot config.
I am working on implementing SMP support in Linux kernel for Marvell PXA2128 ARM SoC. I am using Linus Torvald kernel as base kernel. Kernel version is 3.5. I have added SMP support in Linux kernel, I am able to boot with the second core but sometimes my kernel crashes with "Attempted to kill Init" message, that is, init process of initramfs dies somehow, don't know why. I thought L1 cache of second core is corrupted, so I invalidated the L1 cache of second core before it enters the Linux kernel execution. Crash log of kernel is like
[ 16.413024] Freeing init memory: 268K
[ 16.658111] tmpfs: No value for mount option 'strictatime'
[ 16.809997] scsi 0:0:0:0: Direct-Access SanDisk Cruzer Blade 1.27 PQ: 0 ANSI: 6
[ 16.827545] tmpfs: No value for mount option 'strictatime'
[ 16.972473] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 16.972930] sd 0:0:0:0: [sda] 15330304 512-byte logical blocks: (7.84 GB/7.30 GiB)
[ 16.974487] sd 0:0:0:0: [sda] Write Protect is off
[ 16.976104] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 17.369537] sda: sda1 sda2
[ 17.377258] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 17.602966] tmpfs: No value for mount option 'strictatime'
[ 18.074981] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 18.074981]
[ 18.222442] [<c001691c>] (unwind_backtrace+0x0/0x128) from [<c046beb0>] (dump_stack+0x20/0x24)
[ 18.301177] [<c046beb0>] (dump_stack+0x20/0x24) from [<c046bfcc>] (panic+0x94/0x1d4)
[ 18.378265] [<c046bfcc>] (panic+0x94/0x1d4) from [<c002ad08>] (do_exit+0x390/0x7ac)
[ 18.454956] [<c002ad08>] (do_exit+0x390/0x7ac) from [<c002b37c>] (do_group_exit+0x0/0xc4)
[ 18.533111] CPU0: stopping
[ 18.605621] [<c001691c>] (unwind_backtrace+0x0/0x128) from [<c046beb0>] (dump_stack+0x20/0x24)
[ 18.687225] [<c046beb0>] (dump_stack+0x20/0x24) from [<c00145ac>] (handle_IPI+0x104/0x174)
[ 18.770141] [<c00145ac>] (handle_IPI+0x104/0x174) from [<c0008590>] (gic_handle_irq+0x60/0x68)
[ 18.854888] [<c0008590>] (gic_handle_irq+0x60/0x68) from [<c000e6c0>] (__irq_svc+0x40/0x70)
[ 18.940093] Exception stack(0xc0685f38 to 0xc0685f80)
[ 19.022155] 5f20: c06c4c28 a0000093
[ 19.108398] 5f40: 00000001 60400100 c0684000 c06c48c8 c047560c c0691438 c177a080 562f5842
[ 19.188018] SMP: failed to stop secondary CPUs
[ 19.275970] 5f60: 00000000 c0685f9c c0685f40 c0685f80 c0020668 c0010068 60000013 ffffffff
[ 19.363708] [<c000e6c0>] (__irq_svc+0x40/0x70) from [<c0010068>] (cpu_idle+0x94/0xdc)
[ 19.450805] [<c0010068>] (cpu_idle+0x94/0xdc) from [<c0462d24>] (rest_init+0x7c/0x94)
[ 19.537292] [<c0462d24>] (rest_init+0x7c/0x94) from [<c063f89c>] (start_kernel+0x328/0x380)
Now when my kernel boots successfully, the output of cat /proc/interrupts is like
bash-4.2# cat /proc/interrupts
CPU0 CPU1
39: 101 0 GIC pxa_i2c-i2c
45: 0 26692 GIC timer0
46: 26578 0 GIC timer1
52: 27 0 GIC olpc-ec-1.75
58: 0 0 GIC mmp-vmeta
60: 1142 0 GIC UART3
71: 0 0 GIC mmc2
72: 0 0 GIC olpc-kbd
73: 25978 0 GIC pxa168fb-dss
76: 2282 0 GIC ehci_hcd:usb1
84: 246 0 GIC mmc0
85: 9820 0 GIC mmc1
132: 0 0 ICU rtc Alrm
133: 0 0 ICU rtc 1Hz
137: 0 0 ICU galcore interrupt service
139: 88 0 ICU galcore interrupt service for 2D
141: 20 0 ICU pxa_i2c-i2c
143: 0 0 ICU pxa_i2c-i2c
145: 328 0 ICU pxa_i2c-i2c
186: 52 0 ICU mmc3
252: 0 0 GPIO hsdet-gpio
253: 0 0 GPIO hdmi-hpd
270: 0 0 GPIO d4280000.sdhci cd
278: 0 0 GPIO sdhci_wakeup_irq, sdhci_wakeup_irq, sdhci_wakeup_irq, sdhci_wakeup_irq
335: 0 0 GPIO micdet-gpio
365: 0 0 GPIO DCON
368: 0 0 GPIO olpc-switch-1.75-lid
369: 0 0 GPIO olpc-switch-1.75-ebook
393: 0 0 GPIO olpc-ec-1.75-wake
IPI0: 0 0 Timer broadcast interrupts
IPI1: 3508 4069 Rescheduling interrupts
IPI2: 0 0 Function call interrupts
IPI3: 7 429 Single function call interrupts
IPI4: 0 0 CPU stop interrupts
Err: 0
bash-4.2#
Please give me a clue what important I am missing in SMP implementation.