Understanding exception inside linux kernel - linux

I am trying to debug our embedded linux system under very low temperatures (<40C). The problem is that it does not always boot correctly and I am trying to figure out why. After some analysis I saw that the kernel goes into panic during the start-up with the following output:
can: controller area network core (rev 20090105 abi 8)
NET: Registered protocol family 29
can: raw protocol (rev 20090105)
/opt/elinos-5.1/linux/linux-ppc-2.6.34/drivers/rtc/hctosys.c: unable to open rtc
device (rtc0)
ADDRCONF(NETDEV_UP): eth0: link is not ready
IP-Config: Complete:
device=eth0, addr=192.168.100.100, mask=255.255.255.0, gw=255.255.255.255,
host=192.168.100.100, domain=, nis-domain=(none),
bootserver=192.168.100.20, rootserver=192.168.100.20, rootpath=
Freeing unused kernel memory: 156k init
init started: BusyBox v1.6.1 (2013-06-03 11:53:03 CEST) multi-call binary
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
udevd-work[84]: '/sbin/modprobe -bv of:NioctlT<NULL>Cfsl,mpc5125-ioctl' unexpect
ed exit with status 0x000b
------------[ cut here ]------------
Badness at /opt/elinos-5.1/linux/linux-ppc-2.6.34/kernel/sched.c:3574
NIP: c001acfc LR: c001ace4 CTR: c01c5fa4
REGS: c790f7c0 TRAP: 0700 Not tainted (2.6.34.7-ELinOS-146-ipipe)
MSR: 00021032 <ME,CE,IR,DR> CR: 28000482 XER: 20000000
TASK = c3ba6cb0[71] 'udevd' THREAD: c78e0000
GPR00: 00000000 c790f870 c3ba6cb0 00000001 c790f8b8 00000008 00000000 00000000
GPR08: 00000000 c0370000 00000001 c0370000 5d0fabd2 10033420 10019a6c 00000000
GPR16: 10019328 100194d4 100194c0 1002bad0 10019328 10019474 bfa35428 100192fc
GPR24: 100321f0 00000000 00000000 c649e840 00000000 00000901 00000000 00000000
NIP [c001acfc] add_preempt_count+0x48/0xac
LR [c001ace4] add_preempt_count+0x30/0xac
Call Trace:
Instruction dump:
409e0038 54290024 8009000c 2f800000 40bc0028 48133285 2f830000 419e0068
3d20c037 8009d298 2f800000 409e0058 <0fe00000> 48000050 3d20c037 8009d130
Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0x00000000
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT LTT NESTING LEVEL : 0
ORION
last sysfs file: /sys/devices/platform/gpio-keys-polled/input/input0/uevent
NIP: 00000000 LR: 00000000 CTR: c01c7778
REGS: c790f990 TRAP: 0400 Tainted: G W (2.6.34.7-ELinOS-146-ipipe)
MSR: 20009032 <EE,ME,IR,DR> CR: 28000484 XER: 20000000
TASK = c3ba6cb0[71] 'udevd' THREAD: c78e0000
GPR00: 00000000 c790fa40 c3ba6cb0 00000008 00000008 00000008 c7912804 c0348ba4
GPR08: 00000047 c78a5414 00000000 00000004 28000482 10033420 10019a6c 00000000
GPR16: 10019328 100194d4 100194c0 1002bad0 10019328 10019474 bfa35428 100192fc
GPR24: 100321f0 00000000 c790f618 00200200 00000000 c790fa34 00200200 00000000
NIP [00000000] (null)
LR [00000000] (null)
Call Trace:
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
Kernel panic - not syncing: Fatal exception in interrupt
Call Trace:
Rebooting in 180 seconds..
INFO: RCU detected CPU 0 stall (t=2500 jiffies)
INFO: RCU detected CPU 0 stall (t=10000 jiffies)
INFO: RCU detected CPU 0 stall (t=17500 jiffies)
INFO: RCU detected CPU 0 stall (t=25000 jiffies)
INFO: RCU detected CPU 0 stall (t=32500 jiffies)
INFO: RCU detected CPU 0 stall (t=40000 jiffies)
System Halted, OK to turn off power
------------[ cut here ]------------
kernel BUG at /opt/elinos-5.1/linux/linux-ppc-2.6.34/mm/vmalloc.c:1228!
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT LTT NESTING LEVEL : 0
ORION
last sysfs file: /sys/devices/platform/gpio-keys-polled/input/input0/uevent
NIP: c009b0cc LR: c0013518 CTR: 00000000
REGS: c790f7c0 TRAP: 0700 Tainted: G D W (2.6.34.7-ELinOS-146-ipipe)
MSR: 00029032 <EE,ME,CE,IR,DR> CR: 28000484 XER: 20000000
TASK = c3ba6cb0[71] 'udevd' THREAD: c78e0000
GPR00: 078fe000 c790f870 c3ba6cb0 00001000 00000001 00000001 c9000000 fddff000
GPR08: ffffffff 000000d0 c001018c c790e000 48000488 10033420 10019a6c 00000000
GPR16: 10019328 100194d4 100194c0 1002bad0 10019328 10019474 bfa35428 100192fc
GPR24: c001018c 000000d0 00001000 ffffffff c9000000 fddff000 00000001 00000001
NIP [c009b0cc] __get_vm_area_node+0x68/0x204
LR [c0013518] __ioremap_caller+0x90/0x134
Call Trace:
Instruction dump:
7c9e2378 93e1003c 7cbf2b78 90010044 9261000c 92810010 92a10014 92c10018
92e1001c 93410028 800b000c 5400016e <0f000000> 70a00001 41820030 7c7e0034
Kernel panic - not syncing: Fatal exception in interrupt
Call Trace:
Rebooting in 180 seconds..:
Could anybody give me some hints how to approach the problem? What I want is to understand which component of the system (i.e. memory chip, etc.) causes this failure. I will be very happy to hear any ideas. Thanks.


All the OOPS/panic information show the exception happened in udevd context, I think it may be triggered by "/sbin/modprobe -bv of:NioctlTCfsl,mpc5125-ioctl".
To verify this, you can remove the "/sbin/modprobe -bv of:NioctlTCfsl,mpc5125-ioctl" entry in your root file system to see whether the system can boot up successfully.
I guess your platform CPU is PowerPC architecture. If so, the exception vector is 0x700, it means the instruction fetch exception. CPU tried to fetch one instruction from invalid address. The instruction flow is incremented if there are no jump/branch instructions. If option 1 is verified that it is related to "/sbin/modprobe", please check the kernel module to analysis the instruction fetch exception.
Good luck!

Related

using printk cause error in embedded linux driver programming

printk is one of the core debug technique we use in Linux driver development. I have recently experience a weird problem. I read somewhere saying printk can be used in any context. However, when I use printk inside driver "read" function implementation, I see kernel error after it runs for a while. The error message is as following:
[ 113.017140] cdns-i2c e0005000.i2c: timeout waiting on completion
[ 113.023177] edt_ft5426 1-0038: edt_ft5426_ts_read: read error, addr=0x2 len=29.
[ 113.030585] Unable to handle kernel NULL pointer dereference at virtual address 00000043
[ 113.038688] pgd = c0004000
[ 113.041392] [00000043] *pgd=00000000
[ 113.044964] Internal error: Oops - BUG: 17 [#1] PREEMPT SMP ARM
[ 113.050872] Modules linked in: bfcore(O)
[ 113.054803] CPU: 1 PID: 690 Comm: irq/60-edt-ft54 Tainted: G O 4.14.0-xilinx #1
[ 113.058533] dma_desc_p[MIC_DMA].ready = 0
[ 113.058539] dma_desc_p[SMAP_DMA].ready = 0
[ 113.071478] Hardware name: Xilinx Zynq Platform
[ 113.076004] task: df7e86c0 task.stack: df6f6000
[ 113.080535] PC is at irq_finalize_oneshot+0x0/0xf0
[ 113.085318] LR is at irq_thread_fn+0x2c/0x34
[ 113.089577] pc : [<c0153b00>] lr : [<c0153c1c>] psr: 600f0113
[ 113.095835] sp : df6f7f48 ip : 00000000 fp : df77751c
[ 113.101051] r10: 00000000 r9 : 00000001 r8 : ded48624
[ 113.106268] r7 : c0ffffff r6 : ffffffff r5 : 00000001 r4 : ffffffff
[ 113.112787] r3 : 00000000 r2 : 00000000 r1 : ffffffff r0 : ffffffff
[ 113.119307] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 113.126433] Control: 18c5387d Table: 1ec0c04a DAC: 00000051
[ 113.132171] Process irq/60-edt-ft54 (pid: 690, stack limit = 0xdf6f6210)
[ 113.138862] Stack: (0xdf6f7f48 to 0xdf6f8000)
[ 113.143215] 7f40: df50b600 ffffe000 ded48600 c0153edc 00000000 c0153d04
[ 113.151383] 7f60: 00000000 df777500 df7e4980 df6f6000 00000000 ded48600 c0153d88 df43db70
[ 113.159553] 7f80: df77751c c01332c4 df6f6000 df7e4980 c0133194 00000000 00000000 00000000
[ 113.167727] 7fa0: 00000000 00000000 00000000 c0106fb0 00000000 00000000 00000000 00000000
[ 113.175894] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 113.184062] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 113.192243] [<c0153b00>] (irq_finalize_oneshot) from [<c0153d04>] (irq_thread_dtor+0x0/0x7c)
[ 113.196794] dma_desc_p[MIC_DMA].ready = 0
[ 113.196799] dma_desc_p[SMAP_DMA].ready = 0
[ 113.208925] [<c0153d04>] (irq_thread_dtor) from [<df6f6000>] (0xdf6f6000)
[ 113.215713] Code: 13a00001 e12fff1e e3a00000 e12fff1e (e5903044)
[ 113.221937] ---[ end trace 97dcf8cac1432a55 ]---
[ 113.226572] genirq: exiting task "irq/60-edt-ft54" (690) is an active IRQ thread (irq 60)
The setup is on a Xilinx Zynq 7020 device. I run a UI program which need touch screen as input device. So I need I2C for that. My driver responsible for reading data from FPGA. After it runs for a while, I got the error. In the log, I have:
[ 113.196794] dma_desc_p[MIC_DMA].ready = 0
[ 113.196799] dma_desc_p[SMAP_DMA].ready = 0
These are generated by my printk statements from the "read" function. If I comment away all the printk statements, the program runs ok. Any idea why there is a conflict between printk and I2C driver?

Unable to handle kernel paging request at virtual address...custom board

I have taken rpi as a reference and trying to boot kernel to our custom board, modified the "dts" file by removing all peripherals except UART and Interrupt controller by changing the base address of the rpi board to our board specific address.
Seeting the env variables as follows,
$ setenv initrd 0xc3000000;setenv initrd_high 0xc4000000;setenv fdt_high 0xc1001000;setenv fdt_addr_r 0xc1000000
$ setenv bootargs earlyprintk console=ttyAMA0 mem=128M noinitrd root=/dev/mtdblock3 rw rootfstype=jffs2 rw init=/sbin/init
This is the error log we got....
SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] Unable to handle kernel paging request at virtual address 48000fe0
[ 0.000000] pgd = c0004000
[ 0.000000] [48000fe0] *pgd=00000000
[ 0.000000] Internal error: Oops: 5 [#1] ARM
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.9.22+ #81
[ 0.000000] Hardware name: BCM2835
[ 0.000000] task: c0497f58 task.stack: c0494000
[ 0.000000] PC is at __vic_init+0x3c/0x178
[ 0.000000] LR is at 0x48000fe0
[ 0.000000] pc : [<c0473b74>] lr : [<48000fe0>] psr: a00000d3
[ 0.000000] sp : c0495f50 ip : 00000000 fp : c0495f7c
[ 0.000000] r10: 00000001 r9 : 410fb767 r8 : 48000000
[ 0.000000] r7 : 00000000 r6 : 00000000 r5 : ffffffff r4 : 00000000
[ 0.000000] r3 : 48000fe0 r2 : 00000000 r1 : 00000000 r0 : 00000000
[ 0.000000] Flags: NzCv IRQs off FIQs off Mode SVC_32 ISA ARM
Segment user
[ 0.000000] Control: 00c5387d Table: c0004008 DAC: 00000055
[ 0.000000] Process swapper (pid: 0, stack limit = 0xc0494188)
[ 0.000000] Stack: (0xc0495f50 to 0xc0496000)
Please help me,Thanks in advance.

Seems like the problem occured in interrupt context in procedure __vic_init. This procedure tried to access virtual address 48000fe0.
By the way - are you sure that this is complete calltrace taken from dmesg?
If this is only a snippet taken from dmesg - put the whole calltrace here.
If you have sources for this procedure __vic_init you can debug it - I mean find the number of the line in source file and do the further debug.

Linux kernel crash and unable to boot (console related)?

Having a hard time booting the tinycore linux kernel for an ARM A10 here, on boot the device crashes.
3.0.42 config found here: http://distro.ibiblio.org/tinycorelinux/5.x/armv7/Allwinner-A10/a10Core-kernel-3.0.42.config
U-Boot SPL 2013.01 (Feb 11 2013 - 15:19:28)
Board: mk802ii
DRAM: 1024MB
SUNXI SD/MMC: 0
U-Boot 2013.01 (Feb 11 2013 - 15:19:28) Allwinner Technology
CPU: SUNXI Family
Board: mk802ii
I2C: ready
DRAM: 1 GiB
MMC: SUNXI SD/MMC: 0
*** Warning - bad CRC, using default environment
In: serial
Out: serial
Err: serial
Net: wemac
Hit any key to stop autoboot: 0
reading uEnv.txt
117 bytes read in 2 ms (56.6 KiB/s)
Loaded environment from uEnv.txt
reading boot.scr
304 bytes read in 3 ms (98.6 KiB/s)
Jumping to boot.scr
## Executing script at 44000000
reading script.bin
42132 bytes read in 6 ms (6.7 MiB/s)
reading uImage
4109016 bytes read in 213 ms (18.4 MiB/s)
reading uCore
2951575 bytes read in 154 ms (18.3 MiB/s)
## Booting kernel from Legacy Image at 48000000 ...
Image Name: Linux-3.0.42
Created: 2015-02-16 20:40:40 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4108952 Bytes = 3.9 MiB
Load Address: 40008000
Entry Point: 40008000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 43100000 ...
Image Name: uCore for Allwinner A10
Created: 2014-12-26 21:12:42 UTC
Image Type: ARM Linux RAMDisk Image (uncompressed)
Data Size: 2951511 Bytes = 2.8 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK
Starting kernel ...
<6>Initializing cgroup subsys cpuset
<5>Linux version 3.0.42 (root#localhost.localdomain) (gcc version 4.9.1 20140717 (Red Hat Cross 4.9.1-1) (GCC) ) #3 PREEMPT Mon Feb 16 15:40:29 EST 2015
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: sun4i
<6>Memory Reserved:
<6> SYS : 0x43000000 - 0x4300ffff ( 64 kB)
<6> VE : 0x44000000 - 0x48ffffff ( 80 MB)
<6> G2D : 0x58000000 - 0x58ffffff ( 16 MB)
<6> LCD : 0x5a000000 - 0x5bffffff ( 32 MB)
Memory policy: ECC disabled, Data cache writeback
<6>BROM Ver: 1100 1100 1623
<6>chip-id: A10 (AW1623 revision C)
<7>On node 0 totalpages: 262144
<7>free_area_init_node: node 0, pgdat c07e60e4, node_mem_map c08ab000
<7> Normal zone: 1280 pages used for memmap
<7> Normal zone: 0 pages reserved
<7> Normal zone: 162560 pages, LIFO batch:31
<7> HighMem zone: 768 pages used for memmap
<7> HighMem zone: 97536 pages, LIFO batch:31
<7>pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
<7>pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
<5>Kernel command line: console=tty0 init=/init rootwait panic=10 loglevel=3 dis p.screen0_output_mode=EDID:1280x720p60 hdmi.audio=EDID:0 nozswap nortc
<6>PID hash table entries: 4096 (order: 2, 16384 bytes)
<6>Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
<6>Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
<6>Memory: 1024MB = 1024MB total
<5>Memory: 896644k/896644k available, 151932k reserved, 393216K highmem
<5>Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
DMA : 0xffc00000 - 0xffe00000 ( 2 MB)
vmalloc : 0xe8800000 - 0xf0000000 ( 120 MB)
lowmem : 0xc0000000 - 0xe8000000 ( 640 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.init : 0xc0008000 - 0xc0035000 ( 180 kB)
.text : 0xc0035000 - 0xc07a3000 (7608 kB)
.data : 0xc07a4000 - 0xc07ef9f0 ( 303 kB)
.bss : 0xc07ef9f0 - 0xc08aa768 ( 748 kB)
<6>SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>NR_IRQS:96 nr_irqs:96 96
<6>timer0: Periodic Mode
<6>Console: colour dummy device 80x30
<6>console [tty0] enabled
<6>Calibrating delay loop... <c>1001.88 BogoMIPS (lpj=5009408)
<6>pid_max: default: 32768 minimum: 301
<6>Mount-cache hash table entries: 512
<6>Initializing cgroup subsys cpuacct
<6>Initializing cgroup subsys devices
<6>Initializing cgroup subsys freezer
<6>Initializing cgroup subsys blkio
<6>CPU: Testing write buffer coherency: ok
<6>hw perfevents: enabled with ARMv7 Cortex-A8 PMU driver, 5 counters available
<6>devtmpfs: initialized
<6>print_constraints: dummy:
<6>NET: Registered protocol family 16
<6>hw-breakpoint: debug architecture 0x4 unsupported.
[ccmu] try to set apb1 parent to sata_pll failed!
SOFTWINNER DMA Driver, (c) 2003-2004,2006 Simtec Electronics
<6>Initialize DMAC OK
<6>bio: create slab <bio-0> at 0
<5>SCSI subsystem initialized
<7>libata version 3.00 loaded.
<6>usbcore: registered new interface driver usbfs
<6>usbcore: registered new interface driver hub
<6>usbcore: registered new device driver usb
<6>Advanced Linux Sound Architecture Driver Version 1.0.24.
<6>Bluetooth: Core ver 2.16
<6>NET: Registered protocol family 31
<6>Bluetooth: HCI device and connection manager initialized
<6>Bluetooth: HCI socket layer initialized
<6>Bluetooth: L2CAP socket layer initialized
<6>Bluetooth: SCO socket layer initialized
Init eGon pin module V2.0
<6>Switching to clocksource aw 64bits couter
<6>cfg80211: Calling CRDA to update world regulatory domain
<5>FS-Cache: Loaded
<6>CacheFiles: Loaded
<6>Switched to NOHz mode on CPU #0
<1>Unable to handle kernel NULL pointer dereference at virtual address 00000001
<1>pgd = c0004000
<1>[00000001] *pgd=00000000
<0>Internal error: Oops: 5 [#1] PREEMPT
<d>Modules linked in:
CPU: 0 Not tainted (3.0.42 #3)
PC is at kmem_cache_alloc+0x78/0xd0
LR is at con_insert_unipair+0xc0/0x10c
pc : [<c00f6e7c>] lr : [<c031508c>] psr: 60000093
sp : e783be88 ip : 0000025b fp : e783bea4
r10: c07c37d0 r9 : 00000001 r8 : 00000003
r7 : 00000003 r6 : 000000d0 r5 : e7802200 r4 : 00000001
r3 : 20000013 r2 : 00000000 r1 : c10b0040 r0 : 00000001
Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c5387d Table: 40004019 DAC: 00000015
PC: 0xc00f6dfc:
6dfc eafffff2 c0596580 e1a0c00d e92dd878 e24cb004 e1a05000 e1a06001 e5953000
6e1c e593c004 e5934000 e3540000 0a000018 e10f3000 f10c0080 e5951000 e5910000
6e3c e1540000 0a000007 e3a00000 e121f003 e3500000 0afffff0 e3160902 1a000017
6e5c e1a00004 e89da878 e5912004 e15c0002 1afffff4 e5952014 e28cc001 e3a00001
6e7c e7942002 e5812000 e5952000 e582c004 eaffffed e1a01006 e1a0200e e1a00005
6e9c e7e067d6 ebffff2a e3500000 03a06000 12066001 e1a04000 e3560000 0affffe7
6ebc e5951010 e3510000 0affffe4 e1a00004 eb06e873 eaffffe1 e1a0c00d e92dd878
6edc e24cb004 e3074dc0 e1a06000 e34c4088 e1a05001 e5943054 e3530003 0a00000f
LR: 0xc031500c:
500c e0844a07 e3a00000 e1a03083 e18c70b3 e5963084 e0834004 e5864084 e89da8f0
502c e3073dd8 e34c3088 e593001c e3500000 0a00001c e3a010d0 ebf7876e e3500000
504c e7860105 0a00001e e2403004 e280e07c e3a0c000 e5a3c004 e153000e 1afffffc
506c eaffffe0 e3073dd8 e34c3088 e593001c e3500000 0a00000f e3a010d0 ebf7875d
508c e3500000 e1a0c000 e5850000 0a00000c e1a0000c e3a010ff e3a02080 ebfe6fcc
50ac e1a0c000 eaffffd4 e3a01010 e7861105 e1a00001 eaffffe3 e3a0c010 e585c000
50cc eafffff2 e3e0000b e89da8f0 e1a0c00d e92ddbf0 e24cb004 e2506000 089dabf0
50ec e5964098 e3540000 0a000024 e1a00004 e3a01b01 ebfe6fe6 e2466004 e3a07000
SP: 0xe783be08:
be08 e783c980 ffd23940 ffffffff 0a21fe80 e783be5c 0000040f 00000005 000000d0
be28 00000003 00000003 e783bea4 e783be40 c003b750 c0035214 00000001 c10b0040
be48 00000000 20000013 00000001 e7802200 000000d0 00000003 00000003 00000001
be68 c07c37d0 e783bea4 0000025b e783be88 c031508c c00f6e7c 60000093 ffffffff
be88 c0887dd8 00002665 e78bc5e4 e786d780 e783bec4 e783bea8 c031508c c00f6e10
bea8 00000000 00000001 e786d780 00000003 e783bf04 e783bec8 c0315d70 c0314fd8
bec8 c08952b4 e7803200 c07c3a28 c07c37ce e783befc 00000001 00000014 c08954d8
bee8 00000002 00000004 00000000 00000000 e783bf24 e783bf08 c001e220 c0315c88
FP: 0xe783be24:
be24 000000d0 00000003 00000003 e783bea4 e783be40 c003b750 c0035214 00000001
be44 c10b0040 00000000 20000013 00000001 e7802200 000000d0 00000003 00000003
be64 00000001 c07c37d0 e783bea4 0000025b e783be88 c031508c c00f6e7c 60000093
be84 ffffffff c0887dd8 00002665 e78bc5e4 e786d780 e783bec4 e783bea8 c031508c
bea4 c00f6e10 00000000 00000001 e786d780 00000003 e783bf04 e783bec8 c0315d70
bec4 c0314fd8 c08952b4 e7803200 c07c3a28 c07c37ce e783befc 00000001 00000014
bee4 c08954d8 00000002 00000004 00000000 00000000 e783bf24 e783bf08 c001e220
bf04 c0315c88 00000000 00000000 c08953b4 00000001 e783bf54 e783bf28 c001e734
R1: 0xc10affc0:
ffc0 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffe0 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
0000 e78015e0 0000002e c0d9b020 00000000 e7802580 0000000b c0d9b040 00000000
0020 e786d840 00000299 c0d9bda0 00000000 e78bd880 000009d2 c0d9c7a0 00000000
0040 00000001 0000025a c0d9c780 00000000 e785c700 0000000d c0d9bb80 00000000
0060 e78bea00 000002d2 c0d9c7c0 00000000 e78a0400 0000000c c0d9c400 00000000
0080 e785a000 00000004 c0d9bb00 00000000 e7812000 0000025e c0d9b200 00000000
00a0 e780a000 00000001 c0d9b100 00000000 e7823130 00000291 c0d9b460 00000000
R5: 0xe7802180:
2180 c10b0030 00000000 00000006 00000040 00000040 00000000 00000040 00000040
21a0 00000040 00000000 00000004 00000000 00000040 00000040 00000000 e7800040
21c0 e7802140 e7802240 00000000 00000000 00000000 00000000 00000000 00000000
21e0 00000000 00000000 00000000 e7801060 ffffffff ffffffff ffffffff ffffffff
2200 c10b0040 00000000 00000007 00000080 00000080 00000000 00000020 00000020
2220 00000020 00000000 00000008 00000000 00000080 00000040 00000000 e7800080
2240 e78021c0 e78022c0 00000000 00000000 00000000 00000000 00000000 00000000
2260 00000000 00000000 00000000 e7801080 ffffffff ffffffff ffffffff ffffffff
R10: 0xc07c3750:
3750 f0c5f0c4 f0c7f0c6 f0c9f0c8 f0cbf0ca f0cdf0cc f0cff0ce f0d1f0d0 f0d3f0d2
3770 f0d5f0d4 f0d7f0d6 f0d9f0d8 f0dbf0da f0ddf0dc f0dff0de f0e1f0e0 f0e3f0e2
3790 f0e5f0e4 f0e7f0e6 f0e9f0e8 f0ebf0ea f0edf0ec f0eff0ee f0f1f0f0 f0f3f0f2
37b0 f0f5f0f4 f0f7f0f6 f0f9f0f8 f0fbf0fa f0fdf0fc f0fff0fe 263a0000 2665263b
37d0 25c62666 26602663 25d82022 25d925cb 26402642 266b266a 00a4263c 25ba25b6
37f0 25c425c0 203c2195 00a700b6 21a825ac 21932191 21902192 2194221f 25bc25b2
3810 00210020 00a80022 00240023 00260025 00b40027 00290028 002b002a 00b8002c
3830 00ad002d 002f002e 00310030 00330032 00350034 00370036 00390038 003b003a
<0>Process swapper (pid: 1, stack limit = 0xe783a2e8)
<0>Stack: (0xe783be88 to 0xe783c000)
<0>be80: c0887dd8 00002665 e78bc5e4 e786d780 e783bec4 e783bea8
<0>bea0: c031508c c00f6e10 00000000 00000001 e786d780 00000003 e783bf04 e783bec8
<0>bec0: c0315d70 c0314fd8 c08952b4 e7803200 c07c3a28 c07c37ce e783befc 00000001
<0>bee0: 00000014 c08954d8 00000002 00000004 00000000 00000000 e783bf24 e783bf08
<0>bf00: c001e220 c0315c88 00000000 00000000 c08953b4 00000001 e783bf54 e783bf28
<0>bf20: c001e734 c001e1e0 c0740fc8 e783bf38 c015d534 c0894ef8 00000000 c0627944
<0>bf40: 00000013 00000000 e783bf74 e783bf58 c001dcd8 c001e5ac c06f9560 e783bf7c
<0>bf60: 0000000c c0896500 e783bf9c e783bf78 c001f2f8 c001dbb0 c070e9d4 00000001
<0>bf80: 00000013 c07efa00 e783a020 00000001 e783bfe4 e783bfa0 c003549c c001f234
<0>bfa0: e783bfbc e783bfb0 00000000 c001f228 e783bfcc e783bfc0 c00ae8f4 c002d83c
<0>bfc0: c002de00 00000001 00000013 00000000 00000000 00000000 e783bff4 e783bfe8
<0>bfe0: c0008b44 c0035380 00000000 e783bff8 c006c9fc c0008a44 ffffffff ffffffff
Backtrace:
[<c00f6e04>] (kmem_cache_alloc+0x0/0xd0) from [<c031508c>] (con_insert_unipair+0 xc0/0x10c)
r6:e786d780 r5:e78bc5e4 r4:00002665 r3:c0887dd8
[<c0314fcc>] (con_insert_unipair+0x0/0x10c) from [<c0315d70>] (con_set_default_u nimap+0xf4/0x18c)
r7:00000003 r6:e786d780 r5:00000001 r4:00000000
[<c0315c7c>] (con_set_default_unimap+0x0/0x18c) from [<c001e220>] (console_map_i nit+0x4c/0x58)
[<c001e1d4>] (console_map_init+0x0/0x58) from [<c001e734>] (vty_init+0x194/0x1a4 )
r6:00000001 r5:c08953b4 r4:00000000 r3:00000000
[<c001e5a0>] (vty_init+0x0/0x1a4) from [<c001dcd8>] (tty_init+0x134/0x14c)
r8:00000000 r7:00000013 r6:c0627944 r5:00000000 r4:c0894ef8
[<c001dba4>] (tty_init+0x0/0x14c) from [<c001f2f8>] (chr_dev_init+0xd0/0xdc)
r5:c0896500 r4:0000000c
[<c001f228>] (chr_dev_init+0x0/0xdc) from [<c003549c>] (do_one_initcall+0x128/0x 180)
r6:00000001 r5:e783a020 r4:c07efa00
[<c0035374>] (do_one_initcall+0x0/0x180) from [<c0008b44>] (kernel_init+0x10c/0x 190)
[<c0008a38>] (kernel_init+0x0/0x190) from [<c006c9fc>] (do_exit+0x0/0x754)
<0>Code: 1afffff4 e5952014 e28cc001 e3a00001 (e7942002)
<4>---[ end trace 1871642cfdaefb45 ]---
<0>Kernel panic - not syncing: Attempted to kill init!
Backtrace:
[<c003f900>] (dump_backtrace+0x0/0x10c) from [<c058702c>] (dump_stack+0x18/0x1c)
r6:e783c000 r5:c07afc64 r4:c07f2bb0 r3:00000000
[<c0587014>] (dump_stack+0x0/0x1c) from [<c0587124>] (panic+0x78/0x188)
[<c05870ac>] (panic+0x0/0x188) from [<c006d150>] (complete_and_exit+0x0/0x24)
r3:00000000 r2:e783bbd0 r1:e783bbc8 r0:c0703720
r7:00000001
[<c006c9fc>] (do_exit+0x0/0x754) from [<c003fd8c>] (die+0x298/0x300)
r7:00000001
[<c003faf4>] (die+0x0/0x300) from [<c058709c>] (__do_kernel_fault.part.5+0x6c/0x 7c)
[<c0587030>] (__do_kernel_fault.part.5+0x0/0x7c) from [<c004465c>] (do_page_faul t+0x12c/0x3a8)
r7:00000001 r3:e783be40
[<c0044530>] (do_page_fault+0x0/0x3a8) from [<c0044a20>] (do_translation_fault+0 xa4/0xa8)
[<c004497c>] (do_translation_fault+0x0/0xa8) from [<c0035248>] (do_DataAbort+0x4 0/0xa0)
r6:c004497c r5:00000005 r4:c07aaa68 r3:60000093
[<c0035208>] (do_DataAbort+0x0/0xa0) from [<c003b750>] (__dabt_svc+0x70/0xa0)
Exception stack(0xe783be40 to 0xe783be88)
be40: 00000001 c10b0040 00000000 20000013 00000001 e7802200 000000d0 00000003
be60: 00000003 00000001 c07c37d0 e783bea4 0000025b e783be88 c031508c c00f6e7c
be80: 60000093 ffffffff
r8:00000003 r7:00000003 r6:000000d0 r5:00000005 r4:0000040f
[<c00f6e04>] (kmem_cache_alloc+0x0/0xd0) from [<c031508c>] (con_insert_unipair+0 xc0/0x10c)
r6:e786d780 r5:e78bc5e4 r4:00002665 r3:c0887dd8
[<c0314fcc>] (con_insert_unipair+0x0/0x10c) from [<c0315d70>] (con_set_default_u nimap+0xf4/0x18c)
r7:00000003 r6:e786d780 r5:00000001 r4:00000000
[<c0315c7c>] (con_set_default_unimap+0x0/0x18c) from [<c001e220>] (console_map_i nit+0x4c/0x58)
[<c001e1d4>] (console_map_init+0x0/0x58) from [<c001e734>] (vty_init+0x194/0x1a4 )
r6:00000001 r5:c08953b4 r4:00000000 r3:00000000
[<c001e5a0>] (vty_init+0x0/0x1a4) from [<c001dcd8>] (tty_init+0x134/0x14c)
r8:00000000 r7:00000013 r6:c0627944 r5:00000000 r4:c0894ef8
[<c001dba4>] (tty_init+0x0/0x14c) from [<c001f2f8>] (chr_dev_init+0xd0/0xdc)
r5:c0896500 r4:0000000c
[<c001f228>] (chr_dev_init+0x0/0xdc) from [<c003549c>] (do_one_initcall+0x128/0x 180)
r6:00000001 r5:e783a020 r4:c07efa00
[<c0035374>] (do_one_initcall+0x0/0x180) from [<c0008b44>] (kernel_init+0x10c/0x 190)
[<c0008a38>] (kernel_init+0x0/0x190) from [<c006c9fc>] (do_exit+0x0/0x754)
<0>Rebooting in 10 seconds..

Looks like infamous ARM memset() bug. Check this bugreport for solution: https://bugs.linaro.org/show_bug.cgi?id=928#c7
This is a kernel bug, which was fixed in year 2013 by following commits in mainline Linux repository:
455bd4c430b0c0a361f38e8658a0d6cb469942b5 ARM: 7668/1: fix memset-related crashes caused by recent GCC (4.7.2) optimizations
418df63adac56841ef6b0f1fcf435bc64d4ed177 ARM: 7670/1: fix the memset fix
The problem should be fixed by backporting those patches (in most cases patches could be applied without any modifications), or can be worked around with -fno-builtin-memset

I got the same kind of problem with yocto. By disabling the power management in the kernel, I was able to boot correctly. It's a workaround... I did not find the root cause yet.

application pool crashes in IIS 7.5

I have created a asp.net application with framework 4.0.
It has a sort of interview screen with so many questions on it with the paging.
It may have 1000 pages with 10 questions in each page.
when users are using this application screen randomly gets frozen and asks for window credentials.
We also keep getting error for the same application in error log as below:
A process serving application pool 'appicationpoolname' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '4768'. The data field contains the error number.
Not sure if both above error are related or not.
I have created a rule in debugdiag tool for application pool crash and that has generated a dump file. The full call stack for the dump file is as below
Function Arg 1 Arg 2 Arg 3 Arg 4 Source
KERNELBASE!RaiseException+58 04242420 00000000 00000003 010ce504
clr!Debugger::SendRawEvent+9b 010ce530 0000002f 0000015c 000012a0
clr!Debugger::RaiseStartupNotification+48 6fa92811 00000001 01000000 00000000
clr!Debugger::Startup+73 6fa92bf9 00000001 01000000 00000000
clr!DoAdditionalPEChecks+e5 6fa92ac5 00000001 00000000 00000000
clr!EEStartupHelper+5f4 00000000 6fa92a9d 00000001 00000000
clr!EEStartup+52 00000000 6fa92a41 00000001 00000000
clr!EnsureEEStarted+c4 00000000 6fa92f9d 014bb2c0 00000000
clr!ClrCreateManagedInstance+41 708d4bb8 708d4ca8 010ceeb4 00000000
webengine4!LegacyActivationShim::ClrCreateManagedInstance+61 708d4bb8 708d4ca8 010ceeb4 00c6e268
webengine4!GetIsapiProcessHost+48 00c6e268 01c44e8c 6d8e0000 01c44e68
webengine!GetIsapiProcessHost+d0 00c6e268 01c44e8c 00b907d8 00000000
wbhst_pm+2e2a 01c4fed8 01c44e68 00000000 010cf56c
wbhst_pm+335f 00b907d8 01c4fed8 01c4fdf0 010cf5e4
wbhst_pm!GetProtocolManager+35 00b907d8 01c4fed8 00000000 00b90898
w3wphost!AppHostInitialize+235e 7534415b 00000000 00000000 00000000
w3wphost!AppHostInitialize+2698 010cf69c 708e199c 010cf618 00000001
w3wphost!IPM_MESSAGE_PIPE::operator=+1c03 010cf69c 708e199c 00000001 708f1274
iiscore!W3_SERVER::GetProtocolManagerCustomInterface+36 010cf69c 708e199c 00000001 708f1274
webengine4!InitClrHost+186 708f1274 00000000 014c8004 00000400
webengine4!CMgdEngGlobalModule::OnGlobalApplicationResolveModules+31 01c4efac 01c40330 010cf77c 7210a67a
iiscore!VIRTUAL_MODULE::GlobalDoWork+152 00000400 01c4efac 01c4efa8 01c4efa8
iiscore!W3_SERVER::GlobalNotify+98 00000400 01c4efac 00000000 72122f3a
iiscore!W3_APPLICATION::ResolveModules+22 014cc280 00000000 014cc284 00000001
iiscore!W3_APPLICATION::SetupNotificationContext+95 00000000 00000001 014cc284 014cc2e0
iiscore!W3_CONTEXT::SetupStateMachinePhase2+2ab 014cc280 014cc284 00000000 72103358
iiscore!W3_CONTEXT::SetupStateMachine+241 014cc284 014cc280 75341400 010cf894
iiscore!W3_MAIN_CONTEXT::StartNotificationLoop+3f 014cc284 00000000 00c712a8 014cb828
iiscore!W3_MAIN_CONTEXT::OnNewRequest+47 014cb828 014cb828 72a914e6 753413f0
w3dt!UL_NATIVE_REQUEST::DoStateProcess+26 753413f0 010cf8c0 72a9154c 00000444
w3dt!UL_NATIVE_REQUEST::DoWork+60 00000444 00000000 014cb82c 010cf8f8
w3dt!OverlappedCompletionRoutine+1a 00000000 00000444 014cb82c 00c712b0
w3tp!THREAD_POOL_DATA::ThreadPoolThread+89 00000000 00be7fd8 72cf0000 010cf924
w3tp!THREAD_POOL_DATA::ThreadPoolThread+24 00c712a8 00000000 00000000 00be7fd8
w3tp!THREAD_MANAGER::ThreadManagerThread+39 00be7fd8 010cf970 770f9ed2 00be7fd8
kernel32!BaseThreadInitThunk+12 00be7fd8 590a45de 00000000 00000000
ntdll!__RtlUserThreadStart+70 72cf1e5c 00be7fd8 ffffffff 771872ff
ntdll!_RtlUserThreadStart+1b 72cf1e5c 00be7fd8 00000000 00000000
Exception Information
In W3WP__~1.DMP the assembly instruction at KERNELBASE!RaiseException+58 in C:\Windows\SysWOW64\KERNELBASE.dll from Microsoft Corporation has caused an unknown exception (0x04242420) on thread 5
I am not sure how to use this to fix the issue. I tried using Winddbg with .loadby sos mscorwks
command but it says invalid module mscorwks
Any help would be appriciated.
Thanks,
MMC

Is this a first chance exception dump or a second chance exception dump ? When you created the rule in Debug Diagnostic tool, did you change something in the Unconfigured first chance exceptions section ?
Based on the stack it seems this is a first chance exception dump. Make sure that you change the Action time for un-configured first chance exceptions to NONE and then let the tool generate a second chance exception dump and that would be worth to look at.
If it generates one, analyze it using debugdiag and paste the stack and we can see why it is crashing

JRE crash - A fatal error has been detected by the Java Runtime Environment

I am using an open source JAVA based crawler which juggles millions of URI objects between the memory and disk. Basically the program is memory intensive.
The crash occurs once in a blue moon, I haven't been able to find the reason.
Pasting a part of the error-log, any pointers what possibly is causing this?
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002b97ee1a07df, pid=1445, tid=1102055744
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x6227df]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0000000059db5000): GCTaskThread [stack: 0x0000000041a00000,0x0000000041b01000] [id=1460]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000000000000d5
Registers:
RAX=0x0000000000000008, RBX=0x0000000059df95f8, RCX=0x0000000000000005, RDX=0x00002aabee55cec0
RSP=0x0000000041affb60, RBP=0x0000000041affbd0, RSI=0x0000000059df95f0, RDI=0x00002aaab5c758b8
R8 =0x000000000000004b, R9 =0x0000000000000001, R10=0x00002aaaafbdfb01, R11=0x00002aabe4c0df38
R12=0x0000000000000001, R13=0x00002aabee55cec0, R14=0x00002aabe4c0e068, R15=0x0000000059df95f0
RIP=0x00002b97ee1a07df, EFL=0x0000000000010297, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x0000000041affb60)
0x0000000041affb60: 0000000000000400 0000000059d9bfa0
0x0000000041affb70: 0000000000000005 01002aaab6b05998
0x0000000041affb80: 0000000041affbb0 0000000000000009
.
0x0000000041affd40: 0000000059db5810 0000000059db5820
0x0000000041affd50: 0000000059db5bf8 0000000041affd80
Instructions: (pc=0x00002b97ee1a07df)
0x00002b97ee1a07cf: 41 5e 41 5f c9 c3 48 8b 4a 10 4c 89 fe 4c 89 ea
0x00002b97ee1a07df: ff 91 d0 00 00 00 eb dc 49 8b 55 08 eb c9 4c 89
Stack: [0x0000000041a00000,0x0000000041b01000], sp=0x0000000041affb60, free space=3fe0000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6227df]
V [libjvm.so+0x6230b0]
V [libjvm.so+0x6222d4]
V [libjvm.so+0x23d872]
V [libjvm.so+0x62583b]
V [libjvm.so+0x365dda]
V [libjvm.so+0x5da2af]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00002aac4c093800 JavaThread "JEEvictor" daemon [_thread_blocked, id=24783, stack(0x0000000046504000,0x0000000046605000)]
0x000000005a3a2000 JavaThread "JEEvictor" daemon [_thread_blocked, id=24782, stack(0x0000000046605000,0x0000000046706000)]
0x00002aac4c093000 JavaThread "JEEvictor" daemon [_thread_blocked, id=24781, stack(0x0000000046b0a000,0x0000000046c0b000)]
Other Threads:
0x0000000059dfb000 VMThread [stack: 0x0000000041c02000,0x0000000041d03000] [id=1462]
0x0000000059e32800 WatcherThread [stack: 0x00000000423c3000,0x00000000424c4000] [id=1469]
=>0x0000000059db5000 (exited) GCTaskThread [stack: 0x0000000041a00000,0x0000000041b01000] [id=1460]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0000000059d8b860] Threads_lock - owner thread: 0x0000000059dfb000
[0x0000000059d8bd60] Heap_lock - owner thread: 0x00002aac3c2dd000
Heap
PSYoungGen total 835840K, used 825021K [0x00002aabb3600000, 0x00002aabeffc0000, 0x00002aac33600000)
eden space 809024K, 100% used [0x00002aabb3600000,0x00002aabe4c10000,0x00002aabe4c10000)
from space 26816K, 59% used [0x00002aabe4c10000,0x00002aabe5baf538,0x00002aabe6640000)
to space 27136K, 6% used [0x00002aabee540000,0x00002aabee6ec138,0x00002aabeffc0000)
PSOldGen total 4096448K, used 3687089K [0x00002aaab3600000, 0x00002aabad670000, 0x00002aabb3600000)
object space 4096448K, 90% used [0x00002aaab3600000,0x00002aab946ac6b0,0x00002aabad670000)
PSPermGen total 44608K, used 44199K [0x00002aaaae200000, 0x00002aaab0d90000, 0x00002aaab3600000)
object space 44608K, 99% used [0x00002aaaae200000,0x00002aaab0d29ec0,0x00002aaab0d90000)
Dynamic libraries:
40000000-40009000 r-xp 00000000 fd:00 1116699 /usr/java/jdk1.6.0_18/bin/java
40108000-4010a000 rwxp 00008000 fd:00 1116699 /usr/java/jdk1.6.0_18/bin/java
40384000-40387000 ---p 40384000 00:00 0
40387000-40485000 rwxp 40387000 00:00 0
40647000-4064a000 ---p 40647000 00:00 0
4064a000-40748000 rwxp 4064a000 00:00 0
2aac4cb1b000-2aac50000000 ---p 2aac4cb1b000 00:00 0
2b97eda60000-2b97eda61000 rwxp 2b97eda60000 00:00 0
2b97eda71000-2b97eda72000 rwxp 2b97eda71000 00:00 0
2b97eda72000-2b97eda79000 r-xp 00000000 fd:00 1119007 /usr/java/jdk1.6.0_18/jre/lib/amd64/jli/libjli.so
2b97eda79000-2b97edb7a000 ---p 00007000 fd:00 1119007 /usr/java/jdk1.6.0_18/jre/lib/amd64/jli/libjli.so
2b97edb7a000-2b97edb7c000 rwxp 00008000 fd:00 1119007 /usr/java/jdk1.6.0_18/jre/lib/amd64/jli/libjli.so
2b97edb7c000-2b97edb7e000 rwxp 2b97edb7c000 00:00 0
2b97edb7e000-2b97ee332000 r-xp 00000000 fd:00 1119054 /usr/java/jdk1.6.0_18/jre/lib/amd64/server/libjvm.so
2b97ee332000-2b97ee432000 ---p 007b4000 fd:00 1119054 /usr/java/jdk1.6.0_18/jre/lib/amd64/server/libjvm.so
2b97ee432000-2b97ee5bc000 rwxp 007b4000 fd:00 1119054 /usr/java/jdk1.6.0_18/jre/lib/amd64/server/libjvm.so
2b97ee5bc000-2b97ee5f5000 rwxp 2b97ee5bc000 00:00 0
7fffc4df8000-7fffc4e0e000 rwxp 7ffffffe9000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
VM Arguments:
jvm_args: -Dheritrix.home=/heritrix/heritrix-3.1.0 -Djava.protocol.handler.pkgs=org.archive.net -Dheritrix.out=/heritrix/heritrix-3.1.0/heritrix_out.log -Xmx6144M
java_command: org.archive.crawler.Heritrix -p 8544 -b / -a user:pass
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/default
PATH=/apps/pig-0.9.0/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lpp/mmfs/bin:/home/ppra/bin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/bin
LD_LIBRARY_PATH=/usr/java/jdk1.6.0_18/jre/lib/amd64/server:/usr/java/jdk1.6.0_18/jre/lib/amd64:/usr/java/jdk1.6.0_18/jre/../lib/amd64
SHELL=/bin/bash
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.4 (Tikanga)
uname:Linux 2.6.18-164.11.1.el5 #1 SMP Wed Jan 6 13:26:04 EST 2010 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 268287, NOFILE 1024, AS infinity
load average:1.31 0.80 0.67
CPU:total 16 (8 cores per cpu, 1 threads per core) family 6 model 29 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1
Memory: 4k page, physical 32960544k(162620k free), swap 6291448k(6290568k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for linux-amd64 JRE (1.6.0_18-b07), built on Dec 17 2009 13:42:22 by "java_re" with gcc 3.2.2 (SuSE Linux)
time: Sat Aug 4 07:27:56 2012
elapsed time: 242358 seconds

Well, SIGSEGV denotes a segfault. Likely you are either trying to read something that wasn't ever initialized, or the garbage collector is screwing you.

Resources