AWS Greengrass daemon failed to start in Amazon Linux - linux

I am trying to setup AWS Greengrass core on Amazon Linux with respect to the documentation(followed all the prerequisites) of AWS(http://docs.aws.amazon.com/greengrass/latest/developerguide/start-core.html).
I am getting the below error on starting the Greengrass core daemon
./greengrassd start
Setting up greengrass daemon
Validating execution environment
Found cgroup subsystem: cpuset
Found cgroup subsystem: cpu
Found cgroup subsystem: cpuacct
Found cgroup subsystem: blkio
Found cgroup subsystem: memory
Found cgroup subsystem: devices
Found cgroup subsystem: freezer
Found cgroup subsystem: net_cls
Found cgroup subsystem: perf_event
Found cgroup subsystem: net_prio
Found cgroup subsystem: hugetlb
Found cgroup subsystem: pids
Starting greengrass daemon
Greengrass daemon 23933 failed to start
Failed to create overlay fs for container rootfs no such file or directory
I could not find any troubleshooting method for this and I am stuck in the middle of the process. It would be more helpful If any of you solved for me.

Have you tried executing the ./greengrassd start command using sudo ?
When I omit sudo I can an error that says this command needs to be run with using sudo e.g.
sudo ./greengrassd start

Related

Cannot load cgroup config in docker

I was trying to load my cgroup config with cgconfigparser, but it gives the following error:
cgconfigparser; error loading /etc/cgconfig.conf: Cgroup is not mounted
Then I tried grep cgroup /proc/mounts, which gives:
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
I'm in a docker container based on centos7, did some googling and looks like my cgroup is using cgroup2 instead of cgroup1? If this is the case how do I switch back to cgorup1 in docker without tempering with the docker image? If not, what can I do to resolve this error?

docker startup failure due to cgroups misconfiguration

I am trying to start docker on a Centos-6-ish OS. It is failing for cgroups reasons. I believe the mount is correctly structured (docker recommends https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount) so the final error message is unclear to me.
thrashin(bash):/base/data/tmp# ./cgroups-mount
thrashin(bash):/base/data/tmp# grep cgroup /proc/mounts
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
thrashin(bash):/base/data/tmp# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 4 1 1
blkio 5 1 1
thrashin(bash):/base/data/tmp# dockerd &
[1] 7201
thrashin(bash):/base/data/tmp# WARN[0000] could not change group /var/run/docker.sock to docker: group docker not found
INFO[0000] libcontainerd: new containerd process, pid: 7214
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
WARN[0001] unable to modify root key limit, number of containers could be limited by this quota: open /proc/sys/kernel/keys/root_maxkeys: no such file or directory
INFO[0001] [graphdriver] using prior storage driver: overlay2
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup memory limit
WARN[0001] Unable to find cpu cgroup in mounts
WARN[0001] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[0001] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[0001] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[0001] Your kernel does not support cgroup blkio throttle.write_iops_device
WARN[0001] mountpoint for pids not found
Error starting daemon: Devices cgroup isn't mounted
^C
[1]+ Exit 1 dockerd
cgroup device is mounted.
Is the failure due to the warning that the cpu subsystem is not provided? If so, how do I provide this? Is this a kernel build option?
Solved this by rebuilding the kernel to provide the cpu cgroup subsystem.
It's not immediately clear that that should be the solution to solving
Error starting daemon: Devices cgroup isn't mounted
Given that there are other cgroup warnings, aside from the warning about the cpu subsystem.

Error running qemu-system-riscv using root.bin and vmlinux

I am following riscv.org guides for toolchain building. When emulate using qemu running local built rootfilesystem (with busybox) and Linux Kernel, encounter the error below:
Running Qemu using local-built root.bin and kernel image
danny#danny:~/test/riscv/work$ qemu-system-riscv -hda root-local.bin -kernel vmlinux-local -nographic
unassigned address was called?
with addr: 102000735F80006E
not implemented for riscv
Running Qemu using riscv.org stocked root.bin and kernel image
danny#danny:~/test/riscv/work$ qemu-system-riscv -hda root.bin -kernel vmlinux -nographic
[ 0.150000] io scheduler cfq registered (default)
[ 0.160000] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 0.160000] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 0.160000] TCP: cubic registered
[ 0.160000] htifbd: detected disk with ID 1
[ 0.160000] htifbd: adding htifbd0
[ 0.160000] VFS: Mounted root (ext2 filesystem) readonly on device 254:0.
[ 0.160000] devtmpfs: mounted
[ 0.160000] Freeing unused kernel memory: 64K (ffffffff80002000 - ffffffff80012000)
[ 0.200000] EXT2-fs (htifbd0): warning: mounting unchecked fs, running e2fsck is recommended
#uname -a
Linux ucbvax 3.14.15-g4073e84-dirty #4 Sun Jan 11 07:17:06 PST 2015 riscv GNU/Linux
If qemu testing using the downloaded root.bin and vmlinux from riscv.org, seem ok but cant see the busybox starting message and the terminal cant Halt :
Have tested qemu using various combination and result as below:
**root.bin vmlinux RESULT**
local-built local-built Unassigned address was called ....
Downloaded Downloaded Seem OK but without busybox starting bar
local-built Downloaded Kernelpanic-not syncing:No working init found
Downloaded local-built Unassigned address was called ....
We are starting a project to build and fabricate a RISCV silicon chip for Makers around the world and testing the toolchain now in order to port Ubuntu Core & Android to RISCV. Any idea what might probably went wrong ?
Thanks.
QEMU hasn't been fully updated to support the new RISC-V privileged spec (github issue). The update is currently underway.
For an ISA simulator, spike is a good alternative. It may not have all of the platform features of QEMU, but it could serve as a starting point while the QEMU update completes.

HDMI support with Yocto and Freescale sabre

I have a Freescale sabre, for which I have generate core-image-x11 with yocto. Booting and using it with serial port is ok, but I can't use the HDMI display. The HDMI is detected by u-boot, but not after. Here are some informations given by u-boot :
U-Boot 2014.10+fslc+g5fd0b60 (Feb 27 2015 - 11:26:20)
CPU: Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: MX6-SabreSD
I2C: ready
DRAM: 1 GiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
auto-detected panel HDMI
Display: HDMI (1024x768)
In: serial
Out: serial
Err: serial
PMIC: PFUZE100 ID=0x10
Net: FEC [PRIME]
I have tried this in u-boot :
setenv mmcargs 'setenv bootargs console=ttymxc0,115200 root=/dev/mmcblk0p2 rootwait rw video=mxcfb0'
But it doesn't change anything, even if the modification can be seen in /proc/cmdline after the booting ; and
setenv mmcargs 'setenv bootargs console=ttymxc0,115200 root=/dev/mmcblk0p2 rootwait rw video=mxcfb0:dev=hdmi'
makes the booting stopping at starting kernel.
You may need to add a complete video specification:
video=mxcfb0:dev=hdmi,1024x768M#60,bpp=32
The mxcfb driver can be picky about its parameters. If it still doesn't work, you'll probably have to convince the driver to tell you more about what's going wrong to get more useful advice.

Init process not found when booting with ramdisk as rootfs

I'm trying to setup a ramdisk for a setup I have with AT91SAM9261 and 2.6.30 kernel.
I know you would say why don't I use initramfs? I will use also that one, as of now I would like to check if I can boot with a ramdisk as rootfs.
I have already a root fs running fine on this system in jffs2 format.
I used below tutorial:
http://www.ibm.com/developerworks/library/l-initrd/
So I created the ramdisk directly on the embedded target, with 4MB size.
I used "dd" and "mkefs" and the ramdisk image works fine if I mount it as loop when rootfs is my jffs2.
I can browse mount_point and do stuff inside.
The content was created according to the section "Manually building initial RAM disk" in the above link.
In /dev I placed null and console, in /bin I copied busybox and created the links.
I set the execute attribute to init script as well, which works fine when I run it here (as loop device with jffs2 as rootfs)
In my kernel config I enabled ramdisk support as follows:
CONFIG_BLK_DEV_INITRD=y
CONFIG_RD_GZIP=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
Using tftp I download the ramdisk image to 0x21100000.
At 0x20008000 I have the kernel uncompressed and at 0x22000000 the zImage copied.
However, I'm not able to boot successfully the system as it hangs when checking init process.
As I mentioned, I checked several times that my init can be executed,
I also used in the bootargs init=/bin/busybox or other scripts/binaries to check if any can be used as init.
Command line I use now:
bootargs=64M console=ttyS0,115200 root=/dev/ram0 rw
initrd=0x21100000,4194304 init=/init
For some I set ownership to "nobody" just to check if it makes any difference.
As it can be seen in the log output the ramdisk is mounted, but init fails.
It gives a warning to check the ext2 image, which I did before copying it and I have no error reported.
The tftp server is on Windows7. I though I might have some incompatibility if I copy the ramdisk image from linux to Windows then do tftp from Windows server.
So I copied the image again from Windows to linux, mounted it and checked if the FS reports any error and scripts can be executed - everything looks normal.
So I do have all files in place, attributes set, ramdisk is reported as being mounted ok as rootfs, but init cannot be found.
Uncompressing Linux......................................................................................................... done, booting the kernel.
Linux version 2.6.30-olimex (mishu#KubuntuVM) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #1 Mon Dec 29 13:53:54 CET 2014
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
CPU: VIVT data cache, VIVT instruction cache
Machine: Olimex SAM9-L9261
Ignoring unrecognised tag 0x54410008
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 198 MHz, master 99 MHz, main 18.432 MHz
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16256
Kernel command line: 64M console=ttyS0,115200 root=/dev/ram0 rw initrd=0x21100000,4194304 init=/init
NR_IRQS:192
AT91: 96 gpio irqs in 3 banks
PID hash table entries: 256 (order: 8, 1024 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 64MB = 64MB total
Memory: 57384KB available (2924K code, 246K data, 132K init, 0K highmem)
Calibrating delay loop... 99.12 BogoMIPS (lpj=495616)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 716 bytes
NET: Registered protocol family 16
AT91: Power Management
AT91: Starting after user reset
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 2048 (order: 2, 16384 bytes)
TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (junk in compressed archive); looks like an initrd
Freeing initrd memory: 4096K
NetWinder Floating Point Emulator V0.97 (double precision)
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 120
io scheduler noop registered
io scheduler anticipatory registered (default)
atmel_lcdfb atmel_lcdfb.0: backlight control is not available
atmel_lcdfb atmel_lcdfb.0: 225KiB frame buffer at 23140000 (mapped at ffc00000)
Console: switching to colour frame buffer device 40x30
atmel_lcdfb atmel_lcdfb.0: fb0: Atmel LCDC at 0x00600000 (mapped at c4866000), irq 21
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffb0000 (irq = 6) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
ssc ssc.1: Atmel SSC device at 0xc4878000 (irq 15)
Driver 'sd' needs updating - please use bus_type methods
dm9000 Ethernet Driver, V1.31
eth0 (dm9000): not using net_device_ops yet
eth0: dm9000e at c486a000,c486e044 IRQ 107 MAC: 3a:1f:34:08:54:64 (chip)
NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
Scanning device for bad blocks
Bad eraseblock 1478 at 0x00000b8c0000
Creating 2 MTD partitions on "atmel_nand":
0x000000000000-0x000000040000 : "Partition 1"
0x000000040000-0x000020000000 : "Partition 2"
atmel_spi atmel_spi.0: Atmel SPI Controller at 0xfffc8000 (irq 12)
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 20, io mem 0x00500000
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: AT91 OHCI
usb usb1: Manufacturer: Linux 2.6.30-olimex ohci_hcd
usb usb1: SerialNumber: at91
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
udc: at91_udc version 3 May 2006
g_serial gadget: Gadget Serial v2.4
g_serial gadget: g_serial ready
mice: PS/2 mouse device common for all mice
input: gpio-keys as /class/input/input0
ads7846 spi0.2: touchscreen, irq 29
input: ADS784x Touchscreen as /class/input/input1
rtc-at91sam9 at91_rtt.0: rtc core: registered at91_rtt as rtc0
IRQ 1/rtc0: IRQF_DISABLED is not guaranteed on shared IRQs
rtc-at91sam9 at91_rtt.0: rtc0: SET TIME!
i2c /dev entries driver
i2c-gpio: probe of i2c-gpio failed with error -16
Registered led device: led1
Registered led device: led2
Registered led device: led3
Advanced Linux Sound Architecture Driver Version 1.0.20.
at73c213 spi0.3: at73c213: supported bitrate is 48500 (64 divider)
usb 1-1: new full speed USB device using at91_ohci and address 2
usb 1-1: New USB device found, idVendor=0457, idProduct=0151
usb 1-1: New USB device strings: Mfr=0, Product=2, SerialNumber=3
usb 1-1: Product: USB Mass Storage Device
usb 1-1: SerialNumber: 8d54bd8186b42c
usb 1-1: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
ALSA device list:
#0: SAM9-L9261 external DAC on irq 15
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
rtc-at91sam9 at91_rtt.0: hctosys: unable to read the hardware clock
RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 4096KiB [1 disk] into ram disk... done.
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
VFS: Mounted root (ext2 filesystem) on device 1:0.
Freeing init memory: 132K
Failed to execute /init. Attempting defaults...
Kernel panic - not syncing: No init found. Try passing init= option to kernel.
[<c002f414>] (unwind_backtrace+0x0/0xdc) from [<c025bd34>] (panic+0x40/0x110)
[<c025bd34>] (panic+0x40/0x110) from [<c0029500>] (init_post+0xd0/0xf8)
[<c0029500>] (init_post+0xd0/0xf8) from [<c00085a0>] (kernel_init+0xbc/0xe4)
[<c00085a0>] (kernel_init+0xbc/0xe4) from [<c0041998>] (do_exit+0x0/0x5ac)
[<c0041998>] (do_exit+0x0/0x5ac) from [<00000001>] (0x1)
I plan to rebuild busybox and try again, but as my current ramdisk image works when I mount it as loop I don't have too much confidence it will change something.
I'm out of ideas. I've been trying the whole day to get this working.
Help is much appreciated.
Thanks.
Missing libraries were the root cause for the non-executable init placed on the ramdisk.
Using chroot turned out to be extremely useful. With this issue I learned how to use chroot and simulate a jail.
Another trick was to use user_debug=31 in the command line so that the kernel will output a message saying that the file is not found.
It looks like the "file not found" does not really point to a missing file, but to a failure to read the file system due to busybox not executing properly (missing libraries).
In parallel I have compiled dynamically the busybox 1.20.02 for my arm architecture and the ramdisk worked properly (without libs).
This user_debug=31 turned out to be again useful when I compiled and ran busybox, as it was also failing after mounting the ramdisk, but it thrown out an "undefined exception" which led me to a wrong "arch" value used for compiling busybox. After changing it everything was fine.

Resources