I have a relatively simple question but probably not for me.
I created a bbclass named spi-nand-ubi.bbclass. In this file, I have a few records like
dd if=${DEPLOY_DIR_IMAGE}/${KERNEL_DEVICETREE} of=${SPINAND} bs=1k seek=512 conv=notrunc
dd if=${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE} of=${SPINAND} bs=1k seek=640 conv=notrunc
and above records are fine, but the following record don't
dd if=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubi of=${SPINAND} bs=1k seek=5504 conv=notrunc
I get always an error like
dd: failed to open '/home/mw/yocto/tmp/work/indus-poky-linux-gnueabi/console-image/1.0-r0/deploy-console-image-image-complete/console-image-indus-20230120084720.rootfs.ubi': No such file or directory
When I type manually to the
cd /home/mw/yocto/tmp/work/indus-poky-linux-gnueabi/console-image/1.0-r0/deploy-console-image-image-complete
the file with provided name exists.
What I missed, misunderstood. Please give me a hint on how to resolve this issue.
Thanks
directory content of
/home/mw/yocto/tmp/work/indus-poky-linux-gnueabi/console-image/1.0-r0/deploy-console-image-image-complete
console-image-indus-20230120082905.rootfs.ext4
console-image-indus-20230120082905.rootfs.jffs2
console-image-indus-20230120082905.rootfs.manifest
console-image-indus-20230120082905.rootfs.sunxi-spinand
console-image-indus-20230120082905.rootfs.tar
console-image-indus-20230120082905.rootfs.tar.gz
console-image-indus-20230120082905.rootfs.tar.xz
console-image-indus-20230120082905.rootfs.ubi
console-image-indus-20230120082905.rootfs.ubifs
console-image-indus-20230120082905.testdata.json
console-image-indus-20230120084720.rootfs.sunxi-spinand
console-image-indus-20230120084720.rootfs.tar
console-image-indus-20230120084720.rootfs.tar.gz
console-image-indus-20230120084720.rootfs.tar.xz
console-image-indus.ext4
console-image-indus.jffs2
console-image-indus.manifest
console-image-indus.testdata.json
console-image-indus.ubi
console-image-indus.ubifs
ubinize-console-image-indus-20230120082905.cfg
the content of my spi-nand-ubi.bbclass
inherit image_types
#
# Create an image that can by written into a SPI NAND (128 MBytes) flash technology using dd application.
# Written for Indus board to simplify programming process and to write only one combined image file.
#
# The image layout(layout is valid for 128MBytes Flashes) used is:
#
# OFFSET PARTITION SIZE PARTITION
# 0 -> 458752(0x80000) - SPL U-Boot with NAND offset
# 512*1024 -> 65536(0x20000) - The dtb file
# 640*1024 -> 4980736(0x4C0000) - Kernel
# 5504*1024 -> 122494976(0x7AA0000) - Ubifs rootfs (*.ubi for NAND)
#
# SUM of all partition should give all flash memory size,
# SUM = 0x70000+0x10000+0x4C0000+0x7AA0000= 0x7A12000(128000000)
#
# Before change partition offsets here, do it first for U-Boot DTS, defconifg and Kernel DTS
# This image depends on the rootfs image
RDEPENDS_mtd-utils-tests += "bash"
SPINAND_ROOTFS_TYPE ?= "ubifs"
IMAGE_TYPEDEP_sunxi-spinand= "${SPINAND_ROOTFS_TYPE}"
do_image_sunxi_spinand[depends] += " \
mtools-native:do_populate_sysroot \
dosfstools-native:do_populate_sysroot \
virtual/kernel:do_deploy \
virtual/bootloader:do_deploy \
"
# The NAND SPI Flash image name
SPINAND = "${IMGDEPLOYDIR}/${IMAGE_NAME}.rootfs.sunxi-spinand "
IMAGE_CMD:sunxi-spinand () {
${DEPLOY_DIR_IMAGE}/mknanduboot.sh ${DEPLOY_DIR_IMAGE}/${SPL_BINARY} ${DEPLOY_DIR_IMAGE}/${NAND_SPL_BINARY}
dd if=/dev/zero of=${SPINAND} bs=1M count=16
dd if=${DEPLOY_DIR_IMAGE}/${NAND_SPL_BINARY} of=${SPINAND} bs=1k conv=notrunc
dd if=${DEPLOY_DIR_IMAGE}/${KERNEL_DEVICETREE} of=${SPINAND} bs=1k seek=512 conv=notrunc
dd if=${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE} of=${SPINAND} bs=1k seek=640 conv=notrunc
mkfs.ubifs -F -r ${WORKDIR}/rootfs -m 2048 -e 126976 -c 2048 -o ${DEPLOY_DIR_IMAGE}/rootfs.ubifs
#ubinize -vv -o ${DEPLOY_DIR_IMAGE}/rootfs.ubi -m 2048 -p 131072 -s 2048 ${DEPLOY_DIR_IMAGE}/config.ini
#ubinize -vv -o ${DEPLOY_DIR_IMAGE}/rootfs.ubi -m 2048 -p 131072 -s 2048 ${DEPLOY_DIR_IMAGE}/ubinize${vname}-${IMAGE_NAME}.cfg
dd if=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubi of=${SPINAND} bs=1k seek=5504 conv=notrunc
#dd if=${DEPLOY_DIR_IMAGE}/rootfs.ubi of=${SPINAND} bs=1k seek=5504 conv=notrunc
}
I did a few experiments and I think I found a reason. The problem is that ubi and ubifs are generated later than the receipt which uses my bbclass. This is the reason why it doesn't see demanded files. The question is what should I add to wait on ubi and ubifs to be finalized?
Related
Inside the docker container I test the
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2> >( grep copied )
line execution. I drill into container using 2 ways.
1) the classical one is docker exec -it 2b65c84ddce2 /bin/sh
the execution the line inside the contained inherinted from the alpine I'm greeting
/bin/sh: syntax error: unexpected redirection beacuse of something near >(
2) when I enter the container into the bash executor like docker exec -it 2b65c84ddce2 /bin/bash
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2> >( grep copied ) returns no output
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 returns 2 lines output only, while the expectation is 3:
1+0 records in
1+0 records out
At the host level the same dd command is returning 3 lines like this:
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 0.0109968 s, 46.6 MB/s
and with redirection the output is the last line:
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2> >( grep copied )
512000 bytes (512 kB, 500 KiB) copied, 0.0076261 s, 67.1 MB/s
So how can I get the last line of dd output from the inside of the docker container?
PS.
The redirecting stderr to stdout doesn't help in general:
/ # dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2>&1 | grep copied
/ # dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2>&1
1000+0 records in
1000+0 records out
while at the host system it works
$ dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 2>&1 | grep copied
512000 bytes (512 kB, 500 KiB) copied, 0.00896706 s, 57.1 MB/s
host:
dd --v
dd (coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Rubin, David MacKenzie, and Stuart Kemp.
container:
/ # dd --v
BusyBox v1.31.1 () multi-call binary.
Usage: dd [if=FILE] [of=FILE] [ibs=N obs=N/bs=N] [count=N] [skip=N] [seek=N]
[conv=notrunc|noerror|sync|fsync]
[iflag=skip_bytes|fullblock] [oflag=seek_bytes|append]
Copy a file with converting and formatting
if=FILE Read from FILE instead of stdin
of=FILE Write to FILE instead of stdout
bs=N Read and write N bytes at a time
ibs=N Read N bytes at a time
obs=N Write N bytes at a time
count=N Copy only N input blocks
skip=N Skip N input blocks
seek=N Skip N output blocks
conv=notrunc Don't truncate output file
conv=noerror Continue after read errors
conv=sync Pad blocks with zeros
conv=fsync Physically write data out before finishing
conv=swab Swap every pair of bytes
iflag=skip_bytes skip=N is in bytes
iflag=fullblock Read full blocks
oflag=seek_bytes seek=N is in bytes
oflag=append Open output file in append mode
status=noxfer Suppress rate output
status=none Suppress all output
N may be suffixed by c (1), w (2), b (512), kB (1000), k (1024), MB, M, GB, G
they are in fact, different
For anyone searching this question:
the DD used was being used with BusyBox. The third line is an optional output which is defined when compiling BusyBox from Source. The pre compiled versions have this disabled
ENABLE_FEATURE_DD_THIRD_STATUS_LINE must be defined.
see https://git.busybox.net/busybox/tree/coreutils/dd.c line 166.
#if ENABLE_FEATURE_DD_THIRD_STATUS_LINE
# if ENABLE_FEATURE_DD_STATUS
if (G.flags & FLAG_STATUS_NOXFER) /* status=noxfer active? */
return;
//TODO: should status=none make dd stop reacting to USR1 entirely?
//So far we react to it (we print the stats),
//status=none only suppresses final, non-USR1 generated status message.
# endif
fprintf(stderr, "%llu bytes (%sB) copied, ",
G.total_bytes,
/* show fractional digit, use suffixes */
make_human_readable_str(G.total_bytes, 1, 0)
);
/* Corner cases:
* ./busybox dd </dev/null >/dev/null
* ./busybox dd bs=1M count=2000 </dev/zero >/dev/null
* (echo DONE) | ./busybox dd >/dev/null
* (sleep 1; echo DONE) | ./busybox dd >/dev/null
*/
seconds = (now_us - G.begin_time_us) / 1000000.0;
bytes_sec = G.total_bytes / seconds;
fprintf(stderr, "%f seconds, %sB/s\n",
seconds,
/* show fractional digit, use suffixes */
make_human_readable_str(bytes_sec, 1, 0)
);
#endif
}
I am trying to destroy the luks header on one of my logical volume data1, I am still able to read the file inside data1 after I delete the luks header. I suppose it should not be the case right? Can someone help me in understanding this case?
lsblk output
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 894.2G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 893.8G 0 part
├─vg0-root 251:0 0 758.7G 0 lvm
│ └─luks-45f803e5-3c17-4aaf-a9ad-d66c8b5458de 251:2 0 758.7G 0 crypt /
├─vg0-swap 251:1 0 75G 0 lvm [SWAP]
├─vg0-data3 251:3 0 20G 0 lvm
│ └─luks-6e168d35-26dc-429c-a3d6-8cb4f1c1d39e 251:7 0 20G 0 crypt /data3
├─vg0-data2 251:4 0 20G 0 lvm
│ └─luks-75727dd1-a332-423d-8c37-4cedf9cbe83c 251:8 0 20G 0 crypt /data2
└─vg0-data1 251:5 0 20G 0 lvm
└─luks-cf2d9729-2d1b-48b8-8502-dea937ef602f 251:6 0 20G 0 crypt /data1
Luksdump output to check if the luks header is exists:
-130-sapam#test-host:~ $ sudo cryptsetup luksDump /dev/mapper/vg0-data1
LUKS header information for /dev/mapper/vg0-data1
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 256
MK digest: 9f e7 1a b3 0e fb 4e bc 6d 1b 9e 46 f8 bd 15 22 ea 04 6e c3
MK salt: 83 5e 90 5b b3 a1 c5 a5 d4 22 a0 3e 23 25 51 50
fc cd a8 ac db 9f d0 a8 8b 81 6e 9a 92 1f d8 d3
MK iterations: 43750
UUID: cf2d9729-2d1b-48b8-8502-dea937ef602f
Key Slot 0: ENABLED
Iterations: 439102
Salt: f1 6d 23 b0 b7 ee fc 09 8c 6b 92 ef b2 17 ef d9
0c 83 64 29 bf bc 98 3f f6 93 4b 45 06 49 a9 21
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
Destroying the luks header:
-130-sapam#test-host:~ $ sudo dd bs=512 count=4096 if=/dev/zero of=/dev/mapper/vg0-data1
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 0.00444235 s, 472 MB/s
-0-sapam#test-host:~ $ sudo cryptsetup luksDump /dev/mapper/vg0-data1
-1-sapam#test-host:~ $
I still able to read the file inside /data1/
-1-sapam#test-host:~ $ cat /data1/foo
james
-0-sapam#test-host:~ $
From my understanding is once the header is destroyed, the /data1 should not be able to read right?
It seems you are destroying already mounted partition.
Encryption/decryption keys are hold in the memory while the partition is mounted. You should unmout your LUKS partition first:
# umount /data1
and then erase the LUKS header. You won't be able to mount it again.
Please note cryptsetup utility has a command to erase LUKS header:
# cryptsetup luksErase /dev/mapper/vg0-data1
The advantage of this operation is that you can restore LUKS header from the backup if you done it before.
from cryptsetup(8):
erase <device>
luksErase <device>
Erase all keyslots and make the LUKS container permanently inac‐
cessible. You do not need to provide any password for this op‐
eration.
WARNING: This operation is irreversible.
While luksErase is nice to wipe the keyslots area, note that it doesn't actually destroy the entire LUKS Header. It leaves the metadata intact.
I submitted a feature request asking for the luksErase command to have the ability to wipe the plaintext metadata in the header as well, but the developer rejected & closed it :(
LUKS Header Shredder
You can use the following BASH script to find every LUKS device on the system, wipe the headers, and shutdown the machine.
Disclaimer
The script below contains experimental software that may or may not lead to corruption or total permanent deletion of some or all of your data. I cannot be responsible for any data loss that has occurred as a result of following this guide.
The contents of this answer is provided openly and is licensed under the CC-BY-SA license. The software included in this guide is licensed under the GNU GPLv3 license. All content here is consistent with the limitations of liabilities outlined in its respective licenses.
I highly recommend that any experiments with the script included in this answer is used exclusively on a disposable machine containing no valuable data.
If data loss is a concern for you, then leave now and do not proceed. You have been warned.
#!/bin/bash
#set -x
################################################################################
# File: buskill-selfdestruct.sh
# Purpose: Self-destruct trigger script for BusKill Kill Cord
# For more info, see: https://buskill.in/
# WARNING: THIS IS EXPERIMENTAL SOFTWARE THAT IS DESIGNED TO CAUSE PERMANENT,
# COMPLETE AND IRREVERSIBLE DATA LOSS!
# Note : This script will *not* execute unless it's passed the '--yes'
# argument. Be sure to test this trigger before depending on it!
# Authors: Michael Altfield <michael#buskill.in>
# Created: 2020-03-11
# Updated: 2020-03-11
# Version: 0.1
################################################################################
############
# SETTINGS #
############
BUSKILL_LOCK='/usr/local/bin/buskill-lock.sh'
[ -f ${BUSKILL_LOCK} ] || echo "ERROR: Unable to find buskill-lock.sh"
CRYPTSETUP=`which cryptsetup` || echo "ERROR: Unable to find cryptsetup"
LS=`which ls` || echo "ERROR: Unable to find ls"
CAT=`which cat` || echo "ERROR: Unable to find cat"
GREP=`which grep` || echo "ERROR: Unable to find grep"
ECHO=`which echo` || echo "ERROR: Unable to find echo"
AWK=`which awk` || echo "ERROR: Unable to find awk"
HEAD=`which head` || echo "ERROR: Unable to find head"
LSBLK=`which lsblk` || echo "ERROR: Unable to find lsblk"
OD=`which od` || echo "ERROR: Unable to find od"
##############
# ROOT CHECK #
##############
# re-run as root
if [[ $EUID -ne 0 ]]; then
exec sudo /bin/bash "$0" "$#"
fi
###########
# CONFIRM #
###########
# for safety, exit if this script is executed without a '--yes' argument
${ECHO} "${#}" | ${GREP} '\--yes' &> /dev/null
if [ $? -ne 0 ]; then
${ECHO} "WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING"
${ECHO} "================================================================================"
${ECHO} "WARNING: THIS IS EXPERIMENTAL SOFTWARE THAT IS DESIGNED TO CAUSE PERMANENT, COMPLETE AND IRREVERSIBLE DATA LOSS!"
${ECHO} "================================================================================"
${ECHO} "WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING"
${ECHO}
${ECHO} "cowardly refusing to execute without the '--yes' argument for your protection. If really you want to proceed with damaging your system, retry with the '--yes' argument"
exit 1
fi
###########################
# (DELAYED) HARD SHUTDOWN #
###########################
# The most secure encrypted computer is an encrypted computer that is *off*
# This is our highest priority; initiate a hard-shutdown to occur in 5 minutes regardless
# of what happens later in this script
nohup sleep 60 && echo o > /proc/sysrq-trigger &
nohup sleep 61 && shutdown -h now &
nohup sleep 62 && poweroff --force --no-sync &
###############
# LOCK SCREEN #
###############
# first action: lock the screen!
${BUSKILL_LOCK} &
#####################
# WIPE LUKS VOLUMES #
#####################
# overwrite luks headers
${ECHO} "INFO: shredding LUKS header (plaintext metadata and keyslots with encrypted master decryption key)"
writes=''
IFS=$'\n'
for line in $( ${LSBLK} --list --output 'PATH,FSTYPE' | ${GREP} 'crypt' ); do
device="`${ECHO} \"${line}\" | ${AWK} '{print \$1}'`"
${ECHO} -e "\t${device}"
###########################
# OVERWRITE LUKS KEYSLOTS #
###########################
# erases all keyslots, making the LUKS container "permanently inaccessible"
${CRYPTSETUP} luksErase --batch-mode "${device}" || ${HEAD} --bytes 20M /dev/urandom > ${device} &
# store the pid of the above write tasks so we can try to wait for it to
# flush to disk later -- before triggering a brutal hard-shutdown
writes="${writes} $!"
#####################################
# OVERWRITE LUKS PLAINTEXT METADATA #
#####################################
luksVersion=`${OD} --skip-bytes 6 --read-bytes 2 --format d2 --endian=big --address-radix "n" "${device}"`
# get the end byte to overwrite. For more info, see:
# https://security.stackexchange.com/questions/227359/how-to-determine-start-and-end-bytes-of-luks-header
if [[ $luksVersion -eq 1 ]]; then
# LUKS1: https://gitlab.com/cryptsetup/cryptsetup/-/wikis/LUKS-standard/on-disk-format.pdf
# in LUKS1, the whole header ends at 512 * the `payload-offset`
# this is actually more than we need (includes keyslots), but
# it's the fastest/easiest to bound to fetch in LUKS1
payloadOffset=`${OD} --skip-bytes 104 --read-bytes 4 --format d4 --endian=big --address-radix "n" "${device}"`
luksEndByte=$(( 512 * ${payloadOffset} ))
elif [[ $luksVersion -eq 2 ]]; then
# LUKS2: https://gitlab.com/cryptsetup/LUKS2-docs/blob/master/luks2_doc_wip.pdf
# in LUKS2, the end of the plaintext metadata area is twice the
# size of the `hdr_size` field
hdr_size=`${OD} --skip-bytes 8 --read-bytes 8 --format d8 --endian=big --address-radix "n" "${device}"`
luksEndByte=$(( 2 * ${hdr_size} ))
else
# version unclear; just overwrite 20 MiB
luksEndByte=20971520
fi
# finally, shred that plaintext metadata; we do this in a new file descriptor
# to prevent bash from truncating if ${device} is a file
exec 5<> "${device}"
${HEAD} --bytes "${luksEndByte}" /dev/urandom >&5 &
writes="${writes} $!"
exec 5>&-
done
#######################
# WAIT ON DISK WRITES #
#######################
# wait until all the write tasks above have completed
# note: do *not* put quotes around this arg or the whitespace will break wait
wait ${writes}
# clear write buffer to ensure headers overwrites are actually synced to disks
sync; echo 3 > /proc/sys/vm/drop_caches
#################################
# WIPE DECRYPTION KEYS FROM RAM #
#################################
# suspend each currently-decrypted LUKS volume
${ECHO} "INFO: removing decryption keys from memory"
for device in $( ${LS} -1 "/dev/mapper" ); do
${ECHO} -e "\t${device}";
${CRYPTSETUP} luksSuspend "${device}" &
# clear page caches in memory (again)
sync; echo 3 > /proc/sys/vm/drop_caches
done
#############################
# (IMMEDIATE) HARD SHUTDOWN #
#############################
# do whatever works; this is important.
echo o > /proc/sysrq-trigger &
sleep 1
shutdown -h now &
sleep 1
poweroff --force --no-sync &
# exit cleanly (lol)
exit 0
Sources
LUKS Header Shredder (BusKill Self-Destruct Trigger)
https://github.com/BusKill/buskill-linux/blob/master/triggers/buskill-selfdestruct.sh
I am trying to build my own Linux derivative to run on an TI-AR7 board. I took the board from an old Telekom Speedport W 501V router. To understand how firmware is flashed onto the device I have downloaded the most recent official firmware. Using the Linux file command I determined the image is a tar archive, which can be extracted easily.
ubuntu#ip-172-31-23-210:~/reverse$ ls
fw_speedport_w501v_v_28.04.38.image
ubuntu#ip-172-31-23-210:~/reverse$ file fw*
fw_speedport_w501v_v_28.04.38.image: POSIX tar archive (GNU)
ubuntu#ip-172-31-23-210:~/reverse$ tar -xvf fw*
./var/
./var/tmp/
./var/tmp/kernel.image
./var/tmp/filesystem.image
./var/flash_update.ko
./var/flash_update.o
./var/info.txt
./var/install
./var/chksum
./var/regelex
./var/signature
ubuntu#ip-172-31-23-210:~/reverse$
According to a wiki (Firmware-Image) that I have found, ./var/tmp/kernel.image contains the actual firmware. During the update process this image is written to the mtd1 device. As stated in the wiki (LZMA-Kernel) the lzma compressed kernel starts with the magic number 0xfeed1281. A hexdump of kernel.image contains that number at its beginning.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ hexdump -n 4 kernel.image
0000000 1281 feed
0000004
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$
The following script given on the last wiki entry should decompress the kernel.
#! /usr/bin/perl
use Compress::unLZMA;
use Archive::Zip;
open INPUT, "<$ARGV[0]" or die "can't open $ARGV[0]: $!";
read INPUT, $buf, 4;
$magic = unpack("V", $buf);
if ($magic != 0xfeed1281) {
die "bad magic";
}
read INPUT, $buf, 4;
$len = unpack("V", $buf);
read INPUT, $buf, 4*2; # address, unknown
read INPUT, $buf, 4;
$clen = unpack("V", $buf);
read INPUT, $buf, 4;
$dlen = unpack("V", $buf);
read INPUT, $buf, 4;
$cksum = unpack("V", $buf);
printf "Archive checksum: 0x%08x\n", $cksum;
read INPUT, $buf, 1+4; # properties, dictionary size
read INPUT, $dummy, 3; # alignment
$buf .= pack('VV', $dlen, 0); # 8 bytes of real size
#$buf .= pack('VV', -1, -1); # 8 bytes of real size
read INPUT, $buf2, $clen;
$crc = Archive::Zip::computeCRC32($buf2);
printf "Input CRC32: 0x%08x\n", $crc;
if ($cksum != $crc) {
die "wrong checksum";
}
$buf .= $buf2;
$data = Compress::unLZMA::uncompress($buf);
unless (defined $data) {
die "uncompress: $#";
}
open OUTPUT, ">$ARGV[1]" or die "can't write $ARGV[1]";
print OUTPUT $data;
#truncate OUTPUT, $dlen;
To use the script you may need to install Compress::unLZMA and Archive::Zip perl modules.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ tar -xvf Compress*
Compress-unLZMA-0.04/
Compress-unLZMA-0.04/Makefile.PL
Compress-unLZMA-0.04/ppport.h
Compress-unLZMA-0.04/Changes
Compress-unLZMA-0.04/lzma_sdk/
[...]
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ cd Compress*
ubuntu#ip-172-31-23-210:~/reverse/var/tmp/Compress-unLZMA-0.04$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Compress::unLZMA
Writing MYMETA.yml and MYMETA.json
ubuntu#ip-172-31-23-210:~/reverse/var/tmp/Compress-unLZMA-0.04$ make
cp lib/Compress/unLZMA.pm blib/lib/Compress/unLZMA.pm
/usr/bin/perl /usr/share/perl/5.18/ExtUtils/xsubpp -typemap /usr/share/perl/5.18/ExtUtils/typemap unLZMA.xs > unLZMA.xsc && mv unLZMA.xsc unLZMA.c
cc -c -I. -Ilzma_sdk/Source -D_REENTRANT -D_GNU_SOURCE
[...]
ubuntu#ip-172-31-23-210:~/reverse/var/tmp/Compress-unLZMA-0.04$ sudo make install
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/local/lib/perl/5.18.2/auto/Compress/unLZMA/unLZMA.bs
Installing /usr/local/lib/perl/5.18.2/auto/Compress/unLZMA/unLZMA.so
Installing /usr/local/lib/perl/5.18.2/Compress/unLZMA.pm
Installing /usr/local/man/man3/Compress::unLZMA.3pm
Appending installation info to /usr/local/lib/perl/5.18.2/perllocal.pod
ubuntu#ip-172-31-23-210:~/reverse/var/tmp/Compress-unLZMA-0.04$ # same for Archive::Zip module
After installing these dependencies the script decompressed the kernel successfully.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ ./decompress.pl kernel.image kernel.decompressed
Archive checksum: 0x29176e12
Input CRC32: 0x29176e12
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$
But what kind of file is kernel.decompressed and how do I generate a similar file from my Linux kernel source? I continued analyzing it using file and binwalk.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ file kernel.decompressed
kernel.decompressed: data
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ binwalk kernel.decompressed
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
1509632 0x170900 Linux kernel version "2.6.13.1-ohio (686) (gcc version 3.4.6) #9 Wed Apr 4 13:48:08 CEST 2007"
1516240 0x1722D0 CRC32 polynomial table, little endian
1517535 0x1727DF Copyright string: "Copyright 1995-1998 Mark Adler "
1549488 0x17A4B0 Unix path: /usr/gnemul/irix/
1550920 0x17AA48 Unix path: /usr/lib/libc.so.1
1618031 0x18B06F Neighborly text, "neighbor %.2x%.2x.%.2x:%.2x:%.2x:%.2x:%.2x:%.2x lost on port %d(%s)(%s)"
1966080 0x1E0000 gzip compressed data, maximum compression, from Unix, last modified: 2007-04-04 11:45:13
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$
So the Linux kernel starts at 1509632 and ends at 1516240. What kind of data is stored in front the Linux kernel (0 to 1509632)? I extracted the kernel and that piece of unknown data using dd.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ dd if=kernel.decompressed of=unknown.data bs=1 count=1509632
1509632+0 records in
1509632+0 records out
1509632 bytes (1.5 MB) copied, 1.62137 s, 931 kB/s
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ dd if=kernel.decompressed of=kernel bs=1 skip=1509632 count=6608
6608+0 records in
6608+0 records out
6608 bytes (6.6 kB) copied, 0.0072771 s, 908 kB/s
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$
I need to ask again: What kind of file is kernel and how do I generate a similar file from my Linux kernel source? I used xxd and strings to look at the file more closely.
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ xxd -l 100 kernel
0000000: 4c69 6e75 7820 7665 7273 696f 6e20 322e Linux version 2.
0000010: 362e 3133 2e31 2d6f 6869 6f20 2836 3836 6.13.1-ohio (686
0000020: 2920 2867 6363 2076 6572 7369 6f6e 2033 ) (gcc version 3
0000030: 2e34 2e36 2920 2339 2057 6564 2041 7072 .4.6) #9 Wed Apr
0000040: 2034 2031 333a 3438 3a30 3820 4345 5354 4 13:48:08 CEST
0000050: 2032 3030 370a 0000 0000 0000 0000 0000 2007...........
0000060: 0000 0000 ....
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$ strings kernel
Linux version 2.6.13.1-ohio (686) (gcc version 3.4.6) #9 Wed Apr 4 13:48:08 CEST 2007
do_be
do_bp
do_tr
do_ri
do_cpu
nmi_exception_handler
do_ade
emulate_load_store_insn
do_page_fault
context_switch
__put_task_struct
do_exit
local_bh_enable
run_workqueue
2.6.13.1-ohio gcc-3.4
enable_irq
__free_pages_ok
free_hot_cold_page
prep_new_page
kmem_cache_destroy
kmem_cache_create
pageout
vunmap_pte_range
vmap_pte_range
__vunmap
__brelse
sync_dirty_buffer
bio_endio
queue_kicked_iocb
proc_get_inode
remove_proc_entry
sysfs_get
sysfs_fill_super
kref_get
kref_put
0123456789abcdefghijklmnopqrstuvwxyz
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
vsnprintf
{zt^f
pw0Gm
0cIZ-
68BG+
QC]S%
v,;Zk
ubuntu#ip-172-31-23-210:~/reverse/var/tmp$
This Github repository contains the extracted files to use for further analysis.
If you look up how to clone an entire disk to another one on the web, you will find something like that:
dd if=/dev/sda of=/dev/sdb conv=notrunc,noerror
While I understand the noerror, I am getting a hard time understanding why people think that notrunc is required for "data integrity" (as ArchLinux's Wiki states, for instance).
Indeed, I do agree on that if you are copying a partition to another partition on another disk, and you do not want to overwrite the entire disk, just one partition. In thise case notrunc, according to dd's manual page, is what you want.
But if you're cloning an entire disk, what does notrunc change for you? Just time optimization?
TL;DR version:
notrunc is only important to prevent truncation when writing into a file. This has no effect on a block device such as sda or sdb.
Educational version
I looked into the coreutils source code which contains dd.c to see how notrunc is processed.
Here's the segment of code that I'm looking at:
int opts = (output_flags
| (conversions_mask & C_NOCREAT ? 0 : O_CREAT)
| (conversions_mask & C_EXCL ? O_EXCL : 0)
| (seek_records || (conversions_mask & C_NOTRUNC) ? 0 : O_TRUNC));
/* Open the output file with *read* access only if we might
need to read to satisfy a `seek=' request. If we can't read
the file, go ahead with write-only access; it might work. */
if ((! seek_records
|| fd_reopen (STDOUT_FILENO, output_file, O_RDWR | opts, perms) < 0)
&& (fd_reopen (STDOUT_FILENO, output_file, O_WRONLY | opts, perms) < 0))
error (EXIT_FAILURE, errno, _("opening %s"), quote (output_file));
We can see here that if notrunc is not specified, then the output file will be opened with O_TRUNC. Looking below at how O_TRUNC is treated, we can see that a normal file will get truncated if written into.
O_TRUNC
If the file already exists and is a regular file and the open
mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated
to length 0. If the file is a FIFO or terminal device file, the
O_TRUNC flag is ignored. Otherwise the effect of O_TRUNC is
unspecified.
Effects of notrunc / O_TRUNC I
In the following example, we start out by creating junk.txt of size 1024 bytes. Next, we write 512 bytes to the beginning of it with conv=notrunc. We can see that the size stays the same at 1024 bytes. Finally, we try it without the notrunc option and we can see that the new file size is 512. This is because it was opened with O_TRUNC.
$ dd if=/dev/urandom of=junk.txt bs=1024 count=1
$ ls -l junk.txt
-rw-rw-r-- 1 akyserr akyserr 1024 Dec 11 17:08 junk.txt
$ dd if=/dev/urandom of=junk.txt bs=512 count=1 conv=notrunc
$ ls -l junk.txt
-rw-rw-r-- 1 akyserr akyserr 1024 Dec 11 17:10 junk.txt
$ dd if=/dev/urandom of=junk.txt bs=512 count=1
$ ls -l junk.txt
-rw-rw-r-- 1 akyserr akyserr 512 Dec 11 17:10 junk.txt
Effects of notrunc / O_TRUNC II
I still haven't answered your original question of why when doing a disk-to-disk clone, why conv=notrunc is important. According to the above definition, O_TRUNC seems to be ignored when opening certain special files, and I would expect this to be true for block device nodes too. However, I don't want to assume anything and will attempt to prove it here.
openclose.c
I've written a simple C program here which opens and closes a file given as an argument with the O_TRUNC flag.
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <time.h>
int main(int argc, char * argv[])
{
if (argc < 2)
{
fprintf(stderr, "Not enough arguments...\n");
return (1);
}
int f = open(argv[1], O_RDWR | O_TRUNC);
if (f >= 0)
{
fprintf(stderr, "%s was opened\n", argv[1]);
close(f);
fprintf(stderr, "%s was closed\n", argv[1]);
} else {
perror("Opening device node");
}
return (0);
}
Normal File Test
We can see below that the simple act of opening and closing a file with O_TRUNC will cause it to lose anything that was already there.
$ dd if=/dev/urandom of=junk.txt bs=1024 count=1^C
$ ls -l junk.txt
-rw-rw-r-- 1 akyserr akyserr 1024 Dec 11 17:26 junk.txt
$ ./openclose junk.txt
junk.txt was opened
junk.txt was closed
$ ls -l junk.txt
-rw-rw-r-- 1 akyserr akyserr 0 Dec 11 17:27 junk.txt
Block Device File Test
Let's try a similar test on a USB flash drive. We can see that we start out with a single partition on the USB flash drive. If it get's 'truncated', perhaps the partition will go away (considering it's defined in the first 512 bytes of the disk)?
$ ls -l /dev/sdc*
brw-rw---- 1 root disk 8, 32 Dec 11 17:22 /dev/sdc
brw-rw---- 1 root disk 8, 33 Dec 11 17:22 /dev/sdc1
$ sudo ./openclose /dev/sdc
/dev/sdc was opened
/dev/sdc was closed
$ sudo ./openclose /dev/sdc1
/dev/sdc1 was opened
/dev/sdc1 was closed
$ ls -l /dev/sdc*
brw-rw---- 1 root disk 8, 32 Dec 11 17:31 /dev/sdc
brw-rw---- 1 root disk 8, 33 Dec 11 17:31 /dev/sdc1
It looks like it has no affect whatsoever to open either the disk or the disk's partition 1 with the O_TRUNC option. From what I can tell, the filesystem is still mountable and the files are accessible and intact.
Effects of notrunc / O_TRUNC III
Okay, for my final test I will use dd on my flash drive directly. I will start by writing 512 bytes of random data, then writing 256 bytes of zeros at the beginning. For the final test, we will verify that the last 256 bytes remained unchanged.
$ sudo dd if=/dev/urandom of=/dev/sdc bs=256 count=2
$ sudo hexdump -n 512 /dev/sdc
0000000 3fb6 d17f 8824 a24d 40a5 2db3 2319 ac5b
0000010 c659 5780 2d04 3c4e f985 053c 4b3d 3eba
0000020 0be9 8105 cec4 d6fb 5825 a8e5 ec58 a38e
0000030 d736 3d47 d8d3 9067 8db8 25fb 44da af0f
0000040 add7 c0f2 fc11 d734 8e26 00c6 cfbb b725
0000050 8ff7 3e79 af97 2676 b9af 1c0d fc34 5eb1
0000060 6ede 318c 6f9f 1fea d200 39fe 4591 2ffb
0000070 0464 9637 ccc5 dfcc 3b0f 5432 cdc3 5d3c
0000080 01a9 7408 a10a c3c4 caba 270c 60d0 d2f7
0000090 2f8d a402 f91a a261 587b 5609 1260 a2fc
00000a0 4205 0076 f08b b41b 4738 aa12 8008 053f
00000b0 26f0 2e08 865e 0e6a c87e fc1c 7ef6 94c6
00000c0 9ced 37cf b2e7 e7ef 1f26 0872 cd72 54a4
00000d0 3e56 e0e1 bd88 f85b 9002 c269 bfaa 64f7
00000e0 08b9 5957 aad6 a76c 5e37 7e8a f5fc d066
00000f0 8f51 e0a1 2d69 0a8e 08a9 0ecf cee5 880c
0000100 3835 ef79 0998 323d 3d4f d76b 8434 6f20
0000110 534c a847 e1e2 778c 776b 19d4 c5f1 28ab
0000120 a7dc 75ea 8a8b 032a c9d4 fa08 268f 95e8
0000130 7ff3 3cd7 0c12 4943 fd23 33f9 fe5a 98d9
0000140 aa6d 3d89 c8b4 abec 187f 5985 8e0f 58d1
0000150 8439 b539 9a45 1c13 68c2 a43c 48d2 3d1e
0000160 02ec 24a5 e016 4c2d 27be 23ee 8eee 958e
0000170 dd48 b5a1 10f1 bf8e 1391 9355 1b61 6ffa
0000180 fd37 7718 aa80 20ff 6634 9213 0be1 f85e
0000190 a77f 4238 e04d 9b64 d231 aee8 90b6 5c7f
00001a0 5088 2a3e 0201 7108 8623 b98a e962 0860
00001b0 c0eb 21b7 53c6 31de f042 ac80 20ee 94dd
00001c0 b86c f50d 55bc 32db 9920 fd74 a21e 911a
00001d0 f7db 82c2 4d16 3786 3e18 2c0f 47c2 ebb0
00001e0 75af 6a8c 2e80 c5b6 e4ea a9bc a494 7d47
00001f0 f493 8b58 0765 44c5 ff01 42a3 b153 d395
$ sudo dd if=/dev/zero of=/dev/sdc bs=256 count=1
$ sudo hexdump -n 512 /dev/sdc
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000100 3835 ef79 0998 323d 3d4f d76b 8434 6f20
0000110 534c a847 e1e2 778c 776b 19d4 c5f1 28ab
0000120 a7dc 75ea 8a8b 032a c9d4 fa08 268f 95e8
0000130 7ff3 3cd7 0c12 4943 fd23 33f9 fe5a 98d9
0000140 aa6d 3d89 c8b4 abec 187f 5985 8e0f 58d1
0000150 8439 b539 9a45 1c13 68c2 a43c 48d2 3d1e
0000160 02ec 24a5 e016 4c2d 27be 23ee 8eee 958e
0000170 dd48 b5a1 10f1 bf8e 1391 9355 1b61 6ffa
0000180 fd37 7718 aa80 20ff 6634 9213 0be1 f85e
0000190 a77f 4238 e04d 9b64 d231 aee8 90b6 5c7f
00001a0 5088 2a3e 0201 7108 8623 b98a e962 0860
00001b0 c0eb 21b7 53c6 31de f042 ac80 20ee 94dd
00001c0 b86c f50d 55bc 32db 9920 fd74 a21e 911a
00001d0 f7db 82c2 4d16 3786 3e18 2c0f 47c2 ebb0
00001e0 75af 6a8c 2e80 c5b6 e4ea a9bc a494 7d47
00001f0 f493 8b58 0765 44c5 ff01 42a3 b153 d395
Summary
Through the above experimentation, it seems that notrunc is only important for when you have a file you want to write into, but don't want to truncate it. This seems to have no effect on a block device such as sda or sdb.
You can use
dd if=/dev/zero of=file count=1024 bs=1024
to zero fill a file.
Instead of that I want to one fill a file. How do I do that?
There is no /dev/one file, so how can I simulate that effect via on bash shell?
tr '\0' '\377' < /dev/zero | dd bs=64K of=/dev/sdx
This should be much faster. Choose your blocksizes (or add counts) like you need at. Writing ones to a SSD-Disk till full with a blocksize of 99M gave me 350M/s write performance.
Try this:
dd if=<(yes $'\01' | tr -d "\n") of=file count=1024 bs=1024
Substitute $'\377' or $'\xFF' if you want all the bits to be ones.
MacOS tr may complain about "Illegal byte sequence". Setting LC_CTYPE=C will prevent that. This version can also be used in Linux:
dd if=<(yes $'\01' | LC_CTYPE=C tr -d "\n") of=file count=1024 bs=1024
Well, you could do this:
dd if=/dev/zero count=1024 bs=1024 |
tr '\000' '\001' > file
pv /dev/zero |tr \\000 \\377 >targetfile
...where \377 is the octal representation of 255 (a byte with all bits set to one). Why tr only works with octal numbers, I don't know -- but be careful not to subconsciously translate this to 3FF.
The syntax for using tr is error prone. I recommend verifying that it is making the desired translation...
cat /dev/zero |tr \\000 \\377 |hexdump -C
Note: pv is a nice utility that replaces cat and adds a progress/rate display.
I created a device driver in my github. Installing it creates a file /dev/one that is writing only bits set to 1.
The c file called one.c (the only interesting part is in device_file_read):
// File Driver to create a devince /dev/one like the /dev/zero
#include <linux/init.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/uaccess.h>
MODULE_LICENSE("GPL");
static int device_file_major_number = 0;
static const char device_name[] = "one";
static ssize_t device_file_read(
struct file *file_ptr,
char __user *user_buffer,
size_t count,
loff_t *position) {
printk( KERN_NOTICE "One: Device file is read at offset = %i, read bytes count = %u\n" , (int)*position , (unsigned int)count );
// Allocate Kernel buffer
char* ptr = (char*) vmalloc(count);
// Fill it with one, byte per byte
// -- Note that byte is the smallest accesible data unit
memset(ptr, 0xFF, count);
char res = copy_to_user(user_buffer, ptr, count);
if (res != 0){ return -EFAULT; }
// Return number of byte read
return count;
}
static struct file_operations simple_driver_fops = {
.owner = THIS_MODULE,
.read = device_file_read,
};
int register_device(void) {
int res = 0;
printk( KERN_NOTICE "One: register_device() is called.\n" );
res = register_chrdev( 0, device_name, &simple_driver_fops );
if( res < 0 ) {
printk( KERN_WARNING "One: can\'t register character device with error code = %i\n", res );
return res;
}
device_file_major_number = res;
printk( KERN_NOTICE "One: registered character device with major number = %i and minor numbers 0...255\n", device_file_major_number );
return 0;
}
void unregister_device(void) {
printk( KERN_NOTICE "One: unregister_device() is called\n" );
if(device_file_major_number != 0) {
unregister_chrdev(device_file_major_number, device_name);
}
}
static int my_init(void) {
register_device();
return 0;
}
static void my_exit(void) {
unregister_device();
return;
}
// Declare register and unregister command
module_init(my_init);
module_exit(my_exit);
The Makefile
TARGET_MODULE:=one
BUILDSYSTEM_DIR:=/lib/modules/$(shell uname -r)/build
PWD:=$(shell pwd)
obj-m := $(TARGET_MODULE).o
# See: https://stackoverflow.com/questions/15910064/how-to-compile-a-linux-kernel-module-using-std-gnu99
ccflags-y := -std=gnu99 -Wno-declaration-after-statement
build:
# run kernel build system to make module
$(MAKE) -C $(BUILDSYSTEM_DIR) M=$(PWD) modules
clean:
# run kernel build system to cleanup in current directory
$(MAKE) -C $(BUILDSYSTEM_DIR) M=$(PWD) clean
rm -f MOK.priv MOK*.der
key:
echo "Creating key"
openssl req -new -x509 -newkey rsa:2048 -days 36500 -keyout MOK.priv -outform DER -out MOK.der -nodes -subj "/CN=TinmarinoUnsafe/"
#
echo "\e[31;1mPlease enter a password you will be asked for on reboot:\e[0m"
mokutil --import MOK.der
echo "\e[31;1mNow you must: 1/ reboot, 2/ Select Unroll MOK, 3/ Enter password you previously gave\e[0m"
sign:
cp one.ko one.ko.bck
/usr/src/linux-headers-$(shell uname -r)/scripts/sign-file sha256 MOK.priv MOK.der one.ko
load:
insmod ./$(TARGET_MODULE).ko
unload:
rmmod ./$(TARGET_MODULE).ko
create:
mknod /dev/one c $(shell cat /proc/devices | grep one$ | cut -d ' ' -f1) 0
delete:
rm /dev/one
test:
[ "$(shell xxd -p -l 10 /dev/one)" = "ffffffffffffffffffff" ] \
&& echo "\e[32mSUCCESS\e[0m" \
|| echo "\e[31mFAILED\e[0m"
The instalation is long (3min) due to the driver signature enforcement. Froget this part if you disabled it in your UEFI.
git clone https://github.com/tinmarino/dev_one.git DevOne && cd DevOne # Download
make build # Compile
make key # Generate key for signing
sudo make sign # Sign driver module to permit MOK enforcement (security)
sudo reboot now # Reboot and enable Mok
A blue screen (MOK manager) will appear
Choose "Enroll MOK"
Choose "Continue"
Choose "Yes" (when asked "Enroll the key")
Enter the password you gave at make sign
Choose "Reboot" (again)
sudo make load # Load
sudo make device # Create /dev/one
make test # Test if all is ok
You can simulate a /dev/one without a special device, with a FIFO + yes:
mkfifo ddfifo
dd if=ddfifo of=<file> iflag=fullblock count=1024 bs=1024 status=progress & yes "" | tr '\n' '\1' > ddfifo
tee may be used to double the throughput:
mkfifo ddfifo
dd if=ddfifo of=<file> iflag=fullblock count=1024 bs=1024 status=progress & yes "" | tr '\n' '\1' | tee ddfifo > ddfifo
If you'd like bytes with all bits set to one, swap '\1' for '\377'.