Destroying luks header on dm-crypt linux - linux

I am trying to destroy the luks header on one of my logical volume data1, I am still able to read the file inside data1 after I delete the luks header. I suppose it should not be the case right? Can someone help me in understanding this case?
lsblk output
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 894.2G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 893.8G 0 part
├─vg0-root 251:0 0 758.7G 0 lvm
│ └─luks-45f803e5-3c17-4aaf-a9ad-d66c8b5458de 251:2 0 758.7G 0 crypt /
├─vg0-swap 251:1 0 75G 0 lvm [SWAP]
├─vg0-data3 251:3 0 20G 0 lvm
│ └─luks-6e168d35-26dc-429c-a3d6-8cb4f1c1d39e 251:7 0 20G 0 crypt /data3
├─vg0-data2 251:4 0 20G 0 lvm
│ └─luks-75727dd1-a332-423d-8c37-4cedf9cbe83c 251:8 0 20G 0 crypt /data2
└─vg0-data1 251:5 0 20G 0 lvm
└─luks-cf2d9729-2d1b-48b8-8502-dea937ef602f 251:6 0 20G 0 crypt /data1
Luksdump output to check if the luks header is exists:
-130-sapam#test-host:~ $ sudo cryptsetup luksDump /dev/mapper/vg0-data1
LUKS header information for /dev/mapper/vg0-data1
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 256
MK digest: 9f e7 1a b3 0e fb 4e bc 6d 1b 9e 46 f8 bd 15 22 ea 04 6e c3
MK salt: 83 5e 90 5b b3 a1 c5 a5 d4 22 a0 3e 23 25 51 50
fc cd a8 ac db 9f d0 a8 8b 81 6e 9a 92 1f d8 d3
MK iterations: 43750
UUID: cf2d9729-2d1b-48b8-8502-dea937ef602f
Key Slot 0: ENABLED
Iterations: 439102
Salt: f1 6d 23 b0 b7 ee fc 09 8c 6b 92 ef b2 17 ef d9
0c 83 64 29 bf bc 98 3f f6 93 4b 45 06 49 a9 21
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
Destroying the luks header:
-130-sapam#test-host:~ $ sudo dd bs=512 count=4096 if=/dev/zero of=/dev/mapper/vg0-data1
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 0.00444235 s, 472 MB/s
-0-sapam#test-host:~ $ sudo cryptsetup luksDump /dev/mapper/vg0-data1
-1-sapam#test-host:~ $
I still able to read the file inside /data1/
-1-sapam#test-host:~ $ cat /data1/foo
james
-0-sapam#test-host:~ $
From my understanding is once the header is destroyed, the /data1 should not be able to read right?

It seems you are destroying already mounted partition.
Encryption/decryption keys are hold in the memory while the partition is mounted. You should unmout your LUKS partition first:
# umount /data1
and then erase the LUKS header. You won't be able to mount it again.
Please note cryptsetup utility has a command to erase LUKS header:
# cryptsetup luksErase /dev/mapper/vg0-data1
The advantage of this operation is that you can restore LUKS header from the backup if you done it before.
from cryptsetup(8):
erase <device>
luksErase <device>
Erase all keyslots and make the LUKS container permanently inac‐
cessible. You do not need to provide any password for this op‐
eration.
WARNING: This operation is irreversible.

While luksErase is nice to wipe the keyslots area, note that it doesn't actually destroy the entire LUKS Header. It leaves the metadata intact.
I submitted a feature request asking for the luksErase command to have the ability to wipe the plaintext metadata in the header as well, but the developer rejected & closed it :(
LUKS Header Shredder
You can use the following BASH script to find every LUKS device on the system, wipe the headers, and shutdown the machine.
Disclaimer
The script below contains experimental software that may or may not lead to corruption or total permanent deletion of some or all of your data. I cannot be responsible for any data loss that has occurred as a result of following this guide.
The contents of this answer is provided openly and is licensed under the CC-BY-SA license. The software included in this guide is licensed under the GNU GPLv3 license. All content here is consistent with the limitations of liabilities outlined in its respective licenses.
I highly recommend that any experiments with the script included in this answer is used exclusively on a disposable machine containing no valuable data.
If data loss is a concern for you, then leave now and do not proceed. You have been warned.
#!/bin/bash
#set -x
################################################################################
# File: buskill-selfdestruct.sh
# Purpose: Self-destruct trigger script for BusKill Kill Cord
# For more info, see: https://buskill.in/
# WARNING: THIS IS EXPERIMENTAL SOFTWARE THAT IS DESIGNED TO CAUSE PERMANENT,
# COMPLETE AND IRREVERSIBLE DATA LOSS!
# Note : This script will *not* execute unless it's passed the '--yes'
# argument. Be sure to test this trigger before depending on it!
# Authors: Michael Altfield <michael#buskill.in>
# Created: 2020-03-11
# Updated: 2020-03-11
# Version: 0.1
################################################################################
############
# SETTINGS #
############
BUSKILL_LOCK='/usr/local/bin/buskill-lock.sh'
[ -f ${BUSKILL_LOCK} ] || echo "ERROR: Unable to find buskill-lock.sh"
CRYPTSETUP=`which cryptsetup` || echo "ERROR: Unable to find cryptsetup"
LS=`which ls` || echo "ERROR: Unable to find ls"
CAT=`which cat` || echo "ERROR: Unable to find cat"
GREP=`which grep` || echo "ERROR: Unable to find grep"
ECHO=`which echo` || echo "ERROR: Unable to find echo"
AWK=`which awk` || echo "ERROR: Unable to find awk"
HEAD=`which head` || echo "ERROR: Unable to find head"
LSBLK=`which lsblk` || echo "ERROR: Unable to find lsblk"
OD=`which od` || echo "ERROR: Unable to find od"
##############
# ROOT CHECK #
##############
# re-run as root
if [[ $EUID -ne 0 ]]; then
exec sudo /bin/bash "$0" "$#"
fi
###########
# CONFIRM #
###########
# for safety, exit if this script is executed without a '--yes' argument
${ECHO} "${#}" | ${GREP} '\--yes' &> /dev/null
if [ $? -ne 0 ]; then
${ECHO} "WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING"
${ECHO} "================================================================================"
${ECHO} "WARNING: THIS IS EXPERIMENTAL SOFTWARE THAT IS DESIGNED TO CAUSE PERMANENT, COMPLETE AND IRREVERSIBLE DATA LOSS!"
${ECHO} "================================================================================"
${ECHO} "WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING"
${ECHO}
${ECHO} "cowardly refusing to execute without the '--yes' argument for your protection. If really you want to proceed with damaging your system, retry with the '--yes' argument"
exit 1
fi
###########################
# (DELAYED) HARD SHUTDOWN #
###########################
# The most secure encrypted computer is an encrypted computer that is *off*
# This is our highest priority; initiate a hard-shutdown to occur in 5 minutes regardless
# of what happens later in this script
nohup sleep 60 && echo o > /proc/sysrq-trigger &
nohup sleep 61 && shutdown -h now &
nohup sleep 62 && poweroff --force --no-sync &
###############
# LOCK SCREEN #
###############
# first action: lock the screen!
${BUSKILL_LOCK} &
#####################
# WIPE LUKS VOLUMES #
#####################
# overwrite luks headers
${ECHO} "INFO: shredding LUKS header (plaintext metadata and keyslots with encrypted master decryption key)"
writes=''
IFS=$'\n'
for line in $( ${LSBLK} --list --output 'PATH,FSTYPE' | ${GREP} 'crypt' ); do
device="`${ECHO} \"${line}\" | ${AWK} '{print \$1}'`"
${ECHO} -e "\t${device}"
###########################
# OVERWRITE LUKS KEYSLOTS #
###########################
# erases all keyslots, making the LUKS container "permanently inaccessible"
${CRYPTSETUP} luksErase --batch-mode "${device}" || ${HEAD} --bytes 20M /dev/urandom > ${device} &
# store the pid of the above write tasks so we can try to wait for it to
# flush to disk later -- before triggering a brutal hard-shutdown
writes="${writes} $!"
#####################################
# OVERWRITE LUKS PLAINTEXT METADATA #
#####################################
luksVersion=`${OD} --skip-bytes 6 --read-bytes 2 --format d2 --endian=big --address-radix "n" "${device}"`
# get the end byte to overwrite. For more info, see:
# https://security.stackexchange.com/questions/227359/how-to-determine-start-and-end-bytes-of-luks-header
if [[ $luksVersion -eq 1 ]]; then
# LUKS1: https://gitlab.com/cryptsetup/cryptsetup/-/wikis/LUKS-standard/on-disk-format.pdf
# in LUKS1, the whole header ends at 512 * the `payload-offset`
# this is actually more than we need (includes keyslots), but
# it's the fastest/easiest to bound to fetch in LUKS1
payloadOffset=`${OD} --skip-bytes 104 --read-bytes 4 --format d4 --endian=big --address-radix "n" "${device}"`
luksEndByte=$(( 512 * ${payloadOffset} ))
elif [[ $luksVersion -eq 2 ]]; then
# LUKS2: https://gitlab.com/cryptsetup/LUKS2-docs/blob/master/luks2_doc_wip.pdf
# in LUKS2, the end of the plaintext metadata area is twice the
# size of the `hdr_size` field
hdr_size=`${OD} --skip-bytes 8 --read-bytes 8 --format d8 --endian=big --address-radix "n" "${device}"`
luksEndByte=$(( 2 * ${hdr_size} ))
else
# version unclear; just overwrite 20 MiB
luksEndByte=20971520
fi
# finally, shred that plaintext metadata; we do this in a new file descriptor
# to prevent bash from truncating if ${device} is a file
exec 5<> "${device}"
${HEAD} --bytes "${luksEndByte}" /dev/urandom >&5 &
writes="${writes} $!"
exec 5>&-
done
#######################
# WAIT ON DISK WRITES #
#######################
# wait until all the write tasks above have completed
# note: do *not* put quotes around this arg or the whitespace will break wait
wait ${writes}
# clear write buffer to ensure headers overwrites are actually synced to disks
sync; echo 3 > /proc/sys/vm/drop_caches
#################################
# WIPE DECRYPTION KEYS FROM RAM #
#################################
# suspend each currently-decrypted LUKS volume
${ECHO} "INFO: removing decryption keys from memory"
for device in $( ${LS} -1 "/dev/mapper" ); do
${ECHO} -e "\t${device}";
${CRYPTSETUP} luksSuspend "${device}" &
# clear page caches in memory (again)
sync; echo 3 > /proc/sys/vm/drop_caches
done
#############################
# (IMMEDIATE) HARD SHUTDOWN #
#############################
# do whatever works; this is important.
echo o > /proc/sysrq-trigger &
sleep 1
shutdown -h now &
sleep 1
poweroff --force --no-sync &
# exit cleanly (lol)
exit 0
Sources
LUKS Header Shredder (BusKill Self-Destruct Trigger)
https://github.com/BusKill/buskill-linux/blob/master/triggers/buskill-selfdestruct.sh

Related

Check duplicate disk LABEL before mounting it to the system in bash script

is there a way to check if there is duplicate disk LABEL before mounting it to the system?
i need to make sure that if the user have two external drives, if the two of them have the same label, prompt to the user a warning and asking it to remove the duplicated disk.
my code is in the early stages:
if mountpoint -q "${JOB_MOUNT_DIR}"; then
echo " ${JOB_MOUNT_HD_LABEL} já está montado e está pronto para uso"
else
echo "O dispositivo ""${JOB_MOUNT_HD_LABEL}"" não está montado no diretório ""${JOB_MOUNT_DIR}"""
echo "Deseja montar o diretório?"
echo -n "Qual sua opção? [s/n]: "
read -r "opcao"
if [ "$opcao" == "s" ]; then
mkdir -p "${JOB_MOUNT_DIR}/${JOB_MOUNT_HD_LABEL}"
mount -L "${JOB_MOUNT_HD_LABEL}" "${JOB_MOUNT_DIR}/${JOB_MOUNT_HD_LABEL}"
exit 0
else
echo "Disco não irá ser montado"
exit 0
fi
fi
exit 0
some parts are in pt-br i think that will not be a problem
first it checks if that the disk is already mounted, if not it asks to mount, then there is the problem to know which of the two disk with the same LABEL is to mount
You can use:
lsblk -o name,label
NAME LABEL
mmcblk0
└─mmcblk0p1 eMMC
Or you can use:
blkid
/dev/nvme0n1p1: UUID="36D7-B890" TYPE="vfat" PARTUUID="8614534f-01"
/dev/nvme0n1p5: UUID="65885781-bd9b-4c62-afb0-4a82a0e5759e" TYPE="ext4" PARTUUID="8614534f-05"
/dev/mmcblk0p1: LABEL="eMMC" UUID="79ff33b4-2add-4f2f-844e-d7d242c18578" TYPE="ext4" PARTUUID="d4b36674-ab5f-4f15-bb83-313cce242fe4"
You may need to prefix above commands with sudo
If the above commands get your labels correctly, you can pipe the output roughly as follows to list duplicates:
lsblk -o label | sort | uniq -d

Facing issue in loading UART for Beaglebone debian

I am facing issue in loading uart port for Beaglebone Debian.
Following are my configurations:
I have a beaglebone with Debian OS. I have an sd card with 4 Partitions.
Partition 1 ( mmcblk0p1 ) contains following bootloader configurations:-
/boot/dtbs/4.14.71-ti-r80/am335x-bonegreen-wireless.dtb
/boot/vmlinuz-4.14.71-ti-r80
/boot/uEnv.txt
/boot/initrd.img-4.14.71-ti-r80
/lib/firmware/BB-UART4-00A0.dtbo
/lib/firmware/BB-UART1-00A0.dtbo
/lib/firmware/BB-I2C2-00A0.dtbo
/lib/firmware/AM335X-PRU-UIO-00A0.dtbo
/lib/firmware/BB-BBGW-WL1835-00A0.dtbo
/lib/firmware/BB-BONE-eMMC1-01-00A0.dtbo
/lib/firmware/BB-ADC-00A0.dtbo
/uEnv.txt
Partition 2 ( mmcblk0p2 ) contains the Debian OS.
Partition 3 ( mmcblk0p3 ) contains another Debian OS.
Partition 4 ( mmcblk0p4 ) decides from Which partition to boot?
/uEnv.txt from mmcblk0p1 reads from mmcblk0p4 and decides from which partition to boot.
This is my /uEnv.txt:
rdaddr=0x88080000
initrd_high=0xffffffff
fdt_high=0xffffffff
loadxrd=echo debug: [/boot/initrd.img-${uname_r}] ... ; load mmc 0:1 ${rdaddr} /boot/initrd.img-${uname_r}; setenv rdsize ${filesize}
loaduEnvtxt=load mmc 0:1 ${loadaddr} /boot/uEnv.txt ; env import -t ${loadaddr} ${filesize};
check_dtb=if test -n ${dtb}; then setenv fdtfile ${dtb};fi;
check_uboot_overlays=if test -n ${enable_uboot_overlays}; then setenv enable_uboot_overlays ;fi;
loadall=run loaduEnvtxt; run check_dtb; run check_uboot_overlays; run loadxrd;
rootpart=0:2
flagpart=0:4
bootdir=/boot
bootfile=vmlinuz-4.14.71-ti-r80
console=ttyO0,115200n8
fdtaddr=0x88000000
fdtfile=am335x-bonegreen-wireless.dtb
loadaddr=0x82000000
mmcroot=/dev/mmcblk0p2 ro
mmcrootfstype=ext4 rootwait
mmcargs=setenv bootargs console=${console} ${optargs} ${cape_disable} ${cape_enable} root=${mmcroot} rootfstype=${mmcrootfstype} ${cmdline}
loadfdt=echo debug: [/boot/dtbs/${uname_r}/${fdtfile}] ... ;load mmc 0:1 ${fdtaddr} /boot/dtbs/${uname_r}/${fdtfile}
loadimage=echo debug: [/boot/vmlinuz-${uname_r}] ... ; load mmc 0:1 ${loadaddr} /boot/vmlinuz-${uname_r}
boot_three=setenv rootpart 0:3; setenv mmcroot /dev/mmcblk0p3 ro
findroot=\
if test -e mmc $flagpart three; then \
if test -e mmc $flagpart three_ok; then \
run boot_three; \
elif test ! -e mmc $flagpart three_tried; then \
fatwrite mmc $flagpart $loadaddr three_tried 4; \
run boot_three; \
fi; \
elif test -e mmc $flagpart two; then \
if test ! -e mmc $flagpart two_ok; then \
if test -e mmc $flagpart two_tried; then \
run boot_three; \
else \
fatwrite mmc $flagpart $loadaddr two_tried 4; \
fi; \
fi; \
fi;
uenvcmd=\
run loadall; \
run findroot; \
echo Using root partition ${rootpart}; \
if run loadfdt; then \
echo Loaded ${fdtfile}; \
if run loadimage; then \
run mmcargs; \
bootz ${loadaddr} - ${fdtaddr}; \
fi; \
fi;
In /boot/uEnv.txt I have enabled following Configs:
uname_r=4.14.71-ti-r80
enable_uboot_overlays=1
uboot_overlay_addr0=/lib/firmware/BB-UART4-00A0.dtbo
uboot_overlay_addr1=/lib/firmware/BB-UART1-00A0.dtbo
uboot_overlay_addr2=/lib/firmware/BB-I2C2-00A0.dtbo
uboot_overlay_pru=/lib/firmware/AM335X-PRU-RPROC-4-14-TI-00A0.dtbo
enable_uboot_cape_universal=1
cmdline=coherent_pool=1M net.ifnames=0 quiet
My Boot logs:
U-Boot SPL 2018.09-00002-g0b54a51eee (Sep 10 2018 - 19:41:39 -0500)
Trying to boot from MMC2
Loading Environment from EXT4...
** Unable to use mmc 0:1 for loading the env **
U-Boot 2018.09-00002-g0b54a51eee (Sep 10 2018 - 19:41:39 -0500), Build: jenkins-github_Bootloader-Builder-65
CPU : AM335X-GP rev 2.1
I2C: ready
DRAM: 512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
Reset Source: Power-on reset has occurred.
RTC 32KCLK Source: External.
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from EXT4...
** Unable to use mmc 0:1 for loading the env **
Board: BeagleBone Black
<ethaddr> not set. Validating first E-fuse MAC
BeagleBone Black:
Model: SeeedStudio BeagleBone Green Wireless:
BeagleBone: cape eeprom: i2c_probe: 0x54:
BeagleBone: cape eeprom: i2c_probe: 0x55:
BeagleBone: cape eeprom: i2c_probe: 0x56:
BeagleBone: cape eeprom: i2c_probe: 0x57:
Net: eth0: MII MODE
Could not get PHY for cpsw: addr 0
cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
board_name=[A335BNLT] ...
board_rev=[GW1A] ...
switch to partitions #0, OK
mmc0 is current device
SD/MMC found on device 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
60067 bytes read in 6 ms (9.5 MiB/s)
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc0 is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt ...
2628 bytes read in 2 ms (1.3 MiB/s)
gpio: pin 55 (gpio 55) value is 1
Loaded environment from /uEnv.txt
Importing environment from mmc ...
Checking if uenvcmd is set ...
gpio: pin 56 (gpio 56) value is 1
Running uenvcmd ...
2113 bytes read in 3 ms (687.5 KiB/s)
debug: [/boot/initrd.img-4.14.71-ti-r80] ...
4799493 bytes read in 303 ms (15.1 MiB/s)
Using root partition 0:2
debug: [/boot/dtbs/4.14.71-ti-r80/am335x-bonegreen-wireless.dtb] ...
60067 bytes read in 8 ms (7.2 MiB/s)
Loaded am335x-bonegreen-wireless.dtb
debug: [/boot/vmlinuz-4.14.71-ti-r80] ...
10416640 bytes read in 652 ms (15.2 MiB/s)
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Using Device Tree in place at 88000000, end 88011aa2
Starting kernel ...
[ 0.000749] timer_probe: no matching timers found
[ 0.783193] wkup_m3_ipc 44e11324.wkup_m3_ipc: could not get rproc handle
[ 1.073624] omap_voltage_late_init: Voltage driver support not added
[ 1.080550] PM: Cannot get wkup_m3_ipc handle
Question:
Now when I boot into the partition and check for the UART port I am unable to find them.
ls -l /dev/ttyO*
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO0 -> ttyS0
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO2 -> ttyS2
The output I am expecting is:
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO0 -> ttyS0
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO1 -> ttyS1
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO2 -> ttyS2
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO3 -> ttyS3
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO4 -> ttyS4
lrwxrwxrwx 1 root root 5 Nov 3 2016 /dev/ttyO5 -> ttyS5

Line manipulation & sorting

I am alright at writing Linux scripts but could use some advice. I know the problem is sort of vague, so if you can provide any help whatsoever I will appreciate it!
The following issue is for personal growth, and because I am writing some network tools for fun/learning. No homework involved (I'm a senior in college, none of my classes require this stuff!)
I am using tshark to get information about packet captures. This is what it looks like:
rachel#Ubuntu-1:~/PCAP$ tshark -r LargeTorrent.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
eth frames:4309 bytes:3984321
ip frames:4119 bytes:3969006
icmp frames:1316 bytes:1308988
udp frames:1408 bytes:1350786
data frames:1368 bytes:1346228
dns frames:16 bytes:1176
nbns frames:14 bytes:1300
http frames:8 bytes:1596
nbdgm frames:2 bytes:486
smb frames:2 bytes:486
mailslot frames:2 bytes:486
browser frames:2 bytes:486
tcp frames:1395 bytes:1309232
data frames:1300 bytes:1294800
http frames:6 bytes:3763
data-text-lines frames:2 bytes:324
xml frames:2 bytes:3205
tcp.segments frames:1 bytes:787
nbss frames:34 bytes:5863
smb frames:17 bytes:3047
pipe frames:4 bytes:686
lanman frames:4 bytes:686
smb2 frames:13 bytes:2444
bittorrent frames:10 bytes:1709
tcp.segments frames:2 bytes:433
bittorrent frames:2 bytes:433
bittorrent frames:1 bytes:258
bittorrent frames:2 bytes:221
bittorrent frames:2 bytes:221
arp frames:146 bytes:8760
ipv6 frames:44 bytes:6555
udp frames:40 bytes:6211
dns frames:18 bytes:1711
dhcpv6 frames:14 bytes:2114
http frames:6 bytes:1014
data frames:2 bytes:1372
icmpv6 frames:4 bytes:344
===================================================================
What I would like for it to look like:
rachel#Ubuntu-1:~/PCAP$ tshark -r LargeTorrent.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
=====================================
eth 984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
===================================================================
Edit: I am going to add the original question for the purpose of making sense of the (great) answer that was provided.
Originally, I wanted to only print statistics for "leaves" because eth, ip, etc. are all parents and their statistics are not necessary for my purposes. In addition, instead of having a god-awful block of text with only spaces to show hierarchy, I wanted to erase all the statistics for parents, and show them as breadcrumbs behind the child.
Example:
eth frames:4309 bytes:3984321
ip frames:4119 bytes:3969006
icmp frames:1316 bytes:1308988
udp frames:1408 bytes:1350786
data frames:1368 bytes:1346228
dns frames:16 bytes:1176
Should become
eth:ip:icmp - 1308988 bytes
eth:ip:udp:data - 1346228 bytes
eth:ip:udp:dns - 1176 bytes
To preserve the hierarchy and avoid printing useless statistics.
Anyway, the approved answer by Etan solved this perfectly! And for those of you who are on my level who are unsure of how to proceed after this answer, this will help you finish up:
Save the given script as a filename.awk file
Save the block of text you want to manipulate as a filename.txt file
Call awk -f filename.awk filename.txt
Optionally pipe the output to a file ( awk -f filename.awk filename.txt >> output.txt )
The output I originally thought you wanted could be achieved with this awk script. (I think this can probably be done cleaner but this seems to work well enough.)
function entry() {
# Don't want to print empty entries.
if (ind[0]) {
printf "%s", ind[0]
for (i = 1; i <= ls; i++) {
printf ":%s", ind[i]
}
split(b, a, /:/)
printf " - %s %s\n", a[2], a[1]
}
}
# Found our data marker. Note that and print the current line.
$1 == "Filter:" {d=1; print; next}
# Print lines until we see our data marker.
!d {print; next}
# Print empty lines.
!NF {print; next}
# Save our trailing line for later.
/===/ {suf=$0; next}
{
# Save our previous indentation level.
ls = s
# Find our new indentation level (by where the first field starts).
s = (match($0, /[^[:space:]]/)-1) / 2
# If the current line is at or below the last indent level print the last line.
if (s <= ls) {
entry()
}
# Save the current line's byte count.
b=$NF
# Save the current line's field name.
ind[s] = $1
}
END {
# Print a final line if we had one.
entry()
# Print the suffix line if we have one.
if (suf) {
print suf
}
}
Which, on the sample input, gets you this output.
===================================================================
Protocol Hierarchy Statistics
Filter:
eth:ip:icmp - 1308988 bytes
eth:ip:udp:data - 1346228 bytes
eth:ip:udp:dns - 1176 bytes
eth:ip:udp:nbns - 1300 bytes
eth:ip:udp:http - 1596 bytes
eth:ip:udp:nbdgm:smb:mailslot:browser - 486 bytes
eth:ip:tcp:data - 1294800 bytes
eth:ip:tcp:http:data-text-lines - 324 bytes
eth:ip:tcp:http:xml:tcp.segments - 787 bytes
eth:ip:tcp:nbss:smb:pipe:lanman - 686 bytes
eth:ip:tcp:nbss:smb2 - 2444 bytes
eth:ip:tcp:bittorrent:tcp.segments:bittorrent:bittorrent - 258 bytes
eth:ip:tcp:bittorrent:bittorrent:bittorrent - 221 bytes
eth:arp - 8760 bytes
eth:ipv6:udp:dns - 1711 bytes
eth:ipv6:udp:dhcpv6 - 2114 bytes
eth:ipv6:udp:http - 1014 bytes
eth:ipv6:udp:data - 1372 bytes
eth:ipv6:icmpv6:data - 344 bytes
===================================================================
Output like what you edited to indicate you want is probably more easily handled with sed though.
/Filter:/a \
Protocol Bytes \
=====================================
s/frames:[^ ]*//
s/ b/b/
s/bytes:\([^ ]*\)/\1/
Which ends up with output.
===================================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
=====================================
eth 3984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
===================================================================
A simple script with sed will work as well.
$ printf "\n==========================================================\n"; printf "Protocol Hierarchy Statistics\nFilter:\n\n";printf "\nProtocol\t\t\t\t Bytes\n================================================\n" && sed -e 's/\(frames[:].*bytes[:]\)\(.*$\)/\2/' dat/tshark.txt | tail -n+4 | head -n-1 && printf "================================================\n"
broken down into script form (where dat/tshark.txt is the filename holding the tshark output):
printf "\n==========================================================\n"
printf "Protocol Hierarchy Statistics\nFilter:\n\n"
printf "\nProtocol\t\t\t\t Bytes\n================================================\n"
sed -e 's/\(frames[:].*bytes[:]\)\(.*$\)/\2/' dat/tshark.txt | tail -n+4 | head -n-1
printf "================================================\n"
Output
==========================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
================================================
eth 3984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
================================================
Formatting
Following on from your comment on how to align the bytes info given the variable length of the protocol tags, you can make use of printf to format the output as you have indicated. Like Ethan, I started working on your original question that had the tags consolidated. My initial approach was to read the different levels into different associative arrays that could be combined into what you initially specified. Doing so, I had to produce the output lined up using printf. Here is the first attempt I made working with the first 4-levels of your tshark data:
declare -i ln=0
declare -A l1 l2 l3 l4
## read each line in file and assing to associative arrays for each level
while read -r line; do
ln=${#line} # base level on length of line read
[ $ln -gt 66 ] && continue;
[ $ln -eq 66 ] && { iface="${line%% *}"; l1[${iface}]="${line##* }"; }
[ $ln -eq 64 ] && { proto="${iface}:${line%% *}"; l2[${proto}]="${line##* }"; }
[ $ln -eq 62 ] && { ptype="${proto}:${line%% *}"; l3[${ptype}]="${line##* }"; }
[ $ln -le 60 ] && { data="${ptype}:${line%% *}"; l4[${data}]="${line##* }"; }
done < "$1"
## output a summary of the file
printf "\n4-level deep summary of file '%s':\n\n" "$1"
for i in "${!l1[#]}"; do
for j in "${!l2[#]}"; do
printf " %-32s %s\n" "$j" "${l2[$j]}"
for k in "${!l3[#]}"; do
printf " %-32s %s\n" "$k" "${l3[$k]}"
for l in "${!l4[#]}"; do
[ "${l%:*}" == "$k" ] && printf " %-32s %s\n" "$l" "${l4[$l]}"
done
done
done
done
The output it produced was for example:
eth:ip frames:4119 bytes:3969006
eth:ip:udp frames:1408 bytes:1350786
eth:ip:udp:data frames:1368 bytes:1346228
eth:ip:udp:nbdgm frames:2 bytes:486
eth:ip:udp:nbns frames:14 bytes:1300
You can look at the various printf statements in the code above and see how the alignment is handled. Let me know if you have further questions.
I'm a little surprised that tshark doesn't have a JSON or machine-readable way to get the -z io,phs info, when it has so many ways to extract packet info.
I tried playing with some of the above, but bash seems to have changed over the years (or has different defaults depending on the environment). I am also not sure which shell or version of it was used to produce the above.
The line lengths/output from tshark have also changed: My debugging showed different line lengths, so the trick above using line lengths, e.g. [ $ln -gt 66 ] didn't work for me.
It seems that read -r strips out leading/trailing whitespaces. If you actually want it, you need IFS= to make it give you the spaces:
## read each line in file
while IFS= read -r line ; do
...
done
The "nested" levels associative arrays is clever, but hard to work with - it shows what rabbit holes you can go down with bash - although now when iterating through it, bash produces it in "hash" order and not the order they were added.
Since I actually needed the data in the rest of my script, the nested arrays made it particularly fiddly to deal with. Fine for printf purposes where you just print the line, but what if you actually want to get the frames count for each item and do then do something with it.
Here was my attempt that simplified it a bit. I implemented it as a bash function which gets a few other bits of info from the sample file:
TSHARK=/usr/bin/tshark
CAPINFOS=/usr/bin/capinfos
declare -A fcount
declare -A bcount
declare -A capinfo
function loadcapinfo
{
local sample=$1
local statstofile=$2
local bytes
local frames
local key
if [ ! -f "$sample" ] ; then
echo "FATAL: loadcapinfo: file does not exist: $sample"
exit 1
fi
capinfo[start_time_epoch]=$($CAPINFOS -Tr -Sa $sample | cut -f2)
capinfo[start_time]=$($CAPINFOS -Tr -a $sample | cut -f2)
capinfo[end_time_epoch]=$($CAPINFOS -Tr -Se $sample | cut -f2)
capinfo[end_time]=$($CAPINFOS -Tr -e $sample | cut -f2)
capinfo[size]=$($CAPINFOS -Tr -s $sample | cut -f2)
declare -i ln=0
while IFS= read -r line ; do
ln=${#line} # base level on length of line read
[ $ln -le 1 ] && continue;
pat=".*frames:([0-9]+)\s+bytes:([0-9]+)"
pat_1="^(\w+)"
pat_2="^\s{2}(\w+)"
pat_3="^\s{4}(\w+)"
pat_4="^\s{6}(\w+)"
ethertype="ethertype"
[[ $line =~ $pat ]] && { frames=${BASH_REMATCH[1]}; bytes=${BASH_REMATCH[2]}; } || continue;
[[ $line =~ $pat_1 ]] && { encap="${BASH_REMATCH[1]}:${ethertype}"; key="${encap}"; }
[[ $line =~ $pat_2 ]] && { proto=${BASH_REMATCH[1]}; key="${encap}:${proto}"; }
[[ $line =~ $pat_3 ]] && { ptype=${BASH_REMATCH[1]}; key="${encap}:${proto}:${ptype}"; }
[[ $line =~ $pat_4 ]] && { data=${BASH_REMATCH[1]}; key="${encap}:${proto}:${ptype}:${data}"; }
[ "$proto" = "llc" ] && { key=${key/eth:ethertype:llc/eth:llc} ; }
fcount[${key}]=${frames:=0}
bcount[${key}]=${bytes:=0}
if [ -n "$statstofile" ] ; then
echo "${capinfo[start_time_epoch]},${key},${frames},${bytes}" >> $statstofile
fi
done < <($TSHARK -qr $sample -z io,phs)
unset fcount[0]
}
Now, after this in the script, we can do:
loadcapinfo /my/sample/file.pcap /tmp/stats.txt
Optionally write the counts to a file, /tmp/stats.txt
This uses one associative array for each count, and puts other info into capinfo so now we can do things like:
echo "IPv4 Packet Count is: ${fcount[eth:ethertype:ip]}"
echo "IPv6 Packet Count is: ${fcount[eth:ethertype:ipv6]}"
echo "ARP Count is: ${fcount[eth:ethertype:arp]}"
echo "STP Count is: ${fcount[eth:llc:stp]}"
echo "Start time: ${capinfo[start_time]}"
echo "End time: ${capinfo[end_time]}"
echo "File size: ${capinfo[size]}"
I made the keys match Wireshark's frame.protocols field, which inserts some "pseudo protocol" for most things called "ethertype". This way, if you want to then iterate through the associative array to find the packet(s) in the pcap file, you can use the information to find packets with a given protocol.
tshark -r /my/sample/file.pcap -Y "frame.protocols == eth:ethertype:ip:udp:snmp" -Tfields -e frame.number -e eth.src_resolved -e eth.dst_resolved -e ip.src -e ip.dst -e frame.protocols
for i in "${!fcount[#]}"; do
tshark -r /my/sample/file.pcap -Y "frame.protocols == $i" -Tfields -e frame.number -e eth.src_resolved -e eth.dst_resolved -e ip.src -e ip.dst -e frame.protocols > /tmp/$i.txt
done

Integrity Measurement Architecture(IMA) & Linux Extended Verification Module (EVM)

I am trying to activate IMA appraisal & EVM modules.
After compiling linux kernel 3.10.2 on my bt5R3 and setting kernel boot option in a first time like this:
GRUB_CMDLINE_LINUX="rootflags=i_version ima_tcb ima_appraise=fix ima_appraise_tcb evm=fix"
and after running this command to generate xattr security.ima and security.evm
find / \( -fstype rootfs -o -fstype ext4 \) -type f -uid 0 -exec head -c 1 '{}' \;
like this:
GRUB_CMDLINE_LINUX="rootflags=i_version ima_tcb ima_appraise=enforce ima_appraise_tcb evm=enforce"
I try to create digital signature of xattr like it's recommended on this tutorial
Tutorial to IMA & EVM
Every steps have been followed, creating RSA keys, loading them early at boot in initramfs with keyctl.
Session Keyring
-3 --alswrv 0 65534 keyring: _uid_ses.0
977514165 --alswrv 0 65534 \_ keyring: _uid.0
572301790 --alswrv 0 0 \_ user: kmk-user
126316032 --alswrv 0 0 \_ encrypted: evm-key
570886575 --alswrv 0 0 \_ keyring: _ima
304346597 --alswrv 0 0 \_ keyring: _evm
However as soon as I reboot my OS when I try to read a signed and hashed file I get the error "Permission Denied"
Running dmesg tells me :
[ 5461.175996] type=1800 audit(1375262160.913:57): pid=1756 uid=0 auid=4294967295 ses=4294967295 op="appraise_data" cause="**invalid-HMAC**" comm="sh" name="/root/Desktop/new.sh" dev="sda1" ino=546526 res=0
Have you any idea why i get invalid HMAC ?
They keys are loaded like the tutorial says...
#!/bin/sh -e
PREREQ=""
# Output pre-requisites
prereqs()
{
echo "$PREREQ"
}
case "$1" in
prereqs)
prereqs
exit 0
;;
esac
grep -q "ima=off" /proc/cmdline && exit 1
mount -n -t securityfs securityfs /sys/kernel/security
IMA_POLICY=/sys/kernel/security/ima/policy
LSM_POLICY=/etc/ima_policy
grep -v "^#" $LSM_POLICY >$IMA_POLICY
# import EVM HMAC key
keyctl show |grep -q kmk || keyctl add user kmk "testing123" #u
keyctl add encrypted evm-key "load `cat /etc/keys/evm-key`" #u
#keyctl revoke kmk
# import Module public key
mod_id=`keyctl newring _module #u`
evmctl import /etc/keys/pubkey_evm.pem $mod_id
# import IMA public key
ima_id=`keyctl newring _ima #u`
evmctl import /etc/keys/pubkey_evm.pem $ima_id
# import EVM public key
evm_id=`keyctl newring _evm #u`
evmctl import /etc/keys/pubkey_evm.pem $evm_id
# enable EVM
echo "1" > /sys/kernel/security/evm
# enable module checking
#echo "1" > /sys/kernel/security/module_check
Thanks for your help
Solved, new kernel use HMAC v2 and you have to activate asymmetric key when you compile kernel.
cat .config should have this entries:
CONFIG_EVM_HMAC_VERSION=2
CONFIG_ASYMMETRIC_KEY_TYPE=y
Then when you hash or sign a file use
evmctl -u - -x --imasig/--imahash $file
As well you should have create the asymetric keys and load them in _evm and _ima keyring with keyctl with initramfs.

How can I close a netcat connection after a certain character is returned in the response?

We have a very simple tcp messaging script that cats some text to a server port which returns and displays a response.
The part of the script we care about looks something like this:
cat someFile | netcat somehost 1234
The response the server returns is 'complete' once we get a certain character code (specifically &001C) returned.
How can I close the connection when I receive this special character?
(Note: The server won't close the connection for me. While I currently just CTRL+C the script when I can tell it's done, I wish to be able to send many of these messages, one after the other.)
(Note: netcat -w x isn't good enough because I wish to push these messages through as fast as possible)
Create a bash script called client.sh:
#!/bin/bash
cat someFile
while read FOO; do
echo $FOO >&3
if [[ $FOO =~ `printf ".*\x00\x1c.*"` ]]; then
break
fi
done
Then invoke netcat from your main script like so:
3>&1 nc -c ./client.sh somehost 1234
(You'll need bash version 3 for the regexp matching).
This assumes that the server is sending data in lines - if not you'll have to tweak client.sh so that it reads and echoes a character at a time.
How about this?
Client side:
awk -v RS=$'\x1c' 'NR==1;{exit 0;}' < /dev/tcp/host-ip/port
Testing:
# server side test script
while true; do ascii -hd; done | { netcat -l 12345; echo closed...;}
# Generate 'some' data for testing & pipe to netcat.
# After netcat connection closes, echo will print 'closed...'
# Client side:
awk -v RS=J 'NR==1; {exit;}' < /dev/tcp/localhost/12345
# Changed end character to 'J' for testing.
# Didn't wish to write a server side script to generate 0x1C.
Client side produces:
0 NUL 16 DLE 32 48 0 64 # 80 P 96 ` 112 p
1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
10 LF 26 SUB 42 * 58 : 74
After 'J' appears, server side closes & prints 'closed...', ensuring that the connection has indeed closed.
Try:
(cat somefile; sleep $timeout) | nc somehost 1234 | sed -e '{s/\x01.*//;T skip;q;:skip}'
This requires GNU sed.
How it works:
{
s/\x01.*//; # search for \x01, if we find it, kill it and the rest of the line
T skip; # goto label skip if the last s/// failed
q; # quit, printing current pattern buffer
:skip # label skip
}
Note that this assumes there'll be a newline after \x01 - sed won't see it otherwise, as sed operates line-by-line.
Maybe have a look at Ncat as well:
"Ncat is the culmination of many key features from various Netcat incarnations such as Netcat 1.x, Netcat6, SOcat, Cryptcat, GNU Netcat, etc. Ncat has a host of new features such as "Connection Brokering", TCP/UDP Redirection, SOCKS4 client and server supprt, ability to "Chain" Ncat processes, HTTP CONNECT proxying (and proxy chaining), SSL connect/listen support, IP address/connection filtering, plus much more."
http://nmap-ncat.sourceforge.net
This worked best for me. Just read the output with a while loop and then check for "0x1c" using an if statement.
while read i; do
if [ "$i" = "0x1c" ] ; then # Read until "0x1c". Then exit
break
fi
echo $i;
done < <(cat someFile | netcat somehost 1234)

Resources