Simulate a faulty block device with read errors? - linux

I'm looking for an easier way to test my application against faulty block devices that generate i/o read errors when certain blocks are read. Trying to use a physical hard drive with known bad blocks is a pain and I would like to find a software solution if one exists.
I did find the Linux Disk Failure Simulation Driver which allows creating an interface that can be configured to generate errors when certain ranges of blocks are read, but it is for the 2.4 Linux Kernel and hasn't been updated for 2.6.
What would be perfect would be an losetup and loop driver that also allowed you to configure it to return read errors when attempting to read from a given set of blocks.

It's not a loopback device you're looking for, but rather device-mapper.
Use dmsetup to create a device backed by the "error" target. It will show up in /dev/mapper/<name>.
Page 7 of the Device mapper presentation (PDF) has exactly what you're looking for:
dmsetup create bad_disk << EOF
0 8 linear /dev/sdb1 0
8 1 error
9 204791 linear /dev/sdb1 9
EOF
Or leave out the sdb1 parts to and put the "error" target as the device for blocks 0 - 8 (instead of sdb1) to make a pure error disk.
See also The Device Mapper appendix from "RHEL 5
Logical Volume Manager Administration".
There's also a flakey target - a combo of linear and error that sometimes succeeds. Also a delay to introduce intentional delays for testing.

It seems like Linux's built-in fault injection capabilities would be a good idea to use.
Blog: http://blog.wpkg.org/2007/11/08/using-fault-injection/
Reference: https://www.kernel.org/doc/Documentation/fault-injection/fault-injection.txt

The easiest way to play with block devices is using nbd.
Download the userland sources from git://github.com/yoe/nbd.git and modify nbd-server.c to fail at reading or writing on whichever areas you want it to fail on, or to fail in a controllably random pattern, or basically anything you want.

I would like to elaborate on Peter Cordes answer.
In bash, setup an image on a loopback device with ext4, then write a file to it named binary.bin.
imageName=faulty.img
mountDir=$(pwd)/mount
sudo umount $mountDir ## make sure nothing is mounted here
dd if=/dev/zero of=$imageName bs=1M count=10
mkfs.ext4 $imageName
loopdev=$(sudo losetup -P -f --show $imageName); echo $loopdev
mkdir $mountDir
sudo mount $loopdev $mountDir
sudo chown -R $USER:$USER mount
echo "2ed99f0039724cd194858869e9debac4" | xxd -r -p > $mountDir/binary.bin
sudo umount $mountDir
in python3 (since bash struggles to deal with binary data) search for the magic binary data in binary.bin
import binascii
with open("faulty.img", "rb") as fd:
s = fd.read()
search = binascii.unhexlify("2ed99f0039724cd194858869e9debac4")
beg=0
find = s.find(search, beg); beg = find+1; print(find)
start_sector = find//512; print(start_sector)
then back in bash mount the faulty block device
start_sector=## copy value from variable start_sector in python
next_sector=$(($start_sector+1))
size=$(($(wc -c $imageName|cut -d ' ' -f1)/512))
len=$(($size-$next_sector))
echo -e "0\t$start_sector\tlinear\t$loopdev\t0" > fault_config
echo -e "$start_sector\t1\terror" >> fault_config
echo -e "$next_sector\t$len\tlinear\t$loopdev\t$next_sector" >> fault_config
cat fault_config | sudo dmsetup create bad_drive
sudo mount /dev/mapper/bad_drive $mountDir
finally we can test the faulty block device by reading a file
cat $mountDir/binary.bin
which produces the error:
cat: /path/to/your/mount/binary.bin: Input/output error
clean up when you're done with testing
sudo umount $mountDir
sudo dmsetup remove bad_drive
sudo losetup -d $loopdev
rm fault_config $imageName

Related

Change location of /etc/fstab

I have written a script which requires to read a few entries in /etc/fstab. I have tested the script by manually adding some entries in /etc/fstab and then restored the file to its original contents, also manually. Now I would like to automate those tests and run them as a seperate script. I do, however, not feel comfortable with the idea of changing /etc/fstab altered. I was thinking of making a backup copy of /etc/fstab, then altering it and finally restoring the original file after the tests are done. I would prefer it if I could temporarily alter the location of fstab.
Is there a way to alter the location of fstab to, say, /usr/local/etc/fstab so that when mount -a is run from within a script only the entries in /usr/local/etc/fstab are processed?
UPDATE:
I used bishop's solution by setting LIBMOUNT_FSTAB=/usr/local/etc/fstab. I have skimmed the man page of mount on several occasions in the past but I never noticed this variable. I am not sure if this variable has always been there and I simply overlooked it or if it had been added at some point. I am using mount from util-linux 2.27.1 and at least in this version LIBMOUNT_FSTAB is available and documented in the man-page. It is in the ENVIRONMENT section at the end. This will make my automated tests a lot safer in the future.
UPDATE2:
Since there has been some discussion whether this is an appropriate programming question or not, I have decided to write a small script which demonstrates the usage of LIBMOUNT_FSTAB.
#!/bin/bash
libmount=libmount_fstab
tmpdir="/tmp/test_${libmount}_folder" # temporary test folder
mntdir="$tmpdir/test_${libmount}_mountfolder" # mount folder for loop device
img="$tmpdir/loop.img" # dummy image for loop device
faketab="$tmpdir/alternate_fstab" # temporary, alternative fstab
# get first free loop device
loopdev=$(losetup -f)
# verify there is a free loop device
if [[ -z "$loopdev" ]];then
echo "Error: No free loop device" >&2
exit 1
fi
# check that loop device is not managed by default /etc/fstab
if grep "^$loopdev" /etc/fstab ;then
echo "Error: $loopdev already managed by /etc/fstab" >&2
exit 1
fi
# make temp folders
mkdir -p "$tmpdir"
mkdir -p "$mntdir"
# create temporary, alternative fstab
echo "$loopdev $mntdir ext2 errors=remount-ro 0 1" > "$faketab"
# create dummy image for loop device
dd if=/dev/zero of="$img" bs=1M count=5 &>/dev/null
# setup loop device with dummy image
losetup "$loopdev" "$img" &>/dev/null
# format loop device so it can be mounted
mke2fs "$loopdev" &>/dev/null
# alter location for fstab
export LIBMOUNT_FSTAB="$faketab"
# mount loop device by using alternative fstab
mount "$loopdev" &>/dev/null
# verify loop device was successfully mounted
if mount | grep "^$loopdev" &>/dev/null;then
echo "Successfully used alternative fstab: $faketab"
else
echo "Failed to use alternative fstab: $faketab"
fi
# clean up
umount "$loopdev" &>/dev/null
losetup -d "$loopdev"
rm -rf "$tmpdir"
exit 0
My script primarily manages external devices which are not attached most of the time. I use loop-devices to simulate external devices to test the functionality of my script. This saves a lot of time since I do not have to attach/reattach several physical devices. I think this proves that being able to use an alternative fstab is a very useful feature and allows for scripting safe test scenarios whenever parsing/altering of fstab is required. In fact, I have decided to partially rewrite my script so that it can also use an alternative fstab. Since most of the external devices are hardly ever attached to the system their corresponding entries are just cluttering up /etc/fstab.
Refactor your code that modifies fstab contents into a single function, then test that function correctly modifies the dummy fstab files you provide it. Then you can confidently use that function as part of your mount pipeline.
function change_fstab {
local fstab_path=${1:?Supply a path to the fstab file}
# ... etc
}
change_fstab /etc/fstab && mount ...
Alternatively, set LIBMOUNT_FSTAB per the libmount docs:
LIBMOUNT_FSTAB=/path/to/fake/fstab mount ...

Force unmount of usb drive by closing open applications programatically [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
When I unplug the AC adapter from my laptop I want all USB drives to automatically unmount. If applications are open that block the device, they should be killed. Once everything is killed and unmount a signal tone could be plaid to indicate that it's now safe to unplug it.
The use case is to quickly grab and go your laptop without having to fumble with the ui to get all drives disconnected but avoid unclean unmounts.
Any hints on how to start would be fantastic, thanks you!
ANSWER
For a full copy&paste script see my answer below.
If your USB devices mount to /mount/media do:
kill -9 $(lsof -t $(mount | grep "/mount/media" | cut -d " " -f 1)) # Exit processes blocking umount cleanly
kill $(lsof -t $(mount | grep "/mount/media" | cut -d " " -f 1)) # Force kill remaining open processes
umount $(mount | grep "/mount/media" | cut -d " " -f 1) # Unmount USB drives
Be careful with this since if you don't have blocking applications open lsof will return all pids and you will kill your running OS. See the copy&paste script below for a working implementation that handles this case.
Then call this script whenever the AC adapter is unplugged by adding this line to /etc/udev/rules.d
SUBSYSTEM=="power_supply", ACTION=="change", ATTR{online}=="0" , RUN+="/path/to/script/shown/above"
The answer below by Nuetrino shows how to detect the AC unplug event, this answer: How do I find out which process is preventing a umount? shows how to list and kill all processes blocking the device from unmounting (I had more success with lsof than fuser -c which sometimes didn't list any processes even though umount was still being blocked)
Details
Use udevadm monitor to log the event, e.g.
KERNEL[20154.545075] change /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/ADP0 (power_supply)
then use udevadm info -a -p with the event to get the attributes
udevadm info -a -p /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/ADP0
looking at device '/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/ADP0':
KERNEL=="ADP0"
SUBSYSTEM=="power_supply"
DRIVER==""
ATTR{online}=="0"
ATTR{type}=="Mains"
Now you can set up the udev rules with the attributes you like as answered below.
You can define udev rules to do it.
Just put your rule in /etc/udev/rules.d
Here is a example from me which i use to control brightness when i remove the AC adapter.
SUBSYSTEM=="power_supply", ACTION=="change", ATTR{online}=="0" , RUN+="/usr/local/bin/bright_unplug"
SUBSYSTEM=="power_supply", ACTION=="change", ATTR{online}=="1" , RUN+="/usr/local/bin/bright_replug
I run my custom script 'bright_replug' and 'bright_unplug' when I recieve kernel uvent.
You can use udevadm monitor to
monitor will print the received events for: UDEV - the event which
udev sends out after rule processing KERNEL - the kernel uevent
You can use udevadm info to match more attributes
Step by step solution
1.
Create this script e.g. in /home/user/uall.sh and replace mount_root with the folder where your distribution mounts usb drives, e.g. /media/user
#!/bin/bash
mount_root=/run/media/user
echo "Try unmounting.."
umount $(ls -d -1 $mount_root/*) # Unmount USB drives
mounted=$(ls -d -1 $mount_root/*) # Probe if there are still applications blocking
if ! [ -z "$mounted" ]
then
echo "Found blocked devices: $mounted, killing.."
kill -9 $(lsof -t $mounted) # Exit processes blocking umount cleanly
kill $(lsof -t $mounted) # Force kill remaining open processes
echo "Unmounting.."
umount $(ls -d -1 $mount_root/*) # Unmount USB drives
mounted=$(ls -d -1 $mount_root/*) # Probe if there are still applications blocking
fi
if [ -z "$mounted" ]
then
echo "Success!"
echo "All USB devices umount."
paplay /usr/share/sounds/speech-dispatcher/test.wav
else
echo "Error!"
echo "Tried it all but couldn't umount all USB devices."
echo "These devices are still mounted:"
echo "$mounted"
fi
2.
Create a udev wrapper script (call it /home/user/uall-udev-wrapper) that executes uall.sh as your username:
#!/bin/bash
runuser -l <user> -c '/home/user/uall.sh > /home/user/uall.log'
3.
Create the file /etc/udev/rules.d/99-usb-unmount.rules with the content
SUBSYSTEM=="power_supply", ACTION=="change", ATTR{online}=="0" , RUN+="/home/user/uall-udev-wrapper"
4.
Reboot or run sudo udevadm control --reload-rules && udevadm trigger to load the new udev rule
Optional. Add an alias uall=/home/user/uall.sh to your ~/.bashrc to access the script easily from your terminal and use your Desktop environment to configure a hotkey to quickly unmount all usb drives
Caveats
1.
When udev runs the script mount will not display gvfsd-fuse mount points, neither will cat /proc/mounts or cat /etc/mtab not even if using the runuser -l <user> wrapper. Instead I'm using now ls -d -1 $mount_root/* which only returns the devices mounted by the current user specified in $mount_root, on a multi user system devices mounted by a different user will not be unmounted by this script.
2.
When udev runs the script I do not get audio from paplay or spd-say.
Any input on these caveats would be greatly appreciated.

In Linux terminal, what would "grep -q '/dev/sda1' /proc/mounts || ..." do?

I want to make a bootable USB device by following this page.
http://planet-lab.org/node/172
It asks me to do these to steps:
umount /dev/sda*
grep -q /dev/sda1 /proc/mounts || dd if=PlanetLab-BootCD-3.3.usb of=/dev/sda1
But it makes me confused.
Since I think /dev/sda is my HDD, I thought it should be unmount /dev/sdb* in order to unmount USB device.
And I really don't understand what grep -q /dev/sda1 /proc/mounts is doing.
It seems to check whether "/dev/sda1" is mounted, but I don't exactly know what are the two parameters of grep command is doing. I know -q is for quiet.
And I also know dd can write an image to a drive.
The instructions state
assuming that the device is detected as /dev/sda
If the assumption is different from the actual mount point, you must modify the commands to match your configuration.
The grep -q is used to test for existence without cluttering the screen with the text which is found. The two parameters are
the text sought "/dev/sda1", and
the file in which the text is sought "/proc/mounts".
In other scripts, you may see something like
grep /dev/sda1 /proc/mounts >/dev/null
to achieve the same effect as the -q option.

Detecting USB Thumb Drive when Ready in Linux Shell Script

I am a Windows admin and dev, I do not generally work with Linux so forgive me if this is in some way obvious.
I have a not so good Linux box, some older version of Open SUSE, and I have a script that unmounts the USB thumb drive, formats it, and then waits for the device to become ready again before it runs a script that does a copy/MD5 checksum verification on the source and destination file to ensure the copy was valid. The problem is that on one box the USB thumb drive does not become ready after the format in a consistent way. It takes anywhere from 1 to 2+ minutes before I can access the drive via /media/LABELNAME.
The direct path is /dev/sdb but, of course, I cannot access it directly via this path to copy the files. Here is my shell script as it stands:
#!/bin/bash
set -e
echo "Starting LABELNAME.\n\nUnmounting /dev/sdb/"
umount /dev/sdb
echo "Formatting /dev/sdb/"
mkfs.vfat -I -F32 -n "LABELNAME" /dev/sdb
echo "Waiting on remount..."
sleep 30
echo "Format complete. Running make master."
perl /home/labelname_master.20120830.pl
Any suggestions? How might I wait for the drive to become ready and detect it? I have seen Detecting and Writing to a USB Key / Thumb DriveAutomatically but quite frankly I don't even know what that answer means.
It seems that you have some automatic mounting service running which detects the flash disk and mounts the partition. However, you already know what the partition is, so I recommend that you simply mount the disk in your script, choosing a suitable mount point yourself.
mkfs.vfat -I -F32 -n "LABELNAME" /dev/sdb
echo "Format complete, remounting"
mount /dev/sdb $mountpoint #<-- you would choose $mountpoint
echo "Running make master."
perl /home/labelname_master.20120830.pl

Bash script doesn't wait until commands have been properly executed

I am working on a very simple script but for some reason parts of it seem to run asynchronously.
singlePartDevice() {
# http://www.linuxquestions.org/questions/linux-software-2/removing-all-partition-from-disk-690256/
# http://serverfault.com/questions/257356/mdadm-on-ubuntu-10-04-raid5-of-4-disks-one-disk-missing-after-reboot
# Create single partition
parted -s "$1" mklabel msdos
# Find size of disk
v_disk=$(parted -s "$1" print|awk '/^Disk/ {print $3}'|sed 's/[Mm][Bb]//')
parted -s "$1" mkpart primary ext3 4096 ${v_disk}
parted -s "$1" set 1 raid on
return 0
}
singlePartDevice "/dev/sdc"
singlePartDevice "/dev/sdd"
#/dev/sdc1 exists but /dev/sdd1 does not exist
sleep 5s
#/dev/sdc1 exists AND /dev/sdd1 does also exist
As you see before the call of sleep the script has only partially finished its job. How do I make my script to wait until parted has done its job sucessfully?
(I am assuming that you are working on Linux due to the links in your question)
I am not very familiar with parted, but I believe that the partition device nodes are not created directly by it - they are created by udev, which is by nature an asynchronous procedure:
parted creates a partition
the kernel updates its internal state
the kernel notifies the udev daemon (udevd)
udevd checks its rule files (usually under /etc/udev/) and creates the appropriate device nodes
This procedure allows for clear separation of the device node handling policy from the kernel, which is a Good Thing (TM). Unfortunately, it also introduces relatively unpredictable delays.
A possible way to handle this is to have your script wait for the device nodes to appear:
while [ ! -e "/dev/sdd1" ]; do sleep 1; done
Assuming all you want to do is ensure that the partitions are created before proceeding, there are a couple of different approaches
Check whether process parted has completed before moving to the next step
Check if the devices are ready before moving to the next step (you will need to check the syntax). Eg
until [ -f /dev/sdc && -f /dev/sdd ]
sleep 5

Resources