I tried to use fsync to write some file to SD card ASAP. However fsync does not actually block before the file is physically written to the SD card. It seems to take about 5-6 seconds before the data is actually on the SD card. However mount the file system (I tried ext3, ext4) with commit = 1 or sync option does seem to work, the data is safe after reboot in 1 second. My question is that is there anyway to achieve flushing without resort to partition wide solution? I'm using linux kernel 2.6.37. Thank you
If you want to be sure the content is written on the SD card, you should call blockdev with --flushbufs before exiting the program.
If you want to benchmark the writing process, you can call this after every write.
/sbin/blockdev --flushbufs $dev
Related
I am working on Embedded Linux System which mounts a SDCARD ( formatted as FAT32) and writes about 500KB of Data per second continuously .
Now to handle Power failures, we designed a battery which gives a backup of about 3 seconds .
This enables us to complete graceful shutdown .
We have only one process writing data to the SDCARD .So no other process will access the SDCARD in anyway.
During Power failure and during shutdown process
We use fflush and fsync and then finally fclose of the current file in which data is written to.
We even have verified with oscilloscope by probing the hardware lines that data transaction becomes active with the SDCARD when we call fflush and fsync during shutdown process.
ie cache data is being tried to be written to SDCARD during this time .
That makes me believe that fflush and fsync is implemented correctly .
Only thing i didnot do is umount the SDCARD as it takes longer time.
But still in about 10 tries , we can see the SDCARD root partition getting corrupted and files are missing .
Is umount necessary for a proper graceful shutdown .
What i understand is mount only creates a link of the inode of the root of this filesystem with rest of the file system .
and umount only removes the link .
What is the perfect way or steps to ensure protection of the SDCARD while shutting down in Embedded Linux .
Below is the code which gets executed during the shutdown process currently.
is umount call necessary .
if(fflush(file)<0)
printf("Failed fflush errno=%d",errno);
if(fsync(fileno(file))<0)
printf ("Failed fsync errno=%d",errno);
if(fclose(file)<0)
printf("Failed fclose errno=%d",errno);
file=NULL;
You are doing almost right, however, there is a catch: your SD device might require additional time to flush its own internal buffers.
The needed time is dependent on the precise chipset (and of course, brand, model, etc...).
In order to properly shutdown the system, you need to have a backup battery that allows you to be safe: consider that backup batteries on enterprise RAID cards ranges from 2 to 5 minutes, and the manufacturer knows exactly every hardware details.
Unfortunately, after fsync() you have to wait for more time. In similar circumstances, our RasperryPi devices are safe towards corruption with a backup battery that allows the device to operate for at least 20 seconds (although 10-15 is safe in our case) just to play easy.
Common way to write an image to disk looks like:
dd if=file.img of=/dev/device
After this command, is it necessary to run sync?
sync(2) explains it only flushes filesystem caches. Since dd command is not related to any filesystem, I think it is not necessary to run sync. However, block layer is complex and in doubt, most people prefers to run sync.
Does anyone has a proof that it is useful or useless?
TL;DR: Run blockdev --flushbufs /dev/device after dd.
I tried to follow the different paths in kernel. Here is what I understood:
ioctl(block_dev, BLKFLSBUF, 0) call blkdev_flushbuf(). Considering its name, it should flush caches associated with device (or I think you can consider there is bug in device driver). I think it should also responsible to flush hardware caches if they exist. Notice e2fsprogs use BLKFLSBUF.
fdatasync() (and fsync()) will call blkdev_fsync(). It looks like blkdev_flushbuf() but it only impact range of data written by current process (It use filemap_write_and_wait_range() while BLKFLSBUF use filemap_write_and_wait).
Closing a block device calls blkdev_close() that do not flush buffers.
sync() will call sync_fs(). It will flush filesystem caches and call fsync() on underlying block device.
Command sync /dev/device will call fsync() on /dev/device. However, I think it is useless since dd didn't touch to any filesystem.
So my conclusions is that call to sync has no (direct) impact on block device. However, passing fdatasync (or fsync) to dd is the only way to guarantee that data are correctly written on media.
If have you run dd but you missed to pass fdatasync, running sync /dev/device is not sufficient. You have have to run dd with fdatasync on whole device. Alternatively, you can call BLKFLSBUF to flush whole device. Unfortunately, there is no standard command for that.
EDIT
You can issue a BLKFLSBUF with blockdev --flushbufs /dev/device.
To ensure the data is flushed on a usb device before to unplug, I use the following command :
echo 1 > /sys/block/${device}/device/delete
This way, the data is flushed, and if the device is a hard drive, then the head is parked.
We have a driver writing data directly to sectors of disk by sending the BIO through “make_request_fn” of other block device from block layer.
But somehow the data did not write down to the disk instantly, when I reboot the machine, the data what I wrote to disk before reboot are gone.
The data can reflect correctly on next reboot ONLY if one of following , before reboot,
I use function to flush the block device (from program) after writing
Drop the system cache by “fsyhc ; echo 1 > /proc/sys/vm/drop_caches” after writing
We also tried following ways, but they all did not work.
Adding flag like flush/FUA on BIO flag
Calling blkdev_issue_flush()
This happened on both real machine and virtual environment like VMware.
OS is Ubuntu 14.4.3, with kernel 3.19 and Ext4 file system.
I am wondering if somebody can explain the reason and help me out.
Using linux, I can use raw access to NAND or access to files through filesystem. So, when I need to know, where my file is really located in NAND, what should I do? I cannot found any utilities providing this feature. Moreover, I cannot detect any possibility of this, besides hacking kernel with tons of "printk" (it's not nice way, I guess).
Can anybody enlighten me on this? (I'm using YAFFS2 and JFFS2 filesystems)
You can make a copy of any partition with nanddump. Transfer that partition dump to a PC. The nandsim utility can be used to mount the partitions on a PC.
modprobe nandsim first_id_byte=0x2c second_id_byte=0xda \
third_id_byte=0x90 fourth_id_byte=0x95 parts=2,64,64
flash_erase /dev/mtd3 0 0
ubiformat /dev/mtd3 -f rootfs.ubi
This command emulates a Micron 256MB NAND flash with four partitions. If you just capture the single partition and not the whole device, don't set parts. You can also do nanddump on each partition and then concatenate them all. This code targeted mtd3 with a UbiFs partition. For JFFS2 or YAFFS2, you can try nandwrite or some other appropriate flashing utility on the PC.
How to get real file offset in NAND by file name?
The files may span several NAND sectors and they are almost never contiguous. There is not much of an advantage to keep file data together as there is no disk head that takes physical time to seek. Some flash has marginally better efficiency for sequential reads; yet other flash will give better performance for reads from another erase block.
I would turn on debug at either the MTD layer or in the filesystem. In a live system, the position of the file may migrate over time on the flash even if it is not written. This is active wear leveling.
Title pretty much nailed it, Im copying files to a flash drive and then doing some things to those files. Well I have noticed that after running the dd command the flash drive is still flashing and not all the files are on the device.
Does anyone know how to maybe run a simple loop (in script) to wait on the dd process to finish? I have been googling for about 2-3 hours now trying to figure it out and its beyond me if its even possible.
Thanks in advance!
Try the sync command:
sync writes any data buffered in memory out to disk. This can
include (but is not limited to) modified superblocks, modified inodes,
and delayed reads and writes. This must be implemented by the kernel;
The sync program does nothing but exercise the sync system call.
The kernel keeps data in memory to avoid doing (relatively slow)
disk reads and writes. This improves performance, but if the computer
crashes, data may be lost or the file system corrupted as a result.
The sync command ensures everything in memory is written to disk.
Most likely you are seeing the operating system caching the writes. If you really want to make sure that everything is written to the flash drive so that it is safe to remove, it needs to be unmounted.