How to create loop partition from already existing partition - linux

I believe /images/backups is using the space in /images ?
/dev/sdb1 820G 645G 135G 83% /images
/dev/loop0 296G 296G 0 100% /images/backups
I've a similar kind of partition in another machine /images which is 500G free, and I would want to take out 350G for /images/backups, how to do it ?
Is it right that, it is a simple loop mount which can give specified amount of space or we should create a NULL file of required size and mount ? If so, what are the mount options should be used to specify the size ?

You'll need to create the destination with a fixed size, but can use a "sparse file" which doesn't actually have any blocks written to it yet (and which thus doesn't actually consume space until you write to it).
For instance:
dd if=/dev/zero of=file.img bs=1 count=0 seek=20G
will create a sparse file preallocated to 20GB. That said, actually writing 20GB of zeros to disk up-front (making the file non-sparse) will be faster on writes and lead to less fragmentation.
This can be attached to a loopback device with the losetup command, have a filesystem created, and be mounted:
losetup /dev/loop1 file.img
mke2fs -j /dev/loop1
mount /dev/loop1 /mnt/somewhere
If you want to know if an existing file is sparse, the following will do the trick (on a system with GNU tools; some of the below is not supported in a pure POSIX environment):
{
read block_count block_size file_size
if (( block_count * block_size < file_size )) ; then
echo "Sparse"
else
echo "Non-Sparse"
fi
} < <(stat --format='%b %B %s'$'\n' /images/backups.img)

Related

gcloud instance disk space

I am trying to do some computing on cloud. For this I created a computing instance and then I attached an external storage with about 10TB. But it seemed that I did something wrong and I got only 200GB available for my datalab. Any comment will be helpful
To check this I used
df -h
and
sudo lsblk
Thanks.
As I can see from lsblk command, you have the right size of your datalab-pd disk.
But you can use only 196 Gb.
I think this may be because the file system does not occupy the entire disk space.
Need to extend the file system.
As an example if you have ext3 fs need to do:
- umount /dev/sdb # Unmount your disk
- e2fsck /dev/sdb # Check file system in your disk
- resize2fs /dev/sdb
resize2fs command without any parameters will extend filesystem to all free space on disk.
More info: https://access.redhat.com/articles/1196353

Backup entire disk in Ubuntu

I would like to make a backup of the entire HDD disk.
Step-by-step what I'am trying to do:
1) Check storage capacity (that is going to be backupped):
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 455G 157G 275G 37% /
2) Mount extra, empty hdd to /mnt/backup/
/dev/sdb 294G 63M 279G 1% /mnt/backup
3) Run backup (using lzop as the fastest compressor)
dd if=/dev/sda1 bs=4M conv=noerror iflag=noatime,nofollow | lzop -1 > /mnt/backup/dev-sda1.lzo
But the backup fails with error: lzop: No space left on device: <stdout>
The extra hdd being fulled with dev-sda1.lzo. But the original size of /dev/sda1 "157G" is obviously less than available on /dev/sdb "279G". Even without compression.
In /etc/fstab /dev/sda1 being mounted to "/":
UUID=8a49b90e-6115-43a6-9702-7620182bbbf5 / ext4 errors=remount-ro 0 1
Is it possible that "dd" is doing recursive copy of the "/mnt/backup/" folder and this leads to it fail ?
Please, advice
Thanks to Mark Setchell to show me the correct direction.
Finally the solution to create dump of the whole partition without spaces is:
dump -0a -z1 -f /mnt/hdd1/dev-sda1.dump.gz /dev/sda1
For 157 G partition of Ubuntu 14.04 + development files + database files "dump" takes 45 minutes (on 7200 rpm HDD) and the result file was 80 G (compression level = 1).

How to check a disk for partitions for use in a script in Linux?

I'm scripting something in Bash for Linux systems. How would I check a disk for partitions in a robust manner?
I could use grep, awk, or sed to parse the output from fdisk, sfdisk, etc., but this doesn't seem to be an exact science.
I could also check if there are partitions in /dev, but it is also possible that the partitions exist and haven't been probed yet (via partprobe, as an example).
What would you recommend?
I think I figured out a reliable way. I accidentally learned some more features of partprobe while reading the man page:
-d Don’t update the kernel.
-s Show a summary of devices and their partitions.
Used together, I can scan a disk for partitions without updating the kernel and get a reliable output to parse. It's still parsing text, but at least the output isn't as "human-oriented" as fdisk or sfdisk. This also is information as read from the disk and doesn't rely on the kernel being up-to-date on the partition status for this disk.
Take a look:
On a disk with no partition table:
# partprobe -d -s /dev/sdb
(no output)
On a disk with a partition table but no partitions:
# partprobe -d -s /dev/sdb
/dev/sdb: msdos partitions
On a disk with a partition table and one partition:
# partprobe -d -s /dev/sdb
/dev/sdb: msdos partitions 1
On a disk with a partition table and multiple partitions:
# partprobe -d -s /dev/sda
/dev/sda: msdos partitions 1 2 3 4 <5 6 7>
It is important to note that every exit status was 0 regardless of an existing partition table or partitions. In addition, I also noticed that the options cannot be grouped together (partprobe -d -s /dev/sdb works while partprobe -ds /dev/sdb does not).
Another option is to run:
lsblk
See https://unix.stackexchange.com/a/108951
you could also use:
parted /dev/sda print 1 &> /dev/null echo $?
if a partition (first partition) exist it return true and otherwise false

How To Mount A Hard Disk Of File-System Type "devtmpfs"

I'm trying to recover some data from a hard drive extracted from a broken laptop, and I'm having problems mounting the disk to my current system (Linux Mint). The hard disk I'm recovering from ran Debian. Simply, I'm confused as to how I can mount the hard drive to access the files, however it's not as simple as any other mount I've done. The following details struggles and information I've encountered.
I get the following outputs when trying to mount the hard drive with different file-system tags. I should add that the file-system type isn't automatically detected when using auto, and "sdb" is definitely the correct address for the disk (taken it from dmesg).
$ mount /dev/sdb /mnt/usb -t ntfs
NTFS signature is missing.
Failed to mount '/dev/sdb': Invalid argument
The device '/dev/sdb' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
The following returns the same message when all other common file-system tags are used:
$ sudo mount /dev/sdb usb -t ext2
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
The results from these commands led me to believe that there was an issue with the hard disk and it's partitions, however fdisk proved that it's partition's do seem to be valid and correct:
$ sudo fdisk /dev/sdb -l
Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002da94
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 475920383 237959168 83 Linux
/dev/sdb2 475922430 488396799 6237185 5 Extended
/dev/sdb5 475922432 488396799 6237184 82 Linux swap / Solaris
I then decided to try verify the file-system type of the hard drive, which seems to be "devtmpfs", which I got from the following command using df:
$ df /dev/sdb -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
udev devtmpfs 1014764 4 1014760 1% /dev
And so finally, I mount the hard drive using -t devtmpfs, which is successful in mounting however I'm left with a confusing file system very unlike from what I would expect from what was a standard debian set up.
It contains file folders such as "block","bus","char","disk","dri","mapper"... and files like "sda1","sdb","sdb1","tty","vcs".
I'm totally stumped as to how I should progress, and I'm pretty convinced the hard disk isn't broken and that I'm just mounting it incorrectly. How can I successfully mount the disk so I can access my files? Any help would be greatly appreciated.
Ok, you are trying to mount the entire disk instead of individual partitions, which is why you are getting the error. In short the command you need is:
mount /dev/sdb1 /mnt/usb
The file /dev/sdb references the entire disk as a block file. This includes the partition table at the start, which is why it can't find a filesystem. The file /dev/sdb1 references the first partition, which is where your filesystem will be. From the looks of your fdisk output, this is not an ntfs partition since this is a Windows filesystem and the partition is marked as Linux (most likely you will have ext4 unless you specifically set up something different).
To add a quick explanation of devtmpfs, this is a special filesystem which contains these block files which are specified by udev. You can google both for more information, but by now I'm sure you now know its not what you are looking for.

How to Compress or write zero's /dev/zero to a swap file?

We have a few linux based (Centos) virtual machines which are to be used as distributable virtual appliances. We want to be able to compress them as much as possible for distribution ( via tar.gz, zip, etc).
We've removed all unnecessary files (.log's, /tmp/*, /var/log/, etc) and have written /dev/zero to the free space on the disk.
Is it possible to write zeros via /dev/zero to the swap partitions and files? I know I would need to swapoff -a first. I'm worried about corrupting any internal structures.
Our vm uses both partition swap and file swap.
Also, are there any other strategies for reducing the size of a VM for distribution?
We need to support all of the hypervisor technologies (Xen, VMW, etc), so although the vendors tools maybe useful, I'm looking for strategies that are cross platform.
--- Thanks
You may want to write zeroes and then use mkswap to create an empty swap partition.
$ dd -if=/dev/zero of=/path/to/file bs=512 count=1
adjust the size that you want your files to be.
sudo swapoff -v /dev/sda2 <== The swap partition
sudo dd if=/dev/zero of=/dev/sda2 bs=512 status=progress
sudo mkswap /dev/sda2 UUID=46c1a133-bfdd-4695-a484-08fcf8286896 <== The original UUID of the swap partition

Resources