Paramiko exec_command not working with mkfs? - python-3.x

Some issue executing the following bash with Paramiko:
def format_disk(self, device, size, dformat, mount, name):
stdin_, stdout_, stderr_ = self.client.exec_command(f"pvcreate {device};" \
f"vgcreate {name}-vg {device};" \
f"lvcreate -L {size} --name {name}-lv {name}-vg;" \
f"mkfs.{dformat} /dev/{name}-vg/{name}-lv;" \
f"mkdir {mount};" \
f"echo '/dev/{name}-vg/{name}-lv {mount} {dformat} defaults 0 0' >> /etc/fstab")
print(f"mkfs.{dformat} /dev/{name}-vg/{name}-lv;")
Print statement outputs: mkfs.ext4 /dev/first_try-vg/first_try-lv; If I copy and paste this exact command on the server there are no errors and it formats the disk as expected.
Troubleshooting steps
Server before running python script:
ls: cannot access /first_try: No such file or directory
[root#localhost ~]# vgs
[root#localhost ~]# lvs
[root#localhost ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Feb 25 07:32:51 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=38b7e96a-71e5-4089-a348-bd23828f9dc8 / xfs defaults 0 0
UUID=72fd2a6a-85db-4596-9fc2-6604d0d865a3 /boot xfs defaults 0 0
Server after running python script:
[root#localhost ~]# ls /first_try/
[root#localhost ~]# vgs
VG #PV #LV #SN Attr VSize VFree
first_try-vg 1 1 0 wz--n- <20.00g <15.00g
[root#localhost ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
first_try-lv first_try-vg -wi-a----- 5.00g
[root#localhost ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Feb 25 07:32:51 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=38b7e96a-71e5-4089-a348-bd23828f9dc8 / xfs defaults 0 0
UUID=72fd2a6a-85db-4596-9fc2-6604d0d865a3 /boot xfs defaults 0 0
/dev/first_try-vg/first_try-lv /first_try ext4 defaults 0 0
[root#localhost ~]# mount -a
mount: wrong fs type, bad option, bad superblock on /dev/mapper/first_try--vg-first_try--lv,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
The error from mount -a indicates that the disk is not formatted.
If I format the disk manually and run mount -a it works.
Example:
[root#localhost ~]# mkfs.ext4 /dev/first_try-vg/first_try-lv
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310720 blocks
65536 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): mdone
Writing superblocks and filesystem accounting information: done
[root#localhost ~]# mount -a
[root#localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 18G 4.7G 14G 27% /
devtmpfs 471M 0 471M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 487M 8.4M 478M 2% /run
tmpfs 487M 0 487M 0% /sys/fs/cgroup
/dev/sda1 297M 147M 151M 50% /boot
tmpfs 98M 12K 98M 1% /run/user/42
tmpfs 98M 0 98M 0% /run/user/0
/dev/mapper/first_try--vg-first_try--lv 4.8G 20M 4.6G 1% /first_try

Pariminko could not handle the output from mkfs. I changed the command to use the -q quiet flag and was able to get the script to run successfully.
New commmand mkfs -q -t {dformat} /dev/{name}-vg/{name}-lv

Related

Podman on RHEL 8 running out of space during import

I am having issues with Podman running out of space when importing. This is happening on a RHEL 8 VM that has been deployed for our group. We do have a 80GB /docker partition available, but I am missing some Podman configuration that says to use /docker. This VM
Can you all help me identify?
Here is part of my /etc/containers/storage.conf:
[storage]
# Default Storage Driver, Must be set for proper operation.
driver = "overlay"
# Temporary storage location
runroot = "/docker/temp"
# Primary Read/Write location of container storage
# When changing the graphroot location on an SELINUX system, you must
# ensure the labeling matches the default locations labels with the
# following commands:
# semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH
# restorecon -R -v /NEWSTORAGEPATH
# graphroot = "/var/lib/containers/storage"
graphroot = "/docker"
We are running SELinux, so I did run these commands:
semanage fcontext -a -e /var/lib/containers/storage /docker
restorecon -R -v /docker
and restart the podman service. However, if I run
podman import docker.tar
We receive the error:
Getting image source signatures
Copying blob 848eb673668a [=>------------------------------------] 1.8GiB / 41.3GiB
Error: writing blob: storing blob to file "/var/tmp/storage2140624383/1": write /var/tmp/storage2140624383/1: no space left on device
df -H shows:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 84K 3.9G 1% /dev/shm
tmpfs 3.9G 9.3M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/rhel_rhel86--svr-root 38G 7.2G 31G 20% /
/dev/mapper/rhel_rhel86--svr-tmp 4.7G 66M 4.6G 2% /tmp
/dev/mapper/rhel_rhel86--svr-home 43G 1.4G 42G 4% /home
/dev/sda2 495M 276M 220M 56% /boot
/dev/sdb1 79G 42G 33G 56% /docker
/dev/sda1 500M 5.9M 494M 2% /boot/efi
/dev/mapper/rhel_rhel86--svr-var 33G 1.6G 32G 5% /var
/dev/mapper/rhel_rhel86--svr-var_log 4.7G 109M 4.6G 3% /var/log
/dev/mapper/rhel_rhel86--svr-var_tmp 1.9G 47M 1.9G 3% /var/tmp
/dev/mapper/rhel_rhel86--svr-var_log_audit 9.4G 132M 9.2G 2% /var/log/audit
tmpfs 785M 8.0K 785M 1% /run/user/42
tmpfs 785M 0 785M 0% /run/user/1000
Do you guys know what I'm missing to tell Podman to use /docker instead of /var/tmp/storage2140624383 ?
################################################
Edited December 29:
I was able to change the tmpdir to /docker. However, upon import of this 54GB docker.tar file, it is still telling me I am running out of space. We were able to import a small .tar (around 800MB) successfully, so we know podman is working.
$ podman import docker.tar
Getting image source signatures
Copying blob b45265b317a7 done
Error: writing blob: adding layer with blob "sha256:b45265b317a7897670ff015b177bac7b9d5037b3cfb490d3567da959c7e2cf70": Error processing tar file(exit status 1): write /a65be6ac39ddadfec332b73d772c49d5f1b4fffbe7a3a419d00fd58fcb4bb752/layer.tar: no space left on device
This might be a pretty easy one:
Copying blob 848eb673668a [=>------------------------------------] 1.8GiB / 41.3GiB
vs
/dev/mapper/rhel_rhel86--svr-var_tmp 1.9G 47M 1.9G 3% /var/tmp
As you can see, the image will not fit into the desired temp space directory.
This is somewhat explained in the docs, which states, you can adjust this by changing the TMPDIR environment variable.

How to increase available filesystem size in /dev/root in Linux Centos 7.7 (AltArch)?

I am running a fresh installation of CentOS 7.7. on my Raspberry Pi 3 and unfortunatly I am running out of size when installing yum packages:
[root#centos7 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 1.4G 1.3G 104M 93% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 12M 452M 3% /run
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/mmcblk0p1 286M 57M 230M 20% /boot
tmpfs 93M 0 93M 0% /run/user/0
The OS is installed on a 16gb SD card, so in principle I should have more than enough space:
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 * 8192 593919 292864 c W95 FAT32 (LBA)
/dev/mmcblk0p2 593920 1593343 499712 82 Linux swap / Solaris
/dev/mmcblk0p3 1593344 4524031 1465344 83 Linux
I figured that I should be able to increase the size of the file system with growpart or resize2fs but I am not sure how to do this exactly and I don't want to break anything. What would be the best way to do this? Thanks!
You are on the right track. This is what I did for my Pi4 on a 64GB SD card:
[root#localhost ~]# df -h | head -n 2
Filesystem Size Used Avail Use% Mounted on
/dev/root 1.7G 1.4G 199M 88% /
[root#localhost ~]# fdisk -l
...
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 * 8192 593919 292864 c W95 FAT32 (LBA)
/dev/mmcblk0p2 593920 1593343 499712 82 Linux swap / Solaris
/dev/mmcblk0p3 1593344 5109759 1758208 83 Linux
[root#localhost ~]# growpart /dev/mmcblk0 3
CHANGED: partition=3 start=1593344 old: size=3516416 end=5109760 new: size=120041439 end=121634783
[root#localhost ~]# fdisk -l
...
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 * 8192 593919 292864 c W95 FAT32 (LBA)
/dev/mmcblk0p2 593920 1593343 499712 82 Linux swap / Solaris
/dev/mmcblk0p3 1593344 121634782 60020719+ 83 Linux
[root#localhost ~]# resize2fs /dev/mmcblk0p3
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mmcblk0p3 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 8
The filesystem on /dev/mmcblk0p3 is now 15005179 blocks long.
[root#localhost ~]# df -h | head -n 2
Filesystem Size Used Avail Use% Mounted on
/dev/root 57G 1.4G 55G 3% /
Well, this is an easy thing to do. I share with you my script to extend the /, https://github.com/sbaiidrissiyoussef/STORAGE/blob/master/extend_slash.sh
Upvote if this helps you.

"No space left on device" when using dd to create a disk image

I am trying to trying to create a disk image of my Raspberry Pi Model 3 B+ onto a USB drive using dd. I know there are easier ways to do this on a Raspberry Pi, but I want to try this to test the procedure on a 'sacrificial' system, which I hope to then use on another linux computer running a much larger Ubuntu disk to create a backup. OS is Raspbian Buster 10.
I have been following a procedure I found on an article here: https://www.makeuseof.com/tag/easily-clone-restore-linux-disk-image-dd/
The USB drive has 64GB capacity and has been formatted, initially as exFAT but I also tried NTFS thinking maybe that was the issue. The command ended with the same error, however each time i have tried this the file size transferred has been different, varying from 2-8GB in size before the error occurred.
This is to identify my drives - the SD card is "mmcblk" and my USB drive is "sda", called "NINJA":
pi#raspberrypi:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 57.9G 0 disk
└─sda1 8:1 1 57.9G 0 part
mmcblk0 179:0 0 14.9G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 14.6G 0 part /
This my command I tried to use:
pi#raspberrypi:~ $ sudo dd bs=4M if=/dev/mmcblk0 of=/media/pi/NINJA/raspibackup.img
and this is the output:
dd: error writing '/media/pi/NINJA/raspibackup.img': No space left on device
605+0 records in
604+0 records out
2535124992 bytes (2.5 GB, 2.4 GiB) copied, 325.617 s, 7.8 MB/s
Check how much disk space is "Avail" on the target device.
Example:
[jack#server1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 484M 0 484M 0% /dev
tmpfs 496M 41M 456M 9% /dev/shm
tmpfs 496M 6.9M 489M 2% /run
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root 6.2G 6.2G 172K 100% /
/dev/sda1 1014M 166M 849M 17% /boot
tmpfs 100M 24K 100M 1% /run/user/1000
/dev/sr0 552M 552M 0 100% /run/media/jack/CentOS 7 x86_64
Terminology:
df: DiskFree
-h: Human Readable Sizes (Ex: 6.2G instead of 6485900)
In this example, let's say I want to make a backup of the Boot drive (/dev/sda1) and save it in my Local User Home Folder on my Root Drive (/dev/mapper/centos-root).
When I so this, I will get an error that looks like:
[jack#server1 ~]$ sudo dd if=/dev/sda1 of=boot.img
dd: error writing 'boot.img': No space left on device
1736905+0 records in
1736904+0 records out
889294848 bytes (889 MB) copied, 4.76575 s, 187 MB/s
Terminology:
sudo: Super User Do
dd: Disk Duplicate
if: Input File (source)
of: Output File (destination)
The system is trying to copy ALL of /dev/sda1 (to include freespace) to boot.img, which is impossible at this because /dev/sda1 is 1014M and there is only 172K space left on /dev/mapper/centos-root.
With that said, the actual size of the /dev/sda is actually 16G total! Which means that there is 8G not allocated.
My /dev/sda1 should be 1G where my /dev/sda2 (centos-root) should be 15G... in which it is currently 6.2G
[jack#server1 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 16G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 15G 0 part
├─centos-root 253:0 0 6.2G 0 lvm /
└─centos-swap 253:1 0 820M 0 lvm [SWAP]
sr0 11:0 1 552M 0 rom /run/media/jack/CentOS 7 x86_64
This partition can be extended by doing the following:
[jack#server1 ~]$ sudo lvextend -L +8G /dev/mapper/centos-root
[jack#server1 ~]$ sudo xfs_growfs /dev/mapper/centos-root
Now that my partition is extended, I can do another DiskFree command to double check.
[jack#server1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 484M 0 484M 0% /dev
tmpfs 496M 33M 463M 7% /dev/shm
tmpfs 496M 6.9M 489M 2% /run
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root 15G 7.0G 7.3G 49% /
/dev/sda1 1014M 166M 849M 17% /boot
tmpfs 100M 24K 100M 1% /run/user/1000
/dev/sr0 552M 552M 0 100% /run/media/jack/CentOS 7 x86_64
My root partition is now 15G! Now I can perform my backup of the /dev/sda1 partition!
[jack#server1 ~]$ sudo dd if=/dev/sda1 of=boot.img
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 5.59741 s, 192 MB/s
Mission Complete!
sda1 is not mounted in /media/pi/NINJA/, the image you create is therefore stored on the mmcblk0p2 partition.
Since mmcblk0 is by definition larger than mmcblk0p2, you logically run out of space on it.
Solution :
You need to first mount sda1 using sudo mount /dev/sda1 /media/pi/NINJA/ and try your dd command again after.

Not able to resize the AWS EC2 volume

I created an AWS EC2 Linux instance with 8GB root volume. Then I increased the EBS volume to 9GB and it went to the completed state. It's a small volume, so the resize took a couple of minutes to complete.
Now I try to extend extend the linux file system after resizing the volume using the instructions mentioned here. But, I get the below error message. I tried two times, the entire process. But it's all the same.
The filesystem is already 2096635 (4k) blocks long. Nothing to do!
Here is the screen shot of the image.
Can someone help me?
Just reboot the instance because it automatically resizes your root filesystem on boot.
I tried it myself. Here is the instance with an 8GB volume:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 7.8G 985M 6.7G 13% /
After modifying the EBS Volume:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 9G 0 disk
└─xvda1 202:1 0 8G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 7.8G 985M 6.7G 13% /
After the reboot:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 9G 0 disk
└─xvda1 202:1 0 9G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 8.8G 984M 7.7G 12% /
See also: increase EC2 EBS volume after cloning - resize2fs not working
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage_expand_partition.html
# Before goging to do this, detach and attach the root volume to anothe instance
# Note:
# 1) Before detach the volume, please make a note of device name which going to
# detch from the machine, why because we should mention same name when attaching back, otherwise data will be lost
# 2)
# Identifying device name which we want to expand
lsblk
# Running parted command on the device
sudo parted /dev/xvdf
# Changing the parted units of measure to sectors.
unit s
# Run the print command to list the partitions on the device
print
# if it shows warning, chose fix
# Delete the partition entry for the partition using the number (1) from the previous step
rm 1 # number 1 will change based the partition we want to delete
# Create a new partition that extends to the end of the volume
mkpart Linux 4096s 100%
# Run the print command again to verify your partition
print
# Check to see that any flags that were present earlier are still
# present for the partition that you expanded. In some cases the boot
# flag may be lost. If a flag was dropped from the partition when it was expanded,
# add the flag with the following command, substituting your partition number and the flag name.
# For example, the following command adds the boot flag to partition 1
set 1 boot on
#Run the quit command to exit parted.
quit
# verfiying the device
sudo e2fsck -f /dev/xvdf1

xfs fllesystem remount doesn't work when modify quota configure

1.At the begining
mount | grep home
/dev/sdb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
2.try to modify
mount -o remount,rw,relatime,attr2,inode64,prjquota /dev/sbd1 /home
3.check it again
mount | grep home
/dev/sdb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
It doesn' work.
cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Aug 9 15:24:43 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=8f1038a3-6c31-4ce1-a9ef-3d7325e10bef / ext4 defaults 1 1
UUID=c687eab8-3ddd-4756-b91e-ad562b522f7c /boot ext4 defaults 1 2
UUID=7ae72a46-1407-49e6-8669-95bb9e592794 /home xfs rw,relatime,attr2,inode64,prjquota 0 0
UUID=3ccea12f-25d0-437b-9c4b-6ad6a9bd724c /tmp xfs defaults 0 0
UUID=b8ab4016-49bd-4f48-9620-5bda76f4d8b1 /var/log xfs defaults 0 0
UUID=8b9a7ada-3f02-4ee5-8010-ad32a5d7461e swap swap defaults 0 0
I can modify the /etc/fstab then restart machine make it work. But,is there any way I can change the quota configure without reboot?
Quotas
XFS quotas are not a remountable option. The -o quota option must be specified on the initial mount for quotas to be in effect.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/ch06s09.html
BTW if you need to enable quota for root partition /etc/fstab
does not help you only need to tweak kernel boot options

Resources