Related to: How do I add instance storage to an existing Windows EC2 instance?
My root 60G ebs volume fill up very quickly, and I can't find the culprit in the file system. My actual files only take up about 10G. I've found out that if I "Stop" and then "Start" the instance, it frees up the remaining 50G.
Note: I started the instance as free micro and then later upgraded to m3.medium. Apparantly micro instances don't have ephemeral storage and you can only add "instance store" upon launching an instance. So I'm thinking I don't have access to ephemeral storage and that it is instead eating up my root ebs volume space with temporary files. Is that possible?
#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 59G 47G 13G 80% /
devtmpfs 1.9G 12K 1.9G 1% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
#du -sh /* | sort -n
0 /proc
0 /sys
4.0K /local
4.0K /media
4.0K /mnt
4.0K /selinux
4.0K /srv
7.3M /etc
8.0K /tmp
8.1M /bin
8.6G /var
9.7M /sbin
12K /dev
16K /lost+found
17M /root
21M /lib64
24K /run
26M /home
49M /boot
59M /opt
122M /lib
858M /usr
The problem had nothing to do with ephemeral storage. It was that httpd wasn't restarting after logrotate.
Related
I am having issues with Podman running out of space when importing. This is happening on a RHEL 8 VM that has been deployed for our group. We do have a 80GB /docker partition available, but I am missing some Podman configuration that says to use /docker. This VM
Can you all help me identify?
Here is part of my /etc/containers/storage.conf:
[storage]
# Default Storage Driver, Must be set for proper operation.
driver = "overlay"
# Temporary storage location
runroot = "/docker/temp"
# Primary Read/Write location of container storage
# When changing the graphroot location on an SELINUX system, you must
# ensure the labeling matches the default locations labels with the
# following commands:
# semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH
# restorecon -R -v /NEWSTORAGEPATH
# graphroot = "/var/lib/containers/storage"
graphroot = "/docker"
We are running SELinux, so I did run these commands:
semanage fcontext -a -e /var/lib/containers/storage /docker
restorecon -R -v /docker
and restart the podman service. However, if I run
podman import docker.tar
We receive the error:
Getting image source signatures
Copying blob 848eb673668a [=>------------------------------------] 1.8GiB / 41.3GiB
Error: writing blob: storing blob to file "/var/tmp/storage2140624383/1": write /var/tmp/storage2140624383/1: no space left on device
df -H shows:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 84K 3.9G 1% /dev/shm
tmpfs 3.9G 9.3M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/rhel_rhel86--svr-root 38G 7.2G 31G 20% /
/dev/mapper/rhel_rhel86--svr-tmp 4.7G 66M 4.6G 2% /tmp
/dev/mapper/rhel_rhel86--svr-home 43G 1.4G 42G 4% /home
/dev/sda2 495M 276M 220M 56% /boot
/dev/sdb1 79G 42G 33G 56% /docker
/dev/sda1 500M 5.9M 494M 2% /boot/efi
/dev/mapper/rhel_rhel86--svr-var 33G 1.6G 32G 5% /var
/dev/mapper/rhel_rhel86--svr-var_log 4.7G 109M 4.6G 3% /var/log
/dev/mapper/rhel_rhel86--svr-var_tmp 1.9G 47M 1.9G 3% /var/tmp
/dev/mapper/rhel_rhel86--svr-var_log_audit 9.4G 132M 9.2G 2% /var/log/audit
tmpfs 785M 8.0K 785M 1% /run/user/42
tmpfs 785M 0 785M 0% /run/user/1000
Do you guys know what I'm missing to tell Podman to use /docker instead of /var/tmp/storage2140624383 ?
################################################
Edited December 29:
I was able to change the tmpdir to /docker. However, upon import of this 54GB docker.tar file, it is still telling me I am running out of space. We were able to import a small .tar (around 800MB) successfully, so we know podman is working.
$ podman import docker.tar
Getting image source signatures
Copying blob b45265b317a7 done
Error: writing blob: adding layer with blob "sha256:b45265b317a7897670ff015b177bac7b9d5037b3cfb490d3567da959c7e2cf70": Error processing tar file(exit status 1): write /a65be6ac39ddadfec332b73d772c49d5f1b4fffbe7a3a419d00fd58fcb4bb752/layer.tar: no space left on device
This might be a pretty easy one:
Copying blob 848eb673668a [=>------------------------------------] 1.8GiB / 41.3GiB
vs
/dev/mapper/rhel_rhel86--svr-var_tmp 1.9G 47M 1.9G 3% /var/tmp
As you can see, the image will not fit into the desired temp space directory.
This is somewhat explained in the docs, which states, you can adjust this by changing the TMPDIR environment variable.
I have a server running Centos 7. This is the result of df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 139M 1.4G 10% /run
/dev/vda1 46G 44G 0 100% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
/dev/vda15 99M 3.6M 95M 4% /boot/efi
/dev/mapper/LVMVolGroup-DATA_VOLUME 138G 17G 114G 13% /mnt/data
tmpfs 1.5G 0 1.5G 0% /run/user/0
Even if there are 2GB of free space on / , it shows that the filesystem is at 100% of usage, and I can't install new packages because it tells me there's no space left on device.
Besides, if I type sudo du -sh /* | sort -rh | head -15
the result is:
17G /mnt
1.1G /usr
292M /var
208M /root
139M /run
49M /boot
48M /tmp
32M /etc
28K /home
16K /lost+found
12K /anaconda-post.log
4.0K /srv
4.0K /opt
4.0K /media
0 /sys
So it seems that there are no big files filling up the disk, and the sum of the sizes of the directories is not even equal to 44GB.
Additional info: the only service running on the server is Jenkins, but its home is under /mnt/data/jenkins.
How can I solve the problem?
Found the solution.
The problem was related to some deleted files kept open by Jenkins.
Restarting the service the problem was solved.
The problem was related to the system cache/temp storage. Linux system created the cache files and its archive from time to time, especially when some long option is run like DB import or crone job etc.. or sometimes server up from sines long.
Restarting the service or server
so due to that, the cache/ temp files were deleted and the problem was solved.
even in windows, we faced that kind of performance issue when RAM is low, and restarting the system is the primary solution for that.
I'm using a DS4 Azure VM (Ubuntu 14.04). It comes with a 56GB local SSD.
I need to set up a 25GB swapfile in this local SSD. When I do df -h in the VM, I can see that it seems to be mapped to the /mnt/ folder. Following is the entire output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 29G 22G 6.4G 77% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 14G 4.0K 14G 1% /dev
tmpfs 2.8G 472K 2.8G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 14G 0 14G 0% /run/shm
none 100M 0 100M 0% /run/user
none 64K 0 64K 0% /etc/network/interfaces.dynamic.d
/dev/sdb1 56G 97M 56G 1% /mnt
However, if I try to initialize a swapfile in /mnt, it still gets added to the available disk space in /dev/sda1.
What do I need to do to set up my swap file? An illustrative example would be great. Thanks in advance.
I normally use the following commands to set up a swapfile:
sudo fallocate -l 25G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Update:
I went into /etc/waagent.conf, and tweaked the followed:
# Format if unformatted. If 'n', resource disk will not be mounted.
ResourceDisk.Format=y
# File system on the resource disk
# Typically ext3 or ext4. FreeBSD images should use 'ufs2' here.
ResourceDisk.Filesystem=ext4
# Mount point for the resource disk
ResourceDisk.MountPoint=/mnt
# Create and use swapfile on resource disk.
ResourceDisk.EnableSwap=y
# Size of the swapfile.
ResourceDisk.SwapSizeMB=26000
After this, I resized (and consequently rebooted) my Azure VM from the portal. Currently I can't tell whether the settings have taken effect. Are my settings correct and what's the best way to ensure they've taken effect?
You are right, we should modify /etc/waagent.conf to add a swap file.
By modifying the /etc/waagent.conf file and setting the following 3 parameters a swap file will be created in the directory defined by ResourceDisk.MountPoint
ResourceDisk.Format=y
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=26000
Then we should restart walinuxagent:
service walinuxagent restart
Commands to show the new swap space in use after agent restart:
dmesg | grep swap
root#ubuntu:~# swapon -s
Filename Type Size Used Priority
/mnt/swapfile file 26623996 0 -1
root#ubuntu:~# df -Th
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.4G 12K 3.4G 1% /dev
tmpfs tmpfs 697M 412K 697M 1% /run
/dev/sda1 ext4 29G 869M 27G 4% /
none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup
none tmpfs 5.0M 0 5.0M 0% /run/lock
none tmpfs 3.5G 0 3.5G 0% /run/shm
none tmpfs 100M 0 100M 0% /run/user
/dev/sdb1 ext4 99G 26G 68G 28% /mnt
I resized (and consequently rebooted) my Azure VM from the portal
I resized my VM, and the swap file does not lose.
Are my settings correct and what's the best way to ensure they've
taken effect?
After modify the /etc/waagent.conf and restart walinuxagent, we can use swapon -s to check it.
Hey guys I am new to this Git and Linux..please can you let me know, how to copy data backup (here is backup.tar) to a new disk, as my root directory is showing 100%.
my backup path is : /opt/gitlab-6.5.1-0/apps/gitlab/htdocs/tmp/backups$ls
1419371767_gitlab_backup.tar 1428271073_gitlab_backup.tar
1419458174_gitlab_backup.tar 1428357490_gitlab_backup.tar
1419544598_gitlab_backup.tar 1428443884_gitlab_backup.tar
1419631012_gitlab_backup.tar 1428530316_gitlab_backup.tar
1419717436_gitlab_backup.tar 1428616692_gitlab_backup.tar
1419803830_gitlab_backup.tar 1428703211_gitlab_backup.tar
1419890198_gitlab_backup.tar 1428789555_gitlab_backup.tar
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/server4--vg-root 451G 427G 1.2G 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 1.9G 4.0K 1.9G 1% /dev
tmpfs 376M 1016K 375M 1% /run
none 50M 0 50M 0% /run/lock
none 1.9G 4.0K 1.9G 1% /run/shm
none 100M 4.0K 100M 1% /run/user
/dev/sdb1 236M 65M 159M 29% /boot
/dev/sda1 917G 340G 531G 39% /mnt/disk2
/dev/sdc1 917G 75M 871G 1% /mnt/disk3
How do copy to disk3 and remove the backup from the old drive.
Downvoted because it's not a question about git per se, rather a question about Linux...
Anyway, use the mv command to move (or rename) files from one place to another. Or use the cp command to copy them then rm to delete.
Take a look at this tutorial for more help on how to move files around.
I am new to ubuntu , please help me i need create /data and mount this file system /dev/sdb
I have no clue as to how to do it . I read about mount and unmount , but I am still unable to create.
Below is the current structure .
Pdpie#ubuntu:/dev$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 18G 4.0G 13G 24% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 984M 4.0K 984M 1% /dev
tmpfs 199M 1.5M 198M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 994M 152K 994M 1% /run/shm
none 100M 44K 100M 1% /run/user
I wanted to see like this below
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 18G 4.0G 13G 24% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 984M 4.0K 984M 1% /dev
tmpfs 199M 1.5M 198M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 994M 152K 994M 1% /run/shm
none 100M 44K 100M 1% /run/user
/dev/sdb 50M /data
Can anyone please help me step by step.
Thanks
You need to create your mount point first:
sudo mkdir /data
Then mount the sdb1 (if sdb1 is what you want) to it:
sudo mount /dev/sdb1 /data
Done
PS: To check which one you want to mount run sudo fdisk -l
Follow these steps
1.Create a directory which you want that is mkdir /data
2.Then you will use mount command to mount /data directory to sdb1 which is like this mount /dev/sdb1
3.Then to take it immediate effect use this command mount -a
4.Then to check it you can use df-h
What about the following?
# mkdir /data
# mount /dev/sdb1 /data
If you don't have /dev/sdb<number>, you'll have to create partitions with e.g. parted or fdisk.