I am not able to identify what is causing my ec2 disk space to reach 100% of capacity.
I have a script which deletes files in tmp folder.But still randomly sometimes my disk capacity reaches 100%.
I have attached the logs of df -i to show disk utilization.
Error
PM2 | Error: ENOSPC: no space left on device, write
PM2 | at Object.writeSync (fs.js:679:3)
PM2 | at Object.writeFileSync (fs.js:1393:26)
PM2 | at ProcessContainer (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:70:10)
PM2 | at Object.<anonymous> (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:103:3)
PM2 | at Module._compile (internal/modules/cjs/loader.js:999:30)
I am using command df -i to find
[![enter image description here][1]][1]
[![enter image description here][2]][2]
du -h -d 1
Check the user .pm2/logs directory, if your node app as errors or many regular logs this can increase disk space used.
I think that 8 Go is too small. I think you should upgrade your server to allocate more space. This will solved your problem.
If you can't or if you don't want to add disk space, you can take a look at the /var/log directory to delete some extra log. On long term, you can use logrotate to compress log files and upload compressed one to another place in order to keep /var/log as small a possible.
UPDATE
Also, i am not a specialist of ubuntu and snap, but your /snap directory is 2,1Go in size. You can check this to see if snap retain old version of snap package or if there is some cache that can be cleared.
Here is a bash script to remove old snaps version that i find here : https://www.debugpoint.com/clean-up-snap/
#!/bin/bash
#Removes old revisions of snaps
#CLOSE ALL SNAPS BEFORE RUNNING THIS
set -eu
LANG=en_US.UTF-8 snap list --all | awk '/disabled/{print $1, $3}' |
while read snapname revision; do
snap remove "$snapname" --revision="$revision"
done
You can also delete files in /var/lib/snapd/cache it's a snap cache that can be cleared.
But as i say, not a specialist of Ubuntu, so not tested.
You can use the dh utility
cd /
du -h -d 1
it will show the disk usage for every folder in /, then you can cd in the biggest ones and repeat the same.
You can also run
du | sort -n
and you'll get (after a while) all the folders size in the filesystem (ordered by ascending size). By my experience I'd take a first look at /home, /tmp and /var.
I have an odroid (raspberry-like) machine with an arch linux system installed. Now I want to move the system from one microsd (A) to another microsd (B). When I tried this, the system became corrupted, information about files attributes were lost:
Copy files from A to osx-host cp -R /Volume/microsd_a/* ~/Desktop/backup
Copy files from osx-host to B cp -R ~/Desktop/backup/* /Volume/microsd_b
Is it real to copy linux-system using osx-host with preserving integrity?
Update:
dd. I tried this way, but there is a problem. My sd cards have different sizes, 64 Gb and 16 Gb, but system installed on 64 Gb disk has no more than 8 Gb. So when I launched the copying process, output image file exceed 16 Gb and I killed the process. By the way, the MBR contains information about partition table which should be different (one partition 64Gb / one partition 16 gb). And notice, I do not need to copy bootloader from MBR, I have an ability to flash disk bootloader by other ways.
cp. What I wanted to listen as the answer is the list of flags I need to make this operation. Reading man cp didn't help me. cp -a does not copy all files because of Cannot allocate memory error. Tried cp -aX, no attributes were restored after copying data to second sdcard.
tar. I tried multiple times with flags, last one was tar -cvpf; tar --same-owner -xpf. But file attributes were still corrupted.
Again:
- Are you sure, it is possible to preserve file attributes through copying ext4 -> APFS -> ext4?
- If this is possisble, how does it work and which command with which flags should I use?
cp -R results in change of permissions, time stamps and missing of hidden files, you can't use that command to create a disk image.
what you need is a disk copy/clone. The command to use is dd.
Check out this webpage:
https://pbxbook.com/other/dd_clone.html
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last year.
The community reviewed whether to reopen this question last year and left it closed:
Original close reason(s) were not resolved
Improve this question
I've got some samba drives that are being accessed by multiple users daily. I already have code to recognize shared drives (from a SQL table) and mount them in a special directory where all users can access them.
I want to know, if I remove a drive from my SQL table (effectively taking it offline) how, or even is, there a way to unmount a busy device? So far I've found that any form of umount does not work.
Ignoring the possibility of destroying data - is it possible to unmount a device that is currently being read?
YES!! There is a way to detach a busy device immediately - even if it is busy and cannot be unmounted forcefully. You may cleanup all later:
umount -l /PATH/OF/BUSY-DEVICE
umount -f /PATH/OF/BUSY-NFS (NETWORK-FILE-SYSTEM)
NOTE/CAUTION
These commands can disrupt a running process, cause data loss OR corrupt open files. Programs accessing target DEVICE/NFS files may throw errors OR could not work properly after force unmount.
Do not execute above umount commands when inside mounted path (Folder/Drive/Device) itself. First, you may use pwd command to validate your current directory path (which should not be the mounted path), then use cd command to get out of the mounted path - to unmount it later using above commands.
If possible, let us locate/identify the busy process, kill that process and then unmount the samba share/ drive to minimize damage:
lsof | grep '<mountpoint of /dev/sda1>' (or whatever the mounted device is)
pkill target_process (kills busy proc. by name | kill PID | killall target_process)
umount /dev/sda1 (or whatever the mounted device is)
Make sure that you aren't still in the mounted device when you are trying to umount.
Avoid umount -l
At the time of writing, the top-voted answer recommends using umount -l.
umount -l is dangerous or at best unsafe. In summary:
It doesn't actually unmount the device, it just removes the filesystem from the namespace. Writes to open files can continue.
It can cause btrfs filesystem corruption
Work around / alternative
The useful behaviour of umount -l is hiding the filesystem from access by absolute pathnames, thereby minimising further moutpoint usage.
This same behaviour can be achieved by mounting an empty directory with permissions 000 over the directory to be unmounted.
Then any new accesses to filenames in the below the mountpoint will hit the newly overlaid directory with zero permissions - new blockers to the unmount are thereby prevented.
First try to remount,ro
The major unmount achievement to be unlocked is the read-only remount. When you gain the remount,ro badge, you know that:
All pending data has been written to disk
All future write attempts will fail
The data is in a consistent state, should you need to physcially disconnect the device.
mount -o remount,ro /dev/device is guaranteed to fail if there are files open for writing, so try that straight up. You may be feeling lucky, punk!
If you are unlucky, focus only on processes with files open for writing:
lsof +f -- /dev/<devicename> | awk 'NR==1 || $4~/[0-9]+[uw -]/'
You should then be able to remount the device read-only and ensure a consistent state.
If you can't remount read-only at this point, investigate some of the other possible causes listed here.
Read-only re-mount achievement unlocked 🔓☑
Congratulations, your data on the mountpoint is now consistent and protected from future writing.
Why fuser is inferior to lsof
Why not use use fuser earlier? Well, you could have, but fuser operates upon a directory, not a device, so if you wanted to remove the mountpoint from the file name space and still use fuser, you'd need to:
Temporarily duplicate the mountpoint with mount -o bind /media/hdd /mnt to another location
Hide the original mount point and block the namespace:
Here's how:
null_dir=$(sudo mktemp --directory --tmpdir empty.XXXXX")
sudo chmod 000 "$null_dir"
# A request to remount,ro will fail on a `-o bind,ro` duplicate if there are
# still files open for writing on the original as each mounted instance is
# checked. https://unix.stackexchange.com/a/386570/143394
# So, avoid remount, and bind mount instead:
sudo mount -o bind,ro "$original" "$original_duplicate"
# Don't propagate/mirror the empty directory just about hide the original
sudo mount --make-private "$original_duplicate"
# Hide the original mountpoint
sudo mount -o bind,ro "$null_dir" "$original"
You'd then have:
The original namespace hidden (no more files could be opened, the problem can't get worse)
A duplicate bind mounted directory (as opposed to a device) on which
to run fuser.
This is more convoluted[1], but allows you to use:
fuser -vmMkiw <mountpoint>
which will interactively ask to kill the processes with files open for writing. Of course, you could do this without hiding the mount point at all, but the above mimicks umount -l, without any of the dangers.
The -w switch restricts to writing processes, and the -i is interactive, so after a read-only remount, if you're it a hurry you could then use:
fuser -vmMk <mountpoint>
to kill all remaining processes with files open under the mountpoint.
Hopefully at this point, you can unmount the device. (You'll need to run umount on the mountpoint twice if you've bind mounted a mode 000 directory on top.)
Or use:
fuser -vmMki <mountpoint>
to interactively kill the remaining read-only processes blocking the unmount.
Dammit, I still get target is busy!
Open files aren't the only unmount blocker. See here and here for other causes and their remedies.
Even if you've got some lurking gremlin which is preventing you from fully unmounting the device, you have at least got your filesystem in a consistent state.
You can then use lsof +f -- /dev/device to list all processes with open files on the device containing the filesystem, and then kill them.
[1] It is less convoluted to use mount --move, but that requires mount --make-private /parent-mount-point which has implications. Basically, if the mountpoint is mounted under the / filesystem, you'd want to avoid this.
Try the following, but before running it note that the -k flag will kill any running processes keeping the device busy.
The -i flag makes fuser ask before killing.
fuser -kim /address # kill any processes accessing file
unmount /address
Before unmounted the filesysem. we need to check is any process holding or using the filesystem. That's why it show device is busy or filesystem is in use.
run below command to find out the processes using by a filesystem:
fuser -cu /local/mnt/
It will show how many processes holding/using the filesystem.
local/mnt: 1725e(root) 5645c(shasankarora)
ps -ef | grep 1725 <--> ps -ef | grep <pid>
kill -9 pid
Kill all the processes and then you will able to unmount the partition/busy device.
Check for exported NFS file systems with exportfs -v. If found, remove with exportfs -d share:/directory. These don't show up in the fuser/lsof listing, and can prevent umount from succeeding.
Just in case someone has the same pb. :
I couldn't unmount the mount point (here /mnt) of a chroot jail.
Here are the commands I typed to investigate :
$ umount /mnt
umount: /mnt: target is busy.
$ df -h | grep /mnt
/dev/mapper/VGTout-rootFS 4.8G 976M 3.6G 22% /mnt
$ fuser -vm /mnt/
USER PID ACCESS COMMAND
/mnt: root kernel mount /mnt
$ lsof +f -- /dev/mapper/VGTout-rootFS
$
As you can notice, even lsof returns nothing.
Then I had the idea to type this :
$ df -ah | grep /mnt
/dev/mapper/VGTout-rootFS 4.8G 976M 3.6G 22% /mnt
dev 2.9G 0 2.9G 0% /mnt/dev
$ umount /mnt/dev
$ umount /mnt
$ df -ah | grep /mnt
$
Here it was a /mnt/dev bind to /dev that I had created to be able to repair my system inside from the chroot jail.
After umounting it, my pb. is now solved.
Check out umount2:
Linux 2.1.116 added the umount2() system call, which, like umount(),
unmounts a target, but allows additional flags controlling the
behaviour of the operation:
MNT_FORCE (since Linux 2.1.116) Force unmount even if busy. (Only for
NFS mounts.)
MNT_DETACH (since Linux 2.4.11) Perform a lazy unmount:
make the mount point unavailable for new accesses, and actually
perform the unmount when the mount point ceases to be busy.
MNT_EXPIRE (since Linux 2.6.8) Mark the mount point as expired. If a mount point
is not currently in use, then an initial call to umount2() with this
flag fails with the error EAGAIN, but marks the mount point as
expired. The mount point remains expired as long as it isn't accessed
by any process. A second umount2() call specifying MNT_EXPIRE unmounts
an expired mount point. This flag cannot be specified with either
MNT_FORCE or MNT_DETACH.
I recently had a similar need to unmount in order to change it's label with gparted.
/dev/sda1 was being mounted via /etc/fstab as /media/myusername. When attempts to unmount failed, I researched the error. I had forgotten to unmount a dual partitioned thumb drive with a mountpoint on /dev/hda1 first.
I gave 'lsof' a go as recommended.
$ sudo lsof | grep /dev/sda1
The output of which was:
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
lsof: WARNING: can't stat() fuse file system /run/user/1000/doc
Output information may be incomplete.
Since lsof burped up two fuse warnings, I poked around in /run/user/1000/*, and took a guess that it could be open files or mount points (or both) interfering with things.
Since the mount points live in /media/, I tried again with:
$ sudo lsof | grep /media
The same two warnings, but this time it returned additional info:
bash 4350 myusername cwd DIR 8,21 4096 1048577 /media
sudo 36302 root cwd DIR 8,21 4096 1048577 /media
grep 36303 myusername cwd DIR 8,21 4096 1048577 /media
lsof 36304 root cwd DIR 8,21 4096 1048577 /media
lsof 36305 root cwd DIR 8,21 4096 1048577 /media
Still scratching my head, it was at this point I remembered the thumb drive sticking out of the USB port. Maybe the scratching helped.
So I unmounted the thumb drive partitions (unmounting one automatically unmounted the other) and safefly unplugged the thumb drive. After doing so, I was able to unmount /dev/sda1 (having nothing mounted on it anymore), relabel it with gparted, remount both the drive and thumb drive with no issues whatsoever.
Bacon saved.
Someone has mentioned that if you are using terminal and your current directory is inside the path which you want to unmount, you will get the error.
As a complementary, in this case, your lsof | grep path-to-be-unmounted must have below output:
bash ... path-to-be-unmounted
sudo fusermount -u -z <mounted path>
NB: do not use completition for the path as this will also freeze the terminal.
Another alternative when anything works is editing /etc/fstab, adding noauto flag and rebooting the machine. The device won't be mounted, and when you're finished doing whatever, remove flag and reboot again.
Niche Answer:
If you have a zfs pool on that device, at least when it's a file-based pool, lsof will not show the usage. But you can simply run
sudo zpool export mypool
and then unmount.
Multiple mounts inside a folder
An additional reason could be a secondary mount inside your primary mount folder, e.g. after you worked on an SD card for an embedded device:
# mount /dev/sdb2 /mnt # root partition which contains /boot
# mount /dev/sdb1 /mnt/boot # boot partition
Unmounting /mnt will fail:
# umount /mnt
umount: /mnt: target is busy.
First we have to unmount the boot folder and then the root:
# umount /mnt/boot
# umount /mnt
In my case, I couldn't unmount a partition that was mounted to a directory that was an AFP share. (sharing into an Apple bonjour/avahi mdns world)
I moved all the logins on the server to their home directory; I moved all the remotely connected Macs to some other directory.
I still couldn't unmount the partition even with umount -f
So I restarted the netatalk daemon on the server.
(/etc/netatalk/afp.conf has in it the share assignment)
After the netatalk restart, umount succeeded without the -f.
I am running Ubuntu 10.4 and am mounting a drive using cifs. The command I'm using is:
'sudo mount -t cifs -o workgroup="workgroup",username="username",noserverino,ro //"drive" "mount_dir"'
(Obviously with "" values substituted for actual values)
When I then run the command ls -i I get: 394070
Running it a second time I get: 12103522782806018
Is there any reason to expect the inode value to change?
Running ls -i --full-time shows no change in modification time.
noserverino tells your mount not to use server-generated inode numbers, and instead use client-generated temporary inode numbers, to make up for them. Try with serverino, if your server and the exported filesystem support inode numbers, they should be persistent.
I found that using the option "nounix" before the "noserverino" kept the inodes small and persistent. I'm not really sure why this happened. The server is AIX and I'm running it from Ubuntu. Thank you for your response.