After running following command, related: How do I unpin and remove all IPFS content from my machine?:
# unpin and remove all IPFS content from my machine
ipfs pin ls --type recursive | cut -d' ' -f1 | ifne xargs -n1 ipfs pin rm
ipfs repo gc
Size of the .ipfs repo still remains as it is:
$ du -sh ~/.ipfs
1.4G .ipfs
I can do rm -rf ~/.ipfs && ipfs init but my IPFS id is re-initialized where I want to keep my original peer identity without removing it.
I want to completely clean content of the IPFS repo. How can I achieve it? I observe that after removing pinned objects still size of the ~/.ipfs is not changed.
The ipfs garbage collector isn’t enabled by default and caches can grow unrestrained unless garbage collection is run manually or enabled to run on a schedule.
Try running Garbage Collection via the command
ipfs repo gc
Note: pinned resources are never deleted by the garbage collector.
Read more over here: https://docs.ipfs.io/concepts/persistence/#garbage-collection.
removing pinned objects do not change repo size. Blocks will be deleted during GC later if they are not used by anything else like MFS.
If you are using flatfs datastore, then storage space will be freed. In badgerds they will be overwritten by newer content but usually not freed.
Related
I am not able to identify what is causing my ec2 disk space to reach 100% of capacity.
I have a script which deletes files in tmp folder.But still randomly sometimes my disk capacity reaches 100%.
I have attached the logs of df -i to show disk utilization.
Error
PM2 | Error: ENOSPC: no space left on device, write
PM2 | at Object.writeSync (fs.js:679:3)
PM2 | at Object.writeFileSync (fs.js:1393:26)
PM2 | at ProcessContainer (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:70:10)
PM2 | at Object.<anonymous> (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:103:3)
PM2 | at Module._compile (internal/modules/cjs/loader.js:999:30)
I am using command df -i to find
[![enter image description here][1]][1]
[![enter image description here][2]][2]
du -h -d 1
Check the user .pm2/logs directory, if your node app as errors or many regular logs this can increase disk space used.
I think that 8 Go is too small. I think you should upgrade your server to allocate more space. This will solved your problem.
If you can't or if you don't want to add disk space, you can take a look at the /var/log directory to delete some extra log. On long term, you can use logrotate to compress log files and upload compressed one to another place in order to keep /var/log as small a possible.
UPDATE
Also, i am not a specialist of ubuntu and snap, but your /snap directory is 2,1Go in size. You can check this to see if snap retain old version of snap package or if there is some cache that can be cleared.
Here is a bash script to remove old snaps version that i find here : https://www.debugpoint.com/clean-up-snap/
#!/bin/bash
#Removes old revisions of snaps
#CLOSE ALL SNAPS BEFORE RUNNING THIS
set -eu
LANG=en_US.UTF-8 snap list --all | awk '/disabled/{print $1, $3}' |
while read snapname revision; do
snap remove "$snapname" --revision="$revision"
done
You can also delete files in /var/lib/snapd/cache it's a snap cache that can be cleared.
But as i say, not a specialist of Ubuntu, so not tested.
You can use the dh utility
cd /
du -h -d 1
it will show the disk usage for every folder in /, then you can cd in the biggest ones and repeat the same.
You can also run
du | sort -n
and you'll get (after a while) all the folders size in the filesystem (ordered by ascending size). By my experience I'd take a first look at /home, /tmp and /var.
Context:
I am transferring a backup dir from Server A to Server B.(RHEL)
Directory size (to be transferred) on Server A: 48GB
Available space on Server B: 154GB
Command I'm using on Server A(user: root):
scp -r -C <nameof-backup-dir> user#severB:/path
Unexpected Behaviour:
The backup directory appears on the target server B #/path occupying all available 154GB of space.
Meanwhile the SCP run on the source server A terminates with an "Insufficent space message" for the remaining files.
Question/Help needed:
What am I doing wrong here?
What changes do I need to make to the SCP command to achieve the result?
One thing I can think of is that block sizes are different.
If block size on the destination machine is bigger, small files will occupy more space.
To find out block size :
sudo tune2fs -l /dev/sda1 | grep -i 'block size'
# Replace /dev/sda1 with your device (found out with command [df])
If it's indeed the case, you can recreate destination file system with the same block size as the source file system.
Ever since upgrading to Linux kernel 5.8 and later, I've been having problems with my system freezing up from running out of RAM, and it's all going to the pagecache.
I have a program that reorganizes the data from the OpenStreetMap planet_latest.osm.pbf file into a structure that's more efficient for my usage. However, because this file is larger than the amount of RAM on my system (60GB file versus 48GB RAM), the page cache fills up. Before kernel 5.8, the cache would reach full, and then keep chugging along (with an increase in disk thrashing). Since 5.8, the system freezes because it won't ever automatically release a page from the page cache (such as 30GB earlier in my sequential read of the planet_latest.osm.pbf file). I don't need to use my reorganizing program to hang the system; I found the following unprivileged command would do it:
cat planet_latest.osm.pbf >/dev/null
I have tried using the fadvise64() system service to manually force releases of pages in the planet file I have already passed; it helps, but doesn't entirely solve the problem with the various output files my program creates (especially when those temporary output files are randomly read back later).
So, what does it take to get the 5.8 through 5.10 Linux kernel to actually automatically release old pages from the page cache when system RAM gets low?
To work around the problem, I have been using a script to monitor cache size and write to /proc/sys/vm/drop_caches when the cache gets too large, but of course that also releases new pages I am currently using along with obsolete pages.
while true ; do
H=`free | head -2 | tail -1 | awk '{print $6}'`
if [ $H -gt 35000000 ]; then
echo -n $H " # " ; date
echo 1 >/proc/sys/vm/drop_caches
sensors | grep '°C'
H=`free | head -2 | tail -1 | awk '{print $6}'`
echo -n $H " # "; date
fi
sleep 30
done
(the sensors stuff is to watch out for CPU overheating in stages of my program that are multi-threaded CPU-intensive rather than disk-intensive).
I have filed a bug report at kernel.org, but they haven't looked at it yet.
When I run nodetool clearsnapshot I get the normal "Requested clearing snapshot(s)" message, but the snapshot is never removed. What can I do to troubleshoot why this is occurring? Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
nodetool clearsnapshot 1472489985541
Requested clearing snapshot(s) for [1472489985541]
nodetool listsnapshots | awk '{print $1}' | grep ^1 | sort -u
1472489985541
1473165734236
1473690660090
1474296554367
Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
Yes, you can always safely remove the snapshots directories manually. They are just hard links to actual SSTables
In order to delete a snapshot from all keyspaces using the snapshot name, you must specify the -t flag in your clearsnapshot command.
I need to do it for more predictable benchmarking.
Sounds like you want the sync command, or the sync() function.
If you want disk cache flushing: echo 3 | sudo tee /proc/sys/vm/drop_caches
You can do it like this:
# sync # (move data, modified through FS -> HDD cache) + flush HDD cache
# echo 3 > /proc/sys/vm/drop_caches # (slab + pagecache) -> HDD (https://www.kernel.org/doc/Documentation/sysctl/vm.txt)
# blockdev --flushbufs /dev/sda
# hdparm -F /dev/sda
# NEXT COMMAND IS NOT FOR BENCHMARKING:
# should be run before unplug, flushes everything possible guaranteed.
# echo 1 > /sys/block/sdX/device/delete
You may use strace to see that these are three different syscalls
Also, it may be desirable to turn off HDD cache using hdparm, not sure what thing you benchmarking.
In any way, you cannot prevent HDD to cache last 64/32/16 MB of recently used data. In order to kill that cache, just write some amount of zeroes (and flush) + read some unrelated place from HDD. This is required since cache may be divided to read-part and write-part. After that you can benchmark HDD.
Disk cache purging: echo 3 | sudo tee /proc/sys/vm/drop_caches
Command documentation: https://www.kernel.org/doc/Documentation/sysctl/vm.txt
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches
As this is a non-destructive operation, and dirty objects are not freeable, the user should run "sync" first in order to make sure all cached objects are freed.
Short good enough answer: (copy paste friendly)
DISK=/dev/sdX # <===ADJUST THIS===
sync
echo 3 > /proc/sys/vm/drop_caches
blockdev --flushbufs $DISK
hdparm -F $DISK
Explanation:
sync: From the man page: flush file system buffers. Force changed blocks to disk, update the super block.
echo 3 > /proc/sys/vm/drop_cache: from the kernel docs this will cause the kernel to drop clean caches
blockdev --flushbufs /dev/sda: from the man page: call block device ioctls [to] flush buffers.
hdparm -F /dev/sda: from the man page: Flush the on-drive write cache buffer (older drives may not implement this)
Although the blockdev and hdparm commands look similar according to an answer above they issue different ioctls to the device.
Long probably better way:
(I'll assume that you have formatted the disk but you can adapt these commands if you want to write directly to the disk)
Run this only once before the 1st benchmark:
MOUNT=/mnt/test # <===ADJUST THIS===
# create a file with psuedo-random data. We will read it
# to fill the read cache of the HDD with garbage
dd if=/dev/urandom of=$MOUNT/temp-hddread.tmp bs=64M count=16
Run this every time you want to empty the caches:
DISK=/dev/sdX # <===ADJUST THIS===
MOUNT=/mnt/test # <===AND THIS===
# create a file with psuedo-random data to fill the write cache
# of the disk with garbage. Delete it afterwards it's not useful anymore
dd if=/dev/urandom of=$MOUNT/temp-hddwrite.tmp bs=64M count=16
rm $MOUNT/temp-hddwrite.tmp
# see short good enough answer above
sync
echo 3 > /proc/sys/vm/drop_caches
blockdev --flushbufs $DISK
hdparm -F $DISK
# read the file with pseudo-random data to fill any read-cache
# the disk may have with garbage
dd if=$MOUNT/temp-hddread.tmp of=/dev/null
Run this when you're done.
MOUNT=/mnt/test # <===ADJUST THIS===
# delete the temporary file with pseudo-random data
rm $MOUNT/temp-hddread.tmp
Explanation:
The disk will probably have some H/W cache. Some disks by design or due to bugs may not clear their caches when you issue the blockdev and hdparm commands. To compensate we write and read pseudo-random data hopping to fill these caches so that any cached data are removed from them. How much data you need to fill the cache depends on its size. In the commands above I'm using dd to read/write 16*64MB=1024MB, adjust the arguments if your HDD may have bigger cache (data sheets and experimentation are your friend and it doesn't hurt to specify values above the actual size of the cache). I'm using /dev/urandom as a source for random data because it's fast and we don't care about true randomness (we only care for high entropy because the disk firmware may be using compression before storing data to the cache). I'm creating /mnt/test/temp-hddread.tmp from the start and use it every time I want to read enough random data. I'm creating and deleting /mnt/test/temp-hddwrite.tmp each time I want to write enough random data.
Credits
I've wrote this answer based on the best parts of the existing answers.
Unmounting and re-mounting the disk under test will reset all caches and buffers.