AWS EC2 ERROR: -bash: cannot create temp file for here-document: No space left on device - linux

I'm getting following error while doing tab. And unable to change the files, can open the files but not able to save the files. It looks like the server is working fine and long scripts are continuously executing and never ending.
ubuntu#ip-172-31-49-89:~$ cat /etc/issue.net
Ubuntu 14.04.4 LTS
ubuntu#ip-172-31-49-89:~$ uname -a
Linux ip-172-31-49-89 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ubuntu#ip-172-31-49-89:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 492M 12K 492M 1% /dev
tmpfs 100M 376K 99M 1% /run
/dev/xvda1 50G 47G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
ubuntu#ip-172-31-49-89:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 125803 387 125416 1% /dev
tmpfs 127040 315 126725 1% /run
/dev/xvda1 3276800 134435 3142365 5% /
none 127040 2 127038 1% /sys/fs/cgroup
none 127040 1 127039 1% /run/lock
none 127040 1 127039 1% /run/shm
none 127040 2 127038 1% /run/user
ubuntu#ip-172-31-49-89:~$ cd new
-bash: cannot create temp file for here-document: No space left on device
What should I do now? I just deleted my unnecessary founders and files. Still the same error after some time.
I tried solving this issue and my tmp and log files are smaller:
ubuntu#ip-172-31-49-89:~$ du -sh /tmp
1.3M /tmp
ubuntu#ip-172-31-49-89:~$ du -sh /var/log
6.2M /var/log

Well, you said it yourself: you disk is full:
/dev/xvda1 50G 47G 0 100% /
Start investigating where you disk space is gone from the root folder:
du -hs /* | sort -h
That should give a pretty good idea what's occupying most of the disk drive.

Related

Temp folder runs out of inodes

I have a LiteSpeed server with a Wordpress/woocommerce website on it.
Every few days my /tmp folder runs out of inodes, which effectively disables the website.
Here is what I get, after running df -i, please note the /dev/loop0 part.
root#openlitespeed:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 501653 385 501268 1% /dev
tmpfs 504909 573 504336 1% /run
/dev/vda1 7741440 888567 6852873 12% /
tmpfs 504909 6 504903 1% /dev/shm
tmpfs 504909 3 504906 1% /run/lock
tmpfs 504909 18 504891 1% /sys/fs/cgroup
/dev/vda15 0 0 0 - /boot/efi
/dev/loop0 96000 78235 17765 82% /tmp
tmpfs 504909 11 504898 1% /run/user/0
The output from df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 395M 616K 394M 1% /run
/dev/vda1 58G 43G 16G 74% /
tmpfs 2.0G 128K 2.0G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/loop0 1.5G 13M 1.4G 1% /tmp
tmpfs 395M 0 395M 0% /run/user/0
The folder is full of "sess_" files.
Right now I constantly monitor the folder and issue:
find /tmp/ -mindepth 1 -mtime +5 -delete
Which kinda helps, but is not ideal.
How can I reconfigure to increase the inodes?
Where to look?
Edit:
I have redis, and it's enabled in LiteSpeed Cache plugin.
It's all on DigitalOcean VPS, with 2CPUs and 4 GB RAM. Ubuntu 18.04.
Why does it even have just 96k inodes with the size of 1.5G, when tmpfs has much more?

/dev/vda1 is full but cannot find why

I have a server running Centos 7. This is the result of df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 139M 1.4G 10% /run
/dev/vda1 46G 44G 0 100% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
/dev/vda15 99M 3.6M 95M 4% /boot/efi
/dev/mapper/LVMVolGroup-DATA_VOLUME 138G 17G 114G 13% /mnt/data
tmpfs 1.5G 0 1.5G 0% /run/user/0
Even if there are 2GB of free space on / , it shows that the filesystem is at 100% of usage, and I can't install new packages because it tells me there's no space left on device.
Besides, if I type sudo du -sh /* | sort -rh | head -15
the result is:
17G /mnt
1.1G /usr
292M /var
208M /root
139M /run
49M /boot
48M /tmp
32M /etc
28K /home
16K /lost+found
12K /anaconda-post.log
4.0K /srv
4.0K /opt
4.0K /media
0 /sys
So it seems that there are no big files filling up the disk, and the sum of the sizes of the directories is not even equal to 44GB.
Additional info: the only service running on the server is Jenkins, but its home is under /mnt/data/jenkins.
How can I solve the problem?
Found the solution.
The problem was related to some deleted files kept open by Jenkins.
Restarting the service the problem was solved.
The problem was related to the system cache/temp storage. Linux system created the cache files and its archive from time to time, especially when some long option is run like DB import or crone job etc.. or sometimes server up from sines long.
Restarting the service or server
so due to that, the cache/ temp files were deleted and the problem was solved.
even in windows, we faced that kind of performance issue when RAM is low, and restarting the system is the primary solution for that.

cronjob : No space left on device

I have attached a new volume to an EC2 instance. Volume was attached successfully.Below the output of command.
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 32G 8.1G 22G 27% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 12K 2.0G 1% /dev
tmpfs 396M 340K 395M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 1.0M 0 100% /tmp
When i tried to add new cronjob it shows the error that there is no space left.
sudo crontab -e
/tmp/crontab.jVOoWT/crontab: No space left on device
Your /tmp directory is full, first remove the files from your temp directory by issuing the command below
rm -rf /tmp/*
Run your crontab again
sudo crontab -e
Please execute df -i may be inode's 100% full Remove unnecessary file from /var
run your crontab again
crontab -e
I had the same issue on AWS and ultimately the solution was to boost the capacity of the hard drive. Solved the issue.

Git Data backup from temp filder to a new disk (Disk3)

Hey guys I am new to this Git and Linux..please can you let me know, how to copy data backup (here is backup.tar) to a new disk, as my root directory is showing 100%.
my backup path is : /opt/gitlab-6.5.1-0/apps/gitlab/htdocs/tmp/backups$ls
1419371767_gitlab_backup.tar 1428271073_gitlab_backup.tar
1419458174_gitlab_backup.tar 1428357490_gitlab_backup.tar
1419544598_gitlab_backup.tar 1428443884_gitlab_backup.tar
1419631012_gitlab_backup.tar 1428530316_gitlab_backup.tar
1419717436_gitlab_backup.tar 1428616692_gitlab_backup.tar
1419803830_gitlab_backup.tar 1428703211_gitlab_backup.tar
1419890198_gitlab_backup.tar 1428789555_gitlab_backup.tar
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/server4--vg-root 451G 427G 1.2G 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 1.9G 4.0K 1.9G 1% /dev
tmpfs 376M 1016K 375M 1% /run
none 50M 0 50M 0% /run/lock
none 1.9G 4.0K 1.9G 1% /run/shm
none 100M 4.0K 100M 1% /run/user
/dev/sdb1 236M 65M 159M 29% /boot
/dev/sda1 917G 340G 531G 39% /mnt/disk2
/dev/sdc1 917G 75M 871G 1% /mnt/disk3
How do copy to disk3 and remove the backup from the old drive.
Downvoted because it's not a question about git per se, rather a question about Linux...
Anyway, use the mv command to move (or rename) files from one place to another. Or use the cp command to copy them then rm to delete.
Take a look at this tutorial for more help on how to move files around.

dont know as how to mount the file-system in ubuntu

I am new to ubuntu , please help me i need create /data and mount this file system /dev/sdb
I have no clue as to how to do it . I read about mount and unmount , but I am still unable to create.
Below is the current structure .
Pdpie#ubuntu:/dev$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 18G 4.0G 13G 24% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 984M 4.0K 984M 1% /dev
tmpfs 199M 1.5M 198M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 994M 152K 994M 1% /run/shm
none 100M 44K 100M 1% /run/user
I wanted to see like this below
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 18G 4.0G 13G 24% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 984M 4.0K 984M 1% /dev
tmpfs 199M 1.5M 198M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 994M 152K 994M 1% /run/shm
none 100M 44K 100M 1% /run/user
/dev/sdb 50M /data
Can anyone please help me step by step.
Thanks
You need to create your mount point first:
sudo mkdir /data
Then mount the sdb1 (if sdb1 is what you want) to it:
sudo mount /dev/sdb1 /data
Done
PS: To check which one you want to mount run sudo fdisk -l
Follow these steps
1.Create a directory which you want that is mkdir /data
2.Then you will use mount command to mount /data directory to sdb1 which is like this mount /dev/sdb1
3.Then to take it immediate effect use this command mount -a
4.Then to check it you can use df-h
What about the following?
# mkdir /data
# mount /dev/sdb1 /data
If you don't have /dev/sdb<number>, you'll have to create partitions with e.g. parted or fdisk.

Resources