I copied one of the existing project and created a new project in Hudson. While running build it says "returned status code 141: fatal: write error: No space left on device"
Like suggested in other forums I checked free space and inode used in file system and nothing seems problematic here. Hudson is running as service and Hudons user has been given sudo privilege. Older job can be run so nothing different in new cloned job.
Disk Space
bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root
20G 19G 28K 100% /
tmpfs 1.9G 192K 1.9G 1% /dev/shm
/dev/sda1 485M 83M 377M 19% /boot
/dev/mapper/vg_dev-lv_home
73G 26G 44G 38% /home
i-nodes used
bash-4.1$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_dev-lv_root
1310720 309294 1001426 24% /
tmpfs 490645 4 490641 1% /dev/shm
/dev/sda1 128016 46 127970 1% /boot
/dev/mapper/vg_dev-lv_home
4833280 117851 4715429 3% /home
Hudson build log
bash-4.1$ cat log
Started by user anonymous
Checkout:workspace / /var/lib/hudson/jobs/Demo/workspace - hudson.remoting.LocalChannel#1d4ab266
Using strategy: Default
Checkout:workspace / /var/lib/hudson/jobs/Demo/workspace - hudson.remoting.LocalChannel#1d4ab266
Fetching changes from the remote Git repository
Fetching upstream changes from ssh://demouser#10.10.10.10:20/home/git-repos/proj.git
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ERROR: (Underlying report) : Error performing command: git fetch -t ssh://demouser#10.10.10.10:20/home/git-repos/proj.git +refs/heads/*:refs/remotes/origin/*
Command "git fetch -t ssh://demouser#10.10.10.10:20/home/git-repos/proj.git +refs/heads/*:refs/remotes/origin/*" returned status code 141: fatal: write error: No space left on device
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ERROR: Could not fetch from any repository
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=FATAL: Could not fetch from any repository
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=hudson.plugins.git.GitException: Could not fetch from any repository
at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:887)
at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:845)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.plugins.git.GitSCM.gerRevisionToBuild(GitSCM.java:845)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:622)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
at hudson.model.Run.run(Run.java:1366)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
Your error message is quite clear: There is no space left on device.
This is verified by your df output:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root 20G 19G 28K 100% /
This tells you, you have a root partition / with a total size of 20GB which is use by 100%.
20GB is probably a bit small in your case. As this "partition" is managed by LVM (/dev/mapper/vg...) it is possible to extend it to create more space for your data.
Otherwise you have to check, if there is some "garbage" laying around which can be removed.
You can use something like xdiskusage / to find out, what is occupying your precious disk space.
But if you don't understand the concept of a file system, maybe it is easier to find someone else to do it for you.
I had a very similar issue, it turned out to be a 40 gig log file from a "neverending" build which had been running for 8 hours
Related
when i'm writing df -h in my instance i'm getting this data:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.7G 0 7.7G 0% /dev
tmpfs 7.7G 0 7.7G 0% /dev/shm
tmpfs 7.7G 408K 7.7G 1% /run
tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup
/dev/nvme0n1p1 32G 24G 8.5G 74% /
tmpfs 1.6G 0 1.6G 0% /run/user/1000
but when i'm clicking sudo du -sh / i'm getting:
11G /
So in df -h, / size is 24G but in du -sh same directory the size is 11G.
I'm trying to get some free space on my instance and can't find the files that cause that.
What i'm missing?
did df -h is really giving fake data?
This question comes up quite often. The file system allocates disk blocks in the file system to record its data. This data is referred to as metadata which is not visible to most user-level programs (such as du). Examples of metadata are inodes, disk maps, indirect blocks, and superblocks.
The du command is a user-level program that isn't aware of filesystem metadata, while df looks at the filesystem disk allocation maps and is aware of file system metadata. df obtains true filesystem statistics, whereas du sees only a partial picture.
There are many causes on why the disk space used or available when running the du or df commands differs.
Perhaps the most common is deleted files. Files that have been deleted may still be open by at least one process. The entry for such files is removed from the associated directory, which makes the file inaccessible. Therefore the command du which only counts files does not take these files into account and comes up with a smaller value. As long as a process still has the deleted file in use, however, the associated blocks are not yet released in the file system, so df which works at the kernel level correctly displays these as occupied. You can find out if this is the case by running the following:
lsof | grep '(deleted)'
The fix for this issue would be to restart the services that still have those deleted files open.
The second most common cause is if you have a partition or drive mounted on top of a directory with the same name. For example, if you have a directory under / called backup which contains data and then you mount a new drive on top of that directory and label it /backup but it contains no data then the space used will show up with the df command even though the du command shows no files.
To determine if there are any files or directories hidden under an active mount point, you can try using a bind-mount to mount your / filesystem which will enable me to inspect underneath other mount points. Note, this is recommended only for experienced system administrators.
mkdir /tmp/tmpmnt
mount -o bind //tmp/tmpmnt
du /tmp/tmpmnt
After you have confirmed that this is the issue, the bind mount can be removed by running:
umount /tmp/tmpmnt/
rmdir /tmp/tmpmnt
Another possible cause might be filesystem corruption. If this is suspected, please make sure you have good backups, and at your convenience, please unmount the filesystem and run fsck.
Again, this should be done by experienced system administrators.
You can also check the calculation by running:
strace -e statfs df /
This will give you output similar to:
statfs("/", {f_type=XFS_SB_MAGIC, f_bsize=4096, f_blocks=20968699, f_bfree=17420469,
f_bavail=17420469, f_files=41942464, f_ffree=41509188, f_fsid={val=[64769, 0]},
f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 83874796 14192920 69681876 17% /
+++ exited with 0 +++
Notice the difference between f_bfree and f_bavail? These are the free blocks in the filesystem vs free blocks available to an unprivileged user. The used column is merely a calculation between the two.
Hope this will make your idea clear. Let me know if you still have any doubts.
I don't know actually if this is more a "classic" linux or a docker question but:
On an VM where some of my docker containers are running I've a strange thing. /var/lib/docker is an own partitionwith 20GB. When I look over the partition with df -h I see this:
eti-gwl1v-dockerapp1 root# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 815M 7.0G 11% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda2 12G 3.2G 8.0G 29% /
/dev/sda7 3.9G 17M 3.7G 1% /tmp
/dev/sda5 7.8G 6.8G 649M 92% /var
/dev/sdb2 20G 47M 19G 1% /usr2
/dev/sdb1 20G 2.9G 16G 16% /var/lib/docker
So usage is at 16%. But when I now navigate to /var/lib and do a du -sch docker I see this:
eti-gwl1v-dockerapp1 root# cd /var/lib
eti-gwl1v-dockerapp1 root# du -sch docker
19G docker
19G total
eti-gwl1v-dockerapp1 root#
So same directory/partition but two sizes? How is that going?
This is really a question for unix.stackexchange.com, but there is filesystem overhead that makes the partition larger than the total size of the individual files within it.
du and df show you two different metrics:
du shows you the (estimated) file space usage, i.e. the sum of all file sizes
df shows you the disk space usage, i.e. how much space on the disk is actually used
These are distinct values and can often diverge:
disk usage may be bigger than the mere sum of file sizes due to additional meta data: e.g. the disk usage of 1000 empty files (file size = 0) is >0 since their file names and permissions need to be stored
the space used by one or multiple files may be smaller than their reported file size due to:
holes in the file - block consisting of only null bytes are not actually written to disk, see sparse files
automatic file system compression
deduplication through hard links or copy-on-write
Since docker uses the image layers as a means of deduplication the latter is most probably the cause of your observation - i.e. the sum of the files is much bigger because most of them are shared/deduplicated through hard links.
du estimates filesystem usage through summing the size of all files in it. This does not deal well with the usage of overlay2: there will be many directories which contain the same files as contained in another, but overlaid with additional layers using overlay2. As such, du will show a very inflated number.
I have not tested this since my Docker daemon is not using overlay2, but using du -x to avoid going into overlays could give the right amount. However, this wouldn't work for other Docker drivers, like btrfs, for example.
I am currently using the last version of owncloud. Since the installation, I cannot login anymore. A quick look at /var/log/apache2/error.log explains why :
WARNING: could not create relation-cache initialization file "global/pg_internal.init.7826": No space left on device
DETAIL: Continuing anyway, but there's something wrong.
WARNING: could not create relation-cache initialization file "base/17999/pg_internal.init.7826": No space left on device
DETAIL: Continuing anyway, but there's something wrong.
WARNING: could not create relation-cache initialization file "global/pg_internal.init.7827": No space left on device
DETAIL: Continuing anyway, but there's something wrong.
WARNING: could not create relation-cache initialization file "base/17999/pg_internal.init.7827": No space left on device
DETAIL: Continuing anyway, but there's something wrong.
WARNING: could not create relation-cache initialization file "global/pg_internal.init.7828": No space left on device
But I cannot figure where I do not have enough space. If I try df -h as root, everything seems ok to me :
:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 20G 20G 0 100% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 4.0K 3.9G 1% /dev/shm
tmpfs 3.9G 82M 3.8G 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 898G 912M 851G 1% /home
tmpfs 788M 0 788M 0% /run/user/0
Excepted the first line which I hardly understand what it represents. I installed owncloud into /home/owncloud so I bet everything should be ok.
Any idea?
Edit :
Results of findmnt :
~# findmnt /
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda1 ext4 rw,relatime,errors=remount-ro,data=ordered
~# findmnt /dev/sda1
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda1 ext4 rw,relatime,errors=remount-ro,data=ordered
~# findmnt /dev/sda2
TARGET SOURCE FSTYPE OPTIONS
/home /dev/sda2 ext4 rw,relatime,data=ordered
Often, these programs store their data under /var, In your case, you don't have a separate mountpoint for /var so it's a directory on your root file system /. This is full and so the program is not working.
Before you attempt a resize or anything, I think you should find out what is hogging 20GB. du / | sort -n should give you a rough idea of the guilty parties or you can use a graphical tool like xdiskusage. Clean it up and you'll be good to go.
The other alternative is to look through the config files for owncloud and make it use your home directory to store its data. That way, it will work. But you should clean up your /. Various things will misbehave if you don't.
Maybe you are out of inodes: No space left on device – running out of Inodes.
Use df -i to check that. It happened to me as my backup used to have millions of small files. So there was space left but no inodes left.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
relevant software:
Red Hat Enterprise Linux Server release 6.3 (Santiago)
cpanel installed 11.34.0 (build 7)
background and problem:
I was getting a disk usage warning (via cpanel) because /var seemed to be filling up on my server. The assumption would be that there was a log file growing too large and filling up the partition. I recently removed a large log file and changed my syslog config to rotate the log files more regularly. I removed something like /var/log/somefile and edited /etc/rsyslog.conf. This is the reason I was suspicious of the disk usage report warning issued by cpanel that I was getting because it didn't seem right.
This is what df was reporting for the partitions:
$ [/var]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.9G 511M 8.9G 6% /
tmpfs 5.9G 0 5.9G 0% /dev/shm
/dev/sda1 99M 53M 42M 56% /boot
/dev/sda8 883G 384G 455G 46% /home
/dev/sdb1 9.9G 151M 9.3G 2% /tmp
/dev/sda3 9.9G 7.8G 1.6G 84% /usr
/dev/sda5 9.9G 9.3G 108M 99% /var
This is what du was reporting for /var mount point:
$ [/var]# du -sh
528M .
clearly something funky was going on. I had a similar kind of reporting inconsistency in the past and I restarted the server and df reporting seemed to be correct after that. I decided to reboot the server to see if the same thing would happpen.
This is what df reports now:
$ [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.9G 511M 8.9G 6% /
tmpfs 5.9G 0 5.9G 0% /dev/shm
/dev/sda1 99M 53M 42M 56% /boot
/dev/sda8 883G 384G 455G 46% /home
/dev/sdb1 9.9G 151M 9.3G 2% /tmp
/dev/sda3 9.9G 7.8G 1.6G 84% /usr
/dev/sda5 9.9G 697M 8.7G 8% /var
This looks more like what I'd expect to get.
For consistency this is what du reports for /var:
$ [/var]# du -sh
638M .
question:
This is a nuisance. I'm not sure where the disk usage reports issued by cpanel get their info but it clearly isn't correct. How can I avoid this inaccurate reporting in the future? It seems like df reporting wrong disk usage is a strong indicator of the source problem but I'm not sure. Is there a way to 'refresh' the filesystem somehow so that the df report is accurate without restarting the server? Any other ideas for resolving this issue?
If you remove a file, but it is still open by some process, the disk space is not recovered -- the process continues to access that file. This is a common problem with log files, because syslogd keeps them all open.
The disk space reported by du doesn't include this file because it works by walking down the directory hierarchy adding up the sizes of all the files it finds. But this file can't be found in any directory, so it's not counted. df reports the actual space used in the filesystem.
The logfile rotation script sends a signal to syslogd telling it to close an reopen all its log files. You can accomplish this with:
killall -HUP syslogd
You also need to do this to get syslogd to use your modified syslog.conf.
I am relatively new to Linux. In one of our projects, we use amazon's EC2 instance for processing of some files. We upload files to S3 server after processing. EC2 instance is booted using an existing AMI
Recently I got an error no space left on disk, hence processing of files was halted. I cleaned up some older files and the processing continued.
Now when I look at available space using df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 9.9G 5.7G 3.7G 61% /
none 3.7G 0 3.7G 0% /dev/shm
/dev/xvdb 414G 199M 393G 1% /mnt
/dev/xvdc 414G 199M 393G 1% /data
I can see my files are effecting only /dev/xvda1.
I have following queries
What is the use of other partitions when I can see my files only effecting /dev/xvda1
It looks like we are only using 10 GB of space effectively and other is being wasted. How can I use other space? Can I move some disk space to /dev/xvda1 or directly store files in other areas?
As you can see from the output of df -h, there are two large partitions mouted on /mnt and /data respectively. I suggest that you use those partitions by processing the files in one of those directories. If you cannot move where the processing happens for some reason, you can remount the partitions in the appropriate place.
If for example your files are processed in the directory /var/mydir and you cannot change that, do the following (as root):
umount /mnt
mount /dev/xvdb /var/mydir
You can use the other partition as well of course if you prefer that.