I have a test application which runs every hour and uses unique log file at each execution. To clean the logs the following logrotate configuration has been set:
{
# Daily rotation with 1 week of backlog
daily
rotate 7
maxage 7
dateext
compress
}
The first day the log file compressed (which is ok) but a empty file is left and every other day that files is "emptied" and compressed. And that makes 6 files of every logfiles which fills the inodes table of the FS. Here's two examples :
-rw-r--r-- 1 root root 1752 Feb 11 01:36 J20190211013601_Status.txt-20190212.gz
-rw------- 1 root root 20 Feb 12 03:33 J20190211013601_Status.txt-20190213.gz
-rw------- 1 root root 20 Feb 13 03:37 J20190211013601_Status.txt-20190214.gz
-rw------- 1 root root 20 Feb 14 03:10 J20190211013601_Status.txt-20190215.gz
-rw------- 1 root root 20 Feb 15 03:12 J20190211013601_Status.txt-20190216.gz
-rw------- 1 root root 20 Feb 16 03:36 J20190211013601_Status.txt-20190217.gz
-rw------- 1 root root 20 Feb 17 03:44 J20190211013601_Status.txt-20190218.gz
-rw------- 1 root root 0 Feb 18 03:24 J20190211013601_Status.txt
-rw-r--r-- 1 root root 1752 Feb 11 02:36 J20190211023601_Status.txt-20190212.gz
-rw------- 1 root root 20 Feb 12 03:33 J20190211023601_Status.txt-20190213.gz
-rw------- 1 root root 20 Feb 13 03:37 J20190211023601_Status.txt-20190214.gz
-rw------- 1 root root 20 Feb 14 03:10 J20190211023601_Status.txt-20190215.gz
-rw------- 1 root root 20 Feb 15 03:12 J20190211023601_Status.txt-20190216.gz
-rw------- 1 root root 20 Feb 16 03:36 J20190211023601_Status.txt-20190217.gz
-rw------- 1 root root 20 Feb 17 03:44 J20190211023601_Status.txt-20190218.gz
-rw------- 1 root root 0 Feb 18 03:24 J20190211023601_Status.txt
How can i correct this, in order to delete the files after being compressed
Thanks for time and help,
This is how logrotate is supposed to function; your issue stems from the fact that you're using unique filenames every time your appplication runs.
When logrotate runs for the first time on each log, it's moving the log file from "J20190211023601_Status.txt" to "J20190211023601_Status.txt-20190212.gz" and then creating a new file named J20190211023601_Status.txt.
Logrotate has no inherent idea that those filenames are unique and thus will never be populated again; all it sees is a log it's rotated in the past, so it must be rotated again as per your configuration.
Your easiest solution here is to pass the nocreate directive for this logrotation; this will prevent that new log file from being created and subsequently rotated while still respecting the 7-day age limit on previously rotated files:
{
daily
maxage 7
dateext
compress
nocreate
}
I'm trying to run a bash script which should go in a specific directory.
The Problem is that the Script wont go in the newest Folder.
The Folder looks like:
root#raspberry ~/jdownloader/logs # ls -lha
total 104K
drwxr-xr-x 9 root root 4.0K Nov 30 11:52 .
drwxr-xr-x 14 root root 4.0K Nov 30 11:52 ..
drwxr-xr-x 2 root root 4.0K Nov 30 11:18 1479843940152_Tue, Nov 22, 2016 20.45 +0100
drwxr-xr-x 2 root root 4.0K Nov 30 11:21 1480501204839_Wed, Nov 30, 2016 11.20 +0100
drwxr-xr-x 2 root root 4.0K Nov 30 11:22 1480501242752_Wed, Nov 30, 2016 11.20 +0100
drwxr-xr-x 2 root root 4.0K Nov 30 11:30 1480501308071_Wed, Nov 30, 2016 11.21 +0100
drwxr-xr-x 2 root root 4.0K Nov 30 11:56 1480503116574_Wed, Nov 30, 2016 11.51 +0100
drwxr-xr-x 3 root root 12K Nov 23 11:25 extracting
drwxr-xr-x 2 root root 64K Nov 30 11:22 updatehistory
The Important Snippet from my Script is:
#!/bin/bash
declare dir=/var/log/scriptlog/jdstate
declare dir2=~/jdownloader/logs
NewFolder=`ls -rt1 ~/jdownloader/logs -I extracting -I updatehistory | tail -1 > /var/log/scriptlog/jdstate/newfolder.log`
OutputNewFolder=`head $dir/newfolder.log -n 1`
cd\ "\"$dir2/$OutputeNewFolder\""
When I try to run the script it shows me the error that it can't find the directory.
But when I copy/paste it, it will go to the Directory.
Any idea how it is possible to go to the directory?
For every one how is searching for an Answer:
this is my latest code snippet wich worked gladly for me
#!/bin/bash
declare dir=/var/log/scriptlog/jdstate
declare dir2=~/jdownloader/logs
NewFolder=`ls -rt1 ~/jdownloader/logs -I extracting -I updatehistory | tail -1 > /var/log/scriptlog/jdstate/newfolder.log`
OutputNewFolder=`head $dir/newfolder.log -n 1`
cd "$dir2/$OutputNewFolder"
I´m sure the can be improvements for the newest folder but htis works just finde for me
I was able to connect to my school server via SSH. I had an assignment in which I was supposed to use the touch command to create a new file. Yet it keeps returning permission denied. Others were able to do the same thing. Though why do I keep getting this error?
Below is what was the input from the terminal.
Last login: Tue Aug 23 09:16:18 on ttys000
Dominiks-Air:~ fsociety95$ ssh djaneka1#navajo.dtcc.edu
djaneka1#navajo.dtcc.edu's password:
Last login: Tue Aug 23 09:16:35 2016 from pool-72-94-210-193.phlapa.fios.verizon.net
Navajo is Linux shell server provided to staff, faculty, and students. The
operating system is RedHat Enterprise Linux 5.
Alpine, a Pine replacement, has been provided as a mail client. Run "pine"
at the command prompt.
This server also provides web space to users. Web pages can be stored in
the ~/www directory. This is also accessible by mapping a drive in Windows
to \navajo\homepage. The URL for your homepage is
http://user.dtcc.edu/~username/.
Your home directory is also accessible in Windows by mapping to
\navajo\.
If something appears broken or missing, please email path#dtcc.edu.
Could not chdir to home directory /u/d/j/djaneka1: No such file or directory
-bash-3.2$ touch today
touch: cannot touch `today': Permission denied
-bash-3.2$ pwd
/
-bash-3.2$ touch today
touch: cannot touch `today': Permission denied
-bash-3.2$
Edit: here is the result of ls -al
-bash-3.2$ ls -al
total 204
drwxr-xr-x 25 root root 4096 Aug 22 16:50 .
drwxr-xr-x 25 root root 4096 Aug 22 16:50 ..
-rw-r--r-- 1 root root 0 Aug 3 14:01 .autofsck
-rw-r--r-- 1 root root 0 Jan 30 2009 .autorelabel
-rw------- 1 root root 2050 Aug 3 14:00 .bash_history
drwxr-xr-x 2 root root 4096 May 4 04:14 bin
drwxr-xr-x 4 root root 3072 Aug 3 13:57 boot
drwxr-xr-x 11 root root 4060 Aug 3 14:02 dev
drwxr-xr-x 87 root root 12288 Aug 23 10:05 etc
drwxr-xr-x 3 root root 4096 Oct 1 2009 home
drwxr-xr-x 13 root root 12288 Jun 1 04:09 lib
drwx------ 2 root root 16384 Mar 24 2008 lost+found
drwxr-xr-x 3 root root 4096 Oct 1 2009 media
drwxr-xr-x 2 root root 0 Aug 3 14:02 misc
drwxr-xr-x 4 root root 4096 May 26 2012 mnt
drwxr-xr-x 2 root root 0 Aug 3 14:02 net
drwxr-xr-x 9 root root 4096 Jan 5 2009 nsr
drwxrwxr-x 3 root root 4096 Oct 12 2015 opt
dr-xr-xr-x 219 root root 0 Aug 3 14:01 proc
drwxr-x--- 12 root root 4096 Apr 22 10:06 root
drwxr-xr-x 2 root root 12288 Aug 4 04:02 sbin
drwxr-xr-x 2 root root 4096 Oct 1 2009 selinux
drwxr-xr-x 2 root root 4096 Oct 1 2009 srv
drwxr-xr-x 11 root root 0 Aug 3 14:01 sys
drwxrwxrwt 38 root root 4096 Aug 23 10:07 tmp
drwxr-xr-x 34 root root 4096 Jun 21 08:29 u
drwxr-xr-x 14 root root 4096 Apr 16 2010 usr
drwxr-xr-x 24 root root 4096 Apr 16 2010 var
-rw------- 1 root root 2865 Dec 16 2008 .viminfo
-bash-3.2$
EDIT:
Here is what I see after trying touch today in /home
So to try and create a new document in the root directory you need to be recognised as root. That means using the sudo command.
However for that you would need a password that you may not have. If you do perfect. But in any case I would not recommend adding files to the root directory.
Instead try the following:
cd home
touch today
This should work just fine and answer your question.
Still if you need/want to create today in your root directory try the following
sudo touch today
You will then be prompted for the root password that you can type (if you have it obviously)
In any case I suggest reading this which may be very helpful for you.
I wonder if this was ever truly answered.
If I was looking at it, I would try to see what the system thinks is the home directory of djaneka1, since it may have been setup partway and not completed, leaving stuff owned by root that should have been owned by djaneka1.
If you use the pwd command, and get back the "/" (root) directory there is something wrong with your setup.
The message: Could not chdir to home directory /u/d/j/djaneka1: No such file or directory
tells you it can't find your home directory.
-bash-3.2$ pwd
/
the command "pwd" revealing "/" is just an artifact of the system not being able to find your home directory.
To find what the system thinks is one's home directory,
one can search the file named '/etc/passwd' for one's login name.
I expect this is a possible result if you do that:
$ fgrep 'djaneka1' /etc/passwd
djaneka1:x:1505:1506::/u/d/j/djaneka1:/bin/bash
since it complained that it couldn't find that directory.
This needs to be fixed by someone who has more rights to the system, like root.
there is nothing djaneka1 can do a
> git gc --aggressive --prune=now
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66685/66685), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_cO6T53' write error: No space left on device
sigh, ok
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 15G 3.0G 84% /
udev 485M 4.0K 485M 1% /dev
tmpfs 99M 296K 99M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 494M 0 494M 0% /run/shm
cgroup 494M 0 494M 0% /sys/fs/cgroup
doesn't look that bad
ls -lh .git/objects/pack/
total 580M
-r--r--r-- 1 foouser root 12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root 11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root 40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root root 16M Feb 27 08:19 tmp_pack_cO6T53
So, git gc bails out on a tmp pack that's only 16MB big while my disk appears to have 3GB free. What am I missing? How can I get git gc to work more reliably? I've tried without aggressive option and --prune instead of --prune=now as well, same story.
Update
Doing a df -h during the repack action it shows that it is now using all my disk (100% usage). A little while later the repack action fails and it leaves another 14MB file in the .git/objects/pack/ folder. So, to recap, my packs use a total of 580MB. git repack somehow manages to use up 3GB to repack that. I have ~800MB free in the RAM after it's done btw. - maybe it's using so much working memory that it clogs up the swap? I guess my question comes down to: Are there options to make git repack less resource hungry?
versions: git version 1.7.9.5 on Ubuntu 12.04
Update 2
I've updated git to 2.3. Didn't change anything unfortunately.
> git --version
git version 2.3.0
> git repack -Ad && git prune
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (36893/36893), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_N9jyVJ' write error: No space left on device
Update 3
Ok, so I just noticed something that is curious: the .git directory actually uses much more disk space than the 508MB previously reported.
> du -h -d 1 ./.git
8.0K ./.git/info
40K ./.git/hooks
24M ./.git/modules
28K ./.git/refs
4.0K ./.git/branches
140K ./.git/logs
5.0G ./.git/objects
5.0G ./.git
Upon further inspection .git/objects/pack actually uses 4.5GB. The differences lies in hidden temp files I didn't notice before:
ls -lha ./.git/objects/pack/
total 4.5G
drwxr-xr-x 2 foouser root 56K Feb 27 15:40 .
drwxr-xr-x 260 foouser root 4.0K Oct 26 14:24 ..
-r--r--r-- 1 foouser root 12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root 11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root 40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:37 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.idx
-r--r--r-- 1 root root 14M Feb 27 15:29 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:32 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.idx
-r--r--r-- 1 root root 41M Feb 27 15:30 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:37 .tmp-7729-pack-051980133b8f0052b66dce418b4d3899de0d1342.idx
(continuing for a *long* while).
Now I'd like to know: Is it safe to just delete those?
So here is what I found out so far: I couldn't find any documentation about these hidden '.tmp-XXXX-pack' in the .git/objects/pack folder. All other threads I can find are about non-hidden files with tmp_ prefix in the same folder. The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well. I can't confirm whether that's still possible in git 2.3.0 (which I've updated to since), but at least the disk space requirement doesn't seem to have changed in this newer version - it still can't complete gc/repack. By deleting these .tmp-files I was able to recover my last 4GB and git still seems to behave fine afterwards - your results may vary though, so please make sure you have a backup before doing this. Finally, even 4GB wasn't enough to repack with gc --agressive. My .git folder is 1.1GB after the cleanup, my entire repository is 1.7GB. So 2x the size of your repository is possibly not enough for git gc, even with the aggressive option (which should save space). So I had to recover more space from elsewhere first.
Here is the command I used to clean up (again, have backups!):
git gc --aggressive --prune=now || rm -f .git/objects/*/tmp_* && rm -f .git/objects/*/.tmp-*
Similar scenario (about 2.3G available), except git gc itself would also fail with fatal: Unable to create '/home/ubuntu/my-app-here/.git/gc.pid.lock': No space left on device
What worked was to git prune first, and then run the gc.
I had an instance of this problem. I was able to free up a considerable amount of disk space, but of course, that didn't solve the problem of what to do about the .tmp-* files. I ran git fsck and the Git repository wasn't damaged in that way.
I did the conventional pack-and-garbage-collect operations
git repack -Ad
git prune
but that didn't remove the .tmp-* files, though it would have ensured that all of the necessary objects were in the standard pack-* files if they needed to be copied from transient files left over from the crashed Git processes in the past.
Eventually I realized that I could safely move the .tmp-* files to a scratch directory, then run git fsck to see if what remained in the .git directory was complete. It turned out that it was, so I deleted the scratch directory and the files it contained. If git fsck had reported problems, I could have moved the .tmp-* files back into the .git directory and researched another solution.
The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well
The way "git repack"(man) created temporary files when it received a signal was prone to deadlocking, which has been corrected with Git 2.39 (Q4 2022).
See commit 9b3fadf (23 Oct 2022), and commit 1934307, commit 9cf10d8, commit a4880b2, commit b639606, commit d3d9c51 (21 Oct 2022) by Jeff King (peff).
(Merged by Taylor Blau -- ttaylorr -- in commit c88895e, 30 Oct 2022)
repack: use tempfiles for signal cleanup
Reported-by: Jan Pokorný
Signed-off-by: Jeff King
When git-repack(man) exits due to a signal, it tries to clean up by calling its remove_temporary_files() function, which walks through the packs dir looking for ".tmp-$$-pack-*" files to delete (where "$$" is the pid of the current process).
The biggest problem here is that remove_temporary_files() is not safe to call in a signal handler.
It uses opendir(), which isn't on the POSIX async-signal-safe list.
The details will be platform-specific, but a likely issue is that it needs to allocate memory; if we receive a signal while inside malloc(), etc, we'll conflict on the allocator lock and deadlock with ourselves.
We can fix this by just cleaning up the files directly, without walking the directory.
We already know the complete list of .tmp-* files that were generated, because we recorded them via populate_pack_exts().
When we find files there, we can use register_tempfile() to record the filenames.
If we receive a signal, then the tempfile API will clean them up for us, and it's async-safe and pretty battle-tested.
And:
repack: drop remove_temporary_files()
Signed-off-by: Jeff King
After we've successfully finished the repack, we call remove_temporary_files(), which looks for and removes any files matching ".tmp-$$-pack-*", where $$ is the pid of the current process.
But this is pointless.
If we make it this far in the process, we've already renamed these tempfiles into place, and there is nothing left to delete.
Nor is there a point in trying to call it to clean up when we aren't successful.
It's not safe for using in a signal handler, and the previous commit already handed that job over to the tempfile API.
It might seem like it would be useful to clean up stray .tmp files left by other invocations of git-repack.
But it won't clean those files; it only matches ones with its pid, and leaves the rest.
Fortunately, those are cleaned up naturally by successive calls to git-repack; we'll consider .tmp-*.pack the same as normal packfiles, so "repack -ad", etc, will roll up their contents and eventually delete them.
I need to write rotation of files shell script. I have following format data in a target directory(/backup/store_id/dates_folders)
Like :
cd /backup/
drwxr-xr-x 5 root root 4096 Mar 25 12:30 44
drwxr-xr-x 3 root root 4096 Mar 25 12:30 45
drwxr-xr-x 4 root root 4096 Mar 25 12:30 48
drwxr-xr-x 3 root root 4096 Mar 25 12:30 49
cd /backup/44/
drwxr-xr-x 2 root root 4096 Mar 25 12:30 22032014
drwxr-xr-x 2 root root 4096 Mar 25 12:30 23032014
drwxr-xr-x 2 root root 4096 Mar 25 12:30 24032014
drwxr-xr-x 2 root root 4096 Mar 25 12:30 25032014
now 44 (store_id) contain four dates folders. I want each store_id( like 44 folder) contain only three recent dates folder like 23,24,25 & 22 should be deleted. Please help me how to write in shell script. Please give me some hint
This should work:
cd /backup && ls -d */ | while read storeId; do rm -r `ls -r $storeId | tail -3`; done
I assume here that directory names are more important than their timestamps...
If that is not the case, you should use ls -tr instead of ls -r, to let ls command sort on timestamps...