$ls -lrt test*log*
-rwxr----- 1 root xyzgroup 54231265 Sep 4 16:06 test.log.9
-rwxr----- 1 root xyzgroup 53990979 Sep 4 16:06 test.log.8
-rwxr----- 1 root xyzgroup 53372511 Sep 4 16:06 test.log.7
$cat test.log.9|head -5
Sep 4 12:59:01.701796 <sometext>
Sep 4 12:59:01.701796 <sometext>
Sep 4 12:59:01.720171 <sometext>
Sep 4 12:59:01.720171 <sometext>
Sep 4 12:59:01.720699 <sometext>
$cat test.log.9|tail -5
Sep 4 13:16:01.880489 <sometext>
Sep 4 13:16:01.880489 <sometext>
Sep 4 13:16:01.880489 <sometext>
Sep 4 13:16:01.880489 <sometext>
Sep 4 13:16:01.880489 <sometext>
$cat test.log.8|head -5
Sep 4 13:16:01.898749 <sometext>
Sep 4 13:16:01.898749 <sometext>
Sep 4 13:16:01.898749 <sometext>
Sep 4 13:16:01.898948 <sometext>
Sep 4 13:16:01.898948 <sometext>
$cat test.log.8|tail -5
Sep 4 13:35:02.513804 <sometext>
Sep 4 13:35:02.513804 <sometext>
Sep 4 13:35:02.513804 <sometext>
Sep 4 13:35:02.514136 <sometext>
Sep 4 13:35:02.514136 <sometext>
Log format at the beginning of each line:
Command:
sed -rn '/Sep 4 09:38:*/,/Sep 4 11:23:*/p' test*log* > temp_test.log
Observation:
When performing sed operation on the listed files, required log content between start and end time is not fetched. Instead it will fetch partial log lines or sometime the output file size increases(in GB). Also some time for shorter duration log lines are properly fetched.
Expected:
Log lines between start and end time stamps.
Edit on behalf of the OP after reverting:
When trying the solution in bodo's answer as a remote command using ssh, an error occurs:
local machine$ sudo ssh user#1.2.3.4 "sudo sed -rn '/Sep 4 09:38:*/,/Sep 4 11:23:*/p' $(ls -vr /user/xyz/test*log*)" > temp_test.log
ls: cannot access /user/xyz/test*log*: No such file or directory
If the problem is caused only by the order of the input files as passed to sed you can use the sorting option -v of the ls command (in your specific use case).
-v natural sort of (version) numbers within text
Compare
echo test.log*
echo $(ls -vr test.log*)
In your specific case you can try
sed -rn '/Sep 4 09:38:*/,/Sep 4 11:23:*/p' $(ls -vr test.log*) > temp_test.log
Note: This solution will not work in all cases.
It requires that the file names resulting from the pattern test*log* do not contain any whitespace or special characters. The file names shown in the question fulfill this requirement.
In general, performing word splitting to extract file names from command output is problematic and should be avoided.
Without knowing the contents of the input files it is impossible to check if there are other causes that could lead to the problems described in the question.
Answer for additional question about remote execution:
The command
sudo ssh user#1.2.3.4 "sudo sed -rn '/Sep 4 09:38:*/,/Sep 4 11:23:*/p' $(ls -vr /user/xyz/test*log*)" > temp_test.log
does not work because ls -vr /user/xyz/test*log* is run on the local machine.
You can try quoting to get this part executed on the remote machine, but this can get complicated.
I suggest to create a shell script on the remote machine, e.g. /home/user/logfilter
#!/bin/sh
if [ $# lt 2 ]
then
echo "usage: logfilter from to" >&2
exit 1
fi
from="$1"
to="$2"
sed -rn "/^$from/,/^$to/p" $(ls -vr /user/xyz/test*log*)
and run it like this
sudo ssh user#1.2.3.4 "sudo ./logfilter 'Sep 4 09:38:' 'Sep 4 11:23:'" > temp_test.log
(untested)
Related
There are 3 txt files called
1.txt 2.txt 3.txt
I want to batch copy with the name
1.txt.cp 2.txt.cp 3.txt.cp
using the wildcard *
I entered the command cp *.txt *.txt.cp
but it wasn't working...
cp : target *.txt.cp : is not a directory
what was the problem???
Use: for i in *.txt; do cp "$i" "$i.cp"; done
Example:
$ ls -l *.txt
-rw-r--r-- 1 halley halley 20 out 27 08:14 1.txt
-rw-r--r-- 1 halley halley 25 out 27 08:14 2.txt
-rw-r--r-- 1 halley halley 33 out 27 08:15 3.txt
$ ls -l *.cp
ls: could not access '*.cp': File or directory does not exist
$ for i in *.txt; do cp "$i" "$i.cp"; done
$ ls -l *.cp
-rw-r--r-- 1 halley halley 20 out 27 08:32 1.txt.cp
-rw-r--r-- 1 halley halley 25 out 27 08:32 2.txt.cp
-rw-r--r-- 1 halley halley 33 out 27 08:32 3.txt.cp
$ for i in *.txt; do diff "$i" "$i.cp"; done
$
If you are used to MS/Windown CMD shell, it is important to note that Unix system handle very differently the wild cards. MS/Windows has kept the MS/DOS rule that said that wild cards were not interpreted but were passed to the command. The command sees the wildcard characters and can handle the second * in the command as noting where the match from the first should go, making copy ab.* cd.* sensible.
In Unix (and derivatives like Linux) the shell is in charge of handling the wildcards and it replaces any word containing one with all the possible matches. The good news is that the command has not to care about that. But the downside is that if the current folder contains ab.txt ab.md5 cd.jpg, a command copy ab.* cd.* will be translated into copy ab.txt ab.md5 cd.jpg which is probably not want you would expect...
The underlying reason is Unix shells are much more versatile than the good old MS/DOS inherited CMD.EXE and do have simple to use for and if compound commands. Just look at #Halley Oliveira's answer for the syntax for your use case.
With sed I try to replace the value 0.1.233... On the command line there is no problem; however, when putting this command in a shell script, I get an error:
sed: couldn't open temporary file ../project/cas-dp-ap/sedwi3jVw: Permission denied
I don't understand where this temporary sedwi file comes from.
Do you have any idea why I have this temporary file and how I can pass it?
$(sed -i "s/$current_version/$version/" $PATHPROJET$CREATE_PACKAGE/Chart.yaml)
++ sed -i s/0.1.233/0.1.234/ ../project/cas-dp-ap/Chart.yaml
sed: couldn't open temporary file ../project/cas-dp-ap/sedwi3jVw: Permission denied
+ printf 'The version has been updated to : 0.1.234 \n\n \n\n'
The version has been updated to : 0.1.234
+ printf '***********************************'
sed -i is "in-place editing". However "in-place" isn't really. What happens is more like:
create a temporary file
run sed on original file and put changes into temporary file
delete original file
rename temporary file as original
For example, if we look at the inode of an edited file we can see that it is changed after sed has run:
$ echo hello > a
$ ln a b
$ ls -lai a b
19005916 -rw-rw-r-- 2 jhnc jhnc 6 Jan 31 12:25 a
19005916 -rw-rw-r-- 2 jhnc jhnc 6 Jan 31 12:25 b
$ sed -i 's/hello/goodbye/' a
$ ls -lai a b
19005942 -rw-rw-r-- 1 jhnc jhnc 8 Jan 31 12:25 a
19005916 -rw-rw-r-- 1 jhnc jhnc 6 Jan 31 12:25 b
$
This means that your script has to be able to create files in the folder where it is doing the "in-place" edit.
The proper syntax is identical on the command line and in a script. If you used $(...) at the prompt then you would have received the same error.
sed -i "s/$current_version/$version/" "$PATHPROJET$CREATE_PACKAGE/Chart.yaml"
(Notice also the quoting around the file name. Probably your private variables should use lower case.)
The syntax
$(command)
takes the output from command and tries to execute it as a command. Usually you would use this construct -- called a command substitution -- to interpolate the output of a command into a string, like
echo "Today is $(date)"
(though date +"Today is %c" is probably a better way to do that particular thing).
If I do ls -o I get
-rw-rw-r-- 1 louise 347967 Aug 28 2017 Screenshot from 2017-08-28 09-33-01.png
-rw-rw-r-- 1 louise 377739 Aug 29 2017 Screenshot from 2017-08-29 10-39-49.png
-rw-rw-r-- 1 louise 340682 Aug 29 2017 Screenshot from 2017-08-29 10-40-02.png
I really want to remove the first 3 columns, so I get
347967 Aug 28 2017 Screenshot from 2017-08-28 09-33-01.png
377739 Aug 29 2017 Screenshot from 2017-08-29 10-39-49.png
340682 Aug 29 2017 Screenshot from 2017-08-29 10-40-02.png
ls can't do this, it seems. There are other questions here at SO about removing multiple columns, but not from the beginning.
ls is an interactive tool, whose output is not supposed to be parsed.
Consider using an alternative tool such as stat (GNU version recommended):
stat -c '%s %y %n' *
The output isn't quite the same but you have full control over the format. stat --help gives more information about the possible format sequences.
With GNU stat you can also use --printf to add escape characters such as newlines or tabs in the format string, to make parsing easier:
stat --printf '%s\t%Y\t%n\n' *
%Y (last modification, seconds since Epoch) is more readily suited to parsing than %y (human-readable).
This would still break in cases where the filename contained a newline, so depending on how you plan on using this information, you may want to use a \0 instead of a \n at the end of the format string and process records terminated with a null-byte instead of a newline.
Alternatively, you may find it easier to just loop through the files and call stat on them one by one, extracting whatever you need:
for file in *; do
read -r size modified name < <(stat '%s %Y %n' "$file")
# do whatever with $size, $modified and $name here
done
Assuming you go with the loop-based approach, you can convert the date to any format you want using date, for example:
date -d #"$modified" +'%b %d %H:%m'
Is it possible?
For example, say I want to run the command ll:
My output would look something like this:
josh#zeitgeist ~ ll
total 41148
drwxr-xr-x 42 josh josh 4096 Aug 4 22:52 ./
drwxr-xr-x 4 root root 4096 Jul 9 21:18 ../
-rw-rw-r-- 1 josh josh 3523718 Jul 11 00:17 2017-07-11-001710_3840x2160_scrot.png
but I want it to look like this:
josh#zeitgeist ~ ll
XXXtotal 41148
XXXdrwxr-xr-x 42 josh josh 4096 Aug 4 22:52 ./
XXXdrwxr-xr-x 4 root root 4096 Jul 9 21:18 ../
XXX-rw-rw-r-- 1 josh josh 3523718 Jul 11 00:17 2017-07-11-XXX001710_3840x2160_scrot.png
I already know about using PS1='XXX' to change the prompt; is there a way to change every line of the output that gets displayed, specifically in the terminal(not changing the output and putting it in a file)?
I would like to do this to have a unified line of characters going down the left side of my terminal.
You can easily do it with sed:
ll | sed 's/./XXX&/'
Prepend XXXto all lines including empty lines.
ll | sed 's/^/XXX/'
Edit:
After changing your PS1 you can invoke the solution for all commands using
bash | sed 's/^/XXX/'
You could do something like this I suppose:
while read; do echo "xxx $REPLY"; done < <(ls -l)
Not really sure what the purpose of this would be though.
Using awk:
$ ll|awk '$0="XXX"$0'
There are multiple directories which contain a file with the same name:
direct_afaap/file.txt
direct_fgrdw/file.txt
direct_sardf/file.txt
...
Now I want to extract them to another directory, direct_new and with a different file name such as:
[mylinux~ ]$ ls direct_new/
file_1.txt file_2.txt file_3.txt
How can I do this?
BTW, if I want to put part of the name in original directory into the file name such as:
[mylinux~ ]$ ls direct_new/
file_afaap.txt file_fgrdw.txt file_sardf.txt
What can I do?
This little BaSH script will do it both ways:
#!/bin/sh
#
# counter
i=0
# put your new directory here
# can't be similar to dir_*, otherwise bash will
# expand it too
mkdir newdir
for file in `ls dir_*/*`; do
# gets only the name of the file, without directory
fname=`basename $file`
# gets just the file name, without extension
name=${fname%.*}
# gets just the extention
ext=${fname#*.}
# get the directory name
dir=`dirname $file`
# get the directory suffix
suffix=${dir#*_}
# rename the file using counter
fname_counter="${name}_$((i=$i+1)).$ext"
# rename the file using dir suffic
fname_suffix="${name}_$suffix.$ext"
# copy files using both methods, you pick yours
cp $file "newdir/$fname_counter"
cp $file "newdir/$fname_suffix"
done
And the output:
$ ls -R
cp.sh*
dir_asdf/
dir_ljklj/
dir_qwvas/
newdir/
out
./dir_asdf:
file.txt
./dir_ljklj:
file.txt
./dir_qwvas:
file.txt
./newdir:
file_1.txt
file_2.txt
file_3.txt
file_asdf.txt
file_ljklj.txt
file_qwvas.txt
while read -r line; do
suffix=$(sed 's/^.*_\(.*\)\/.*$/\1/' <<<$line)
newfile=$(sed 's/\.txt/$suffix\.txt/' <<<$line)
cp "$line" "~/direct_new/$newfile"
done <file_list.txt
where file_list is a list of your files.
You can achieve this with Bash parameter expansion:
dest_dir=direct_new
# dir based naming
for file in direct_*/file.txt; do
[[ -f "$file" ]] || continue # skip if not a regular file
dir="${file%/*}" # get the dir name from path
cp "$file" "$dest_dir/file_${dir#*direct_}.txt"
done
# count based naming
counter=0
for file in direct_*/file.txt; do
[[ -f "$file" ]] || continue # skip if not a regular file
cp "$file" "$dest_dir/file_$((++counter)).txt"
done
dir="${file%/*}" removes all characters starting from /, basically, giving us the dirname
${dir#*direct_} removes the direct_ prefix from dirname
((++counter)) uses Bash arithmetic expression to pre-increment the counter
See also:
Why you shouldn't parse the output of ls(1)
Get file directory path from file path
How to use double or single brackets, parentheses, curly braces
It may not be quite what you want, but it will do the job. Use cp --backup=numbered <source_file> <destination_directory:
$ find . -name test.sh
./ansible/test/integration/roles/test_command_shell/files/test.sh
./ansible/test/integration/roles/test_script/files/test.sh
./Documents/CGI/Code/ec-scripts/work/bin/test.sh
./Documents/CGI/Code/ec-scripts/trunk/bin/test.sh
./Test/test.sh
./bin/test.sh
./test.sh
$ mkdir BACKUPS
$ find . -name test.sh -exec cp --backup=numbered {} BACKUPS \;
cp: './BACKUPS/test.sh' and 'BACKUPS/test.sh' are the same file
$ ls -l BACKUPS
total 28
-rwxrwxr-x. 1 jack jack 121 Jun 9 10:29 test.sh
-rwxrwxr-x. 1 jack jack 34 Jun 9 10:29 test.sh.~1~
-rwxrwxr-x. 1 jack jack 34 Jun 9 10:29 test.sh.~2~
-rwxrwxr-x. 1 jack jack 388 Jun 9 10:29 test.sh.~3~
-rwxrwxr-x. 1 jack jack 388 Jun 9 10:29 test.sh.~4~
-rwxrwxr-x. 1 jack jack 20 Jun 9 10:29 test.sh.~5~
-rwxrwxr-x. 1 jack jack 157 Jun 9 10:29 test.sh.~6~
If you really want to put part of the folder name in, you have to decide exactly what part you want. You could, of course, just replace the directory separator character with some other character, and put the whole path into the filename.