Identify pattern in text files in a single unix directory - linux

Question

If i want to identify a pattern in Unix in one single directory, may i know which unix utility will be helpful ( like awk )
Input :
$ ls
a_20171007_001.txt
a_20171007_002.txt
b_20171007_001.txt
c_20180101_001.txt
expecting output :
a_20171007_002.txt
b_20171007_001.txt
The output should return latest version of file based on filename irrespective of file creation time
The output file shouldn't have future dated file ( e.g., current date :20171008 so 20180101 shouldn't come in output )
any suggestions on how to achieve this easily in unix ( awk or sed )
Thanks alot for all your solutions. But unfortunately if the file name is not follow any pattern it is not helping.
eg, input :
ab_bc_all_20171008_001.txt
bc_cd_ad_all_20171008_001.txt
ab_bc_all_20171008_002.txt
ad_dc_cd_ed_all_20180101_001.txt
ae_bc_zx_ed_ac_all_20170918_001.txt
output :
bc_cd_ad_all_20171008_001.txt
ab_bc_all_20171008_002.txt
ae_bc_zx_ed_ac_all_20170918_001.txt
in above case only pattern after 'all' the date field is appearing.
Can you please suggest in above case..
Thanks in advance.

Answer 1

Something like this in Perl:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
use Time::Piece;
my $today = localtime->ymd("");
my %latest;
for my $file (glob '*.txt') {
my ($id, $date, $num) = split /[_.]/, $file;
$latest{$id}{$date} = $num
if $date <= $today
&& (! exists $latest{$id}
|| ! exists $latest{$id}{$date}
|| $num > $latest{$id}{$date});
}
for my $id (keys %latest) {
for my $date (keys %{ $latest{$id} }) {
say "$id\_$date\_$latest{$id}{$date}.txt";
}
}

Answer 2

a simple awk solution
$ awk -F_ -vdate=`date +%Y%m%d` ' !($1 in file) && $2<=date {file[$1]=$0} ($1 in file){if($0>=file[$1]){file[$1]=$0}} END{ for(i in file)print file[i] }' f1
a_20171007_002.txt
b_20171007_001.txt
Explanation:
Store the current date in date variable in the format yyyymmdd
While iterating though records/filenames, if the date in filename i.e $2 is less than or equal to current date and the prefix (for eg. a,b etc) doesn't exist in array file then store it in file array for eg. file['a']=a_20171007_001.txt else it won't be stored and in this example c_20180101_001.txt would be straight forwardly rejected.
For next records, if the prefix i.e $1 exists in array file then check if the whole record is greater than the existing record (lexicographically). If yes, overwrite the record in file array.

Answer 3

Could you please try following and let me know if this helps you.
ls -ltr *.txt | awk -v date=$(date +%Y) -F"_" 'prev != $1 && val && date_val<=date{print val} {prev=$1;val=$0;date_val=substr($2,1,4)} END{if(date_val<=date){print val}}'
Adding a better readable form of solution too now.
ls -ltr *.txt | awk -v date=$(date +%Y) -F"_" '
prev != $1 && val && date_val<=date{
print val
}
{
prev=$1;
val=$0
date_val=substr($2,1,4)
}
END{
if(date_val<=date){
print val
}
}'

Answer 4

GNU Awk solution for static filename format <prefix>_<date>_<version>.txt:
Exemplary ls -1 output (extended):
a_20171007_001.txt
a_20171007_002.txt
b_20171007_001.txt
c_20180101_001.txt
a_20171007_0010.txt
b_20171007_004.txt
ls -1 | awk -F'[_.]' '{ k=$1"_"$2 }{ if (a[k]<$3) a[k]=$3 }
END{
for (i in a) {
split(substr(i, index(i,"_")+1), b, "");
ts=mktime(sprintf("%d %d %d 00 00 00",b[1]b[2]b[3]b[4],b[5]b[6],b[7]b[8]));
if (systime() >= ts) print i"_"a[i]".txt"
}
}'
The output:
b_20171007_004.txt
a_20171007_0010.txt

Answer 5

This one is ok only in shell (dash)
d=$(date +%Y%m%d)
ls -1r *_*_*.txt|while IFS='_' read w x y
do
[ "$x" -le "$d" ] && [ "$v" != "$w$x" ] && { echo "$w"_"$x"_"$y";v="$w$x";}
done
The spec change ???
Try this one
d=$(date +%Y%m%d)
ls -1r *_*_*.txt|while read l
do
b="${l%_*_*}"
a="${l#$b*_}"
c="${a%_*}"
[ "$c" -le "$d" ] && [ "$v" != "$b$c" ] && { echo "$l";v="$b$c";}
done

Answer 6

$ ls -1r | awk -v today="$(date +%Y%m%d)" -F'_' '($2 <= today) && !seen[$1,$2]++'
b_20171007_001.txt
a_20171007_002.txt