I recently sorted a bunch of files into directories by subject. I'm trying to change the date of the directory to be the same as the modification time of the newest file in the directory. I have a script that I think should work, but it doesn't.
for DIRECTORY in * ; do FILE=`ls -1 --sort=time "$DIRECTORY" | tail -n 1` ; DATE=`stat -c %y "$DIRECTORY/$FILE" | tr '.' ' ' | awk '{print $1,$2}' ` ; echo $DATE $DIRECTORY ; touch -m -d "$DATE" "$DIRECTORY" ; done
It seems like it should work, but it doesn't. Any help?
Simply run this command :
latestFile=$(ls -1t | sed q)
for i in *; do touch -r "$latestFile"; done
! The command is not compatible with files with spaces !
Related
I am trying to create a script which will list the name of the user, only if that specific user has a specific and available directory in his folder.
#!/bin/bash
for i in `cat /etc/passwd | cut -d : -f5\,6 | cut -d : -f2`
do
cd $i
if [ -d public_html ]
then
echo `cat /etc/passwd | cut -d : -f5,6 | cut -d : -f1`
fi
done
First, I get a list of all the user names that have home folders.
Then, for each user, I enter in his directory
If in his directory, public_html directory is found, echo the user.
When I run this in the terminal, all the users are listed:
cat /etc/passwd | cut -d : -f5,6 | cut -d : -f1
However, I need to somehow get the user i from that whole list.
Can anybody be so kind to explain what I´m doing wrong, and what to look out for?
You're getting the wrong result because you're checking for a file (with -f). Try using -d.
Using cat and cut is not bash either and can easily be replaced with script in most cases.
I'd do something like this:
while IFS=: read -r user pass uid gid desc homedir shell; do [ -d "$homedir"/public_html ] && echo "$user"; done < /etc/passwd
We set the field separator (IFS) to : (since that's what /etc/passwd uses), we read the file in and set the variables we want to use.
Then for each line we do the test and (&&) if the test is successful we echo the result.
The quotes are not really necessary since we know the formatting of the file but good practice.
If users' home directories are set up in the standard way:
cd /home
for pubdir in */public_html/ ; do
echo "${pubdir%%/*}"
done
I did it ghetto style...
#!/bin/bash
counter=0
for i in `cat /etc/passwd | cut -d : -f6`
do
if [ -r $i ]
then
cd $i
if [ -d public_html ]
then
counter2=2
for j in `cat /etc/passwd | cut -d : -f5`
do
counter2=$(($counter2 + 1))
if [ $counter -eq $counter2 ]
then
echo $j
fi
done
fi
counter=$(($counter + 1))
fi
done
#!/bin/bash
getent passwd | awk -F: '{printf "%s\t%s\n",$1, $6}'| while read -r user home
do
if [ -d ${home}/public_html ] ; then
echo ${user}
fi
done
This should work irrespective of use of /etc/passwd or ldap ...
I have the following script cleaning egrep arguments from all .csv files in current folders (used to clean email lists):
#!/bin/bash
for file in $(find . -name "*.csv" ); do
echo "====================================================" >> db_purge_log.txt
echo "$file" >> db_purge_log.txt
echo "----------------------------------------------------" >> db_purge_log.txt
echo "Contacts BEFORE purge:" >> db_purge_log.txt
wc -l $file | cut -d " " -f1 >> db_purge_log.txt
echo " " >> db_purge_log.txt
cat $file | egrep -v "marketing" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
echo "$file is now clean!"
mv tmp_file $file ;
echo "Contacts AFTER purge:" >> db_purge_log.txt
wc -l $file | cut -d " " -f1 >> db_purge_log.txt
done
I would like the egrep -v "marketing" section to be running in a loop on a file called X.csv and taking all the arguments from there. Eventually a list of around 6M contacts will be suppressed from another list of 6M contacts (need 6M*6M queries on the server if even possible).
Any idea how to accomplish that?
Note that your exclusion list will need to be one pattern (email address) per line, i.e. from the egrep man page:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing.
So modify your exclusion line as suggested by Orr, but, also make sure that your x.CSV file is really one email address per line. Also, this should most likely be case insensitive, so, perhaps something like:
cat $file | egrep -vi -f Excludes.txt | \
grep -v -E -i '([0-z])\1{2,}' | sort | uniq > tmp_file
Based on experience, I prefer to use uniq AFTER sort.
:)
Dale
By default, vim spell checker is code aware, so it doesn't spell-check code parts of the file. In effect, in markdown it considers (pandoc multiline) tables to be codes and thus doesn't spell-check their contents.
Is it possible to override this? Or enable spell-check for the entire file including code.
As far as I'm able to determine, there is no way to tell Vim to ignore the
spellcheck suggestions in the syntax file and to just "check everything".
A fairly heavy-handed workaround is to disable syntax entirely with :syn off;
you can re-enable this with :syn on.
Specifically for Markdown, you can disable highlighting of code blocks with
:syn clear markdownCodeBlock; you can reset this with :syn on as well.
Use syntax spell
:syntax spell toplevel
See:
http://usevim.com/2013/05/10/synspell/
In that case I would contact the maintainer of the markdown syntax file and ask him/she if (s)he could fix this issue.
I created bash script fixing syntax files. IT IS NOT PERFECT BUT IT IS GOOD. It can be reversed by running it again. It adds contains=#Spell to syn match and syn region definitions in all files in given directory.
To use it:
Save the script as fix_syntax_files.sh
Give it permissions
Change path at the bottom of the script to one corresponding to your vim plugins location
Run the script
(OPTIONAL) Run script again to revert the changes
The script makes backup of all files before modification so you can assume it is safe to run it. I anyway do not take any responsibility for potential problems caused by the script.
Edit:
You can leave feedback to the script in the following repository:
https://github.com/dominikduda/config_files/blob/master/bin/fix_vim_syntax_files.sh
#!/bin/bash
function fix_file {
sed -i -e '/exe/! {s/contains=/contains=#Spell,/g}' $1
sed -i -e 's/contains=#Spell,ALL/contains=ALL/g' $1
sed -i -e 's/contains=#Spell,ALLBUT/contains=ALLBUT/g' $1
sed -i -e 's/contains=#Spell,TOP/contains=TOP/g' $1
sed -i -e 's/contains=#Spell,CONTAINED/contains=CONTAINED/g' $1
sed -i -e 's/contains=#Spell,NONE/contains=#Spell/g' $1
sed -i -e '/^ *syn match/ {/contains=/! s/$/ contains=#Spell/g}' $1
sed -i -e '/^ *syn region/ {/contains=/! s/$/ contains=#Spell/g}' $1
return 0
}
function revert_file {
mv "$1/$2.spellfix-backup" "$1/$2"
return 0
}
function fix_recursively_in_catalog {
syntax_catalogs_paths="$(find $1 -type d ! -name '*.*' -not -path '*git*' -print)"
syntax_catalogs_count="$(echo "${syntax_catalogs_paths}" | wc -l)"
echo "${syntax_catalogs_count} syntax catalogs found and will be scanned for files"
echo "${syntax_catalogs_paths}" | while read -r catalog_path ; do
echo " Scanning $catalog_path"
ls -p "${catalog_path}" | grep -v / | grep -v .spellfix-backup | grep .vim | while read -r file_name ; do
cp "${catalog_path}/${file_name}" "${catalog_path}/${file_name}.spellfix-backup"
fix_file "${catalog_path}/${file_name}"
echo " Fixing ${file_name} (backup created as ${file_name}.spellfix-backup)"
done
done
echo 'Fix done.'
echo 'Remember to REVERT FIX before updating vim plugins'
return 0
}
function revert_recursively_in_catalog {
syntax_catalogs_paths="$(find $1 -type d ! -name '*.*' -not -path '*git*' -print)"
syntax_catalogs_count="$(echo "${syntax_catalogs_paths}" | wc -l)"
echo "${syntax_catalogs_count} syntax catalogs found and will be scanned for spellfix-backup files"
echo "${syntax_catalogs_paths}" | while read -r catalog_path ; do
echo " Scanning $catalog_path"
ls -p "${catalog_path}" | grep -v / | grep -v .spellfix-backup | grep .vim | while read -r file_name ; do
revert_file "${catalog_path}" "${file_name}"
echo " Reverting ${file_name} (from file ${file_name}.spellfix-backup)"
done
done
echo 'Revert done.'
echo 'Remember to FIX AGAIN after plugins update (or set it as a post update hook)'
return 0
}
function main {
syntax_catalogs_paths="$(find $1 -type d ! -name '*.*' -not -path '*git*' -print)"
while read -r catalog_path ; do
if ls -p "${catalog_path}" | grep -v / | grep .spellfix-backup; then
echo ".spellfix-backup files found, reverting fix!"
echo "--------------------------------------------"
revert_recursively_in_catalog $1
return 0
fi
done < <(echo "${syntax_catalogs_paths}")
echo ".spellfix-backup files NOT found, fixing!"
echo "-----------------------------------------"
fix_recursively_in_catalog $1
}
main ~/PATH/TO/VIM/PLUGINS/
I created a script to download file from URL and I want to download it in the specific directory but the problem is when its time in downloading it will not put to the directory given and also when extracting the file is in the given directory.
diskspace=$(df -h /var/ | sed '1d' | awk '{print $5}' | cut -d'%' -f1)
bundle=$(awk -F = '{print $2}' config.txt)
allowed=10
if [ "${diskspace}" -gt "${allowed}" ]; then
cd `/var/`
wget $bundle
else
echo "Not enough space to download the bundle"
echo $output
exit
fi
while true; do
for f in *.tar.gz; do
case $f in '*.tar.gz') exit 0;; esac
tar zxf "$f"
rm -v "$f"
done
done
Can Someone help me to this problem ? The thing that I want to happen is to download the file in the given directory and also extract it there. Help is greatly appreciated.
I'm not a linux expert, and usually in this situation PHP would be much more suitable... But due to the circumstances it occurred that I wrote it in Bash :)
I have the following .sh which runs over all .csv files in the current folder and execute a bunch of commands.
The goal: Cleaning email lists in .csv files (not actually .csv but just a .txt file in practice).
for file in $(find . -name "*.csv" ); do
echo "====================================================" >> db_purge_log.txt
echo "$file" >> db_purge_log.txt
echo "----------------------------------------------------" >> db_purge_log.txt
echo "Contacts BEFORE purge:" >> db_purge_log.txt
wc -l $file | cut -d " " -f1 >> db_purge_log.txt
echo " " >> db_purge_log.txt
cat $file | egrep -v "xxx|yyy|zzz" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
mv tmp_file $file ;
echo "Contacts AFTER purge:" >> db_purge_log.txt
wc -l $file | cut -d " " -f1 >> db_purge_log.txt
done
Now the trouble is:
I want to add a command, somewhere in the middle of this loop, to use another .csv file as suppression list, meaning - every line found as perfect match in that suppression list - delete from $file.
At this point my brain is stuck and I can't think of a solution. To be honest, I didn't manage using sort or grep on 2 different files and export to a 3rd file without completely eliminating the duplicated lines cross both files, so I end up with much less data.
Any help would be much appreciated!
Clean up
Before adding functionality to the script, the existing script needs to be cleaned up — a lot.
I/O Redirection — Don't Repeat Yourself
When I see wall-to-wall I/O redirections like that, I want to cry — that isn't how you do it! You have three options to avoid all that:
for file in $(find . -name "*.csv" )
do
echo "===================================================="
echo "$file"
echo "----------------------------------------------------"
echo "Contacts BEFORE purge:"
wc -l $file | cut -d " " -f1
echo " "
cat $file | egrep -v "xxx|yyy|zzz" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
mv tmp_file $file ;
echo "Contacts AFTER purge:"
wc -l $file | cut -d " " -f1
done >> db_purge_log.txt
Or:
{
for file in $(find . -name "*.csv" )
do
echo "===================================================="
echo "$file"
echo "----------------------------------------------------"
echo "Contacts BEFORE purge:"
wc -l $file | cut -d " " -f1
echo " "
cat $file | egrep -v "xxx|yyy|zzz" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
mv tmp_file $file ;
echo "Contacts AFTER purge:"
wc -l $file | cut -d " " -f1
done
} >> db_purge_log.txt
Or even:
exec >>db_purge_log.txt # By default, standard output will go to db_purge_log.txt
for file in $(find . -name "*.csv" )
do
echo "===================================================="
echo "$file"
echo "----------------------------------------------------"
echo "Contacts BEFORE purge:"
wc -l $file | cut -d " " -f1
echo " "
cat $file | egrep -v "xxx|yyy|zzz" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
mv tmp_file $file ;
echo "Contacts AFTER purge:"
wc -l $file | cut -d " " -f1
done
The first form is adequate for this script which has a single loop in it to provide I/O redirection to. The second form, using { and } would handle more general sequences of commands. The third form, using exec, is 'permanent'; you can't recover the original standard output, whereas with the { ... } form you can have different sections of the script writing to different places.
One other advantage of all these variations is that you can trivially send errors to the same place that you're sending standard output if that's what you desire. For example:
exec >>db_purge_log.txt 2>&1
Other issues
Suppressing file name from wc — instead of:
wc -l $file | cut -d " " -f1
use:
wc -l < $file
UUOC — Useless use of cat — instead of:
cat $file | egrep -v "xxx|yyy|zzz" | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
use:
egrep -v "xxx|yyy|zzz" $file | grep -v -E -i '([0-z])\1{2,}' | uniq | sort -u > tmp_file
UUOU — Useless use of uniq
It is not at all clear why you need uniq and sort -u; in context, sort -u is sufficient, so:
egrep -v "xxx|yyy|zzz" $file | grep -v -E -i '([0-z])\1{2,}' | sort -u > tmp_file
UUOG — Useless use of grep
egrep is equivalent to grep -E and both are capable of handling multiple regular expressions, and the second will match what is matched by the expression in the parentheses 3 or more times (we really only need to match three times), so in fact the second expression will do the job of the first. And the [0-z] match is dubious. It probably matches sundry punctuation characters as well as the upper and lower case digits, but you're already doing a case-insensitive search because of the -i, so we can regularize all that to:
grep -Eiv '([0-9a-z]){3}' $file | sort -u > tmp_file
File names with spaces
The code is not going to handle file names with spaces, tabs or newlines because of the for file in $(find ...) notation. It probably isn't necessary to deal with that now — be aware of the issue.
Final clean up
for file in $(find . -name "*.csv" )
do
echo "===================================================="
echo "$file"
echo "----------------------------------------------------"
echo "Contacts BEFORE purge:"
wc -l < $file
echo " "
grep -Evi '([0-9a-z]){3}' | sort -u > tmp_file
mv tmp_file $file
echo "Contacts AFTER purge:"
wc -l <$file
done >> db_purge_log.txt
Add the extra functionality
I want to add a command, somewhere in the middle of this loop, to use another .csv file as suppression list — meaning that every line found as perfect match in that suppression list should be deleted from $file.
Since we're already sorting the input files ($file), we can sort the suppression file (call it suppfile='suppressions.txt'too if it is not already sorted. Given that, we then use comm to eliminate the lines that appear in both $file and $suppfile. We're interested in the lines that only appear in $file (or, as will be the case here, in the edited and sorted version of the file), so we want to suppress the common entries and the entries from $suppfile that do not appear in $file. The comm -23 - "$suppfile" command reads the edited, sorted file from standard input - and leaves out the entries from "$suppfile"
suppfile='suppressions.txt' # Must be in sorted order
for file in $(find . -name "*.csv" )
do
echo "===================================================="
echo "$file"
echo "----------------------------------------------------"
echo "Contacts BEFORE purge:"
wc -l < "$file"
echo " "
grep -Evi '([0-9a-z]){3}' | sort -u | comm -23 - "$suppfile" > tmp_file
mv tmp_file "$file"
echo "Contacts AFTER purge:"
wc -l < "$file"
done >> db_purge_log.txt
If the suppression file is not in sorted order, simply sort it into a temporary file. Beware of using the .csv suffix on the suppression file in the current directory; it will catch the file and empty it because every line in the suppression file matches a line in the suppression file, which is not helpful for any files processed after the suppression file.
Oops — I over-simplified the grep regex. It should (probably) be:
grep -Evi '([0-9a-z])\1{2}' $file
The difference is considerable. My original rewrite will look for any three adjacent digits or letters (e.g. 123 or abz); the revision (actually very similar to one of the original commands) looks for a character from [0-9A-Za-z] followed by two occurrences of the same character (e.g. 111 or aaa, but not 123 or abz).
If perchance the alternatives xxx|yyy|zzz were really not 3 repeated characters, you might need two invocations of grep in sequence.
If I understand you correctly, assuming a recent 'nix, grep should do most of the trick for you. The command, grep -vf filterfile input.csv will output the lines in input.csv that do NOT match any regular expression found in filterfile.
A couple of other comments ... uniq needs the input sorted in order to remove dups, so you might want the sort before it in the pipe (unless your input data is sorted).
Or if the input is sorted to start with, grep -u will omit duplicates.
Small suggestion -- you might add a #!/bin/bash as the first line in order to ensure that the script is run by bash rather than the user's login shell (it might not be bash).
HTH.
b