I'm making this tiny program in Shell:
#***************************************************************
# Function.
# NAME: chk_df
# Synopsis:
# Check if a local directory (dirName) exist and has a file (fileName).
#
#
# The return codes are the following:
# 99 : dirName does not exists
# 0 : dirName exists and has fileName
# 1 : dirName exists and has not fileName
#
# Parameters:
# In values: dirName <string> fileName <string>
# Out values: returnCode <int>
#
# How to use:
# chk_df dirName fileName
#***************************************************************
chk_df(){
# Check the number of arguments that could be passed.
# In this case, two, dirName, fileNAme.
if [[ ${##} != 2 ]]; then
echo "Error ...Use [Function]: chk_df <dirName> <fileName>"
echo "Ex: chk_df /foo lola.txt"
exit
fi
DIR=$1
FILE=$2
[[ ! -d $DIR ]] && return 99
[[ -d $DIR && ! -e $DIR/$FILE ]] && return 1
[[ -d $DIR && -e $DIR/$FILE ]] && return 0
}
Because I need to check if a file is in a directory, I did this (horrible?) patch $DIR/$FILE , but things like this could happen:
I) If we do: chk_df /foo lola.txt
We get: /foo/lola.txt
II) If we do: chk_df /foo/ lola.txt
We get: /foo//lola.txt [Notice the //]
In both cases the code seems to work. Why? I read that backslash acts like a space. So, could I put n backslash without unknown problems?
Could I leave it like that or it will bring problems? Is there a difference? UNIX assume it to the right way?
EXTRA QUESTION: why I can not do the returns with negative numbers? This is: return -1
/ , //, or any string of consecutive slashes have the same meaning according to the POSIX standard, with the exception that they may have a different meaning at the beginning of a path (so /foo and //foo may denote different objects). Linux does not use this exception, so any number of consecutive slashes always means the same thing as a single /.
(The exception is there to cater to the needs of other Unix-like systems that use leading // to denote a network path.)
There are no difference.
// = /
You can, in principle, use as many / separators as you want (until you start hitting PATH_MAX or some other hard limitation):
$ ls /usr/bin///////////////less
/usr/bin///////////////less
One problem you'll run into is if you ever want to test that two paths are the same[*], because /usr/bin/less and /usr/bin//less are the same path but are different strings. It can be useful to canonicalise paths before comparison.
[*] Ignoring the fact that different paths can refer to the same object.
Related
NOTE BEFORE READING: The following question is described very precisely and that is the reason for the length of a question. If you want to understand the problem, it's better to read the entire thing. Many thanks for all the answers!
I am working on a bash script (.sh file) which will check certain values in every file of a directory. Bash script will be executed in a pre-commit (pre-commit is not a part of the question).
There is a directory that contains multiple .c files in multiple subdirectories. I want to check a part of two lines which are NOT in every .c file but only in some of them. The structure of a file that contains the useful information is as following:
/*
## SYMBOL = some_symbol1
## A2L_TYPE = PARAMETER
.
.
.
#! DEFAULT = some_value1
## END
*/
some_symbol1 = some_value1
/*
## SYMBOL = some_symbol2
## A2L_TYPE = PARAMETER
.
.
.
#! DEFAULT = some_value2
## END
*/
some_symbol2 = some_value2
This kind of structure is automatically generated by another script.
I want to check if some_value1 (in comment) is equal to some_value1 (in variable).
There are hundreds of these variable in each .c file (not necessarily in each .c file).
The main functionality of a script should be:
Check some_value1 in comment and variable and throw an error if they are not the same. Script has to go through EVERY .c file in a directory (bash is in root) and ALL subdirectories to find previously mentioned structure.
Value of variable can be something as 0.06F, where in comment, there is 0.06 (compare only the numbers)
Value of variable can also be an array: { 0.0F, 0.45F, 0.3F } where in the comment, there is [ 0.0, 0.45, 0.3 ] (without F and difference in braces)
To summarize:
I want to build a check script that compares some_value1 (in comment) and some_value1 (in variable) and throw an error if they don't match
Useful information is not in EVERY .c file but only in some of them (don't know which)
Values after #! DEFAULT is a comment where the value of variable is a number (maybe this is not that important?)
between A2L_TYPE and DEFAULT, there can be desired number of unimportant stuff. (still a comment)
What I tried so far is for loop through every .c file and a nested for loop to read every line in each .c file. What I wanted to implement was a grep command inside for loop to check each line if there is a #! DEFAULT pattern and save it to the variable.
Latest code that I tried:
!/bin/bash
shopt -s globstar
for d in */**/*.c
do
while IFS="" read -r p || [ -n "$p" ]
do
grep -P "#! DEFAULT" $d
done < $d
done
This is currently not working because it gives an error that certain grep targets are directories
If any has any questions, I will try to explain it better.
# search for files with extension ".c"
# execute awk on any matches, using '= ' as field separator
find . -type f -name '*.c' -exec awk -F'=[[:space:]]*' '
# check if first three lines match template
( NR==1 && /^\/\*/ ) ||
( NR==2 && /^## SYMBOL = / ) ||
( NR==3 && /^## A2L_TYPE = PARAMETER/ ) { ok++ }
# template mismatch - skip this file
( NR==4 && ok!=3 ) {
printf "%s : ignored\n", FILENAME
nextfile
}
# store first occurrence of some_value1
# note line number where second occurrence expected
/^#! DEFAULT =/ { v[1]=v1=$2; n=NR+3 }
# test second occurrence
NR==n {
v[2]=v2=$2;
# prune everything except numbers and array delimiters
for (s in v) gsub(/[^0-9.,]/,"",v[s]);
# output result
# match exactly or only number list
printf "%s #(%d,%d) : ", FILENAME,n-3,n
if (v1==v2 || v[1]==v[2])
printf "match (%s)==(%s)\n", v1,v2
else
printf "mismatch (%s)!=(%s)\n", v1,v2
# no need to check rest of this file
# elide to check multiple values per file
nextfile
}
' {} +
I know I can do this to reflect just last 2 directories in the PS1 value.
PS1=${PWD#"${PWD%/*/*}/"}#
but lets say we have a directory name that's really messy and will reduce my working space , like
T-Mob/2021-07-23--07-48-49_xperia-build-20191119010027#
OR
2021-07-23--07-48-49_nokia-build-20191119010027/T-Mob#
those are the last 2 directories before the prompt
I want to set a condition if directory length of either of the last 2 directories is more than a threshold e.g. 10 chars , shorten the name with 1st 3 and last 3 chars of the directory (s) whose length exceeds 10
e.g.
2021-07-23--07-48-49_xperia-build-20191119010027 &
2021-07-23--07-48-49_nokia-build-20191119010027
both gt 10 will be shortened to 202*027 & PS1 will be respectively
T-Mob/202*027/# for T-Mob/2021-07-23--07-48-49_xperia-build-20191119010027# and
202*027/T-Mob# for 2021-07-23--07-48-49_nokia-build-20191119010027/T-Mob#
A quick 1 Liner to get this done ?
I cant post this in comments so Updating here. Ref to Joaquins Answer ( thx J)
PS1=''`echo ${PWD#"${PWD%/*/*}/"} | awk -v RS='/' 'length() <=10{printf $0"/"}; length()>10{printf "%s*%s/", substr($0,1,3), substr($0,length()-2,3)};'| tr -d "\n"; echo "#"`''
see below o/p's
/root/my-applications/bin # it shortened as expected
my-*ons/bin/#cd - # going back to prev.
/root
my-*ons/bin/# #value of prompt is the same but I am in /root
A one-liner is basically always the wrong choice. Write code to be robust, readable and maintainable (and, for something that's called frequently or in a tight loop, to be efficient) -- not to be terse.
Assuming availability of bash 4.3 or newer:
# Given a string, a separator, and a max length, shorten any segments that are
# longer than the max length.
shortenLongSegments() {
local -n destVar=$1; shift # arg1: where do we write our result?
local maxLength=$1; shift # arg2: what's the maximum length?
local IFS=$1; shift # arg3: what character do we split into segments on?
read -r -a allSegments <<<"$1"; shift # arg4: break into an array
for segmentIdx in "${!allSegments[#]}"; do # iterate over array indices
segment=${allSegments[$segmentIdx]} # look up value for index
if (( ${#segment} > maxLength )); then # value over maxLength chars?
segment="${segment:0:3}*${segment:${#segment}-3:3}" # build a short version
allSegments[$segmentIdx]=$segment # store shortened version in array
fi
done
printf -v destVar '%s\n' "${allSegments[*]}" # build result string from array
}
# function to call from PROMPT_COMMAND to actually build a new PS1
buildNewPs1() {
# declare our locals to avoid polluting global namespace
local shorterPath
# but to cache where we last ran, we need a global; be explicit.
declare -g buildNewPs1_lastDir
# do nothing if the directory hasn't changed
[[ $PWD = "$buildNewPs1_lastDir" ]] && return 0
shortenLongSegments shorterPath 10 / "$PWD"
PS1="${shorterPath}\$"
# update the global tracking where we last ran this code
buildNewPs1_lastDir=$PWD
}
PROMPT_COMMAND=buildNewPs1 # call buildNewPs1 before rendering the prompt
Note that printf -v destVar %s "valueToStore" is used to write to variables in-place, to avoid the performance overhead of var=$(someFunction). Similarly, we're using the bash 4.3 feature namevars -- accessible with local -n or declare -n -- to allow destination variable names to be parameterized without the security risk of eval.
If you really want to make this logic only apply to the last two directory names (though I don't see why that would be better than applying it to all of them), you can do that easily enough:
buildNewPs1() {
local pathPrefix pathFinalSegments
pathPrefix=${PWD%/*/*} # everything but the last 2 segments
pathSuffix=${PWD#"$pathPrefix"} # only the last 2 segments
# shorten the last 2 segments, store in a separate variable
shortenLongSegments pathSuffixShortened 10 / "$pathSuffix"
# combine the unshortened prefix with the shortened suffix
PS1="${pathPrefix}${pathSuffixShortened}\$"
}
...adding the performance optimization that only rebuilds PS1 when the directory changed to this version is left as an exercise to the reader.
Probably not the best solution, but a quick solution using awk:
PS1=`echo ${PWD#"${PWD%/*/*}/"} | awk -v RS='/' 'length()<=10{printf $0"/"}; length()>10{printf "%s*%s/", substr($0,1,3), substr($0,length()-2,3)};'| tr -d "\n"; echo "#"`
I got this results with your examples:
T-Mob/202*027/#
202*027/T-Mob/#
I are creating a .sh using bash for validate the api sub folders versions
The objective is validate this strings into APIS_BUILD var and find all .proto files into ./proto folder to compile into protobuffer Go file
# define subfolder apis to build
APIS_BUILD=(
prototests/v1/
prototests2/v2
testfolder
)
# the "testfolder" are a invalid folder
Test cases:
prototestes/v1 # valid
prototestes/v1/cobranca # valid
prototestes/v1/cobrnaca/faturamento # valid
outrapastacomarquivosproto/v1 # valid
prototests # invalid
/prototests # invalid
Then, I created this script for validate the APIS_BUILD string array
#!/usr/bin/env bash
# text color
RED='\033[0;31m' # RED
BLUE='\033[0;34m' # Blue
NC='\033[0m' # No Color
# Underline color
UCyan='\033[4;36m' # Cyan
# define subfolder apis to build
APIS_BUILD=(
prototests/v1
cobrancas/v1
)
DST_DIR="." # define the directory to store the build-in protofiles
SRC_DIR="./proto" # define the proto files folder
# Compile proto file
# $1 = Filename to compile
function compile() {
protoc --go_out=$DST_DIR --proto_path=proto --go_opt=M$1=services \
--go_opt=paths=import --go-grpc_out=. \
$1
}
# Validate api_build's
function validateApiBuilds() {
for t in ${APIS_BUILD[#]}; do
IFS="/"
read -a SUBSTR <<<"$t"
if [ ${#SUBSTR[#]} -lt 2 ]; then
printf "${RED}The API_BUILD value ${UCyan}\"${t}\"${RED} are declare wrong, please declare [api_folder]/[version_folder] (example: prototest/v1)${NC}\n" 1>&2
exit 1
fi
done
}
validateApiBuilds
for filename in $(find $SRC_DIR -name '*.proto'); do
[ -e "$filename" ] || continue
echo $filename
done
The subfolder:
But I getting a strange behavior:
If run the .sh file with the validateApiBuilds function the return for $filename is always .
If run the .sh file without the validateApiBuilds function the return for $filename are getting the testservice.proto file
Pictures:
With validateApiBuilds function:
Without validateApiBuilds function:
All the variables:
# define subfolder apis to build
APIS_BUILD=(
prototests/v1
cobrancas/v1
)
DST_DIR="." # define the directory to store the build-in protofiles
SRC_DIR="./proto" # define the proto files folder
Bash version:
$ bash --version
$ GNU bash, versão 4.4.19(1)-release (x86_64-pc-linux-gnu)
Obs.: I changed the validateApiBuilds function to use a regex validation for strings into API_BUILDS variable. But I really wanted to know the reason for this behavior.
edit 2: The make-proto.config file
# define subfolder apis to build
APIS_BUILD=(
prototests/v1
cobrancas/v1
)
DST_DIR="." # define the directory to store the build-in protofiles
SRC_DIR="./proto" # define the proto files folder
Use find better
for filename in $(anything) is always an antipattern -- it splits values on characters in IFS, and then expands each result as a glob. To make find emit completely unambiguous strings, use -print0:
while IFS= read -r -d '' filename; do
[ -e "$filename" ] || continue
echo "$filename"
done < <(find "$SRC_DIR" -name '*.proto' -print0)
Don't change IFS unnecessarily
Change your code to make the assignment to IFS be on the same line as the read, which will make IFS only be modified for that one command.
That is to say, instead of:
IFS=/
read -a SUBSTR <<<"$t"
...you should write:
IFS=/ read -a SUBSTR <<<"$t"
I am listing the files in a directory and looping through them okay, BUT I need to know how many there are too. ${#dirlist[#]} is always 1, but for loop works?
#!/bin/bash
prefix="xxx"; # as example
len=${#prefix}; # string length
dirlist=`ls ${prefix}*.text`;
qty=${#dirlist[#]}; # sizeof array is always 1
for filelist in $dirlist
do
substring="${filelist:$len:-5}";
echo "${substring}/${qty}";
done
I have files xxx001.text upto xxx013.text
but all I get is 001/1 002/1 003/1
This:
dirlist=`ls ${prefix}*.text`
doesn't make an array. It only makes a string with space separated file names.
You have to do
dirlist=(`ls ${prefix}*.text`)
to make it an array.
Then $dirlist will reference only the first element, so you have to use
${dirlist[*]}
to reference all of them in the loop.
Declare an array of files:
arr=(~/myDir/*)
Iterate through the array using a counter:
for ((i=0; i < ${#arr[#]}; i++)); do
# [do something to each element of array]
echo "${arr[$i]}"
done
You're not creating an array unless you surround it with ( ):
dirlist=(`ls ${prefix}*.text`)
dir=/tmp
file_count=`ls -B "$dir" | wc -l`
echo File count: $file_count
The array syntax in bash is simple, using parentheses ( and ):
# string
var=name
# NOT array of 3 elements
# delimiter is space ' ' not ,
arr=(one,two,three)
echo ${#arr[#]}
1
# with space
arr=(one two three)
# or ' ',
arr=(one, two, three)
echo ${#arr[#]}
3
# brace expansion works as well
# 10 elements
arr=({0..9})
echo ${#arr[#]}
10
# advanced one
curly_flags=(--{ftp,ssl,dns,http,email,fc,fmp,fr,fl,dc,domain,help});
echo ${curly_flags[#]}
--ftp --ssl --dns --http --email --fc --fmp --fr --fl --dc --domain --help
echo ${#curly_flags[#]}
12
if you want to run a command and store the output
# a string of output
arr=$(ls)
echo ${#arr[#]}
1
# wrapping with parentheses
arr=($(ls))
echo ${#arr[#]}
256
A more advanced / handy way is by using built-in bash commands mapfile or readarray and process substitution. here is is an example of using mapfile:
# read the output of ls, save it in the array name: my_arr
# -t Remove a trailing DELIM from each line read (default newline)
mapfile -t my_arr < <(ls)
echo ${#my_arr[#]}
256
Problem: to have a tab completion which takes two words and calculates the best match from them for Man, and then returns the best matches
Example: The following pseudo-code should give me at least Zsh's reverse-menu-complete -command. Right now, I cannot search manuals inside manuals without zgrep.
man zsh:reverse <TAB>
where ":" is the separator which I want.
Initial Problem: Which files does the TAB completion run when I press TAB for one word in searching manuals by Zsh?
I will attempt to provide an insight to how zsh completion system works and an incomplete go at this problem.
The file that runs when you use TAB completion for man in zsh is located under the /usr/share/zsh/${ZSH_VERSION}/functions directory. The tree varies across distributions, but the file is named _man, and provides completion for man, apropos and whatis.
After _man is invoked, it works as following (rough description):
if completing for man and --local-file was specified as first flag, invoke standard files completion (_files)
construct manpath variable from a set of defaults / $MANPATH. This is where the manpages will be searched
determine if we invoked man with a section number parameter, if yes - only those sections will be searched
if the zstyle ':completion:*:manuals' separate-sections true was used, separate sections in output (don't mix between them)
invoke _man_pages to provide a list of man pages for the match
_man_pages now does a bit of magic with compfiles -p pages '' '' "$matcher" '' dummy '*'. pages is the variable with all the directories containing manpages for requested section(s). The actual globbing pattern is constructed from the implicit parameter $PREFIX and the last parameter to compfiles - * in this case. This results in /usr/share/man/man1 to be transformed into /usr/share/man/man1/foo*
The new list of glob patterns is globbed, obtaining all files which match the pattern
_man_pages then strips any suffixes from the files and adds them to the completion widget list of choices by using compadd
Now, as you can see, the list of manpages is directly determined by $PREFIX variable. In order to make zsh:foo to list only man pages of zsh* which contain the word foo, it needs to be split across : character (if any).
The following addition in _man_pages partially solve the issue (zsh 4.3.4):
Original:
_man_pages() {
local matcher pages dummy sopt
zparseopts -E M+:=matcher
if (( $#matcher )); then
matcher=( ${matcher:#-M} )
matcher="$matcher"
else
matcher=
fi
pages=( ${(M)dirs:#*$sect/} )
compfiles -p pages '' '' "$matcher" '' dummy '*'
pages=( ${^~pages}(N:t) )
(($#mrd)) && pages[$#pages+1]=($(awk $awk $mrd))
# Remove any compression suffix, then remove the minimum possible string
# beginning with .<->: that handles problem cases like files called
# `POSIX.1.5'.
[[ $OSTYPE = solaris* ]] && sopt='-s '
if ((CURRENT > 2)) ||
! zstyle -t ":completion:${curcontext}:manuals.$sect" insert-sections
then
compadd "$#" - ${pages%.(?|<->*(|.gz|.bz2|.Z))}
else
compadd "$#" -P "$sopt$sect " - ${pages%.(?|<->*(|.gz|.bz2|.Z))}
fi
}
Modified (look for ##mod comments):
_man_pages() {
local matcher pages dummy sopt
zparseopts -E M+:=matcher
if (( $#matcher )); then
matcher=( ${matcher:#-M} )
matcher="$matcher"
else
matcher=
fi
pages=( ${(M)dirs:#*$sect/} )
##mod
# split components by the ":" character
local pref_words manpage_grep orig_prefix
# save original prefix (just in case)
orig_prefix=${PREFIX}
# split $PREFIX by ':' and make it an array
pref_words=${PREFIX//:/ }
set -A pref_words ${=pref_words}
# if there are both manpage name and grep string, use both
if (( $#pref_words == 2 )); then
manpage_grep=$pref_words[2]
# PREFIX is used internally by compfiles
PREFIX=$pref_words[1]
elif (( $#pref_words == 1 )) && [[ ${PREFIX[1,1]} == ":" ]]; then
# otherwise, prefix is empty and only grep string exists
PREFIX=
manpage_grep=$pref_words[1]
fi
compfiles -p pages '' '' "$matcher" '' dummy '*'
##mod: complete, but don't strip path names
pages=( ${^~pages} )
(($#mrd)) && pages[$#pages+1]=($(awk $awk $mrd))
##mod: grep pages
# Build a list of matching pages that pass the grep
local matching_pages
typeset -a matching_pages
# manpage_grep exists and not empty
if (( ${#manpage_grep} > 0 )); then
for p in ${pages}; do
zgrep "${manpage_grep}" $p > /dev/null
if (( $? == 0 )); then
#echo "$p matched $manpage_grep"
matching_pages+=($p)
fi
done
else
# there's no manpage_grep, so all pages match
matching_pages=( ${pages} )
fi
#echo "\nmatching manpages: "${matching_pages}
pages=( ${matching_pages}(N:t) )
# keep the stripped prefix for now
#PREFIX=${orig_prefix}
# Remove any compression suffix, then remove the minimum possible string
# beginning with .<->: that handles problem cases like files called
# `POSIX.1.5'.
[[ $OSTYPE = solaris* ]] && sopt='-s '
if ((CURRENT > 2)) ||
! zstyle -t ":completion:${curcontext}:manuals.$sect" insert-sections
then
compadd "$#" - ${pages%.(?|<->*(|.gz|.bz2|.Z))}
else
compadd "$#" -P "$sopt$sect " - ${pages%.(?|<->*(|.gz|.bz2|.Z))}
fi
}
However, it's still not fully working (if you uncomment the #echo "$p matched $manpage_grep" line, you can see that it does build the list) - I suspect that somewhere internally, the completion system sees that, for instance, "zshcompctl" is not matched by prefix "zsh:foo", and does not display the resulting matches. I've tried to keep $PREFIX as it is after stripping the grep string, but it still does not want to work.
At any rate, this at least should get you started.