BASH + how to verify the words in array are contain in the variable - linux

Question

Hi friend and colleges
I wrote the following script in order to verify the words in array are contain in the $list variable
#!/bin/bash
list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
array=( sdb sdd sde sdf sdg )
function contain_word
{
contain=false
[[ -z "${list// }" ]] && return
for arr in ${array[*]}
do
echo "$list" | grep -q $arr
[[ $? -eq 0 ]] && (( count ++ ))
done
[[ ${#array[#]} -eq $count ]] && export contain=true
}
contain_word
echo $contain
this script do the job but its long code for this purpose and ugly
I will happy to get good idea how to do it better ( in bash / awk / perl one liner etc )
Example1
For
list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
array=( sdb sdd sde sdf sdg )
it will print true
Example2
For
list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
array=( sdw sdd sde sdf sdg )
it will print false

Answer 1

$ cat tst.sh
contain_word() {
list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
printf '%s\n' "${array[#]}" |
awk -v list="$list" '
BEGIN {
split(list,tmpArr)
for (idx in tmpArr) {
wordSet[tmpArr[idx]]
}
}
!($0 in wordSet) {
exit 1
}
'
}
array=( sdb sdd sde sdf sdg )
contain_word
printf '%s -> %s\n' "${array[*]}" "$?"
array=( sdw sdd sde sdf sdg )
contain_word
printf '%s -> %s\n' "${array[*]}" "$?"
$ ./tst.sh
sdb sdd sde sdf sdg -> 0
sdw sdd sde sdf sdg -> 1
The above uses full string comparison so no partial matches nor false regexp matches are possible. It also won't fail due to globbing and will work using any awk in any shell (that supports arrays using the syntax you provided) on any UNIX box. You can of course tweak the awk code or the calling shell code to print true or false rather than just the awk exit status as appropriate.

Answer 2

Here's a few lines of python that would do it.
[user#local ~/tmp/b] python
>>> list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
>>> array="sdb sdd sde sdf sdg"
>>> set(array.split(" ")).issubset(set(list.split(" ")))
True
>>> list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
>>> array="sdw sdd sde sdf sdg"
>>> set(array.split(" ")).issubset(set(list.split(" ")))
False

Answer 3

You can convert the scalar variable into array variable and compare both arrays (Not a single linear and probably there is a lot of way).
my $list = "sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo";
my #listary = split / /, $list;
my #myarray = qw(sdb sdd sde sdf sdg);
And please compare both arrays

Answer 4

My solution is awk based and it turns the list into a regular expression in order to remove every word in the list from the array. If the result is empty, true is printed; otherwise false is printed.
awk -v list="$list" '
{
gsub(" +","|",list)
gsub(" *("list") *","")
print ($0) ? "false" : "true"
}
' <<<"${array[*]}"

Answer 5

Edit2: Using Perl is nice, simple and very efficient.
perl -e'#h{split/ /,shift}=();exists$h{$_}||exit 1 for#ARGV' "$list" "${array[#]}" && echo "true" || echo "false"
Original:
It would be much simpler if array is in file but anyway
list="sdb sdc sdd sde sdf sdg sdh sdi sdk sdj sdo"
array=( sdb sdd sde sdf sdg )
[[ $(echo $list | sed 's/ /\n/g' | sort -u | grep -Ff <(echo ${array[#]} | sed 's/ /\n/g') | wc -l) -eq ${#array[#]} ]] && echo "true" || echo "false"
or shorter
[[ $(sed 's/ /\n/g' <<<$list | sort -u | grep -Ff <(sed 's/ /\n/g' <<<${array[#]}) | wc -l) -eq ${#array[#]} ]] && echo "true" || echo "false"
Why would anybody cycle around array? There is a difference between O(N*M) and O(N+M).
Edit: It seems that understanding how computers work and what is O notation is less common than I expected, there is a small demonstration.
#!/bin/bash
list="aaa aab aac aad aae aaf aag aah aai aaj aak aal aam aan aao aap aaq aar aas aat aau aav aaw aax aay aaz aba abb abc abd abe abf abg abh abi abj abk abl abm abn abo abp abq abr abs abt abu abv abw abx aby abz aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu acv acw acx acy acz ada adb adc add ade adf adg adh adi adj adk adl adm adn ado adp adq adr ads adt adu adv adw adx ady adz aea aeb aec aed aee aef aeg aeh aei aej aek ael aem aen aeo aep aeq aer aes aet aeu aev aew aex aey aez afa afb afc afd afe aff afg afh afi afj afk afl afm afn afo afp afq afr afs aft afu afv afw afx afy afz aga agb agc agd age agf agg agh agi agj agk agl agm agn ago agp agq agr ags agt agu agv agw agx agy agz aha ahb ahc ahd ahe ahf ahg ahh ahi ahj ahk ahl ahm ahn aho ahp ahq ahr ahs aht ahu ahv ahw ahx ahy ahz aia aib aic aid aie aif aig aih aii aij aik ail aim ain aio aip aiq air ais ait aiu aiv aiw aix aiy aiz aja ajb ajc ajd aje ajf ajg ajh aji ajj ajk ajl ajm ajn ajo ajp ajq ajr ajs ajt aju ajv ajw ajx ajy ajz aka akb akc akd ake akf akg akh aki akj akk akl akm akn ako akp akq akr aks akt aku akv akw akx aky akz ala alb alc ald ale alf alg alh ali alj alk all alm aln alo alp alq alr als alt alu alv alw alx aly alz ama amb amc amd ame amf amg amh ami amj amk aml amm amn amo amp amq amr ams amt amu amv amw amx amy amz ana anb anc and ane anf ang anh ani anj ank anl anm ann ano anp anq anr ans ant anu anv anw anx any anz aoa aob aoc aod aoe aof aog aoh aoi aoj aok aol aom aon aoo aop aoq aor aos aot aou aov aow aox aoy aoz apa apb apc apd ape apf apg aph api apj apk apl apm apn apo app apq apr aps apt apu apv apw apx apy apz aqa aqb aqc aqd aqe aqf aqg aqh aqi aqj aqk aql aqm aqn aqo aqp aqq aqr aqs aqt aqu aqv aqw aqx aqy aqz ara arb arc ard are arf arg arh ari arj ark arl arm arn aro arp arq arr ars art aru arv arw arx ary arz asa asb asc asd ase asf asg ash asi asj ask asl asm asn aso asp asq asr ass ast asu asv asw asx asy asz ata atb atc atd ate atf atg ath ati atj atk atl atm atn ato atp atq atr ats att atu atv atw atx aty atz aua aub auc aud aue auf aug auh aui auj auk aul aum aun auo aup auq aur aus aut auu auv auw aux auy auz ava avb avc avd ave avf avg avh avi avj avk avl avm avn avo avp avq avr avs avt avu avv avw avx avy avz awa awb awc awd awe awf awg awh awi awj awk awl awm awn awo awp awq awr aws awt awu awv aww awx awy awz axa axb axc axd axe axf axg axh axi axj axk axl axm axn axo axp axq axr axs axt axu axv axw axx axy axz aya ayb ayc ayd aye ayf ayg ayh ayi ayj ayk ayl aym ayn ayo ayp ayq ayr ays ayt ayu ayv ayw ayx ayy ayz aza azb azc azd aze azf azg azh azi azj azk azl azm azn azo azp azq azr azs azt azu azv azw azx azy azz baa bab bac bad bae baf bag bah bai baj bak bal bam ban bao bap baq bar bas bat bau bav baw bax bay baz bba bbb bbc bbd bbe bbf bbg bbh bbi bbj bbk bbl bbm bbn bbo bbp bbq bbr bbs bbt bbu bbv bbw bbx bby bbz bca bcb bcc bcd bce bcf bcg bch bci bcj bck bcl bcm bcn bco bcp bcq bcr bcs bct bcu bcv bcw bcx bcy bcz bda bdb bdc bdd bde bdf bdg bdh bdi bdj bdk bdl bdm bdn bdo bdp bdq bdr bds bdt bdu bdv bdw bdx bdy bdz bea beb bec bed bee bef beg beh bei bej bek bel bem ben beo bep beq ber bes bet beu bev bew bex bey bez bfa bfb bfc bfd bfe bff bfg bfh bfi bfj bfk bfl bfm bfn bfo bfp bfq bfr bfs bft bfu bfv bfw bfx bfy bfz bga bgb bgc bgd bge bgf bgg bgh bgi bgj bgk bgl bgm bgn bgo bgp bgq bgr bgs bgt bgu bgv bgw bgx bgy bgz bha bhb bhc bhd bhe bhf bhg bhh bhi bhj bhk bhl bhm bhn bho bhp bhq bhr bhs bht bhu bhv bhw bhx bhy bhz bia bib bic bid bie bif big bih bii bij bik bil bim bin bio bip biq bir bis bit biu biv biw bix biy biz bja bjb bjc bjd bje bjf bjg bjh bji bjj bjk bjl bjm bjn bjo bjp bjq bjr bjs bjt bju bjv bjw bjx bjy bjz bka bkb bkc bkd bke bkf bkg bkh bki bkj bkk bkl bkm bkn bko bkp bkq bkr bks bkt bku bkv bkw bkx bky bkz bla blb blc bld ble blf blg blh bli blj blk bll blm bln blo blp blq blr bls blt blu blv blw blx bly blz bma bmb bmc bmd bme bmf bmg bmh bmi bmj bmk bml"
array=(
aaa aab aac aad aae aaf aag aah aai aaj aak aal aam aan aao aap aaq aar aas
aat aau aav aaw aax aay aaz aba abb abc abd abe abf abg abh abi abj abk abl
abm abn abo abp abq abr abs abt abu abv abw abx aby abz aca acb acc acd ace
acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu acv acw acx
acy acz ada adb adc add ade adf adg adh adi adj adk adl adm adn ado adp adq
adr ads adt adu adv adw adx ady adz aea aeb aec aed aee aef aeg aeh aei aej
aek ael aem aen aeo aep aeq aer aes aet aeu aev aew aex aey aez afa afb afc
afd afe aff afg afh afi afj afk afl afm afn afo afp afq afr afs aft afu afv
afw afx afy afz aga agb agc agd age agf agg agh agi agj agk agl agm agn ago
agp agq agr ags agt agu agv agw agx agy agz aha ahb ahc ahd ahe ahf ahg ahh
ahi ahj ahk ahl ahm ahn aho ahp ahq ahr ahs aht ahu ahv ahw ahx ahy ahz aia
aib aic aid aie aif aig aih aii aij aik ail aim ain aio aip aiq air ais ait
aiu aiv aiw aix aiy aiz aja ajb ajc ajd aje ajf ajg ajh aji ajj ajk ajl ajm
ajn ajo ajp ajq ajr ajs ajt aju ajv ajw ajx ajy ajz aka akb akc akd ake akf
akg akh aki akj akk akl akm akn ako akp akq akr aks akt aku akv akw akx aky
akz ala alb alc ald ale alf alg alh ali alj alk all alm aln alo alp alq alr
als alt alu alv alw alx aly alz ama amb amc amd ame amf amg amh ami amj amk
aml amm amn amo amp amq amr ams amt amu amv amw amx amy amz ana anb anc and
ane anf ang anh ani anj ank anl anm ann ano anp anq anr ans ant anu anv anw
anx any anz aoa aob aoc aod aoe aof aog aoh aoi aoj aok aol aom aon aoo aop
aoq aor aos aot aou aov aow aox aoy aoz apa apb apc apd ape apf apg aph api
apj apk apl apm apn apo app apq apr aps apt apu apv apw apx apy apz aqa aqb
aqc aqd aqe aqf aqg aqh aqi aqj aqk aql aqm aqn aqo aqp aqq aqr aqs aqt aqu
aqv aqw aqx aqy aqz ara arb arc ard are arf arg arh ari arj ark arl arm arn
aro arp arq arr ars art aru arv arw arx ary arz asa asb asc asd ase asf asg
ash asi asj ask asl asm asn aso asp asq asr ass ast asu asv asw asx asy asz
ata atb atc atd ate atf atg ath ati atj atk atl atm atn ato atp atq atr ats
att atu atv atw atx aty atz aua aub auc aud aue auf aug auh aui auj auk aul
aum aun auo aup auq aur aus aut auu auv auw aux auy auz ava avb avc avd ave
avf avg avh avi avj avk avl avm avn avo avp avq avr avs avt avu avv avw avx
avy avz awa awb awc awd awe awf awg awh awi awj awk awl awm awn awo awp awq
awr aws awt awu awv aww awx awy awz axa axb axc axd axe axf axg axh axi axj
axk axl axm axn axo axp axq axr axs axt axu axv axw axx axy axz aya ayb ayc
ayd aye ayf ayg ayh ayi ayj ayk ayl aym ayn ayo ayp ayq ayr ays ayt ayu ayv
ayw ayx ayy ayz aza azb azc azd aze azf azg azh azi azj azk azl azm azn azo
azp azq azr azs azt azu azv azw azx azy azz baa bab bac bad bae baf bag bah
bai baj bak bal bam ban bao bap baq bar bas bat bau bav baw bax bay baz bba
bbb bbc bbd bbe bbf bbg bbh bbi bbj bbk bbl bbm bbn bbo bbp bbq bbr bbs bbt
bbu bbv bbw bbx bby bbz bca bcb bcc bcd bce bcf bcg bch bci bcj bck bcl bcm
bcn bco bcp bcq bcr bcs bct bcu bcv bcw bcx bcy bcz bda bdb bdc bdd bde bdf
bdg bdh bdi bdj bdk bdl bdm bdn bdo bdp bdq bdr bds bdt bdu bdv bdw bdx bdy
bdz bea beb bec bed bee bef beg beh bei bej bek bel bem ben beo bep beq ber
bes bet beu bev bew bex bey bez bfa bfb bfc bfd bfe bff bfg bfh bfi bfj bfk
bfl bfm bfn bfo bfp bfq bfr bfs bft bfu bfv bfw bfx bfy bfz bga bgb bgc bgd
bge bgf bgg bgh bgi bgj bgk bgl bgm bgn bgo bgp bgq bgr bgs bgt bgu bgv bgw
bgx bgy bgz bha bhb bhc bhd bhe bhf bhg bhh bhi bhj bhk bhl bhm bhn bho bhp
bhq bhr bhs bht bhu bhv bhw bhx bhy bhz bia bib bic bid bie bif big bih bii
bij bik bil bim bin bio bip biq bir bis bit biu biv biw bix biy biz bja bjb
bjc bjd bje bjf bjg bjh bji bjj bjk bjl bjm bjn bjo bjp bjq bjr bjs bjt bju
bjv bjw bjx bjy bjz bka bkb bkc bkd bke bkf bkg bkh bki bkj bkk bkl bkm bkn
bko bkp bkq bkr bks bkt bku bkv bkw bkx bky bkz bla blb blc bld ble blf blg
blh bli blj blk bll blm bln blo blp blq blr bls blt blu blv blw blx bly blz
bma bmb bmc bmd bme bmf bmg bmh bmi bmj bmk bml
);
function contain_word
{
[[ -z "${list// }" ]] && return 1
for arr in ${array[*]}
do
echo "$list" | grep -q $arr
[[ $? -eq 0 ]] && (( count ++ ))
done
[[ ${#array[#]} -eq $count ]]
}
function contain_word2
{
[[ $(sed 's/ /\n/g' <<<$list | sort -u | grep -Ff <(sed 's/ /\n/g' <<<${array[#]}) | wc -l) -eq ${#array[#]} ]]
}
contain_word$1 && echo "true" || echo "false"
And simple demonstration what O(M*N) vs O(M+N) means for M=N=1000 which is not too much for modern HW, isn't it?
$ time ./test.sh
true
real 0m0.989s
user 0m1.040s
sys 0m0.319s
$ time ./test.sh 2
true
real 0m0.011s
user 0m0.012s
sys 0m0.000s
Even for M=N=100
list="aaa aab aac aad aae aaf aag aah aai aaj aak aal aam aan aao aap aaq aar aas aat aau aav aaw aax aay aaz aba abb abc abd abe abf abg abh abi abj abk abl abm abn abo abp abq abr abs abt abu abv abw abx aby abz aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu acv acw acx acy acz ada adb adc add ade adf adg adh adi adj adk adl adm adn ado adp adq adr ads adt adu adv"
array=(
aaa aab aac aad aae aaf aag aah aai aaj aak aal aam aan aao aap aaq aar aas
aat aau aav aaw aax aay aaz aba abb abc abd abe abf abg abh abi abj abk abl
abm abn abo abp abq abr abs abt abu abv abw abx aby abz aca acb acc acd ace
acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu acv acw acx
acy acz ada adb adc add ade adf adg adh adi adj adk adl adm adn ado adp adq
adr ads adt adu adv
)
$ time ./test.sh
true
real 0m0.117s
user 0m0.105s
sys 0m0.042s
$ time ./test.sh 2
true
real 0m0.008s
user 0m0.008s
sys 0m0.001s
That's it for how inefficient it is.
BTW using Perl would be more elegant than AWK
function contain_word3
{
perl -e'#h{split/ /,shift}=();exists$h{$_}||exit 1 for#ARGV' "$list" "${array[#]}"
}
and fast (8ms).

Answer 6

A perl one-liner
perl -ape 'BEGIN{$H{$_}=1while$_=shift}$_=!grep!$H{$_},#F' $list <<<"${array[*]}"
prints
1
and
perl -ape 'BEGIN{$H{$_}=1while$_=shift}$_=!grep!$H{$_},#F' $list <<<"${array[*]} not"
prints nothing because not is not in $list
How it works, perl -h for command line switches,
BEGIN{$H{$_}=1while$_=shift} : fill a hash with keys #ARGV and values 1 and empties #ARGV list
$_=!grep!$H{$_},#F : grep return the array of elements not found in the hash, because of scalar context $=!, returns the number of element, and ! returns 1 if =0, nothing if >0.
Otherwise it can also be done in bash >=4.0 with associative arrays:
declare -A hashlist=([sdg]="1" [sdf]="1" [sde]="1" [sdd]="1" [sdc]="1" [sdb]="1" [sdo]="1" [sdk]="1" [sdj]="1" [sdi]="1" [sdh]="1" )
array=( sdb sdd sde sdf sdg )
r=0; for a in "${array[#]}"; do ((r|=\!hashlist[$a])); done ;((r=\!r))
to follow the same logic than perl but can be simplified inverting logic
r=1; for a in "${array[#]}"; do ((r&=\!hashlist[$a])); done
also can be optimized to break the loop when first item is not found
r=1; for a in "${array[#]}"; do ((r&=hashlist[$a])) || break; done
then r=1 if all entries were found, 0 otherwise.
note that ! must be escaped \! only in command line because of -H switch but in a script the \ must be removed.