I am trying to automate string replace thing for my project ...
I need output like this in file
insert into libraries values("Schema_name", "table_name", "table_name", "/data/Projects/Ope/ACT/Domain/Code/Files");
and what I am getting in the file is
`insert into libraries values(Schema_name, table_name, table_name, /data/Projects/Ope/ACT/Domain/Code/Files)`;
replace_script.sh
#!/bin/bash
while read line
do
param1=`echo $line | awk -F ' ' '{print $1}'`
param2=`echo $line | awk -F ' ' '{print $2}'`
echo "insert into libraries values(\"$param1\",\"$param2\",\"$param2\",\"/data/Projects/Ope/ACT/Domain/Code/Files\");" >> input_queries.hql
done <<EOF
schema_name table_name
schema_name table_name
EOF
The exact code given in your question emits as output:
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
This is, as I understand it, exactly what you claim to want.
However, SQL doesn't use double quotes for data -- it uses single quotes for that.
escape_sql() {
local val
val=${1//\\/\\\\}
val=${val//\'/\\\'}
val=${val//\"/\\\"}
printf '%s' "$val"
}
while read -r param1 param2; do
printf $'insert into libraries values(\'%s\', \'%s\', \'%s\', \'/data/Projects/Ope/ACT/Domain/Code/Files\');\\n' \
"$(escape_sql "$param1")" \
"$(escape_sql "$param2")" \
"$(escape_sql "$param2")"
done <<EOF
schema_name table_name
schema_name table_name
EOF
The above makes a rudimentary attempt to prevent malicious values from escaping their quotes -- though you should really use a language with native SQL bindings for your database for the purpose!
That said -- this is not safe escaping against malicious data (for instance, data containing literal quotes). For that, use a language built-to-purpose.
Related
I wrote the following bash code in order to create CSV path with disks partitions , so each partition will get new increment dev disk
number_of_disks=5
mount_p=({a..z})
path=` for i in \`seq 1 $number_of_disks \`; do mount_p="$(echo $mount_p| tr '[a-z]' '[b-z]a')"; echo /home/sd$mount_p/oop/app/data","; done `
but when I print the $path we get space between each partion
echo $path
/home/sdb/oop/app/data, /home/sdc/oop/app/data,
/home/sdd/oop/app/data, /home/sde/oop/app/data,
/home/sdf/oop/app/data,
Second problem is the unnecessary "," at the end of the line
Based on my code how to create the path variable without space and without "," at the end of the CSV line
You are using a very complex way (hacky as hell) to achieve something rather simple:
path=$(echo /home/sd{a..e}/oop/app/data | tr ' ' ,)
You can change your path variable like tihs:
echo $path | sed 's/ //g;s/,$//g'
It will remove last ',' and spaces.
UPD.
Or:
path=( $(echo /home/sd{b..z}/oop/app/data | tr ' ' ',') ); echo "${path[#]}"
I have a file with these arguments and their values this way
# parameters.txt
VAR1 001
VAR2 aaa
VAR3 'Hello World'
and another file to configure like this
# example.conf
VAR1 = 020
VAR2 = kab
VAR3 = ''
when I want to get the values in a function I use this command
while read p; do
VALUE=$(echo $p | awk '{print $2}')
done < parameters.txt
the firsts arguments throw the right values, but the last one just gets the 'Hello for the blank space, my question is how do I get the entire 'Hello World' value?
If you can use bash, there is no need to use awk: read and shell parameter expansion can be combined to solve your problem:
while read -r name rest; do
# Drop the '= ' part, if present.
[[ $rest == '= '* ]] && value=${rest:2} || value=$rest
# $value now contains the line's value,
# but *including* any enclosing ' chars, if any.
# Assuming that there are no *embedded* ' chars., you can remove them
# as follows:
value=${value//\'/}
done < parameters.txt
read by default also breaks a line into fields by whitespace, like awk, but unlike awk it has the ability to assign the remainder of the line to a varaible, namely the last one, if fewer variables than fields found are specified;
read's -r option is generally worth specifying to avoid unexpected interpretation of \ chars. in the input.
As for your solution attempt:
awk doesn't know about quoting in input - by default it breaks input into fields by whitespace, irrespective of quotation marks.
Thus, a string such as 'Hello World' is simply broken into fields 'Hello and World'.
However, in your case you can split each input line into its key and value using a carefully crafted FS value (FS is the input field separator, which can be also be set via option -F; the command again assumes bash, this time for use of <(...), a so-called process substitution, and $'...', an ANSI C-quoted string):
while IFS= read -r value; do
# Work with $value...
done < <(awk -F$'^[[:alnum:]]+ (= )?\'?|\'' '{ print $2 }' parameters.txt)
Again the assumption is that values contain no embedded ' instances.
Field separator regex $'^[[:alnum:]]+ (= )?\'?|\'' splits each line so that $2, the 2nd field, contains the value, stripped of enclosing ' chars., if any.
xargs is the rare exception among the standard utilities in that it does understand single- and double-quoted strings (yet also without support for embedded quotes).
Thus, you could take advantage of xargs' ability to implicitly strip enclosing quotes when it passes arguments to the specified command, which defaults to echo (again assumes bash):
while read -r name rest; do
# Drop the '= ' part, if present.
[[ $rest == '= '* ]] && value=${rest:2} || value=$rest
# $value now contains the line's value, strippe of any enclosing
# single quotes by `xargs`.
done < <(xargs -L1 < parameters.txt)
xargs -L1 process one (1) line (-L) at a time and implicitly invokes echo with all tokens found on each line, with any enclosing quotes removed from the individual tokens.
The default field separator in awk is the space. So you are only printing the first word in the string passed to awk.
You can specify the field separator on the command line with -F[field separator]
Example, setting the field separator to a comma:
$ echo "Hello World" | awk -F, '{print $1}'
Hello World
I have a sqlite3 database. One column has the TEXT type, and contains blobs which I would like to save as file. Those are gzipped files.
The output of the command sqlite3 db.sqlite3 ".dump" is:
INSERT INTO "data" VALUES(1,'objects','object0.gz',X'1F8B080000000000000 [.. a few thousands of hexadecimal characters ..] F3F5EF')
How may I extract the binary data from the sqlite file to a file using the command line ?
sqlite3 cannot output binary data directly, so you have to convert the data to a hexdump, use cut to extract the hex digits from the blob literal, and use xxd (part of the vim package) to convert the hexdump back into binary:
sqlite3 my.db "SELECT quote(MyBlob) FROM MyTable WHERE id = 1;" \
| cut -d\' -f2 \
| xxd -r -p \
> object0.gz
With SQLite 3.8.6 or later, the command-line shell includes the fileio extension, which implements the writefile function:
sqlite3 my.db "SELECT writefile('object0.gz', MyBlob) FROM MyTable WHERE id = 1"
You can use writefile if using the sqlite3 command line tool:
Example usage:
select writefile('blob.bin', blob_column) from table where key='12345';
In my case, I use "hex" instead of "quote" to retrieve image from database, and no need "cut" in the command pipe. For example:
sqlite3 fr.db "select hex(bmp) from reg where id=1" | xxd -r -p > 2.png
I had to make some minor changes on CL's answer, in order to make it work for me:
The structure for the command that he is using does not have the database name in it, the syntax that I am using is something like:
sqlite3 mydatabase.sqlite3 "Select quote(BlobField) From TableWithBlod Where StringKey = '1';" | ...
The way he is using the cut command does not work in my machine. The correct way for me is:
cut -d "'" -f2
So the final command would be something like:
sqlite3 mydatabase.sqlite3 "Select quote(BlobField) From TableWithBlod Where StringKey = '1';" | cut -d "'" -f2 | xxd -r -p > myfile.extension
And in my case:
sqlite3 osm-carto_z14_m8_m.mbtiles "select quote(images.tile_data) from images where images.tile_id = '1';" | cut -d "'" -f2 | xxd -r -p > image.png
I'm going through and reading lines from a file. They have a ton of information that is unnecessary, and I want to reformat the lines for later use so that I can use the necessary information later.
Example line in file (file1)
Name: *name* Date: *date* Age: *age* Gender: *gender* Score: *score*
Say I want to just pull gender and age from the file and use that later
New line
*gender*, *age*
In bash:
while read line; do
<store variable for gender>
<store variable for age>
<overwrite each line in CSV - gender,age>
<use gender/age as inputs for later comparisons>
done < file1
EDIT: There is no stability in the entries. One value can be found using a echo $line | cut and the other value is found using a [ $line =~ "keyValue" ] then setting that value
I was thinking of storing the combination of the two variables as such:
newLine="$val1,$val2"
Then using a sed in-line replace to replace the $line with $newLine.
Is there a better way, though? It may come down to a sed formatting issue with variables.
This will produce your desired output from your posted sample input:
$ cat file
Name: *name* Date: *date* Age: *age* Gender: *gender* Score: *score*
$ awk -F'[: ]+' -v OFS=', ' '{for (i=1;i<NF;i+=2) a[$i]=$(i+1); print a["Gender"], a["Age"]}' file
*gender*, *age*
$ awk -F'[: ]+' -v OFS=', ' '{for (i=1;i<NF;i+=2) a[$i]=$(i+1); print a["Score"], a["Name"], a["Date"] }' file
*score*, *name*, *date*
and you can see above how easy it is to print whatever fields you like in whatever order you like.
If it's not what you want, post some more representative input.
Your example leaves room for interpretation, so I'm assuming that there may be whitespace in the field values, but that there are no colons in the field values and that each field key is followed by a colon. I also assume that the order is stable.
while IFS=: read _ _ _ age gender _; do
age="${age% Gender}" # Use parameter expansion to strip off the key for the *next* field.
gender="${gender% Score}"
printf '"%s","%s"\n' "$gender" "$age"
done < file1 > file1.csv
Update
Since your question now states that there is no stability, you have to iterate through the possible values to get your output:
while IFS=: read -a line; do
unset age key sex
for chunk in "${line[#]}"; do
val="${chunk% *}" # Everything but the key
case "$key" in
Age) age="$val";;
Gender) sex="$val";;
esac
# The key is for the *next* iteration.
key="${chunk##* }"
done
if [[ $age || $sex ]]; then
printf '"%s","%s"\n' "$sex" "$age"
fi
done < file1 > file1.csv
(Also I added quotes around the output values in the csv to be compliant with the actual csv format and in case sex or age happened to have commas in it. Maybe someone is 1,000,000 years old. ;)
This is my below shell script which I am using to query the hive tables and I saved this file as test4.sh
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT1
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT split(ckey, '\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1;"`
Problem Statement:-
I am running the above shell script as-
bash -x test4.sh
If you see this line in the above hive sql query:
**SELECT split(ckey, '\\|') AS t1**.
Do I need to escape slash sign to make it to work in shell script?
Inside backticks and double quotes, you basically need to double your backslashes, yes. If a backslash is not a known escape sequence, it will be preserved, though.
bash$ echo "foo\bar"
foo\bar
bash$ echo "foo\\bar"
foo\bar
bash$ echo "split(ckey, '\\|')"
split(ckey, '\|')
So if you want hive to see a double backslash there, you will need to put four backslashes in the Bash script.
See further e.g. http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_03.html