I am trying to read a file & extract only the base64 encoded part. I am doing so by using sed to find the lines between the pattern's 'base64' & '--'.
I am able to do so using
sed -e '1,/base64/d' -e '/--/,$d' file.txt | sed '/^\s*$/d'
But when i try to decode it using base64 as shown below,
sed -e '1,/base64/d' -e '/--/,$d' file.txt | sed '/^\s*$/d' | base64 -d
I get an error message as :-
BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchangbase64: invalid input
Please let me know what am i doing wrong and is there an alternate, better way to achieve the end result. Any help is appreciated.
I got the issue. the lines in the files were as follows
QkVHSU46VkNBTEVOREFSDQpNRVRIT0Q6UkVRVUVTVA0KUFJPRElEOk1pY3Jvc29mdCBFeGNoYW5nZSBTZ
XJ2ZXIgMjAxMA0KVkVSU0lPTjoyLjANCkJFR0lOOlZUSU1FWk9ORQ0KVFpJRDpBVVMgRWFzdGVybiBTdG
There was an end line character at the end, so all i had to do was remove it. Hence, now my code looks as below & it works :-
sed -e '1,/base64/d' -e '/--/,$d' _12829640_21.12.-.5p.meeting.room_1485234639532_1.mail | sed '/^\s$/d' | tr -d "\n\r" | base64 -d
Related
I'm doing a linux online course but im stuck with a question, you can find the question below.
You will get three files called a.bf, b.bf and c.bf. Merge the contents of these three files and write it to a new file called abc.bf. Respect the order: abc.bf must contain the contents of a.bf first, followed by those of b.bf, followed by those of c.bf.
Example
Suppose the given files have the following contents:
a.bf contains +++.
b.bf contains [][][][].
c.bf contains <><><>.
The file abc.bf should then have
+++[][][][]<><><>
as its content.
I know how to merge the 3 files but when i use cat my output is:
+++
[][][]
<><><>
When i use paste my output is "+++ 'a lot of spaces' [][][][] 'a lot of spaces' <><><>"
My output that i need is +++[][][][]<><><>, i dont want the spaces between the content. Can someone help me?
What you want to do is delete the newline characters.
With tr:
cat {a,b,c}.bf | tr --delete '\n' > abc.bf
With echo & sed:
echo $(cat {a,b,c}.bf) | sed -E 's/ //g' > abc.bf
With xargs & sed:
<{a,b,c}.bf xargs | sed -E 's/ //g' > abc.bf
Note that sed is only used to remove the spaces.
With cat & sed:
cat {a,b,c}.bf | sed -z 's/\n//g'
echo -n "$(cat a.bf)$(cat b.bf)$(cat c.bf)" > abc.bf
echo -n will not output trailing newlines
I have these 3 commands those work correctly and encode the image as json.
encoded_string=$(base64 volunteers.jpg)
payload="{\"instances\": [{\"image\": {\"b64\": \"$encoded_string\"}}]}"
echo $payload >input.json
But how do I convert it back to jpg format? This returns an error "base64: invalid input"
cat '/"' input.json '/"' | jq -r '.instances[0].image.b64' | base64 -d >output.jpg
The problem most likely is due to the embedded newlines that that are created during the encoding part of the image. You can just remove them by using tr -d \\n in your original attempt which an be slightly re-written succinctly with inputs from jq capabilities to read from standard input.
jq -Rn '.instances[0].image.b64 = inputs' < <(base64 volunteers.jpg | tr -d \\n) > input.json
The -n part is to avoid jq reading a separate input stream of its own and -R for reading raw input. Here we feed the encoded content as if it were in a file using bash process substitution syntax <(..) and feed this created file to jq
and then decoding back the created JSON as
jq -r '.instances[0].image.b64' input.json | base64 -d > output.jpg
Re-writing your original attempt with minor enhancements and without using a temporary file for storing the JSON
JSON='{"instances": [{"image": {"b64": "'"$(base64 volunteers.jpg | tr -d \\n)"'" }}]}'
jq -r '.instances[0].image.b64' <<<"$JSON" | base64 -d >output.jpg
or use printf() inplace of the here-strings(<<<)
printf '%s\n' "$JSON" | jq -r '.instances[0].image.b64' | base64 -d >output.jpg
The invocation
cat '/"' input.json '/"'
is hopelessly muddled: cat input.json would suffice. Even better, assuming input.json contains valid JSON, you could write:
< input.json jq -r '.instances[0].image.b64' | base64 -d >output.jpg
I am having problems using sed to change this:
script_summary("Short random text");
script_id(#12345);
Into this:
script_tag(name:"text", value:"Short random text");
script_oid("1.3.6.1.4.1.25623.1.0.#12345");
Could you please, point me out in the right direction?
Thank you very much in advance.
Best regards,
What about:
Using 2 iterations. Testing it:
echo 'script_summary("Short random text");' | sed -e 's/script_summary("\(.*\)");/script_tag(name:"text", value:"\1");/'
echo 'script_id(#12345);' | sed -e 's/script_id(\(#[0-9]\+\));/script_oid("1.3.6.1.4.1.25623.1.0.\1");/'
So you may want to use it with
sed -i -e <regex1> <file>
sed -i -e <regex2> <file>
or
cat <file> | sed -e <regex1> | sed -e <regex2>
depends on how big the files is, and what you want to do with it.
I need some assistance trying to build up a variable using a list of exclusions in a file.
So I have a exclude file I am using for rsync that looks like this:
*.log
*.out
*.csv
logs
shared
tracing
jdk*
8.6_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
**/lost+found*/
dlxwhsr*
regression
tmp
working
investigation
Investigation
dcsserver_weblogic_
dcswebrdtEAR_weblogic_
I need to build up a string to be used as a variable to feed into egrep -v, so that I can use the same exclusion list for rsync as I do when egrep -v from a find -ls.
So I have created this so far to remove all "*" and "/" - and then when it sees certain special characters it escapes them:
cat exclude-list.supt | while read line
do
echo $line | sed 's/\*//g' | sed 's/\///g' | 's/\([.-+_]\)/\\\1/g'
What I need the ouput too look like is this and then export that as a variable:
SEXCLUDE_supt="\.log|\.out|\.csv|logs|shared|PR116PICL|tracing|lost\+found|jdk|8\.6\_Code|rpsupport|dbarchive|inarchive|comms|dlxwhsr|regression|tmp|working|investigation|Investigation|dcsserver\_weblogic\_|dcswebrdtEAR\_weblogic\_"
Can anyone help?
A few issues with the following:
cat exclude-list.supt | while read line
do
echo $line | sed 's/\*//g' | sed 's/\///g' | 's/\([.-+_]\)/\\\1/g'
Sed reads files line by line so cat | while read line;do echo $line | sed is completely redundant also sed can do multiple substitutions by either passing them as a comma separated list or using the -e option so piping to sed three times is two too many. A problem with '[.-+_]' is the - is between . and + so it's interpreted as a range .-+ when using - inside a character class put it at the end beginning or end to lose this meaning like [._+-].
A much better way:
$ sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file
\.log
\.out
\.csv
logs
shared
tracing
jdk
8\.6\_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
lost\+found
dlxwhsr
regression
tmp
working
investigation
Investigation
dcsserver\_weblogic\_
dcswebrdtEAR\_weblogic\_
Now we can pipe through tr '\n' '|' to replace the newlines with pipes for the alternation ready for egrep:
$ sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file | tr "\n" "|"
\.log|\.out|\.csv|logs|shared|tracing|jdk|8\.6\_Code|rpsupport|dbarchive|...
$ EXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file | tr "\n" "|")
$ echo $EXCLUDE
\.log|\.out|\.csv|logs|shared|tracing|jdk|8\.6\_Code|rpsupport|dbarchive|...
Note: If your file ends with a newline character you will want to remove the final trailing |, try sed 's/\(.*\)|/\1/'.
This might work for you (GNU sed):
SEXCLUDE_supt=$(sed '1h;1!H;$!d;g;s/[*\/]//g;s/\([.-+_]\)/\\\1/g;s/\n/|/g' file)
This should work but I guess there are better solutions. First store everything in a bash array:
SEXCLUDE_supt=$( sed -e 's/\*//g' -e 's/\///g' -e 's/\([.-+_]\)/\\\1/g' exclude-list.supt)
and then process it again to substitute white space:
SEXCLUDE_supt=$(echo $SEXCLUDE_supt |sed 's/\s/|/g')
I have some html files and want to extract only lines with containing these tags:
head
p
I used sed to extract these parts of the files, as follows:
grep "<head>" myfile.html | sed -e 's%\(head\)\(.*\)\(/head\)%title\2\/title%'
grep "<p>" myfile.html | sed -e 's%\(<p>\)\(.*\)\(</p\)\(>\)%\2\\%'
Everything is Ok, but I get "\" character at the end of each line. How I can overcome this problem?
In this command, you're telling it to add a backslash by including the double backslash:
sed -e 's%\(<p>\)\(.*\)\(</p\)\(>\)%\2\\%'
Try removing the backslashes:
sed -e 's%\(<p>\)\(.*\)\(</p\)\(>\)%\2%'
Also, you don't need grep:
sed -ne '/<p>/{s%\(<p>\)\(.*\)\(</p\)\(>\)%\2%;p}'
Don't use \ at the end of the replacement string:
grep "<p>" myfile.html | sed -e 's%\(<p>\)\(.*\)\(</p\)\(>\)%\2%'