I have the following bash script
TOKEN=$(aws ecr get-login-password --region us-east-2)
echo $TOKEN
USERNAME="AWS"
ENCODE_ME="${USERNAME}:${TOKEN}"
echo $ENCODE_ME
BASE_64_ENCODED=$(echo -n "${USERNAME}:${TOKEN}" | base64)
# BASE_64_ENCODED=$(printf "%s" "${CONVERT_ME}" | base64)
echo $BASE_64_ENCODED
the value of $ENCODE_ME gives something like below:
AWS:eyJwYXlsb2FkIjoidS9ralRSNTJpQWdJVC9DV1RQbFJOdEduUXUrdkkxcFc2dzYzckhiR0NSdGpLbEZsK0ZFN1YzU1l3ckZjSVh0RnNlcXJmMzJQMFVMTzQ5VWFLNdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOno1NWhIQmhCa3k5LzFnQU1GWFVzS2ZCMUVnZHZENHFOWTlzS0tSN3BjRkVLUUJYY25sa2RJZUJUS2Q0SUVaWDhFeTBWUzNueklkZFc4bmhVaDlTbklhMk1ZenU4V3dVUkR1TTYvUW5KMXkrQ3lKaDFJTGZaaWJnd1c2VHpTaU5UdHpxVkJ3TkdtS2tXOGxQL1h0VG5UVIbzQ2SU1aVFB0TE1FUkFRamJkbit0N0ppeEYzS2U5MGxEUElLUE0wMTA3OUtnV0FydDBvWmZ3RVQwUVZzRGJ4eFZraWVXNmxsampnSlVRaXR4SERKTkhHMDRqOEl4OWhka3NzUUlFd29jUlNdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOenNrRGZiRVM4N3ZraXliTjkzLzQ1Mkk2aUwwWmY0YzVvQWZpMXVGeFpqYVZCTUlKUHJsamJuSy84bDNWbmJaWS80bHRSK0pTemJ0M0tZcldLTUhzUVhEUTRWTlVFd1h2dW9xWXBSVHQvVkJvOTB4UDVMWnFRbDI0SWxyS3kzOU9iYXNzWnduV29rbW5yaDhIMDluaG14N05vMlN4SlVKTXpBUm10K1h3b28yQjZMZWJkK1A1TDlPSk9vSWdRWWZXdlpoQzMrWDZKZ3ZKTjh5M3lzeGVZbW5mejNKTTlmQlNOMFJ1NndRbTNoMjJ5bk5XNGdLclFLWG5EazZQcGZyV1EvdnEiLCJkYXRha2V5IjoiQVFFQkFIakI3L2lnd01nNE5Qd2F1cnhTSVl4NEhmbnh1R2MvNDhiRHd2d0RwTllXWmdBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOHIrc2plRDRUQTArNWRTaTVTTnVBVzJMVjIzNVNLamVQaFBROHpMYXBOdFNrWFVoUHhLd2swSkIxSkdJaWQ1ST0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE2MTE0NDQyNzZ9
and the BASE_64_ENCODED gives this value with spaces:
QVdTOmV5SndZWGxzYjJGa0lqb2lZbmhrUVUwelNVWnRhR1p2ZEhONWNsSlJXVE54YURCdFVrRk9Z VWRhVDBSd2JFVmlprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUb0eVlXaFBTVEpvU1RkTUwyTTNaMjExTDFsT04yUmtWMEkw WVdadmFVMHlVSFJKVEdSNlFteENkM2t2Ykc5QmNVUk5XRXBaU2xkSGQydHZMMU5OZWt4SVNHVlVj R2gxYjNalprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbxWnFSVW8zVERGcWRFTkxlVWR1T0ZaR01FaGhS MnNyVGtock5YcFhjalpFUzB4SWNHWXhObmxaY1c5TVNIWnZiV05rTTNsUlkxSjZhMngxYlhGVWFW RktSeXRoU3l0QlJXSnJaVEpMVjFKSkswZFJSVWxMWmxGUmR6ZDFUR3MzTWxCdVpXMXZjMkUwTDFs TmEzTldkMVJhVVhoc1NrSnJSMVY1VkhaaFlsaHdZbFZYZEVsSVVGRlFiMGhMY2sxWGRXUmthM0kx ZVZsdUwyTm9TbEJpU1RSNE5uUm5lWGx1Yml0a2EyMUljbWRHWVZnd2FXUjZXRFZIVUhWbmNrcExO RmczVXk5aWVWWkNjbFZZWlcxTWF6UnVjRkZyWmtRMk9XY3hNR05FTW14NVVqTTJNVGRFVTA4NVlX NTJhalpRTUdwT1FqaENUazVUTjNwTmRIcG9hSGhWWlRkeVFVZEhObVJqVTNZeWIwMVhjbXQ1V25o TlltdDFRazFKUm1aRVJHaDRRWG95VEdOd0wxa3JaWFJCWmpWMlJYZEdPSFF3YUZWd1UxZzRUV3Rx VTFGb1MycFVTazllprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbuUnlUMWh3UjIxU1VFaExa RGd4TTBGaU5qbFNTVGRhVkd4cWF5OUhaa2s1TW5VM2FXWk5TbXR0TVdObWF6bDRNVEl6T1hSM1Ru aEVlbTQxVVV0T04zWm5jVWhoWjFoeGVXdFhXRU0yYVU0M1RtMWhMMXBWTlRGUlVWQk1OR2hMU0M4 dmVVMU1UM0ZVV1VsbEwyaERjVkI1Y0ZSNlRFRTBWVXREWm1SV1VVdHNUM1pRVUVKTGRWTjNhbk5W Wld4YWVHYzFOVmhuZEhOaE5sbEZVV1ZzTld4UmFYWXhNMXB6VkhGMVYwSkdNazFLUzFVemJqZ3dZ eTk2UVU1VldESk5ia0ZpVlZGWlVXVndWVFZTTkdOSGVqSjRUVVJNZEU0clNXcG5PVXN4VmxsblZt bDVUblprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbGxYV21kQlFVRklOSGRtUVZsS1MyOWFT V2gyWTA1QlVXTkhiMGM0ZDJKUlNVSkJSRUp2UW1kcmNXaHJhVWM1ZHpCQ1FuZEZkMGhuV1VwWlNW cEpRVmRWUkVKQlJYVk5Ra1ZGUkUxQmFIaFlhMEUwZVdkR1dIbDJWM2xSU1VKRlNVRTNNWEZHUWts Sk0zUnNNR1E1Ymtjd2JqZDNRWGhISzBndmN6aFNWbUpDYTJkcVptOUNTVFJFZWtwemFtTjZlbGxq TnpkTVNtMXBObVZUWkZJM2FVOTFWa054TlZrelFVTTVTbGs1VWpoclNUMGlMQ0oyWlhKemFXOXVJ am9pTWlJc0luUjVjR1VpT2lKRVFWUkJYMHRGV1NJc0ltVjRjR2x5WVhScGIyNGlPakUyTVRFME5q QXpNREo5
Why does it produces spaces in-between?
Thank You!
base64(1) by default wraps lines at column 76. What you're seeing is the whitespace of those newlines.
If you add -w0 (disable wrapping altogether) to the base64 options, it will spit out its result without any wrapping.
Do man base64 for further information.
Related
I am writing a bash script which has a json value stored in a variable now i want to extract the values in that json using Jq. The code used is.
json_val={"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}
code_val= echo"$json_val" | jq '.code'
This throws an error of no such file or direcotry.
If i change this to
json_val={"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}
code_val=echo" $json_val " | jq '.code'
This does not throws any error but the value in code_val is null.
If try to do it manually echo {"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"} | jq '.code' it throws parse numeric letter error.
how can i do it in first case.
You may use this:
json_val='{"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}'
code_val=$(jq -r '.code' <<< "$json_val")
echo "$code_val"
lyz1To6ZTWClDHSiaeXyxg
Note following changes:
Wrap complete json string in single quotes
use of $(...) for command substitution
Use of <<< (here-string) to avoid a sub-shell creation
PS: If you're getting json text from a curl command and want to store multiple fields in shell variables then use:
read -r code_val redirect_to < <(curl ... | jq -r '.code + "\t" + .redirect_to')
Where ... is your curl command.
If try to do it manually:
$ echo {"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"} | jq '.code'
...it throws parse numeric letter error.
seems like you did not escape the string of the echo command. in your case, escaping with a singe-quote (apostrophe ') will do - same as you did with the jq json-path argument ('.code')
$ echo '{"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}' | jq '.code'
"lyz1To6ZTWClDHSiaeXyxg"
I need to convert a string into a sequence of decimal ascii code using bash command.
example:
for the string 'abc' the desired output would be 979899 where a=97, b=98 and c=99 in ascii decimal code.
I was able to achieve this with ascii hex code using xxd.
printf '%s' 'abc' | xxd -p
which gives me the result: 616263
where a=61, b=62 and c=63 in ascii hexadecimal code.
Is there an equivalent to xxd that gives the result in ascii decimal code instead of ascii hex code?
If you don't mind the results are merged into a line, please try the following:
echo -n "abc" | xxd -p -c 1 |
while read -r line; do
echo -n "$(( 16#$line ))"
done
Result:
979899
str=abc
printf '%s' $str | od -An -tu1
The -An gets rid of the address line, which od normally outputs, and the -tu1 treats each input byte as unsigned integer. Note that it assumes that one character is one byte, so it won't work with Unicode, JIS or the like.
If you really don't want spaces in the result, pipe it further into tr -d ' '.
Unicode Solution
What makes this problem annoying is that you have to pipeline characters when converting from hex to decimal. So you can't do a simple conversion from char to hex to dec as some characters hex representations are longer than others.
Both of these solutions are compatible with unicode and use a character's code point. In both solutions, a newline is chosen as separator for clarity; change this to '' for no separator.
Bash
sep='\n'
charAry=($(printf 'abc🎶' | grep -o .))
for i in "${charAry[#]}"; do
printf "%d$sep" "'$i"
done && echo
97
98
99
127926
Python (in Bash)
Here, we use a list comprehension to convert every character to a decimal number (ord), join it as a string and print it. sys.stdin.read() allows us to use Python inline to get input from a pipe. If you replace input with your intended string, this solution is then cross-platform.
printf '%s' 'abc🎶' | python -c "
import sys
input = sys.stdin.read()
sep = '\n'
print(sep.join([str(ord(i)) for i in input]))"
97
98
99
127926
Edit: If all you care about is using hex regardless of encoding, use #user1934428's answer
I am trying to write a small shell script, which can read a text file (given as argument), deleting all invalid Base64 chars and then decode this Base64 String into readable Text.
For this Example i can assume, that i have got a valid Base64 String polluted with additional invalid chars. So simply deleting them makes the String valid again.
I am having problems with the "remove al invalid chars" part.
Here is my Script:
#!/bin/bash
args=("$#")
#echo ${args[0]}
# read file
STRING="$(cat ${args[0]})"
echo "Input:"
echo $STRING
echo "\n"
#BASE64_REGEX='!/[^A-Za-z0-9+\/=]/'
STRING=${STRING//[!?_-]/}
echo "Fixed:"
echo $STRING
echo "\n"
# decode String
DECODED=$(base64 -d <<< "$STRING")
echo "Decoded:"
echo $DECODED
echo "\n"
I think my problem is this part here STRING=${STRING//[!?_-]/}. After this Operation the String contains ??___--- + linebreak, so i must somehow be close.
EDIT:
This would be the example String. And i try to remove all Characters, which are NOT part of the Base64 alphapet.
!RGllIGVpbnppZ2VuIFNvbmRlc??nplaWNoZW4gaW0gQmFzZTY0IEFscGhhYmV0IHNpbmQgIisg_L_y_A9Ii4gQWxsZSB3ZWl0ZXJlbi-B-T-b25kZXJ6!ZWljaGVuICIhIsKnJCUiIGtvbW1!lbiBkb3J0IG5pY2h0IHZvci"4=
Thanks for your help!
It' because ! in first position in a character set invert the set like ^ (note: only true for pattern matching (glob) not regex matching, but in this case it's just pattern matching)
maybe you want
STRING=${STRING//[?\!_-]/}
why not use the set in comments
STRING=${STRING//[^A-Za-z0-9+\/=]/}
I have a 2GB file in raw format. I want to search for all appearance of a specific HEX value "355A3C2F74696D653E" AND collect the following 28 characters.
Example: 355A3C2F74696D653E323031312D30342D32365431343A34373A30322D31343A34373A3135
In this case I want the output: "323031312D30342D32365431343A34373A30322D31343A34373A3135" or better: 2011-04-26T14:47:02-14:47:15
I have tried with
xxd -u InputFile | grep '355A3C2F74696D653E' | cut -c 1-28 > OutputFile.txt
and
xxd -u -ps -c 4000000 InputFile | grep '355A3C2F74696D653E' | cut -b 1-28 > OutputFile.txt
But I can't get it working.
Can anybody give me a hint?
As you are using xxd it seems to me that you want to search the file as if it were binary data. I'd recommend using a more powerful programming language for this; the Unix shell tools assume there are line endings and that the text is mostly 7-bit ASCII. Consider using Python:
#!/usr/bin/python
import mmap
fd = open("file_to_search", "rb")
needle = "\x35\x5A\x3C\x2F\x74\x69\x6D\x65\x3E"
haystack = mmap.mmap(fd.fileno(), length = 0, access = mmap.ACCESS_READ)
i = haystack.find(needle)
while i >= 0:
i += len(needle)
print (haystack[i : i + 28])
i = haystack.find(needle, i)
If your grep supports -P parameter then you could simply use the below command.
$ echo '355A3C2F74696D653E323031312D30342D32365431343A34373A30322D31343A34373A3135' | grep -oP '355A3C2F74696D653E\K.{28}'
323031312D30342D32365431343A
For 56 chars,
$ echo '355A3C2F74696D653E323031312D30342D32365431343A34373A30322D31343A34373A3135' | grep -oP '355A3C2F74696D653E\K.{56}'
323031312D30342D32365431343A34373A30322D31343A34373A3135
Why convert to hex first? See if this awk script works for you. It looks for the string you want to match on, then prints the next 28 characters. Special characters are escaped with a backslash in the pattern.
Adapted from this post: Grep characters before and after match?
I added some blank lines for readability.
VirtualBox:~$ cat data.dat
Thisis a test of somerandom characters before thestringI want5Z</time>2011-04-26T14:47:02-14:47:15plus somemoredata
VirtualBox:~$ cat test.sh
awk '/5Z\<\/time\>/ {
match($0, /5Z\<\/time\>/); print substr($0, RSTART + 9, 28);
}' data.dat
VirtualBox:~$ ./test.sh
2011-04-26T14:47:02-14:47:15
VirtualBox:~$
EDIT: I just realized something. The regular expression will need to be tweaked to be non-greedy, etc and between that and awk need to be tweaked to handle multiple occurrences as you need them. Perhaps some of the folks more up on awk can chime in with improvements as I am real rusty. An approach to consider anyway.
I am just studying base64 encoding and decoding algorithms and try some programs. I found some example code online, but the result looks a little weird for me.
Here is the link: http://knol2share.blogspot.com/2011/07/base64-encoding-and-decoding-in-c.html
I tried to use it to encode and decode a string.
Enter a string:
02613
Base64 Encoded value: MDI2MTM=
Base64 Decoded value: 02613% -- I do not know why there is a "%", is there a way to get the correct result
I even tried the Base64 program in linux and got the same result after removing the newline in encoding.
Here is the result:
%echo -n 02613 |base64
MDI2MTM=
%echo -n MDI2MTM= | base64 --decode
02613%
Does anyone know how I can get the exact same result with the input string? Thanks.
It is printed if the decoded text does not end with a newline.
$ printf "foobar\n" | base64 | base64 --decode
foobar
$ printf "foobar" | base64 | base64 --decode
foobar%
Isn't the % sign a command prompt ?
Add new line after decoded b64 and check.
Here's the example with echo command, -n option will remove newline character, that's why in first case we don't have the symbol % and in the second case with have one appended.
➜ ~ echo "HELLO" | base64 | base64 --decode
HELLO
➜ ~ echo -n "HELLO" | base64 | base64 --decode
HELLO%