How can I prevent jq from truncating long decimal values?
For example:
echo '18302628978110292481' | jq .
result: 18302628978110292000
Javascript does not support such big numbers and so does jq. The integer size is 2^53. Check this
To make it work, you'll need to treat them as strings:
echo '"18302628978110292481"' | jq .
# Prints "18302628978110292481"
Related
I have the following bash script
TOKEN=$(aws ecr get-login-password --region us-east-2)
echo $TOKEN
USERNAME="AWS"
ENCODE_ME="${USERNAME}:${TOKEN}"
echo $ENCODE_ME
BASE_64_ENCODED=$(echo -n "${USERNAME}:${TOKEN}" | base64)
# BASE_64_ENCODED=$(printf "%s" "${CONVERT_ME}" | base64)
echo $BASE_64_ENCODED
the value of $ENCODE_ME gives something like below:
AWS:eyJwYXlsb2FkIjoidS9ralRSNTJpQWdJVC9DV1RQbFJOdEduUXUrdkkxcFc2dzYzckhiR0NSdGpLbEZsK0ZFN1YzU1l3ckZjSVh0RnNlcXJmMzJQMFVMTzQ5VWFLNdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOno1NWhIQmhCa3k5LzFnQU1GWFVzS2ZCMUVnZHZENHFOWTlzS0tSN3BjRkVLUUJYY25sa2RJZUJUS2Q0SUVaWDhFeTBWUzNueklkZFc4bmhVaDlTbklhMk1ZenU4V3dVUkR1TTYvUW5KMXkrQ3lKaDFJTGZaaWJnd1c2VHpTaU5UdHpxVkJ3TkdtS2tXOGxQL1h0VG5UVIbzQ2SU1aVFB0TE1FUkFRamJkbit0N0ppeEYzS2U5MGxEUElLUE0wMTA3OUtnV0FydDBvWmZ3RVQwUVZzRGJ4eFZraWVXNmxsampnSlVRaXR4SERKTkhHMDRqOEl4OWhka3NzUUlFd29jUlNdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOenNrRGZiRVM4N3ZraXliTjkzLzQ1Mkk2aUwwWmY0YzVvQWZpMXVGeFpqYVZCTUlKUHJsamJuSy84bDNWbmJaWS80bHRSK0pTemJ0M0tZcldLTUhzUVhEUTRWTlVFd1h2dW9xWXBSVHQvVkJvOTB4UDVMWnFRbDI0SWxyS3kzOU9iYXNzWnduV29rbW5yaDhIMDluaG14N05vMlN4SlVKTXpBUm10K1h3b28yQjZMZWJkK1A1TDlPSk9vSWdRWWZXdlpoQzMrWDZKZ3ZKTjh5M3lzeGVZbW5mejNKTTlmQlNOMFJ1NndRbTNoMjJ5bk5XNGdLclFLWG5EazZQcGZyV1EvdnEiLCJkYXRha2V5IjoiQVFFQkFIakI3L2lnd01nNE5Qd2F1cnhTSVl4NEhmbnh1R2MvNDhiRHd2d0RwTllXWmdBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFREExZ1k3TEtib1A3d1JKN3lBSUJFSUE3c1NtRFNkUC9yUXhLaDhWOHIrc2plRDRUQTArNWRTaTVTTnVBVzJMVjIzNVNLamVQaFBROHpMYXBOdFNrWFVoUHhLd2swSkIxSkdJaWQ1ST0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE2MTE0NDQyNzZ9
and the BASE_64_ENCODED gives this value with spaces:
QVdTOmV5SndZWGxzYjJGa0lqb2lZbmhrUVUwelNVWnRhR1p2ZEhONWNsSlJXVE54YURCdFVrRk9Z VWRhVDBSd2JFVmlprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUb0eVlXaFBTVEpvU1RkTUwyTTNaMjExTDFsT04yUmtWMEkw WVdadmFVMHlVSFJKVEdSNlFteENkM2t2Ykc5QmNVUk5XRXBaU2xkSGQydHZMMU5OZWt4SVNHVlVj R2gxYjNalprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbxWnFSVW8zVERGcWRFTkxlVWR1T0ZaR01FaGhS MnNyVGtock5YcFhjalpFUzB4SWNHWXhObmxaY1c5TVNIWnZiV05rTTNsUlkxSjZhMngxYlhGVWFW RktSeXRoU3l0QlJXSnJaVEpMVjFKSkswZFJSVWxMWmxGUmR6ZDFUR3MzTWxCdVpXMXZjMkUwTDFs TmEzTldkMVJhVVhoc1NrSnJSMVY1VkhaaFlsaHdZbFZYZEVsSVVGRlFiMGhMY2sxWGRXUmthM0kx ZVZsdUwyTm9TbEJpU1RSNE5uUm5lWGx1Yml0a2EyMUljbWRHWVZnd2FXUjZXRFZIVUhWbmNrcExO RmczVXk5aWVWWkNjbFZZWlcxTWF6UnVjRkZyWmtRMk9XY3hNR05FTW14NVVqTTJNVGRFVTA4NVlX NTJhalpRTUdwT1FqaENUazVUTjNwTmRIcG9hSGhWWlRkeVFVZEhObVJqVTNZeWIwMVhjbXQ1V25o TlltdDFRazFKUm1aRVJHaDRRWG95VEdOd0wxa3JaWFJCWmpWMlJYZEdPSFF3YUZWd1UxZzRUV3Rx VTFGb1MycFVTazllprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbuUnlUMWh3UjIxU1VFaExa RGd4TTBGaU5qbFNTVGRhVkd4cWF5OUhaa2s1TW5VM2FXWk5TbXR0TVdObWF6bDRNVEl6T1hSM1Ru aEVlbTQxVVV0T04zWm5jVWhoWjFoeGVXdFhXRU0yYVU0M1RtMWhMMXBWTlRGUlVWQk1OR2hMU0M4 dmVVMU1UM0ZVV1VsbEwyaERjVkI1Y0ZSNlRFRTBWVXREWm1SV1VVdHNUM1pRVUVKTGRWTjNhbk5W Wld4YWVHYzFOVmhuZEhOaE5sbEZVV1ZzTld4UmFYWXhNMXB6VkhGMVYwSkdNazFLUzFVemJqZ3dZ eTk2UVU1VldESk5ia0ZpVlZGWlVXVndWVFZTTkdOSGVqSjRUVVJNZEU0clNXcG5PVXN4VmxsblZt bDVUblprTm5CTmVUTnRObGxCY0RWNlVtcHRVRFJIYW1Ka1JsTkVRMXB5YWxoSFptTlVaV1ZUTW5B clN6SXlUR2hOV1VVaUxDSmtZWFJoYTJWNUlqb2lRVkZGUWtGSWFrSTNMMmxuZDAxbk5FNVFkMkYx Y25oVFNWbDRORWhtYm5oMVIyTXZORGhpUkhkMmQwUndUbGxYV21kQlFVRklOSGRtUVZsS1MyOWFT V2gyWTA1QlVXTkhiMGM0ZDJKUlNVSkJSRUp2UW1kcmNXaHJhVWM1ZHpCQ1FuZEZkMGhuV1VwWlNW cEpRVmRWUkVKQlJYVk5Ra1ZGUkUxQmFIaFlhMEUwZVdkR1dIbDJWM2xSU1VKRlNVRTNNWEZHUWts Sk0zUnNNR1E1Ymtjd2JqZDNRWGhISzBndmN6aFNWbUpDYTJkcVptOUNTVFJFZWtwemFtTjZlbGxq TnpkTVNtMXBObVZUWkZJM2FVOTFWa054TlZrelFVTTVTbGs1VWpoclNUMGlMQ0oyWlhKemFXOXVJ am9pTWlJc0luUjVjR1VpT2lKRVFWUkJYMHRGV1NJc0ltVjRjR2x5WVhScGIyNGlPakUyTVRFME5q QXpNREo5
Why does it produces spaces in-between?
Thank You!
base64(1) by default wraps lines at column 76. What you're seeing is the whitespace of those newlines.
If you add -w0 (disable wrapping altogether) to the base64 options, it will spit out its result without any wrapping.
Do man base64 for further information.
I have a csv file of 100000 ids
wef7efwe1fwe8
wef7efwe1fwe3
ewefwefwfwgrwergrgr
that are being transformed into a json object using jq
output=$(jq -Rsn '
{"id":
[inputs
| . / "\n"
| (.[] | select(length > 0) | . / ";") as $input
| $input[0]]}
' <$FILE)
output
{
"id": [
"wef7efwe1fwe8",
"wef7efwe1fwe3",
....
]
}
currently, I need to manually split the file into smaller 10000 line files... because the API call has a limit.
I would like a way to automatically loop through the large file... and only use 10000 lines as a time as $FILE... up until the end of the list.
I would use the split command and write a little shell script around it:
#!/bin/bash
input_file=ids.txt
temp_dir=splits
api_limit=10000
# Make sure that there are no leftovers from previous runs
rm -rf "${temp_dir}"
# Create temporary folder for splitting the file
mkdir "${temp_dir}"
# Split the input file based on the api limit
split --lines "${api_limit}" "${input_file}" "${temp_dir}/"
# Iterate through splits and make an api call per split
for split in "${temp_dir}"/* ; do
jq -Rsn '
{"id":
[inputs
| . / "\n"
| (.[] | select(length > 0) | . / ";") as $input
| $input[0]]
}' "${split}" > api_payload.json
# now do something ...
# curl -dapi_payload.json http://...
rm -f api_payload.json
done
# Clean up
rm -rf "${temp_dir}"
Here's a simple and efficient solution that at its core just uses jq. It takes advantage of the -c command-line option. I've used xargs printf ... for illustration - mainly to show how easy it is to set up a shell pipeline.
< data.txt jq -Rnc '
def batch($n; stream):
def b: [limit($n; stream)]
| select(length > 0)
| (., b);
b;
{id: batch(10000; inputs | select(length>0) | (. / ";")[0])}
' | xargs printf "%s\n"
Parameterizing batch size
It might make sense to set things up so that the batch size is specified outside the jq program. This could be done in numerous ways, e.g. by invoking jq along the lines of:
jq --argjson n 10000 ....
and of course using $n instead of 10000 in the jq program.
Why “def b:”?
For efficiency. jq’s TCO (tail recursion optimization) only works for arity-0 filters.
Note on -s
In the Q as originally posted, the command-line options -sn are used in conjunction with inputs. Using -s with inputs defeats the whole purpose of inputs, which is to make it possible to process input in a stream-oriented way (i.e. one line of input or one JSON entity at a time).
I am writing a bash script which has a json value stored in a variable now i want to extract the values in that json using Jq. The code used is.
json_val={"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}
code_val= echo"$json_val" | jq '.code'
This throws an error of no such file or direcotry.
If i change this to
json_val={"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}
code_val=echo" $json_val " | jq '.code'
This does not throws any error but the value in code_val is null.
If try to do it manually echo {"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"} | jq '.code' it throws parse numeric letter error.
how can i do it in first case.
You may use this:
json_val='{"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}'
code_val=$(jq -r '.code' <<< "$json_val")
echo "$code_val"
lyz1To6ZTWClDHSiaeXyxg
Note following changes:
Wrap complete json string in single quotes
use of $(...) for command substitution
Use of <<< (here-string) to avoid a sub-shell creation
PS: If you're getting json text from a curl command and want to store multiple fields in shell variables then use:
read -r code_val redirect_to < <(curl ... | jq -r '.code + "\t" + .redirect_to')
Where ... is your curl command.
If try to do it manually:
$ echo {"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"} | jq '.code'
...it throws parse numeric letter error.
seems like you did not escape the string of the echo command. in your case, escaping with a singe-quote (apostrophe ') will do - same as you did with the jq json-path argument ('.code')
$ echo '{"code":"lyz1To6ZTWClDHSiaeXyxg","redirect_to":"http://example.com/client-redirect-uri?code=lyz1To6ZTWClDHSiaeXyxg"}' | jq '.code'
"lyz1To6ZTWClDHSiaeXyxg"
I need to convert a string into a sequence of decimal ascii code using bash command.
example:
for the string 'abc' the desired output would be 979899 where a=97, b=98 and c=99 in ascii decimal code.
I was able to achieve this with ascii hex code using xxd.
printf '%s' 'abc' | xxd -p
which gives me the result: 616263
where a=61, b=62 and c=63 in ascii hexadecimal code.
Is there an equivalent to xxd that gives the result in ascii decimal code instead of ascii hex code?
If you don't mind the results are merged into a line, please try the following:
echo -n "abc" | xxd -p -c 1 |
while read -r line; do
echo -n "$(( 16#$line ))"
done
Result:
979899
str=abc
printf '%s' $str | od -An -tu1
The -An gets rid of the address line, which od normally outputs, and the -tu1 treats each input byte as unsigned integer. Note that it assumes that one character is one byte, so it won't work with Unicode, JIS or the like.
If you really don't want spaces in the result, pipe it further into tr -d ' '.
Unicode Solution
What makes this problem annoying is that you have to pipeline characters when converting from hex to decimal. So you can't do a simple conversion from char to hex to dec as some characters hex representations are longer than others.
Both of these solutions are compatible with unicode and use a character's code point. In both solutions, a newline is chosen as separator for clarity; change this to '' for no separator.
Bash
sep='\n'
charAry=($(printf 'abc🎶' | grep -o .))
for i in "${charAry[#]}"; do
printf "%d$sep" "'$i"
done && echo
97
98
99
127926
Python (in Bash)
Here, we use a list comprehension to convert every character to a decimal number (ord), join it as a string and print it. sys.stdin.read() allows us to use Python inline to get input from a pipe. If you replace input with your intended string, this solution is then cross-platform.
printf '%s' 'abc🎶' | python -c "
import sys
input = sys.stdin.read()
sep = '\n'
print(sep.join([str(ord(i)) for i in input]))"
97
98
99
127926
Edit: If all you care about is using hex regardless of encoding, use #user1934428's answer
Example file.txt:
100 foo
2 bar
300 tuu
When using sort -k 1,1 file.txt, the order of lines will not change, though we are expecting :
2 bar
100 foo
300 tuu
How to sort a field consisting of numbers based on the absolute numerical value?
Take a peek at the man page for sort...
-n, --numeric-sort
compare according to string numerical value
So here is an example...
sort -n filename
If you are sorting strings that are mixed text & numbers, for example filenames of rolling logs then sorting with sort -n doesn't work as expected:
$ ls |sort -n
output.log.1
output.log.10
output.log.11
output.log.12
output.log.13
output.log.14
output.log.15
output.log.16
output.log.17
output.log.18
output.log.19
output.log.2
output.log.20
output.log.3
output.log.4
output.log.5
output.log.6
output.log.7
output.log.8
output.log.9
In that case option -V does the trick:
$ ls |sort -V
output.log.1
output.log.2
output.log.3
output.log.4
output.log.5
output.log.6
output.log.7
output.log.8
output.log.9
output.log.10
output.log.11
output.log.12
output.log.13
output.log.14
output.log.15
output.log.16
output.log.17
output.log.18
output.log.19
output.log.20
from man page:
-V, --version-sort
natural sort of (version) numbers within text
Well, most other answers here refer to
sort -n
However, I'm not sure this works for negative numbers. Here are the results I get with sort version 6.10 on Fedora 9.
Input file:
-0.907928466796875
-0.61614990234375
1.135406494140625
0.48614501953125
-0.4140167236328125
Output:
-0.4140167236328125
0.48614501953125
-0.61614990234375
-0.907928466796875
1.135406494140625
Which is obviously not ordered by numeric value.
Then, I guess that a more precise answer would be to use sort -n but only if all the values are positive.
P.S.: Using sort -g returns just the same results for this example
Edit:
Looks like the locale settings affect how the minus sign affects the order (see here). In order to get proper results I just did:
LC_ALL=C sort -n filename.txt
You have to use the numeric sort option:
sort -n -k 1,1 File.txt
Use sort -n or sort --numeric-sort.
You must do the following command:
sort -n -k1 filename
That should do it :)
Use sort -nr for sorting in descending order. Refer
Sort
Refer the above Man page for further reference
echo " Enter any values to sorting: "
read n
i=0;
t=0;
echo " Enter the n value: "
for(( i=0;i<n;i++ ))
do
read s[$i]
done
for(( i=0;i<n;i++ ))
do
for(( j=i+1;j<n;j++ ))
do
if [ ${s[$i]} -gt ${s[$j]} ]
then
t=${s[$i]}
s[$i]=${s[$j]}
s[$j]=$t
fi
done
done
for(( i=0;i<n;i++ ))
do
echo " ${s[$i]} "
done