Use match from file in string - linux

Question

I have a list of user ids in a file and I'm trying to generate some sql from the file.
cat users.json | awk '/UID/{print "INSERT IGNORE INTO potential_problem_users VALUES ("$2");" }'
But it's not doing what I expected it to do:
);SERT IGNORE INTO potential_problem_users VALUES ("1"
);SERT IGNORE INTO potential_problem_users VALUES ("2"
The query I am trying for is:
INSERT IGNORE INTO potential_problem_users VALUES ([userID]);
example json if it helps:
{
"results": [
{
"UID": "abc"
},
{
"UID": "124"
}
],
"objectsCount": 5,
"totalCount": 10966,
"statusCode": 200,
"errorCode": 0,
"statusReason": "OK"
}
Am I using the right tool? If I am, what am I doing wrong?

Answer 1

Your JSON input has a couple of issues which makes it impossible to parse them using parser like jq which on OS X you can install by brew install jq
The error free JSON from your question would be
{
"results": [
{
"UID": "abc"
},
{
"UID": "124"
}
],
"objectsCount": 5,
"totalCount": 10966,
"statusCode": 200,
"errorCode": 0,
"statusReason": "OK"
}
To parse this JSON and produce two statements as queries containing the UID values, just do
jq --raw-output '"INSERT IGNORE INTO potential_problem_users VALUES (" + (.results[] | .UID) + ")"' users.json
would produce an output as
INSERT IGNORE INTO potential_problem_users VALUES (abc)
INSERT IGNORE INTO potential_problem_users VALUES (124)
The addition operator + in jq allows you to concatenate strings to form the final string.
jq snippet from jqplay.

Answer 2

awk -F: '/UID/ {print "INSERT IGNORE INTO potential_problem_users VALUES ("gensub(" ","","g",$2)");"}' users.json
Use gensub to take the spaces out of second : delimited field. Note also that you don't need to cat the file to awk it.

Answer 3

You have several JSON parsers available to you on OS X that will also work on Linux. You can use Perl, Python or Ruby among others.
Here is a demo in Ruby:
Given:
$ cat json
{
"results": [
{
"UID": "abc"
},
{
"UID": "124"
}
],
"objectsCount": 5,
"totalCount": 10966,
"statusCode": 200,
"errorCode": 0,
"statusReason": "OK"
}
You can parse that file and print to lines of interest this way:
$ ruby -0777 -lane 'require "json"
d=JSON.parse($_)
d["results"].each {|e| puts "INSERT IGNORE INTO potential_problem_users VALUES (#{e["UID"]})" }' json
INSERT IGNORE INTO potential_problem_users VALUES (abc)
INSERT IGNORE INTO potential_problem_users VALUES (124)

Answer 4

users.json contains carriage returns (aka \rs aka control-Ms), run dos2unix or similar on it first then re-run your awk script (or run any other script) on it and then let us know if you still have a problem.
Almost every time you find characters you expect to see at the end of a line showing up at the start instead the problem is control-Ms.