Passing value passing to multiple commands in xargs - aws-cli

Question

I'm trying to execute multiple commands within xargs. The issue I'm seeing here is the piped value '%' is being passed only to the 1st sub-command inside xargs, but not to the 2nd one. Validated the same by interchanging the commands position, and still always the 2nd command never gets the required value for '%'
Command-1
aws ec2 describe-instances --query 'Reservations[].Instances[?(LaunchTime>=`2015-01-01` && LaunchTime<=`2015-02-28`)][].{id: InstanceId, launched: LaunchTime}' | jq --raw-output '.[] | .id' | xargs -n 1 -I % sh -c 'aws cloudwatch get-metric-statistics --metric-name NetworkPacketsIn --start-time 2018-01-01T00:00:00Z --end-time 2018-02-28T23:59:59Z --period 2592000 --namespace AWS/EC2 --statistics Maximum --dimensions Name=InstanceId,Value=%; echo instance: %;'
Output:
{
"Label": "NetworkPacketsIn",
"Datapoints": []
}
instance: %
{
"Label": "NetworkPacketsIn",
"Datapoints": []
}
instance: %
Command-2
aws ec2 describe-instances --query 'Reservations[].Instances[?(LaunchTime>=`2015-01-01` && LaunchTime<=`2015-02-28`)][].{id: InstanceId, launched: LaunchTime}' | jq --raw-output '.[] | .id' | xargs -n 1 -I % sh -c 'echo instance: %; aws cloudwatch get-metric-statistics --metric-name NetworkPacketsIn --start-time 2018-01-01T00:00:00Z --end-time 2018-02-28T23:59:59Z --period 86400 --namespace AWS/EC2 --statistics Maximum --dimensions Name=InstanceId,Value=%;'
Output
instance: i-3e4fab33
{
"Label": "NetworkPacketsIn",
"Datapoints": []
}
instance: i-c2abbac8
{
"Label": "NetworkPacketsIn",
"Datapoints": []
}

Answer 1

TL;DR on a Mac xargs arguments can not grow beyond 255 bytes after replacement is done.
Shortening your argument and leaving the semicolon off the last command fixed the error:
aws ec2 describe-instances --query 'Reservations[].Instances[?(LaunchTime>=`2015-01-01` && LaunchTime<=`2015-02-28`)][].{id: InstanceId, launched: LaunchTime}' | jq --raw-output '.[] | .id' | xargs -I % sh -c 'aws cloudwatch get-metric-statistics --metric-name NetworkPacketsIn --start-time 2018-01-01T00:00:00Z --end-time 2018-02-28T23:59:59Z --period 86400 --namespace AWS/EC2 --statistics Maximum --dimensions Name=InstanceId,Value=%;echo id=%'
Here's the longer answer along with some tests to prove it.
From the xargs man page:
-I replstr
Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or
5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments,
after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenat-
ing as much of the argument containing replstr as possible, to the constructed arguments to utility, up to
255 bytes. The 255 byte limit does not apply to arguments to utility which do not contain replstr, and fur-
thermore, no replacement will be done on utility itself. Implies -x.
The argument the OP is passing to xargs
'aws cloudwatch get-metric-statistics --metric-name NetworkPacketsIn --start-time 2018-01-01T00:00:00Z --end-time 2018-02-28T23:59:59Z --period 2592000 --namespace AWS/EC2 --statistics Maximum --dimensions Name=InstanceId,Value=%; echo instance: %;'
is 250 bytes. When the % is replaced by the AMI ID it grows past the 255 byte limit and blows up.
If you want to test this yourself try the following, the argument has 254 bytes:
echo blah |xargs -I % sh -c 'export blah=%; echo $blah; echo $blah; echo $blah;\
echo $blah; echo $blah; echo $blah;echo $blah; echo $blah; echo $blah;echo $blah;\
echo $blah; echo $blah;echo $blah; echo $blah; echo $blah;echo $blah; echo $blah;\
echo $blah;echo $blah;echo $blah;'
This will pass the word blah to every echo statement correctly.
blah blah blah blah blah blah blah blah blah blah blah blah blah blah
blah blah blah blah blah blah
Add one more echo $blah; to the end, taking the byte total to 265 bytes and it blows up:
% % % % % % % % % % % % % % % % % % % %
To make a long post even longer, I passed the instance id to the describe-instances command with a --instance-ids switch and it worked as expected because the argument expansion was below the 255 limit.
aws ec2 describe-instances --query 'Reservations[].Instances[?(LaunchTime>=`2015-01-01` && LaunchTime<=`2015-02-28`)][].{id: InstanceId, launched: LaunchTime}' | jq --raw-output '.[] | .id' | xargs -I % sh -c 'echo instance: %; aws ec2 describe-instances --instance-ids=%; '