My bash script runs to retrieve lambda functions and their tags.
It runs ok and does what it needed to do, however I need to get the output written to a .txt or a .csv file, which needs to be in a readable format.
Below is the script I have;
#!/bin/bash
while read -r name; do
aws lambda list-functions | jq -r ".Functions[].FunctionArn" | xargs -I {} aws lambda list-tags --resource {} --query '{"{}":Tags}' --output text
done
Below is what a returned value looks like after the script runs;
ARN:AWS:LAMBDA:EU-WEST-1:1939999:FUNCTION:example-lambda EXX dev example-lambda False release-1.1.9 False True
I need to get all the items returned and lined up neatly in a txt or csv file. Any help would be appreciated.
I would recommend to use the resourcegroupstaggingapi API to solve this problem. This API allows you to get all resources of a specific type and their tags.
To get all your Lambda functions for your default region and their tags you can run the following command:
aws resourcegroupstaggingapi get-resources --resource-type-filters "lambda"
The output of this command can now be parsed with jq. The great thing about jq is that you can manipulate the output to be CSV.
To get CSV output with two columns (ARN, Tags) you can run the following command:
aws \
resourcegroupstaggingapi \
get-resources \
--resource-type-filters "lambda" \
| jq -r '.ResourceTagMappingList[] | [.ResourceARN, ((.Tags | map([.Key, .Value] | join("="))) | join(","))] | #csv'
The advantage of this approach is that you only have a single HTTP call making it relatively fast. The disadvantage is that you only get the ARN and the tags.
As shimo mentioned in a comment to the question, a way to save the output of a command to a file is using the > operator.
> operator replaces the existing content of the file with the output of the command. If you want to save the output of multiple commands to the same file, you should use >> operator.
You can also use a pipe and the command tee. The output will be printed in your screen and in a file, which will be at the end of the file.
I found this tutorial helpful. Based on what you written, you could pipe the output of your command into a csv, or concotanate into an array then write it into a file with a newline character at the end of each line.
Related
i run the following function in (git-)bash under windows:
function config_get_container_values() {
local project_name=$1
local container_name=$2
#local container_name="gitea"
echo "###"
buildcmd="jq -r \".containers[]."
echo "$buildcmd"
buildcmd="${buildcmd}${container_name}"
echo "$buildcmd"
buildcmd="${buildcmd}foobar"
echo "$buildcmd"
echo "###"
}
The output of this is the following. Whyever, after using the variable to extend the string, he starts to overwrite $buildcmd. I tried this also with everything in one line as well with the append command (=+). Everytime the same result.
###
jq -r ".containers[].
jq -r ".containers[].gitea
foobar".containers[].gitea
###
The really strange thing is: When i enable the line local container_name="gitea" everything works as expected. The output is:
###
jq -r ".containers[].
jq -r ".containers[].gitea
jq -r ".containers[].giteafoobar
###
When i put this all into a news file, its also works as expected. So i think something goes wrong in the thousands of line before calling this function. Any idea, what could be cause of this behavior?
Regards
Dave
This is not how you should build up the command, DOS line endings aside. Use --arg to pass the name into the filter as a variable. For example,
config_get_container_values() {
local project_name=$1
local container_name=$2
jq -r --arg n "$container_name " '.containers[][$n+"foobar"]'
}
config_get_container foo gitea < some.json
If the function is invoked with
config_get_container_values proj gitea
it produces the "expected" output. If it is invoked with
config_get_container_values proj $'gitea\r'
it produces output that looks like the first output example. $'gitea\r' expands to a string that consists of 'gitea' followed by a Carriage return (CR) character.
One possible cause of the problem is that the container name (gitea) was read from a file that had Windows/DOS line endings (CR-LF). Problems like that are common. See the first question ("Check whether your script or data has DOS style end-of-line characters") in the "Before asking about problematic code" section of the Stack Overflow 'bash' Info page.
I have a ~300 GB zipped vcf file (.vcf.gz) which contains the genomes of about 700 dogs. I am only interested in a few of these dogs and I do not have enough space to unzip the whole file at this time, although I am in the process of getting a computer to do this. Is it possible to unzip only parts of the file to begin testing my scripts?
I am trying to a specific SNP at a position on a subset of the samples. I have tried using bcftools to no avail: (If anyone can identify what went wrong with that I would also really appreciate it. I created an empty file for the output (722g.990.SNP.INDEL.chrAll.vcf.bgz) but it returns the following error)
bcftools view -f PASS --threads 8 -r chr9:55252802-55252810 -o 722g.990.SNP.INDEL.chrAll.vcf.gz -O z 722g.990.SNP.INDEL.chrAll.vcf.bgz
The output type "722g.990.SNP.INDEL.chrAll.vcf.bgz" not recognised
I am planning on trying awk, but need to unzip the file first. Is it possible to partially unzip it so I can try this?
Double check your command line for bcftools view.
The error message 'The output type "something" is not recognized' is printed by bcftools when you specify an invalid value for the -O (upper-case O) command line option like this -O something. Based on the error message you are getting it seems that you might have put the file name there.
Check that you don't have your input and output file names the wrong way around in your command. Note that the -o (lower-case o) command line option specifies the output file name, and the file name at the end of the command line is the input file name.
Also, you write that you created an empty file for the output. You don't need to do that, bcftools will create the output file.
I don't have that much experience with bcftools but generically If you want to to use awk to manipulate a gzipped file you can pipe to it so as to only unzip the file as needed, you can also pipe the result directly through gzip so it too is compressed e.g.
gzip -cd largeFile.vcf.gz | awk '{ <some awk> }' | gzip -c > newfile.txt.gz
Also zcat is an alias for gzip -cd, -c is input/output to standard out, -d is decompress.
As a side note if you are trying to perform operations on just a part of a large file you may also find the excellent tool less useful it can be used to view your large file loading only the needed parts, the -S option is particularly useful for wide formats with many columns as it stops line wrapping, as is -N for showing line numbers.
less -S largefile.vcf.gz
quit the view with q and g takes you to the top of the file.
I have a Jenkins console output that looks like this:
Started by remote host 10.16.17.13
Building remotely on ep9infrajen201 (ep9) in workspace d:\Jenkins\workspace\Tools\Provision
[AWS-NetProvision] $ powershell.exe -NonInteractive -ExecutionPolicy ByPass "& 'C:\Users\user\AppData\Local\Temp\jenkins12345.ps1'"
Request network range: 10.1.0.0/13
{
"networks": [
"10.1.0.0/24"
]
}
Finished: SUCCESS
I get this from a curl command that I run. to check the JENKINS_JOB_URL/lastBuild/consoleText
My question is, for the sake of some other automation I am doing, how do I get just "10.1.0.0/24" so I can assign it to a shell variable using LINUX tools?
Thank you
Since you listed jq among the tags of your duplicate question, I'll assume you have jq installed. You have to clean up your output to get JSON first, then get to the part of JSON you need. awk does the former, jq the latter.
.... | awk '/^{$/{p=1}{if(p){print}}/^}$/{p=0}' | jq -r .networks[0]
The AWK script looks for { on its own on a line to turn on a flag p; prints the current line if the flag is set; and switches off the flag when it encounters } all by itself.
EDIT: Since this output was generated on a DOS machine, it has DOS line endings (\r\n). To convert those before awk, additionally pipe through dos2unix.
Can I register the output of a task? Is there an argument with ansible command for that ?
This is my command:
ansible all -m ios_command -a"commands='show run'" -i Resources/Inventory/hosts
I need this, because the output is a dictionary and I only need the value for one key. If this is not possible, is there a way to save the value of that key to a file?
I have found that you can convert ansible output to json when executing playbooks with "ANSIBLE_STDOUT_CALLBACK=json" preceding the "ansible-playbook" command. Example:
ANSIBLE_STDOUT_CALLBACK=json ansible-playbook Resources/.Scripts/.Users.yml
This will give you a large output because it also shows each host's facts, but will have a key for each host on each task.
This method is not possible with ansible command, but it's output is similar to json. It just shows "10.20.30.111 | SUCCESS =>" before the main bracket.
Source
Set the following in your ansible.cfg under the [defaults] group
bin_ansible_callbacks=True
Then as #D_Esc mentioned, you can use
ANSIBLE_STDOUT_CALLBACK=json ansible all -m ios_command -a"commands='show run'" -i Resources/Inventory/hosts
and can get the json output which you can try to parse.
I have not found a way to register the output to a variable using ad-hoc commands
I want to extract the instance ID and the tags from the result of a command ec2-describe-instances and want to store the result in a text file. The result set gives :
But i want the tags owner and cost.centre also to be fetched
Kindly guide me how to do that
this will help to find instance-id
$ aws ec2 describe-instances --filters Name=vpc-id,Values=vpc-xxx | awk '{ print $8 }' | sort -n | grep "i-"
i-4115d38c
i-5d534697
i-6e679a45
i-7a659851
i-8d6bae40
i-cd6f9000
i-d264ad1e
i-d5888618
i-e2332e2e
ps considering that you already configured / run "aws configure"
If I understand the question correctly, I think you just need to add expressions to your 2nd grep:
ec2-describe-instances | grep -i "tag" | grep -i -e "name" -e "owner" -e "cost.centre"
This is going to be unnecessarily complicated to do in a shell script. Here are some suggestion:
you are using ec2cli. Don't use that. Use AWS-CLI instead. Because parsing the output in ec2cli is a pain. Whereas AWS-CLI provides output in JSON, it is way more easier to parse. Also, AWS is going to support AWS-CLI only henceforth.
The information that you need a perfect use-case for using a hash. You can either install and run AWs-CLI commands via a perl script and then capture the output in a hash. Perl is very powerful for handling such data structure.
OR, you can use one of the SDKs from AWS (I use Ruby SDK) and then capture the whole information in a hash and then print it the way you want.
Bottom line is, you need to capture the tags in a hash to make your life easier. Ans this becomes more and more prominent when you have multiple tags.
Using awk
ec2-describe-instances |awk 'BEGIN{IGNORECASE=1}/(name|owner|cost.center)/&&/tag/'
TAG instance i-c4 Name Rii_Win_SAML
TAG instance i-c42 Owner Rii Pandey
Here is another way to do without using jq and other parsing tools.
ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId,Tags[?Key==`Name`].Value | [0],Tags[?Key==`cost.centre`].Value | [0]]' --output text
This should work:
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "owner"},{Key: "costcenter"}]}))|"Instance ID: \(.InstanceId) Owner: \(.Tags[]|select(.Key=="owner")|.Value), Cost Center: \(.Tags[]|select(.Key=="costcenter")|.Value)"'
TL;DR: The way AWS does tags is a living nightmare, even with jq
Also, use AWS CLI, not old, unsupported tools.