I'm trying to create a github job that transforms a list of changed files to list of their directories in json format, but completely stuck with the
error shown on screenshot (line 8: Packages/test: Is a directory). Echo command at the end shows empty string.
Action step code here:
- name: Get folders
id: get-folders
run: |
echo "Changed packages: ${{ steps.filter.outputs.changed_files }}"
folders=()
for package in ${{ steps.filter.outputs.changed_files }};
do
folder="$(dirname ${package})"
folders+=("${folder}")
done
result=$("${folders[#]}" | jq -R . | jq -s)
#result=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${folders[#]}")
echo '::set-output name=FOLDERS::${result}'
- run: echo "${{ steps.get-folders.outputs.FOLDERS }}"
YOUR_BASH_ARRAY=( a b c );
jsonString="$(jq --compact-output --null-input '$ARGS.positional' --args -- "${YOUR_BASH_ARRAY[#]}")"
Result: jsonString==["a","b","c"]
Credit to pmf for calling out the answer provided by the ops' question.
Related
im currently running into an issue with deleting certain files from our thumbor cache with ansible. After alot of snipping I receive a list with the file names and im running following ansible task to find and delete them:
shell: find . -name {{ item }} -exec rm "{}" \;
args:
chdir: "{{ thumbor_data_path }}"
ignore_errors: true
loop:
- deletion_hash
when: file_url is defined and deletion_hash | length > 0
the list is definitly filled with the correct names of files I know exist and the task itself marks himself as changed, but the files are not getting deleted. The names of the files are sha1 hashes, and are two directories deep.
Is there something wrong with the shell script?
Example of the deletion_hash list:
"msg": [
"115b744b9f6b23bbad3b6181c858cb953136",
"f52f17b2cca937e5586751ff2e938979890b",
"1c39661a0925b3cdb3b524983aaf6cccd6ee",
"1afc79a9e0e3c07ff0e95e1af3b5cb7ae54c",
"424e9159fe652f47c8e01d0aa85a86fbefed",
"11e4994789f24537d6feea085d2bf39c355b",
"a1d2fe0e122d37555df4062d4c0a5d10b651",
"aef976fc897a87091be5a8d5a11698e19591",
"e79f3ee1e6ccb3caff288b0028e031d75d77",
"9448e5e49679c908263922debdffff68eecb",
"a3933be52277a341906751c3da2dfb07ccd8",
"bef3370862a7504f7857be396d5a3139f5c0",
"8cc0cbe847234af96c0463d49c258c85d50f",
"1e7bf6110dcf994d1270682939e14416fc6e",
"d21dae2c047895129e7c462f6ddc4e512a58",
"c107b29b3185171ec46b479352fab6c97ad2"
]
You can try using the file module; this comes with an assumption that the thumbor_data_path variable does not end with a /; if it does, you need to modify this a bit.
- name: Remove file (delete file)
ansible.builtin.file:
path: "{{ thumbor_data_path }}/{{ item }}"
state: absent
loop: deletion_hash
when: file_url is defined and deletion_hash | length > 0
I want to create a cache key from an expression in cache step located in a steps template. Since caches are imutable, I want to start with a clean cache every week or every months. My template file looks like this:
# steps-tpl.yml
steps:
- task: Cache#2
inputs:
key: compiler-cache | "<expression>" | "$(Agent.JobName)" | "$(Build.SourceBranch)" | "$(Build.SourceVersion)"
restoreKeys: |
compiler-cache | "<expression>" | "$(Agent.OS)" | "$(Agent.JobName)" | "$(Build.SourceBranch)" | "$(Build.SourceVersion)"
compiler-cache | "<expression>" | "$(Agent.OS)" | "$(Agent.JobName)" | "$(Build.SourceBranch)"
compiler-cache | "<expression>" | "$(Agent.OS)" | "$(Agent.JobName)" | "refs/heads/master"
path: $(Pipeline.Workspace)/ccache
displayName: 'ccache/clcache: Warm up cache'
The expression should give me a different value each week of the year or for simplicity, the number of the current month.
I tried different things for expression and none of them worked. Firs I tried to put the expression into a parameter default value which didn't work. Then I tried to put the expression directly in place like this:
"${{ format('{0:MM}', pipeline.startTime) }}"
which didn't work either. And neither this:
"$[format('{0:MM}', pipeline.startTime)]"
I guess run-time expressions are not available outside a script context.
I even tried to set an environment variable in a step before the caching:
echo '##vso[task.setvariable variable=COMPILER_CACHE_KEY_PREFIX,isOutput=true]$[ counter(format("{0:yyyMMdd}", pipeline.startTime), 7) ]'
and to use that with $(previousStep.COMPILER_CACHE_KEY_PREFIX) which didn't work either.
How can I put such an expression into the cache key? Is it possible at all?
I had to do something similar. What worked for me was setting a variable in a Powershell task, then using it with the syntax that expands at runtime:
variables:
- name: projectFolder
value: ''
# assign temporary variable resultfolder to variable projectFolder
- powershell: |
Write-Host "##vso[task.setvariable variable=projectFolder]$(resultfolder)"
displayName: Save project folder path
# refer to projectFolder, not the temporary variable resultfolder
- task: Cache#2
inputs:
key: 'nuget | "$(Agent.OS)" | $(projectFolder)/packages.lock.json'
restoreKeys: |
nuget | "$(Agent.OS)"
path: $(NUGET_PACKAGES)
displayName: Cache NuGet packages
The Cache task outputs the following:
Resolving key:
- nuget [string]
- "Windows_NT" [string]
- D:\a\1\s\PTS_Tasks/packages.lock.json [file] --> 811E162D4016AFBA61F7850XXXXXXXXXXXXX366DFAC37C757FC4D4F49992
Resolved to: nuget|"Windows_NT"|/9Pr8Jh9L66tny00XXXXXXXXXXnL23DoLKFCDGhG8=
Showing that the variable projectFolder has been expanded correctly.
So perhaps execute your expression inside the Powershell script, then assign the result to a variable.
I'm not sure why it doesn't work directly in the Cache task, but this worked for me:
variables:
cacheKeyDay: $[format('{0:ddMMyyyy}', pipeline.startTime)]
- task: Cache#2
inputs:
key: $(cacheKeyDay)
I solved the issue with a workaround. First a script to generate the value I need for the cache key, then writing this value to a file und then using the file fingerprint as the cache value.
- bash: |
CACHEKEY=$(find -s ./src -type f -exec cat {} \;|md5)
CACHEKEY="${CACHEKEY}-${{parameters.jobname}}"
echo $CACHEKEY
echo $CACHEKEY > cachekey.txt
displayName: 'get hash of all folders, write value to file'
- task: Cache#2
inputs:
key: 'cachekey.txt'
path: $(build.artifactStagingDirectory)
cacheHitVar: 'CacheRestored'
I'm trying to get for certain text in multiple directories and then to send the output, when I'm using the same shell on the machines it works correctly and sends the requested output, but when trying to use ansible in this case I'm getting error.
- name: Extract logs from server
hosts: all
tasks:
- name: execute
shell: grep -ir {{ my_var }} /var/log/folder/folder/* | mailx -s "Findings" {{ email }}
my_var: 1234-abcd-1a2b3c3d
email: bla.bla#company.com
the error :
fatal: [10.54.229.13]: FAILED! => {"changed": true, "cmd": "grep -ir 37f624c3-c898-4e2b-958a-ab78d20a03fe /var/log/3ds/components/* > /tmp/log", "delta": "0:00:00.005845", "end": "2020-10-05 17:28:17.509948", "msg": "non-zero return code", "rc": 2, "start": "2020-10-05 17:28:17.504103", "stderr": "grep: /var/log/3ds/components/*: No such file or directory", "stderr_lines": ["grep: /var/log/3ds/components/*: No such file or directory"], "stdout": "", "stdout_lines": []}
I also tried to grep > /tmp/tmp.file and then from there to cat the tmp.file | mailx the content.
and it works fine when using on machine iteself but not threw ansible.
while trying :
"find /var/log/folder/components/* -type f -exec grep 37f624c3-c898-
{} \; > /tmp/log "
getting the following:
ERROR! Syntax Error while loading YAML.
found unknown escape character
The error appears to have been in '/tmp/awx_730_htsr76go/project/v2_log_extraction.yml': line 9, column 103, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: execute
shell: "find /var/log/folder/components/* -type f -exec grep 37f624c3-c898 {} \; > /tmp/log "
^ here
while using the var in playbook
- name: Extract logs from server
hosts: all
become: yes
tasks:
- name: execute
shell: "find /var/log/folder/components/* -type f -exec grep "{{ my_var}}" {} \; | mailx -s "log" {{ email }}"
args:
executable: /bin/sh
getiing the following error :
The error appears to have been in '/tmp/awx_733_qiog0f53/project/v2_log_extraction.yml': line 9, column 64, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: execute
shell: "find /var/log/3ds/components/* -type f -exec grep {{ my_var}} {} \; | mailx -s "log" {{ email }}"
^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value
please advice the best way to get to the result
Hi guys,
the first code works perfectly.
The issue was between the chair and the pc :D
I used a wrong inventory with the wrong server there really wasn't /var/log/folder/folder
thanks every one for helping
I'm working on a Makefile which takes a command line parameter, runs a translation (tr) on it, then uses that variable to create a directory and also prints the result. When the variable is printed, it is as I expect (the translated name), however when the variable is used to create the directory, the results are incorrect.
So for example, when I run:
make build=debug
I see:
Building...
***************************
BuilD Type: Debug
Build Target: x86
But the resultant file system is:
out/
└─── debug | tr d D_x86
I thought that running the translation on the input variable build and assigning it to $BUILD_TYPE would have set this new variable to "Debug" but it seems like that is not the case all the time... How does this actually work?
Full Makefile:
BUILD_TYPE=Release
BUILD_TARGET=Arm
OUTPUT_DIR=out
MKDIR_P = mkdir -p
ifeq ($(build),debug)
BUILD_TYPE='$(build) | tr d D'
endif
ifeq ($(target),x86)
BUILD_TARGET=$(target)
endif
TT=$(BUILD_TYPE)_$(BUILD_TARGET)
all: directories
#echo 'Building...'
#echo '***************************'
#echo 'Build Type: $(BUILD_TYPE)'
#echo 'Build Target: $(BUILD_TARGET)'
#echo 'Output Directory: $(PWD)/$(OUTPUT_DIR)'
#echo '***************************'
directories:
${MKDIR_P} $(OUTPUT_DIR)
${MKDIR_P} $(OUTPUT_DIR)/$(TT)
This line:
BUILD_TYPE='$(build) | tr d D'
is not running a shell command. That is a literal string assignment.
You are assigning to the variable BUILD_TYPE the literal string 'debug | tr d D'.
To do what you want you need to actually run a shell command. Something like this for example.
BUILD_TYPE:=$(shell echo '$(build)' | tr d D)\
As to why you are seeing things "correctly" on the output line that's because you happen to be generating a valid shell pipeline there and make is running it.
That is this line
#echo 'Build Type: $(BUILD_TYPE)'
becomes
#echo 'Build Type: 'debug | tr d D''
which, you'll notice, is a valid shell line. Also notice that you get BuilD type and not Build type printed out?
How do I use the when statement based on the standard output of register: result? If standard output exists I want somecommand to run if no standard output exists I want someothercommand to run.
- hosts: myhosts
tasks:
- name: echo hello
command: echo hello
register: result
- command: somecommand {{ result.stdout }}
when: result|success
- command: someothercommand
when: result|failed
Try checking to see it if equals a blank string or not?
- hosts: myhosts
tasks:
- name: echo hello
command: echo hello
register: result
- command: somecommand {{ result.stdout }}
when: result.stdout != ""
- command: someothercommand
when: result.stdout == ""
As of 2018, the recommended way to test if output is empty is just:
when: result.stdout | length > 0
That is the pythonic way of evaluating truth, null, empty strings, empty lists all evaluate as false.
Other older alternatives not recommended or even not working:
result.stdout != "" would not pass ansible-lint check!
result.stdout | bool will NOT work as most strings will evaluate as False, only cases where it would return true is if stdout happens to be one of the true, yes,... kind of strings.
result.stdout used to work but now triggers:
[DEPRECATION WARNING]: evaluating as a bare variable, this
behaviour will go away and you might need to add |bool to the
expression in the future. Also see CONDITIONAL_BARE_VARS configuration
toggle.. This feature will be removed in version 2.12. Deprecation
warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.`
To expand on this answer and address the comment regarding potential problems if stdout isn't defined, the following when statement can be used to ensure stdout is defined before trying to check its length:
when: result.stdout is defined and result.stdout | length > 0