I have 2 Release.ini file:
one I am downloading from S3 storing in /CodeRootFolder/tmp location
and other in /DocRootDir location
I want to verify if there are changes in downloaded (i.e. /CodeRootFolder/tmp/Release.ini) file, and if there is change then execute the rsync command via puppet exec as below.
The error is if there change, it does not executing, it seems to as diff returns 1.
exec {'Actual code deployment with rsync':
command => "rsync ${myclass::CodeRootFolder}/tmp/* ${myclass::DocRootDir}/)",
#cwd => "${myclass::CodeRootFolder}",
onlyif => "diff --changed-group-format='%<' --unchanged-group-format='' ${myclass::CodeRootFolder}/tmp/Release.ini ${myclass::DocRootDir}/Release.ini",
path => ['/opt/rh/php55/root/usr/bin','/opt/rh/php55/root/usr/sbin','/usr/local/sbin','/usr/local/bin', '/sbin/' ,'/bin/', '/usr/sbin/','/usr/bin/'],
}
Is there a good solution to my problem.
Thanks in advance
The error is if there change, it does not executing, it seems to as
diff returns 1.
Yes, that's correct.
Invoking diff
An exit status of 0 means no differences were found, 1 means some
differences were found, and 2 means trouble.
If you want the command executed if there is change, you have to use unless instead of onlyif.
Related
I have the following content inside my p4aliases.txt.
diff-cl $(target-cl) = diff -dl //...#$(EQ)$(target-cl)
Basically it diffs against your files in current workspace toward the target shelved files of changelist.
It is fine. I can execute it. But when I compare the result coming from above aliased command against the direct raw (non-aliased) command as follows
p4 diff -dl //...#=<target-cl>
the output lines of text from aliased command is in wrong order e.g. changes according to a certain file shows up first before a line of file shown, line orders are messed up. This is not the case if you execute with a non-aliased command.
Example
Expected result
==== //depot/common.h#none - x:\mydir\project\src\common.h ====
==== //depot/file.cpp#none - x:\mydir\project\src\file.cpp ====
3a4
> added line 1
==== //depot/file.h#none - x:\mydir\project\src\file.h ====
Actual result
3a4
> added line 1
==== //depot/common.h#none - x:\mydir\project\src\common.h ====
==== //depot/file.cpp#none - x:\mydir\project\src\file.cpp ====
==== //depot/file.h#none - x:\mydir\project\src\file.h ====
I have p4 version as of Rev. P4/NTX64/2021.1/2126753 (2021/05/12).
Perforce server version (got from p4 info) is Server version: P4D/LINUX26X86_64/2017.1/1574018 (2017/10/02).
How can I solve this issue?
Could this be a version too far away between client and server
Update
I have tested p4 client all the way down from 2016-2020 version by downloading old binaries from ftp.perforce.com (in directory perforce). No luck. Output still messed the same. So it's not the problem about version mismatch.
This looks like a bug in the p4 client. When the client does a diff, it's written by the ClientUser::Diff() method, which defaults to writing to stdout (i.e. it does not route the output through ClientUser::OutputText()):
https://workshop.perforce.com/projects/perforce_software-p4/files/2018-2/client/clientuser.cc#436
https://workshop.perforce.com/projects/perforce_software-p4/files/2018-2/client/clientuser.cc#573
Output from commands run as part of an alias go through the ClientUserStrBuf subclass, which buffers all of its output. The file headers, for example, are buffered by ClientUserStrBuf::OutputInfo():
https://workshop.perforce.com/projects/perforce_software-p4/files/2018-2/client/clientaliases.cc#1647
There isn't a ClientUserStrBuf::Diff() implementation, though, so that diff output goes straight to stdout while the headers are buffered and printed at the end (presumably after some post-processing) -- hence the diff output showing up first in the console.
The fix I'd make would be to have the base ClientUser::Diff() implementation route the output through OutputText() when no output file is provided, which seems like the least-surprise behavior; that'd fix the aliases behavior and might even make life a little easier for other client developers who would otherwise hit the same issue. If you have a support contract with Perforce you can file this as a bug report, or since the client is open source you can take a crack at fixing and building it yourself. I don't think there's a workaround that doesn't involve modifying the client source code.
Samwise has the correct approach to truly fix the problem at hands although it might take some effort to understand the code, and conduct the fix itself.
At any rate, if we took such approach we won't be able to take benefits of bug fixes and future updates as we will be stuck with 2018-2 version of p4 as it's the latest as it can be in which we can grab the source.
I would recommend to use WSL then interact with p4.exe (yes, a Windows-based binary) for Windows-based project, and p4 for Linux-based binary. If you didn't use WSL, please find the .bash_aliases-like solution as I have below to seamlessly solve aliases diff operation.
Put the following code into your ~/.bash_aliases
# p4 - fix of aliases diff operation
# platform independent, it will choose a correct binary path to execute properly
p4() { cmd="p4.exe" # default is Windows-based
# get the last argument value, if "-lx" passed in then we know it's linux
if [[ "${#: -1}" == "-lx" ]]; then
cmd="/usr/local/bin/p4"
fi
if [[ $1 == "diff-cl" ]]; then
if [ -z "$2" ]; then
echo "usage: p4 diff-cl <CL>"
return 1
fi
$cmd diff -dl //...#=$2 | diffp4 | less -r
elif [[ $1 == "diff-cl-fonly" ]]; then
if [ -z "$2" ]; then
echo "usage: p4 diff-cl-fonly <CL>"
return 1
fi
$cmd diff -Od -dl -ds //...#=$2 | diffp4 | grep ==== | less -r
else
$cmd "$#"
fi
}
then source ~/.bash_aliases.
What it does is to allow you to still use p4 with all of its original commands & arguments normally with exceptions of diff-cl (which is the same name of alias I've put into p4aliases.txt for Windows or ~/.p4aliases for Linux). You can safely remove diff-cl entry from p4's alias file, or just leave it there. What we have in ~/.bash_aliases file will intercept whenever such argument matches then execute the raw command, just that we don't have to type long command ourselves.
We can later remove such section in our ~/.bash_aliases file when upstream p4 has been fixed.
In else section, we just relay the the whole arguments, and it will be performed just as normally done.
Extra: diff-cl-fonly to list out only files (its depot path, and local workspace path) which have changes.
i try to automate geting differencies at php files on container, but I receive exit code: 1 and kill process.
RUN diff ../../../composer/vendor/leroy-merlin-br/mongolid/src/Mongolid/ActiveRecord.php ../../../diff/ActiveRecord.php > ../../../diff/diff-ActiveRecord.diff
How can I fix is?
diff returns 1 if there are differences between the files, which will fail the RUN directive. You could explicitly return 0 for that line if you don't want to break docker's workflow:
RUN diff ../../../composer/vendor/leroy-merlin-br/mongolid/src/Mongolid/ActiveRecord.php ../../../diff/ActiveRecord.php > ../../../diff/diff-ActiveRecord.diff; exit 0
Within CI, I am running a bash script that calls many bash scripts.
./internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null
This doest not disable the stdout returned by the script.
The Gitlabi-CI runners stop logging after 100MB of log, It says Job's log exceeded limit of 10240000 bytes.
I know the log script can only grow up.
How can I optimize the output log size?
I don't need to have all the stdout, I can have stderr but then it will be a long running script without information.
Is there a way to display the commands which is running like when doing set -x?
Edit
Reading the answers, I was not able to solve my issue. I need to add that I am using nodejs to run the bash script that run the long bash script.
This is how I call my node script within .gitlab-ci.yml:
scripts:
- node my_script.js
Within my_script.js, I have:
exports.handler = () => {
const ls = spawn('bash', [path.join(__dirname, 'release.sh')], { stdio: 'inherit' });
ls.on('close', (code) => {
if (code !== 0) {
console.log(`ps process exited with code ${code}`);
process.exitCode = code;
}
});
};
Within my_script.sh, I have:
./internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null
You can selectively redirect file handles with exec.
exec >stdout 2>stderr
This however loses the connection to the terminal, so there is no simple way to output anything to the terminal after this point.
You can instead duplicate a file handle with m>&n where m is the number of the file descriptor to duplicate and n is the number of the new one (choose a big number like 99 to not accidentally clobber an existing handle).
exec 98<&1 # stdout
exec 99<&2 # stderr
exec >/dev/null 2>&1
:
To re-enable output,
exec 1<&98 2<&99
If you redirected to a temporary file instead of /dev/null you could obviously now show the tail of those files to the caller.
tail -n 100 "$TMPDIR"/stdout "$TMPDIR"/stderr
(On a shared server, probably use mktemp to create a unique temporary directory at the beginning of your script; static hard-coded file names make it impossible to run two builds at the same time.)
As you usually can't predict where the next error will happen, probably put all of this in a wrapper script which performs the redirection, runs the build, and finally displays the tail end of the temporary log files. Some build servers probably want to see some signs of life in the log file every few minutes, so perhaps tail a few lines every once in a while in a loop, too.
On the other hand, if there is just a single build command, the whole build job's stdout and stderr can simply be redirected to a log file, and you don't need to exec things back and forth. If you need to enable output selectively for portions of the script, use exec as above; but for wholesale redirection, just redirect the one command.
In summary, maybe your build script would look something like this.
#!/bin/sh
t=$(mktemp -t -d cibuild.XXXXXXXX) || exit
trap 'kill $buildpid; wait $buildpid; tail -n 500 "$t"/*; rm -rf "$t"' 0 1 2 3 5 15
# Your original commands here
${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}">"$t"/stdout 2>"$t"/stderr &
buildpid=$!
while kill -0 $buildpid; do
sleep 180
date
tail -n 1 "$t"/*
done
wait
A flaw with this approach is that you lose timing information. A proper solution woud let you see when each line was produced, and display standard output and standard error intermixed in the order the messages were printed, perhaps with visible time stamps, and even with coloring hints (red time stamps for stderr?)
Option 1
If your script will output the error message to stderr, you can ignore all output to stdout by using command > /dev/null, where /dev/null is a black hole that will take away any output to it.
Option 2
If there's any pattern on your error message, you can use grep to filter out those error messages.
Edit 1:
To show the command that is running, you can supply -x command to bash; therefore, your command will be
bash -x ${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" > /dev/null
bash will print the command executed to stderr
Edit 2:
If you want to reduce the size of the output file, you can pass it to gzip by using ${initial_process_wd}/internals/declination/create "${RELEASE_VERSION}" "${CI_COMMIT_REF_NAME}" | gzip > logfile.
To read the content of the logfile, you can use zcat logfile.
i've a problem with the commandline command "smbclient" of samba on arm.
I wrote a script to download files from a Windows Share.
Here the smb-part of this script.
smbclient //CNAME/SNAME -I0.0.0.0 -N -c "case_sensitive; cd folder; prompt; mget file"
echo $?
My problem ar the exit codes.
If the file is downloaded completely, the exit code is 0 (OK)
If the file cannot be downloaded, the exit code is 1 (OK)
If the testmaschine loses the connection to the share due downloading a file, the exit code is 0 (NOT GOOD), but error ("Lost connection...etc.") is written to console. (OK)
I tried it with two different versions.
samba-3.0.32
samba-3.6.19
Both the same.
Does someone know a good workaround (or smbclient-argument) to let my script know, that the download failed?
PS. I checked the smbclient sources. It looks like they forgot to set the exitcode. Because everytime there is another error the set the Errormessage and do an (e.g. exit(1)). But for timeouts, they only set the Errormessage.
Thank you in advance!
What would be best is to use the -E argument to smbclient and redirect 2>/errorlog from the command line. You can then check this file to see if any errors occurred.
Warning, the first line is always the Domain=......... so you may need to strip that line out.
Something like this:
smbclient Hostname -A authfile -E 1>log 2>errorlog <<-EOF
get foo
EOF
In the errorlog you should find something like below, your log file will be empty
Domain=[Hostname] OS=[Windows Server 2008 R2 Standard 7601 Service
Pack 1] Server=[Windows Server 2008 R2 Standard 6.1]
NT_STATUS_OBJECT_NAME_NOT_FOUND opening remote file \foo
We have a wrapper script for Teradata TPT utility. The wrapper script is pretty straightforward but the problem is that the exit status of the wrapper is not the same as that of the utility. In many cases, the script returns 0 even if the utility fails. I have saved the exit status in a separate variable because some steps need to be done before exiting but exiting with this variable's value doesn't seem to work. Or is the utility returning status 0 even in case of some failures even if the logs clearly specify some other status?
The worse part is, this behavior is quite random, sometimes the script does fail with the exit status of the utility. I want to be sure if there is some problem with utility's exit status.
The script runs through KSH. The final part of the wrapper script is:
tbuild -f $sql.tmp -j ${id}_$JOB >$out 2>&1
ret_code=$?
cd ${TWB_ROOT}/logs
logpath=`ls -t ${TWB_ROOT}/logs/${id}_${JOB}*.out |head -1`
logpath1=${logpath##*/}
logname=${logpath1%-*}
tlogview -l ${logpath} > /edw/$GROUP/tnl/jobs/$JOB/logs/tpt_logs/${logname}.log
###Mainting 3 tpt binary log files
if [ $ret_code -eq 0 ]
then
binout=$TPTLOGDIR/${logname}.dat
binout1=$TPTLOGDIR/${logname}.dat1
binout2=$TPTLOGDIR/${logname}.dat2
[ -f $binout1 ] && mv $binout1 $binout2
[ -f $binout ] && mv $binout $binout1
mv "$logpath" "/edw/${GROUP}/tnl/jobs/$JOB/logs/tpt_logs/${logname}.dat"
fi
rm -f $sql.tmp
echo ".exit"
exit $ret_code
Thanks in advance for the help and suggestions.
The script looks ok, and should indeed return the same exit code as the tbuild utility.
It comes down to knowledge of the specific product.
I've never worked with any of these products, but Teradata has an ample User Guide for the Parallel Transporter, with an explicit Post-Job Considerations section, warning:
Even if the job completed successfully, action may still be required based on error and warning information in the job logs and error tables.
So technically, a job might complete, but results may vary from time to time.
I guess you have to define your own policies and scan the logfiles for patterns of warnings and error messages, and then generate your own exit codes for semantic failures. Tools like logstash or splunk might come in handy.
BTW, you might consider using logrotate for rotating the $TPTLOGDIR/${logname}.dat files.
Turns out that the issue was in the utility itself, as suspected. Shell script worked fine.