I was trying to extract a summary of contributions from git log and create a concise summary of that and create an excel/csv out of it to present reports.
I did try
git log --after="2020-12-10" --pretty=format:'"%h","%an","%ae","%aD","%s","(trailers:key="Reviewed By")"'
and the CSV looks like with a blank CSV column at the end.
...
"7c87963cc","XYZ","xyz#abc.com","Tue Dec 8 17:40:13 2020 +0000","[TTI] Add support for target hook in compiler.", ""
...
and the git log looks something like
commit 7c87963cc
Author: XYZ <xyz#abc.com>
Date: Tue Dec 8 17:40:13 2020 +0000
[TTI] Add support for target hook in compiler.
This adds some code in the TabeleGen ...
This is my body of commit.
Reviewed By: Sushant
Differential Revision: https://codereviews.com/DD8822
What I couldn't be successful was in extracting the Differential Revision string using the (trailers:key="Reviewed By") command.
I couldn't find much on how to get this working.
I checked the git manual and I did try what it explains.
Is there something I might be missing in this command?
The expected output should have the text
https://codereviews.com/DD8822 at the last position in the above CVS output.
I'm not sure but:
trailer keys cannot have whitespaces (therefore Reviewed By -> Reviewed-By, and Differential Revision -> Differential-Revision);
trailers should not be delimited by new lines, but separated from the commit commit message (therefore Reviewed By from your question is not considered as a trailer).
I would also not recommend using CSV, but using TSV instead: git output is not aware of CSV syntax (semi-colons and commas escaping), therefore the output document may be generated unparsable.
If your commit messages would look like this (- instead of spaces, no new line delimiters):
commit 7c87963cc
Author: XYZ <xyz#abc.com>
Date: Tue Dec 8 17:40:13 2020 +0000
[TTI] Add support for target hook in compiler.
This adds some code in the TabeleGen ...
This is my body of commit.
Reviewed-By: Sushant
Differential-Revision: https://codereviews.com/DD8822
Then the following command would work for you:
git log --pretty=format:'%h%x09%an%x09%ae%x09%aD%x09%s%x09%(trailers:key=Reviewed-By,separator=%x20,valueonly)%x09%(trailers:key=Differential-Revision,separator=%x20,valueonly)'
producing short commit id, author name, author email, date, commit message, trailer Reviewed-By, and trailer Differential-Revision to your tab-separated values output.
If you may not change the old commit messages because your history is not safe for doing this (it's published, pulled by peers, your tools are bound to the published commit hashes), then you have to process the git log output with sed, awk, perl, or any other text-transforming tool to generate your report. Say, process something like git log --pretty=format:'%x02%h%x1F%an%x1F%ae%x1F%aD%x1F%s%x1F%n%B' where lines between ^B (STX) and EOF should be analyzed somehow (filtered for the trailers you are interestged in), then joined to their group lines starting with ^B, and then character replaced to replace field and entry separators with \t and no character respectively.
But again, if you may edit the history by fixing commit message trailers (not sure how much it may affect), I'd recommend you do that and then reject the idea of extra scripts processing trailers that are not recognized by git-interpret-trailers and simply fix the commit messages.
Edit 1 (text tools)
If rewriting the history is not an option, then implementing some scripts may help you out. I'm pretty weak at writing powerful sed/awk/perl scripts, but let me try.
git log --pretty=format:'%x02%h%x1F%an%x1F%ae%x1F%aD%x1F%s%x1F%n%B' \
| gawk -f trailers.awk \
| sed '$!N;s/\n/\x1F/' \
| sed 's/[\x02\x1E]//g' \
| sed 's/\x1F/\x09/g'
How it works:
git generates a log made of data delimited with standard C0 C1 codes assuming there are no such characters your commit messages (STX, RS and US -- I don't really know if it a good place to use them like that and if I apply them semantically correct);
gawk filters the log output trying to parse STX-started groups and extract the trailers, generating "two-rowed" output (each odd line for regular data, each even line for comma-joined trailer values even for missing trailers);
sed joins odd and even lines by pairs (credits go to Karoly Horvath);
sed removes STX and RS;
sed replaces US to TAB.
Here is the trailers.awk (again I'm not an awk guy and have no idea how idiomatic the following script it, but it seems to work):
#!/usr/bin/awk -f
BEGIN {
FIRST = 1
delete TRAILERS
}
function print_joined_array(array) {
if ( !length(array) ) {
return
}
for ( i in array ) {
if ( i > 0 ) {
printf(",")
}
printf("%s", array[i])
}
printf("\x1F")
}
function print_trailers() {
if ( FIRST ) {
FIRST = 0
return
}
print_joined_array(TRAILERS["Reviewed By"])
print_joined_array(TRAILERS["Differential Revision"])
print ""
}
/^\x02/ {
print_trailers()
print $0
delete TRAILERS
}
match($0, /^([-_ A-Za-z0-9]+):\s+(.*)\s*/, M) {
TRAILERS[M[1]][length(TRAILERS[M[1]])] = M[2]
}
END {
print_trailers()
}
A couple of words how the awk script works:
it assumes that records that do not require processing are starting with STX;
it tries to grep each non-"STX" line for a Key Name: Value pattern and saves the found result to a temporary array TRAILERS (that serves actually as a multimap, like Map<String, List<String>> in Java) for each record;
each record is written as is, but trailers are written either before detecting a new record or at EOF.
Edit 2 (better awk)
Well, I'm really weak at awk, so once I read more about awk internal variables, I figured out the awk script can be reimplemented entirely and produce a ready to use TSV-like output itself without any post-processing with sed or perl. So the shorter and improved version of the script is:
#!/bin/bash
git log --pretty=format:'%x1E%h%x1F%an%x1F%ae%x1F%aD%x1F%s%x1F%B%x1E' \
| gawk -f trailers.awk
#!/usr/bin/awk -f
BEGIN {
RS = "\x1E"
FS = "\x1F"
OFS = "\x09"
}
function extract(array, trailer_key, __buffer) {
for ( i in array ) {
if ( index(array[i], trailer_key) > 0 ) {
if ( length(__buffer) > 0 ) {
__buffer = __buffer ","
}
__buffer = __buffer substr(array[i], length(trailer_key))
}
}
return __buffer
}
NF > 1 {
split($6, array, "\n")
print $1, $2, $3, $4, $5, extract(array, "Reviewed By: "), extract(array, "Differential Revision: ")
}
Much more concise, easier to read, understand and maintain.
In a GitHub repository I have a hundred or so autogenerated files with a comment header that specifies things such as date-generated, version, etc.
The trouble is that more often than not the header changes but the actual useful contents of the files don't change.
When this happens GitHub reports a hundred or so changed files while in reality maybe only a couple have changed. This makes it very hard to spot the real changes.
Unfortunately these headers must be present at release time for reasons that are not relevant to the issue at hand. Otherwise I would just strip them out and be done with it.
To work around this issue my plan is to:
Strip the comment headers from the files and save them into a separate file before committing them to the repository.
Reincorporate the headers into the files at release time as a step in a Jenkins pipeline.
For the first part of the plan I am stripping the first 5 lines of the files and saving them into a single versions.txt file.
The original files (*.h) I am working on all have a format like
/* Some comment */
/* More useless text */
/* Version of the file */
/* Maybe a date */
/* More comment */
Actual useful file content
I use the following command
head -5 *.h > versions.txt && sed -i '1,+4d' *.h
to remove the first 5 lines from each file and create a versions.txt file that ends up looking like this:
==> File1.h <==
/* Some comment */
/* More useless text */
/* Version of the file */
/* Maybe a date */
/* More comment */
==> File2.h <==
/* Some comment */
/* More useless text */
/* Version of the file */
/* Maybe a date */
/* More comment */
==> File3.h <==
/* Some comment */
/* More useless text */
/* Version of the file */
/* Maybe a date */
/* More comment */
I would now like to find the way to revert the process and put each set of lines back into it's respective original file.
A better alternative would be to find a way for GitHub to ignore the first 5 lines of the files, but as far as I've been able to tell this is not possible.
Try this solution:
awk '/^==>/ { cnt=1;filname=$2 } { print "sed -i \""cnt"i "$0"\" "filname;cnt++ }' version.txt
This search for lines beginning with "==>" and sets the second space delimited field to a variable filname. This proceeding lines up until the next "==>" are then used to create a sed command to insert the lines at the top of the file, using cnt to terack where in the file to insert the lines.
Once you are happy that the above command prints sed commands as expected (VERY IMPORTANT!) you can then action the commands with awk's system function and so:
awk '/^==>/ { cnt=1;filname=$2 } { system("sed -i \""cnt"i "$0"\" "filname);cnt++ }' version.txt
Raman Sailopal's answer hit the mark except for a couple of details:
The command would also insert the ==> Filename <== line generated by the stripping command head -5 *.h > versions.txt && sed -i '1,+4d' *.h
The command would not insert empty lines.
The command worked on the extra empty line that is inserted by the original stripping command as a separator between files.
I therefore expanded the provided command with if statements so that it
only operates when cnt > 1 && cnt < 7 making it work on exactly five lines while disregarding the first ==> Filename <== line.
Inserts empty lines with sed -i 1G when needed.
Here is the full command I ended up with:
awk '/^==>/ { cnt=1;filename=$2 }
{ if(cnt > 1 && cnt < 7 && $0 == "") system("sed -i \""cnt-1"G\" " filename)}
{ if(cnt > 1 && cnt < 7 && $0 != "") system("sed -i \""cnt-1"i "$0"\" "filename);cnt++}' versions.txt
I have a huge log file and I wanted to copy a log starting from text to my local directory.
log file name
'cktrm.ecg-2015-12-21.gz'
command I ran was
bash-3.2$ gzgrep '2665941' cktrm.ecg-2015-12-21.gz > ~/log.txt
but this only copy all the lines that contains 2665941 to the text file.
What I need is to copy starting from that text to the end of the file.
For example:
...
log.info [id = 2665941]
log.debug ....
log.debug ...
log.debug [add to id 2665941]
...
what currently printed to text file is
log.info [id = 2665941]
log.debug [add to id 2665941]
what I need is
log.info [id = 2665941]
log.debug ....
log.debug ...
log.debug [add to id 2665941]
...
You can use a short awk program that prints its input starting from where it finds a pattern:
zcat cktrm.ecg-2015-12-21.gz | awk "/2665941/ {printLog = 1}
printLog { print }" > ~/log.txt
Note: this is a single command, but it consists of two lines. If you press enter while in the middle of a quoted string bash will give you a secondary prompt and you'll be able to continue to the next line of the command, where of course you'll need to close the quote.
The zcat uncompresses your file and its output is the input of the awk command.
The awk script means:
When the current line matches the given pattern (in this case 2665941), set the printLog parameter to 1.
When the printLog parameter is set to something nonzero, print the current line.
This means that from the moment it finds the first occurrence of the pattern, it will print all lines.
Finally, the output of awk is redirected to your requested file.
If you know the specific format of the start and end markers you can use awk to achieve what you need:
cat > test-file
hear no evil
no evil here
start see no evil
know no evil
no evil known here
end see no evil
no know evil
^D
To only display lines between 'start see no evil' and 'end see no evil'
cat test-file | awk -e '/start/,/end/ {print}'
results in the following output
start see no evil
know no evil
no evil known here
end see no evil
Is there a way we can find out the last changelist number for a file in Perforce?
You're looking for p4 fstat:
$ p4 fstat //depot/path/to/file
... depotFile //depot/path/to/file
... headAction edit
... headType ktext
... headTime 1348571116
... headRev 8
... headChange **496308**
... headModTime 1348571025
...
$
If you mapped the file to a client-workspace, p4 filelog -m 1 works similar.
Is there any way I can get the following output from perforce for files under a certain directory? I'd like: file name, last modified by username, last modified date, created by username, creation date.
So far I've only been able to get a list of changelists and their descriptions, but then I can't pipe that into another P4 command to do anything else with it. I'm in Windows by the way.
You can either go with p4 fstat and a subsequent p4 describe or you could parse the output of p4 filelog:
$ p4 filelog Rakefile
//depot/path/to/Rakefile
... #9 change 421932 edit on 2011/10/27 by user#client (text) '....'
... #8 change 421210 edit on 2011/10/24 by user#client (text) '....'
... #7 change 419771 edit on 2011/10/17 by user#client (text) '....'
... #6 change 393076 edit on 2011/05/26 by user#client (text) '....'
... #5 change 374172 edit on 2011/02/11 by user#client (text) '....'
... #4 change 374083 edit on 2011/02/11 by user#client (text) '....'
... #3 change 374042 edit on 2011/02/11 by user#client (text) '....'
... #2 change 373901 edit on 2011/02/10 by user#client (text) 'need ci/reporter rake-task'
... #1 change 359972 add on 2010/11/23 by user#client (text) 'added first version'
user#client will be the username and the name of the client workspace. Revision #1 is obviously the creation date, and the revision on top of the list (#9 here) is the last modified. So basically you have to parse the first line that begins with '...' for the last modified stuff and the last line that begins with '...' for the creation information.