I would like to save all the variables that are in the directory in a separate file, cut out duplicates
To begin with, I wrote all the lines with global variables in a separate file
grep -rI "\$.*" folder/ >> output.txt
Then I tried to pull out the variables of this file
cat output.txt | sed /\$.*.[{A-Z}]/p
And output was not what I expected
So how can I take needed variables, when file after grep like this:
something.text_text.txt: - export IMAGE_NAME=${MY_REGISTY}/$MY_PR/${MY_PRNNN} something.text_text.txt:
- docker build --network host -t ${IMAGE_NAME}:${VERSION} -f $DILE_PATH --build-arg setupfile=$SET_FIL> something.text_text.txt:
- docker push ${IMAGE_NAME}:${VERSION} something.text_text.txt: - docker tag ${IMAGE_NAME}:${VERSION} ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text_text.txt: - docker push ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text.txt:
- /^rel_.*$/ something.text.txt: - eval $(ssh-agent -s) something.text.txt: - chmod 400 $MY_SSH_KEY something.text.txt:
- ssh-add $MY_KEY something.text.txt: - git checkout ${MY_BUIL_NAME} something.text.txt: - git reset --hard origin/${MY_F_NAME} something.text.txt: - mvn -s MY_settings.xml ${MTS} license:add-third-party something.text.txt: - cat ${LICENSE_LIST_FILE} something.text.txt: POM_XML_COMMIT_HASH_LOCAL=$(git log --oneline --follow -- pom.xml | awk '{ print $1 }' | head -n 1) || true something.text.txt: echo POMIT_HASH_LOCAL=${PCOMMIT_HASH_LOCAL} something.text.txt: POM_XML_COMMIT_HASH_REMOTE=$(git log --oneline origin/${MY_BUILD_REF_NAME} --follow -- pom.xml | awk '{ print $1 }' | h> something.text.txt: echo POM_XML_COMMIT_HASH_REMOTE=${POM_OMMIT_HASH_REMOTE} something.text.txt: if [[ ${POM_XML_COMMIT_HASH_LOCAL} = ${POMMIT_HASH_REMOTE} ]]; then something.text.txt: echo "File pom.xml is the same for local and origin ${MY_BUILD_REF_NAME} branch." something.text.txt: echo "New commits are presented in origin/${MY_BUILD_REF} branch for pom.xml file. Skipping." something.text.txt: - git add -f ${LICENSE_LIST_FILE} something.text.txt: - export MY_PUSH_URL=`echo $MY_REPOSITORY_URL | perl -pe 's#.*#(.+?(\:\d+)?)/#git#\1:#'` something.text.txt: - git remote set-url --push origin "${MY_PUSH_URL}" something.text.txt: - git push -f -o ci.skip origin ${MY_BUILD_REF_NAME} something.text_tests.txt: - docker login -u $MY_REGISTRY_USER -p $MY_REGISTRY_PASSWORD $MY_REGISTRY something.text_tests.txt: - export CONFIG_FILE=${HOME}/.docker/config.json something.text_tests.txt:
- export VERSION=$(cat current_version) something.text_tests.txt: - export MY_PROJECT_NAME_UPPER_CASE=$(echo ${MY_PROJECT_NAME} | tr a-z A-Z) something.text_tests.txt: - export ${MY_PROJECUPPER_CASE}_IMAGE=${MYISTRY}/${MY_PROJECT_PATH}/${MY_PROJECT_NAME}:${VERSION} something.text_tests.txt: - docker pull ${MY_REG}/${MY_PR}/${MY_PROJEC}:${VERS}
Try
grep -Po '\$\.*.[{_A-Z}]+' output.txt
-P makes grep using the Perl syntax
-o outputs only the matching parts
Now, improve your regex. For starters, I have already added _ to it, but it would still find ${X}{Y} (false positive) or not find ${lowercase} (false negative) and just partly find ${DIR#/} (because of extra syntax).
You can tell grep to output only matching parts using --only-matching or simply -o.
Real problem here is what really makes a valid variable identifier. This of course depends on for which shell was script written and how many different styles original author of the script used.
Let's assume something sane, identifiers starts with [a-Z] and can only contain alphanumeric characters and underscore. You can also reference same identifier using $MY_VARIABLE or ${MY_VARIABLE}.
I would go with something like that:
grep -rhIo '\$[a-zA-Z_\{\}]*' directory | sort --unique
But be aware that syntax for arrays and operations above variables will break this very quickly.
To get correct results for ${adjacent}text maybe go for
grep -hrEo '\$\{?[A-Za-z_0-9]+\}?' .
This will still not work correctly for
: <<\_
literal $text in a quoted here document
_
echo '$quoted literal text'
echo \$escaped \$dollar \$signs
etc
but for a quick and dirty attempts, maybe just ignore those corner cases, or add some sort of postprocessing to remove them.
Properly solving this requires you to have a sh parser to figure out which dollar signs are quoted etc; and with eval even that won't be complete.
As a comment on other answers here, grep -P is not portable, and requires GNU grep. If you don't have that, and really require Perl regex extensions, maybe simply go with Perl.
perl -lne 'print($&) while m/\$\{?[A-Za-z_0-9]+\}?/go' **/*
The **/* recursive wildcard is not portable either; if you require a POSIX-compatible script, maybe resort to
find . -type f -exec \
'print($&) while m/\$\{?[A-Za-z_0-9]+\}?/go' {} +
though of course Perl isn't at all POSIX either.
I have file names which I get from git diff command in my shell script
and it's output looks like this :
bewsdfsdf.txt xcvxcv.txt # separated by space
My script:
python3 "unitest.py"
COMMIT_ID=$(git rev-parse --verify HEAD)
echo ${COMMIT_ID}
Files=$(git diff-tree --no-commit-id --name-only -r ${COMMIT_ID})
touch names.txt
echo ${Files} > names.txt
cat names.txt
I want to get each file name separately. How can I do it?
Redirect the output of this command git diff-tree --no-commit-id --name-only -r ${COMMIT_ID} to xargs utility.
Using the -I argument, you can get each file name separately and execute a command for each file name (if replace-str is set to %)
For example : If you want to echo each file name,
git diff-tree --no-commit-id --name-only -r ${COMMIT_ID} | xargs -I % echo %
I have few hundreds of remote and local branches. I wonder whether there is a command to help me find a folder with a specific name in all branches.
My git version is 1.8.3.1. I also have smartgit installed if it matters.
Thanks in advance.
The following command will output all refs (local and remotes) that point to a commit which contains the path specified in the variable SEARCH_PATH
SEARCH_PATH="somePath"
git for-each-ref --format="%(refname)" refs/heads refs/remotes |
while read ref
do
if [[ `git ls-tree -r --name-only $ref` =~ "$SEARCH_PATH" ]] ; then
echo $ref;
fi
done
You can run following to list your required folders/files
for line in `git for-each-ref --format="%(refname)" refs/heads`;
do
git ls-tree -r $line | grep 'file_regex'
done
Let's say I want to rename a method in source code contained in a git repository. I could do this by hand, but the method name might be in multiple places (e.g., unit test, documentation, actual method). To check where the method is used, I use 'git grep'. I get 'git grep' to show only lines that I want to change, and then I don't have a workflow to automatically change those lines.
I'm looking for an automated way (hopefully using git tools) to do this last step. I was hoping there was some sort of 'git sed' or equivalent, but I can't find any.
The interface I'm thinking would be nice: git sed 's/old-method-name/new-method-name/g'
You could use git ls-files in combination with xargs and sed:
git ls-files -z | xargs -0 sed -i -e 's/old-method-name/new-method-name/g'
Thanks to both Noufal and Greg for their posts. I combined their solutions, and found one that uses git grep (more robust than git ls-files for my repo, as it seems to list only the files that have actual src code in them - not submodule folders for example), and also has the old method name and new method name in only one place:
In the [alias] block of my ~/.gitconfig file:
sed = ! git grep -z --full-name -l '.' | xargs -0 sed -i -e
To use:
git sed 's/old-method-name/new-method-name/ig'
You could do a
for i in $(git grep --full-name -l old_method_name)
do
perl -p -i -e 's/old_method_name/new_method_name/g' $i
done
stick that in a file somewhere and then alias it as git sed in your config.
Update: The comment by tchrist below is a much better solution since it prevents perl from spawning repeatedly.
Here's a solution that combines those of of Noufal and claytontstanley and avoids touching files that won't change.
In the [alias] block of my ~/.gitconfig file:
psed = !sh -c 'git grep --null --full-name --name-only --perl-regexp -e \"$1\" | xargs -0 perl -i -p -e \"s/$1/$2/g\"' -
To use:
git psed old_method_name new_method_name
Yes, there's. In Ubuntu the package git-extras provides the command. Install it:
$ sudo apt-get install git-extras
Use it like bellow e.g. to correct a spelling issue quickly:
$ git sed 'qoute' 'quote'
Unfortunately it doesn't support file filters like what git grep does:
$ git grep -e 'class' -- '*.py'
The same functionality also exists on Mac and other operating systems. Checkout its installation page.
Unhappy with most other solutions provided (which is basically just a string-replace on git tracked files) I wrote my own script: git-sed.
It supports any expression sed supports (e.g git sed '1{/^$/d}')
Can run on a subset of paths in the repo (git sed 's/foo/bar' src tests)
Multiple expressions (git sed -e 's/foo/bar' -e '/bar/d').
etc...
Just drop it anywhere in PATH to use it or add an alias pointing to the full path.
Note that starting git 2.1 (Q3 2014), you can set "full-name" by default for git grep.
(See commit 6453f7b by Andreas Schwab)
"git grep" learned grep.fullname configuration variable to force "--full-name" to be default.
This may cause regressions on scripted users that do not expect this new behaviour.
That means the previous solutions can benefit from:
git config grep.full-name true
And use:
psed = !sh -c 'git grep --null --name-only --perl-regexp -e \"$1\" | xargs -0 perl -i -p -e \"s/$1/$2/g\"' -
See git-search-replace on github - it's designed for this exactly.
I have written a git sed which supports file filtering:
#!/bin/bash
split=$(($# + 1))
for i in $(seq 1 $#); do
if [[ "${!i}" = "--" ]]; then
split=$i
fi
done
git ls-files -z "${#:$split:$#}" | xargs -0 sed -b -i "${#:1:$(($split - 1))}"
(You probably don't want the -b parameter on non-Windows platforms; it's necessary on Windows to preserve Windows-style newlines.)
You can then add an alias in your .gitconfig:
[alias]
sed = ! <path to git-sed>
so that you can use it like git sed -e <your expression> -- <path filter>.
I want to version control my web server as described in Version control for my web server, by creating a git repo out of my /var/www directory. My hope was that I would then be able to push web content from our dev server to github, pull it to our production server, and spend the rest of the day at the pool.
Apparently a kink in my plan is that Git won't respect file permissions (I haven't tried it, only reading about it now.) I guess this makes sense in that different boxes are liable to have different user/group setups. But if I wanted to force permissions to propagate, knowing my servers are configured the same, do I have any options? Or is there an easier way to approach what I'm trying to do?
Git is Version Control System, created for software development, so from the whole set of modes and permissions it stores only executable bit (for ordinary files) and symlink bit. If you want to store full permissions, you need third party tool, like git-cache-meta (mentioned by VonC), or Metastore (used by etckeeper). Or you can use IsiSetup, which IIRC uses git as backend.
See Interfaces, frontends, and tools page on Git Wiki.
The git-cache-meta mentioned in SO question "git - how to recover the file permissions git thinks the file should be?" (and the git FAQ) is the more staightforward approach.
The idea is to store in a .git_cache_meta file the permissions of the files and directories.
It is a separate file not versioned directly in the Git repo.
That is why the usage for it is:
$ git bundle create mybundle.bdl master; git-cache-meta --store
$ scp mybundle.bdl .git_cache_meta machine2:
#then on machine2:
$ git init; git pull mybundle.bdl master; git-cache-meta --apply
So you:
bundle your repo and save the associated file permissions.
copy those two files on the remote server
restore the repo there, and apply the permission
This is quite late but might help some others. I do what you want to do by adding two git hooks to my repository.
.git/hooks/pre-commit:
#!/bin/bash
#
# A hook script called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if it wants
# to stop the commit.
SELF_DIR=`git rev-parse --show-toplevel`
DATABASE=$SELF_DIR/.permissions
# Clear the permissions database file
> $DATABASE
echo -n "Backing-up permissions..."
IFS_OLD=$IFS; IFS=$'\n'
for FILE in `git ls-files --full-name`
do
# Save the permissions of all the files in the index
echo $FILE";"`stat -c "%a;%U;%G" $FILE` >> $DATABASE
done
for DIRECTORY in `git ls-files --full-name | xargs -n 1 dirname | uniq`
do
# Save the permissions of all the directories in the index
echo $DIRECTORY";"`stat -c "%a;%U;%G" $DIRECTORY` >> $DATABASE
done
IFS=$IFS_OLD
# Add the permissions database file to the index
git add $DATABASE -f
echo "OK"
.git/hooks/post-checkout:
#!/bin/bash
SELF_DIR=`git rev-parse --show-toplevel`
DATABASE=$SELF_DIR/.permissions
echo -n "Restoring permissions..."
IFS_OLD=$IFS; IFS=$'\n'
while read -r LINE || [[ -n "$LINE" ]];
do
ITEM=`echo $LINE | cut -d ";" -f 1`
PERMISSIONS=`echo $LINE | cut -d ";" -f 2`
USER=`echo $LINE | cut -d ";" -f 3`
GROUP=`echo $LINE | cut -d ";" -f 4`
# Set the file/directory permissions
chmod $PERMISSIONS $ITEM
# Set the file/directory owner and groups
chown $USER:$GROUP $ITEM
done < $DATABASE
IFS=$IFS_OLD
echo "OK"
exit 0
The first hook is called when you "commit" and will read the ownership and permissions for all the files in the repository and store them in a file in the root of the repository called .permissions and then add the .permissions file to the commit.
The second hook is called when you "checkout" and will go through the list of files in the .permissions file and restore the ownership and permissions of those files.
You might need to do the commit and checkout using sudo.
Make sure the pre-commit and post-checkout scripts have execution permission.
We can improve on the other answers by changing the format of the .permissions file to be executable chmod statements, and to make use of the -printf parameter to find. Here is the simpler .git/hooks/pre-commit file:
#!/usr/bin/env bash
echo -n "Backing-up file permissions... "
cd "$(git rev-parse --show-toplevel)"
find . -printf 'chmod %m "%p"\n' > .permissions
git add .permissions
echo done.
...and here is the simplified .git/hooks/post-checkout file:
#!/usr/bin/env bash
echo -n "Restoring file permissions... "
cd "$(git rev-parse --show-toplevel)"
. .permissions
echo "done."
Remember that other tools might have already configured these scripts, so you may need to merge them together. For example, here's a post-checkout script that also includes the git-lfs commands:
#!/usr/bin/env bash
echo -n "Restoring file permissions... "
cd "$(git rev-parse --show-toplevel)"
. .permissions
echo "done."
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on you
r path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n"; exit 2; }
git lfs post-checkout "$#"
In case you are coming into this right now, I've just been through it today and can summarize where this stands. If you did not try this yet, some details here might help.
I think #Omid Ariyan's approach is the best way. Add the pre-commit and post-checkout scripts. DON'T forget to name them exactly the way Omid does and DON'T forget to make them executable. If you forget either of those, they have no effect and you run "git commit" over and over wondering why nothing happens :) Also, if you cut and paste out of the web browser, be careful that the quotation marks and ticks are not altered.
If you run the pre-commit script once (by running a git commit), then the file .permissions will be created. You can add it to the repository and I think it is unnecessary to add it over and over at the end of the pre-commit script. But it does not hurt, I think (hope).
There are a few little issues about the directory name and the existence of spaces in the file names in Omid's scripts. The spaces were a problem here and I had some trouble with the IFS fix. For the record, this pre-commit script did work correctly for me:
#!/bin/bash
SELF_DIR=`git rev-parse --show-toplevel`
DATABASE=$SELF_DIR/.permissions
# Clear the permissions database file
> $DATABASE
echo -n "Backing-up file permissions..."
IFSold=$IFS
IFS=$'\n'
for FILE in `git ls-files`
do
# Save the permissions of all the files in the index
echo $FILE";"`stat -c "%a;%U;%G" $FILE` >> $DATABASE
done
IFS=${IFSold}
# Add the permissions database file to the index
git add $DATABASE
echo "OK"
Now, what do we get out of this?
The .permissions file is in the top level of the git repo. It has one line per file, here is the top of my example:
$ cat .permissions
.gitignore;660;pauljohn;pauljohn
05.WhatToReport/05.WhatToReport.doc;664;pauljohn;pauljohn
05.WhatToReport/05.WhatToReport.pdf;664;pauljohn;pauljohn
As you can see, we have
filepath;perms;owner;group
In the comments about this approach, one of the posters complains that it only works with same username, and that is technically true, but it is very easy to fix it. Note the post-checkout script has 2 action pieces,
# Set the file permissions
chmod $PERMISSIONS $FILE
# Set the file owner and groups
chown $USER:$GROUP $FILE
So I am only keeping the first one, that's all I need. My user name on the Web server is indeed different, but more importantly you can't run chown unless you are root. Can run "chgrp", however. It is plain enough how to put that to use.
In the first answer in this post, the one that is most widely accepted, the suggestion is so use git-cache-meta, a script that is doing the same work that the pre/post hook scripts here are doing (parsing output from git ls-files). These scripts are easier for me to understand, the git-cache-meta code is rather more elaborate. It is possible to keep git-cache-meta in the path and write pre-commit and post-checkout scripts that would use it.
Spaces in file names are a problem with both of Omid's scripts. In the post-checkout script, you'll know you have the spaces in file names if you see errors like this
$ git checkout -- upload.sh
Restoring file permissions...chmod: cannot access '04.StartingValuesInLISREL/Open': No such file or directory
chmod: cannot access 'Notebook.onetoc2': No such file or directory
chown: cannot access '04.StartingValuesInLISREL/Open': No such file or directory
chown: cannot access 'Notebook.onetoc2': No such file or directory
I'm checking on solutions for that. Here's something that seems to work, but I've only tested in one case
#!/bin/bash
SELF_DIR=`git rev-parse --show-toplevel`
DATABASE=$SELF_DIR/.permissions
echo -n "Restoring file permissions..."
IFSold=${IFS}
IFS=$
while read -r LINE || [[ -n "$LINE" ]];
do
FILE=`echo $LINE | cut -d ";" -f 1`
PERMISSIONS=`echo $LINE | cut -d ";" -f 2`
USER=`echo $LINE | cut -d ";" -f 3`
GROUP=`echo $LINE | cut -d ";" -f 4`
# Set the file permissions
chmod $PERMISSIONS $FILE
# Set the file owner and groups
chown $USER:$GROUP $FILE
done < $DATABASE
IFS=${IFSold}
echo "OK"
exit 0
Since the permissions information is one line at a time, I set IFS to $, so only line breaks are seen as new things.
I read that it is VERY IMPORTANT to set the IFS environment variable back the way it was! You can see why a shell session might go badly if you leave $ as the only separator.
In pre-commit/post-checkout an option would be to use "mtree" (FreeBSD), or "fmtree" (Ubuntu) utility which "compares a file hierarchy against a specification, creates a specification for a file hierarchy, or modifies a specification."
The default set are flags, gid, link, mode, nlink, size, time, type, and uid. This can be fitted to the specific purpose with -k switch.
I am running on FreeBSD 11.1, the freebsd jail virtualization concept makes the operating system optimal. The current version of Git I am using is 2.15.1, I also prefer to run everything on shell scripts. With that in mind I modified the suggestions above as followed:
git push: .git/hooks/pre-commit
#! /bin/sh -
#
# A hook script called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if it wants
# to stop the commit.
SELF_DIR=$(git rev-parse --show-toplevel);
DATABASE=$SELF_DIR/.permissions;
# Clear the permissions database file
> $DATABASE;
printf "Backing-up file permissions...\n";
OLDIFS=$IFS;
IFS=$'\n';
for FILE in $(git ls-files);
do
# Save the permissions of all the files in the index
printf "%s;%s\n" $FILE $(stat -f "%Lp;%u;%g" $FILE) >> $DATABASE;
done
IFS=$OLDIFS;
# Add the permissions database file to the index
git add $DATABASE;
printf "OK\n";
git pull: .git/hooks/post-merge
#! /bin/sh -
SELF_DIR=$(git rev-parse --show-toplevel);
DATABASE=$SELF_DIR/.permissions;
printf "Restoring file permissions...\n";
OLDIFS=$IFS;
IFS=$'\n';
while read -r LINE || [ -n "$LINE" ];
do
FILE=$(printf "%s" $LINE | cut -d ";" -f 1);
PERMISSIONS=$(printf "%s" $LINE | cut -d ";" -f 2);
USER=$(printf "%s" $LINE | cut -d ";" -f 3);
GROUP=$(printf "%s" $LINE | cut -d ";" -f 4);
# Set the file permissions
chmod $PERMISSIONS $FILE;
# Set the file owner and groups
chown $USER:$GROUP $FILE;
done < $DATABASE
IFS=$OLDIFS
pritnf "OK\n";
exit 0;
If for some reason you need to recreate the script the .permissions file output should have the following format:
.gitignore;644;0;0
For a .gitignore file with 644 permissions given to root:wheel
Notice I had to make a few changes to the stat options.
Enjoy,
One addition to #Omid Ariyan's answer is permissions on directories. Add this after the for loop's done in his pre-commit script.
for DIR in $(find ./ -mindepth 1 -type d -not -path "./.git" -not -path "./.git/*" | sed 's#^\./##')
do
# Save the permissions of all the files in the index
echo $DIR";"`stat -c "%a;%U;%G" $DIR` >> $DATABASE
done
This will save directory permissions as well.
Another option is git-store-meta. As the author described in this superuser answer:
git-store-meta is a perl script which integrates the nice features of git-cache-meta, metastore, setgitperms, and mtimestore.
Improved version of https://stackoverflow.com/users/9932792/tammer-saleh answer:
It only updates the permissions on changed files.
It handles symlinks
It ignores empty directories (git can not handle them)
.git/hooks/pre-commit:
#!/usr/bin/env bash
echo -n "Backing-up file permissions... "
cd "$(git rev-parse --show-toplevel)"
find . -type d ! -empty -printf 'X="%p"; chmod %m "$X"; chown %U:%G "$X"\n' > .permissions
find . -type f -printf 'X="%p"; chmod %m "$X"; chown %U:%G "$X"\n' >> .permissions
find . -type l -printf 'chown -h %U:%G "%p"\n' >> .permissions
git add .permissions
echo done.
.git/hooks/post-merge:
#!/usr/bin/env bash
echo -n "Restoring file permissions... "
cd "$(git rev-parse --show-toplevel)"
git diff -U0 .permissions | grep '^\+' | grep -Ev '^\+\+\+' | cut -c 2- | /usr/bin/bash
echo "done."