Git grep across multiple repositories in a directory - linux

I have a list of bitbucket repositories on a server:
[user#lonapdbitbucket1 repositories]$ ls
1039 1044 1059 2165 2656 3958 3958 9284 9274 8274 7264 7263 8274
If I cd into one of these repositories and run git grep, to search for Ansible encryption strings, then it works fine - git grep manages to find an Ansible encryption string:
[user#lonapdbitbucket1 repositories]$ cd 1044
[user#lonapdbitbucket1 repositories]$ git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
To do this across multiple repos, I thought to convert it into a bash script:
# secret_scan.sh
repos_root=/var/lib/docker/volumes/bitbucket/_data/shared/data/repositories
git_grep_cmd=git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
for dir in ./*
do
# below line is just to clean up the directory string
repo_dir="$(d{dir#./}"
cd "${repos_root}${repo_dir}"; \
eval "git_grep_cmd"
done
Unfortunately, this does not work:
[user#lonapdbitbucket1 repositories]$ ./secret_scan.sh
fatal: not a git repository (or any parent up to mount point /var/lib)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
[user#lonapdbitbucket1 repositories]$ _
Would anyone be able to suggest a solution here, to essentially cd into multiple repositories and then run git grep on each, replicating results as if i were doing it on the command line?

I'm not sure what you are trying to achieve with eval and repo_dir. This should be as simple as:
repos_root=/var/lib/docker/volumes/bitbucket/_data/shared/data/repositories
for dir in *
do
cd "$repos_root$dir";
git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
done
Since your repos_root is absolute, you don't need to take care of returning to the original directory.
But I'm sceptical that git rev-list --all can be substituted, its output will be huge. Are you trying to find the string in ALL commits? To search the full history for a string, check Search all of Git history for a string and How to grep Git commit diffs or contents for a certain word

Related

How to make bash scripts to automatically add and download files from github?

I have a cloned directory on a linux server. Keyed access is configured. After some processing on the server, at the output, I get csv files that I need to add to the git. I also need to receive files on the server that have been changed on the git itself (these can be either new added files, or some changes in the code). The adding process happens every day, so I want to automate it. On one of the sites I found an example suitable for me. Two scripts that add and clone changes to the server. Startup automation takes place through crontab.
Code file:
#!/bin/bash
# Go to the GIT category
cd '/home/user/www/'
# Submitting changes to the main branch
git checkout main
git add -A
git commit -m "update main"
git push
and...
#!/bin/bash
# Go to the GIT category
cd '/home/user/www/'
# Loading data from the main branch
git checkout main
git pull
But I get the following errors in the output, what could be the problem?
Cronlog:
/home/user/RFFI-V/Work/start_git_download.sh: line 3: cd: too many arguments
/home/user/RFFI-V/Work/start_git_push.sh: line 3: cd: too many arguments
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Run each command manually, to see if you receive any issues.
If you dont receive any issues, then is the bash script in the correct folder for crontab and the script by default would run as root, if you want to run the script as a different user, an example syntax to setup the script to run as a different user in crontab is shown below,
1 2 3 4 5 USERNAME /path/to/script.sh

Files were not listed with git status and got lost after a commit

I lost files from my local repo. by "mistake" after using git. Am trying to understand that mistake and undo it if possible. what I did sequentially is:
git status
git add -A
git commit -m "some message"
git push origin master
The weird part is that I had some new files (since the last commit) that did not get committed and pushed, and now do not exist on my local repo.
Looking more into it, I saw that my git status did not list these files in the first place. These are the files that I lost. Why would that happen ? and is there a way to get them back ?
I have git version 1.8.3.1
If you didn't add the files to git at any point, it's unlikely you can recover them using git. You can see if they are in your repository by this handy "one-liner"
alias gitobjects="git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %
│ (objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | c
│ ut -c 1-12,41- | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest"
gitobjects | grep "myfile"

Prevent git from failing on filesystems without chmod permissions in Linux

I am trying to initialize a git repro on a samba mount with very limited permissions.
Trying to init I will receive:
$ git init .
error: chmod on /mnt/server/subfolder/.git/config.lock failed: Operation not permitted
fatal: could not set 'core.filemode' to 'false'
Which is surprising as filemode is already globally set to false
$ git config --get core.filemode
false
The Problem in general is that /mnt/server is a samba mount to a folder to which I have very limited access.
Also I am not able to change any permission for the /mnt/server mount as I am working on shared server with on which several users need the access to the /mnt/server mount.
So changing mounting permission like suggested here is not an option.
Also creating a symlink like suggested here does not work, as symlinks are not enabled on the samba drive.
So the question is how to prevent git from failing a chmod error or prevent it from doing chmod at all?
Is this possible?
Or how do I init a git in the environment?
A bit hacky solution is:
Init the an empty repro at destiantion with sufficient permission i.e. mktemp -d.
$ tempdir = $(mktemp -d)
$ git init $tempdir
Initialized empty Git repository in /tmp/tmp.pREa198fnx/.git/
Move the created .git folder to target destination.
$ mv $tempdir/.git /srv/server/sub/
mv: preserving times for './.git/branches': Operation not permitted
mv: preserving permissions for ‘./.git/branches’: Operation not permitted
mv: preserving times for './.git/hooks/applypatch-msg.sample': Operation not permitted
mv: preserving permissions for ‘./.git/hooks/applypatch-msg.sample’: Operation not permitted
mv: preserving times for './.git/hooks/commit-msg.sample': Operation not permitted
...
There will some error during moving but it won't stop mv from moving the files.
In the end the git works as expected:
$ echo "Foo" > bar.txt
$ git add bar.txt
$ git commit -m "Added Foobar"
[master (root-commit) e232039] bar.txt
1 file changed, 1 insertion(+)
create mode 100755 bar.txt
$ git status
On branch master
nothing to commit, working tree clean
branch and checkout seems to work to, didn't test push/pull.
Would still appreciate a cleaner solution.

finding the name of git branch from remote server

i am trying to find the name of git branch on remote server using a shell script. I put the following command in a script under the bin directory.
git symbolic-ref --short HEAD
When I execute the script using ssh from another machine
ssh -i keyfile.pem user#ipaddress 'bash -s' /path/to/the/script
I get an error
fatal: Not a git repository (or any of the parent directories): .git
Not sure where I am doing wrong. Any help would be appreciated.
Thanks.
Your call to git is using the wrong working directory (likely your home directory).
In your script, either cd to the path containing the git directory or specify the -C option with git:
git -C /path/to/git/checkout symbolic-ref --short HEAD
The -C option allows you to overwrite the working directory:
-C <path>
Run as if git was started in <path> instead of the current working directory. When
multiple -C options are given, each subsequent non-absolute -C <path> is interpreted
relative to the preceding -C <path>.

Script for root to git pull as another user

I have a script that I would like to have do a git pull inside another user's git directory. This script is run by the root user. For example:
cd /home/username/GitProject
sudo -u username -i git pull
When I run this, I get:
fatal: Not a git repository (or any of the parent directories): .git
Is there a way to have my script do a git pull as username?
Try without the -i option to sudo. That option is documented as first changing to the target user's home directory, which undoes the directory change you so carefully do before that. Alternatively, use the appropriate options to git to specify the directory, something like this:
sudo -u username -i git --git-dir=/home/username/GitProject/.git --work-tree=/home/username/GitProject pull
This can be done without sudo. This assumes you have password-less ssh keys since you are talking about a script. Here's the failure:
# git clone <user>#<host>:/path/to/repo
Cloning into 'repo'...
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
This shows that ~ properly expands to the user's homedir:
# MYUSER=somebody
# su - $MYUSER -c "echo ~"
/home/somebody
And here's the actual command used to clone into the home directory along with some extra proofs:
# su - $MYUSER -c "git clone <user>#<host>:/path/to/repo"
Cloning into 'repo'...
remote: Counting objects: 13, done.
<..>
# ls -l /home/$MYUSER/repo/.git/config
-rw-r--r-- 1 somebody somebody 275 Nov 8 23:55 /home/somebody/repo/.git/config
# su - $MYUSER -c "cd ~/repo; git remote -v"
origin <user>#<host>:/path/to/repo (fetch)
origin <user>#<host>:/path/to/repo (push)
# su - $MYUSER -c "cd ~/repo; git pull"
Already up-to-date.

Resources