I have created a large amount of files using this Python script that I used primarily to benchmark Git.
The result is very surprising especially the differences between Windows and Linux.
Basically my script creates 12 directories with 512 files in each of them. Each file is about 2 to 4 kB. With the Git objects the repository is about 12k files.
I benchmarked:
Time to add all the files to git git add .
Time to checkout a previous branch
Time to copy the repository on the same SSD
Time to copy the repository to an external Sata HDD (NTFS)
I did this with the very same repository on both Windows 10 and Linux:
Operation Linux Windows Ratio
--------- ----- ------- -----
1. git add . 0.47s 21.7s x46
2. git checkout HEAD~1 0.35s 16.2s x46
3. git checkout . 0.40s 20.5s x50
4. cp -r ssd->ssd 0.35s 1m14s x211
5. cp -r ssd->hdd 4.90s 6m25s x78
The operation was done in this order:
$ mkdir test
$ cp test.py test && cd test
$ ./test.py # Creation of the files
$ git init
$ time git add . # (1)
$ git commit -qam 1
$ ./test.py # Alter some files
$ commit -qam 2
$ cd ..
$ time cp -r test /media/hdd/ # (4)
$ time cp -r test test2 # (5)
$ cd test
$ time git checkout HEAD~1 # (2)
$ ./test.py
$ git checkout master
$ git reset --soft head~1
$ time git checkout . # (3)
The benchmark was done on the same PC (using dual boot).
Why such differences? I can not believe it.
Related
I have a list of bitbucket repositories on a server:
[user#lonapdbitbucket1 repositories]$ ls
1039 1044 1059 2165 2656 3958 3958 9284 9274 8274 7264 7263 8274
If I cd into one of these repositories and run git grep, to search for Ansible encryption strings, then it works fine - git grep manages to find an Ansible encryption string:
[user#lonapdbitbucket1 repositories]$ cd 1044
[user#lonapdbitbucket1 repositories]$ git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
To do this across multiple repos, I thought to convert it into a bash script:
# secret_scan.sh
repos_root=/var/lib/docker/volumes/bitbucket/_data/shared/data/repositories
git_grep_cmd=git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
for dir in ./*
do
# below line is just to clean up the directory string
repo_dir="$(d{dir#./}"
cd "${repos_root}${repo_dir}"; \
eval "git_grep_cmd"
done
Unfortunately, this does not work:
[user#lonapdbitbucket1 repositories]$ ./secret_scan.sh
fatal: not a git repository (or any parent up to mount point /var/lib)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
fatal: this operation must be run in a work tree
[user#lonapdbitbucket1 repositories]$ _
Would anyone be able to suggest a solution here, to essentially cd into multiple repositories and then run git grep on each, replicating results as if i were doing it on the command line?
I'm not sure what you are trying to achieve with eval and repo_dir. This should be as simple as:
repos_root=/var/lib/docker/volumes/bitbucket/_data/shared/data/repositories
for dir in *
do
cd "$repos_root$dir";
git grep -P '\$ANSIBLE_VAULT;[0-9]\.[0-];AES256' $(git rev-list --all)
done
Since your repos_root is absolute, you don't need to take care of returning to the original directory.
But I'm sceptical that git rev-list --all can be substituted, its output will be huge. Are you trying to find the string in ALL commits? To search the full history for a string, check Search all of Git history for a string and How to grep Git commit diffs or contents for a certain word
I have a script on a remote machine which contains a for loop as below:
#!/bin/bash -eux
# execute build.sh in each component
for f in workspace/**/**/build.sh ; do
echo $f
REPO=${f%*build.sh}
echo $REPO
git -C ./$REPO checkout master
git -C ./$REPO pull origin master
$f
done
This script is finding all the repos with a build.sh file inside, pulls the latest changes and build them.
This works fine when I execute the scrript on the machine but when I try to trigger this script remotely the for loop just runs once, and I see that it returns a repo which actually doesn't have build.sh at all:
$ ssh devops "~/build.sh"
+ for f in workspace/**/**/build.sh
+ echo 'workspace/**/**/build.sh'
+ REPO='workspace/**/**/'
+ echo workspace/core/auth/
workspace/**/**/build.sh
workspace/core/auth/
+ git -C ./workspace/core/auth/ checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.
+ git -C ./workspace/core/auth/ pull origin master
From https://gitlabe.com/workspace/core/auth
* branch master -> FETCH_HEAD
Already up to date.
+ 'workspace/**/**/build.sh'
/home/devops/build.sh: line 10: workspace/**/**/build.sh: No such file or directory
I tried to make a one-liner of the for loop and use ssh and that also didn't work. How can I solve this problem?
You need to enable globbing on the remote machine. Add this to the beginning of your script:
shopt -s globstar
Also see this thread
Q) What is the proper way to build Wine in 64bit Debian? What is the modern tool chain and how must it be implemented to complete the task? And also.. how do I incorporate Wine Nine into the package?
People have been asking for years and even when the available tutorials are up to date they are incomplete, so only complete answers on this thread please!
A) I will attempt my own answer. Feel free to recycle bits of it as part of an improved or more modern answer! This was originally meant for 17.1 but with the 17.3 changes added after. It's all from sporadic notes/term output so there is definitely room for better answers here. In fact I think I lost a piece somewhere! Hope this helps fellow Debian lovers anyways! Best of luck! This will definitely ruin your weekend and end in abject failure, so at least there is that to look forward to!
Building Wine in Debian > 9
1) Setting up the Environment:
# Start by grabing some development tools and basic dependencies:
sudo apt-get install debhelper apt-utils binutils-multiarch-dev gcc-multilib pkg-config libtool-bin python-mako build-essential devscripts git git-buildpackage
sudo dpkg --add-architecture i386
sudo apt-get build-dep mesa
sudo apt-get build-dep wine
# And you could make git pretty like that one guy does, which debian probably did already:
git config --global color.ui auto
git config --global core.whitespace trailing-space,space-before-tab
# From: https://wiki.debian.org/PbuilderTricks https://wiki.ubuntu.com/PbuilderHowto
# http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.building.html
# Essentially we need to build some libraries then build against those libraries. Even with a true multiarch process in hand we'd be at the mercy of debian packaging, and DFSG policy is not for the faint of heart. At first gbp may seem like yet another tool but infact git-buildpackage is a wrapper that makes dealing with it all easier and solves that irritating little chroot problem for us. Mostly.
# First lets prep our 32bit chroot a bit:
ARCH=i386 git-pbuilder create
ARCH=i386 git-pbuilder login --save-after-login
# We need to build wine against mesa, so we start a private repo for when we've built it:
apt-get install nano apt-utils
nano /etc/apt/sources.list
# Example
deb http://mirrors.accretive-networks.net/debian/ sid main contrib
deb-src http://mirrors.accretive-networks.net/debian/ sid main contrib
# apt-ftparchive:
deb [trusted=yes] file:///home/user/pbuilder/deps ./
apt-get update
apt-get upgrade
apt-get dist-upgrade
# If it wasn't sid it is now. Oh yeah, dependencies:
sudo apt-get build-dep mesa
sudo apt-get build-dep wine
exit
# Now create that directory!
mkdir -p /home/user/pbuilder/deps
# You can do the same with amd64 but realy we only need it for wine64.. it simplifies cross building the wow64 stuff:
ARCH=amd64 git-pbuilder create
# Optional, just saves redownloading if your first build fails:
ARCH=amd64 git-pbuilder login --save-after-login
sudo apt-get build-dep wine
exit
# Install the private repo with a script, so you don't have to log in and do it by hand in the future (even more optional):
sudo mkdir -p /var/cache/pbuilder/hook.d
mkdir -p /home/doc/pbuilder/deps
# This doesn't work on drives using nodev, add 'dev' to defaults or remove for drives /etc/fstab entry
# Somebody should do something about that.
bash -c 'cat > /etc/pbuilderrc << EOF
# the file in /usr/share/pbuilder/pbuilderrc is the default template.
# /etc/pbuilderrc is the one meant for overwriting defaults read pbuilderrc.5
MIRRORSITE=http://mirrors.accretive-networks.net/debian/
OTHERMIRROR="deb [trusted=yes] file:///home/user/pbuilder/deps ./"
BINDMOUNTS="/home/user/pbuilder/deps"
# the hook dir may already be set/populated!
HOOKDIR="/var/cache/pbuilder/hook.d"
# this is necessary for running 'apt-ftp-archive' in the hook below
EXTRAPACKAGES="apt-utils"
EOF'
bash -c 'cat > /var/cache/pbuilder/hook.d/D01mesa << EOF
#!/bin/bash
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788580
# This fix didnt seem to work for test sys: mount -t devpts none /dev/pts
# Added dev to defaults on / and /opt in /etc/fstab, this is potentially a security issue but should be fine temporarily.
# Private Repo
(cd /home/user/pbuilder/deps; apt-ftparchive packages . > Packages)
# Add deps:
apt-get update
#apt-get install -f libegl1-mesa:i386=17.1.5-1 etc, etc,
#apt --fix-broken install
EOF'
# Create the empty list:
touch /home/user/pbuilder/deps/Packages
# updates chroots sources.list
sudo pbuilder --update --override-config --distribution sid
chmod u+x /home/user/pbuilder/D01deps
# The tutorial shows the actual editing of the appropriate debian/ directory. We supply those for the tutorial versions.
# If your version debian/ is older than source, you will need to remove the debian/patches it comes with.
# (i.e wget urltosome.deb; dpkg -x some.deb /some/where)
# We assume you will have them ready in /opt/src/debian-mesa and -wine/ so no git is provided here:
# Lets establish /opt/src as our working directory and populate our gits
mkdir -p /opt/src; cd /opt/src
# old: git clone http://copr-dist-git.fedorainfracloud.org/git/kkofler/qtwebengine/mesa.git mesa.nlp/
git clone git://anongit.freedesktop.org/mesa/mesa mesa.git/
git clone git://source.winehq.org/git/wine.git wine.git/
git clone --depth=1 https://github.com/wine-compholio/wine-staging.git wine-staging/
git clone --depth=1 https://github.com/sarnex/wine-d3d9-patches wine-d3d9/
# Go ahead and skip this bit if you plan to go straight to the Mesa 32bit build:
git clone --depth=1 git://anongit.freedesktop.org/mesa/drm libdrm/
git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
# So thats most the downloading finished anyways!
2) Proper Kernel Config
# This next bit mostly from: https://people.freedesktop.org/~mslusarz/nouveau-wiki-dump/InstallDRM.html
# This kernel worked on the test system, but was not with out issues. Kernel config needs it own tutorial the minimum for nouveau is shown here:
cd /opt/src/linux-2.6
apt-get install xorg-dev kernel-package
# For nouveau:
git remote add nouveau git://anongit.freedesktop.org/nouveau/linux-2.6
git remote update
git checkout -b nouveau-master nouveau/master
# Otherwise -b master
# Ok to just hit enter for all the new stuff...
cp -i /boot/config-`uname -r` ./.config
make oldconfig
# ... because we are looking closer now
make xconfig # needs libqt4-dev or use menuconfig and libncurses5
# Nouveau sanity checks:
#scripts/config --enable CONFIG_FB
scripts/config --enable CONFIG_I2C
scripts/config --enable CONFIG_I2C_ALGOBIT
scripts/config --enable CONFIG_FRAMEBUFFER_CONSOLE
scripts/config --enable CONFIG_BACKLIGHT_LCD_SUPPORT
scripts/config --enable CONFIG_BACKLIGHT_CLASS_DEVICE
scripts/config --enable CONFIG_FB_CFB_FILLRECT
scripts/config --enable CONFIG_FB_CFB_COPYAREA
scripts/config --enable CONFIG_FB_CFB_IMAGEBLIT
scripts/config --enable CONFIG_VT_HW_CONSOLE_BINDING
# Avoid the following framebuffers: offb uvesafb nvidiafb rivafb nvidia
cat .config | grep NVIDIA
# (Outdated) If you need firmware:
# Download file from https://people.freedesktop.org/~pq/nouveau-drm/
# Or follow steps at https://people.freedesktop.org/~mslusarz/nouveau-wiki-dump/NVC0_Firmware.html
# Place in /lib/usr/nouveau
git merge origin
# The fun bit..
search "<<<" keep origin:
#sudo nano drivers/gpu/drm/nouveau/nouveau_fence.c
# OR
git checkout origin drivers/gpu/drm/nouveau/nouveau_fence.c
git add .
git commit
git merge origin
make-kpkg clean
fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image
# To later update gits:
git remote update
git reset --hard nouveau/master
# Libdrm, for reference:
autoreconf -vfi
./configure --prefix=/usr/ --disable-intel --disable-vmwgfx --disable-radeon --enable-omap-experimental-api --enable-install-test-programs
sudo cp src/.libs/nouveau_drv.so /usr/lib/xorg/modules/drivers
3) Build Mesa:
cd /opt/src/mesa.git
git checkout -b master origin/master
cp ../debian-mesa/ debian/ -R
# You can skip these edits if your using debian/ from tutorial or use your and edit:
echo "10" > debian/compat
# Be sure to use tabs not spaces and stay to format in:
nano debian/rules
# You should replace all gallium-llvm with -llvm, but it's not entirely needed.
# find: confflags += \
# add: --enable-nine \
# replace: --disable-omx \
# with: --disable-omx-bellagio \
# replace: dh_install -a --fail-missing
# with: dh_install -a
echo "17.3.0" > VERSION
# You can follow the examples inside, or use dch -i instead:
nano debian/changelog
# echo "NOT_INSTALLED :=" > debian/not-installed # ?
# We need to install st nine somewhere, it may as well be here:
sudo bash -c 'cat > debian/libgl1-mesa-glx.install << EOF
usr/lib/*/d3d/d3dadapter9.so
usr/lib/*/d3d/d3dadapter9.so.*
usr/lib/*/pkgconfig/d3d.pc
usr/include/d3dadapter/drm.h
usr/include/d3dadapter/d3dadapter9.h
usr/include/d3dadapter/present.h
EOF'
# There will be symbol changes, if your lucky, my diff will get you by.
# If not, your first build will fail and generate it's own diff, use that like so:
sudo bash -c 'cat > debian/patches/gbm-addsymbols.diff << EOF
--- debian/libgbm1.symbols (libgbm1_17.3.0_i386)
+++ dpkg-gensymbols4b6R6J 2017-09-18 09:07:51.531786647 +0000
## -3,6 +3,7 ##
gbm_bo_create#Base 7.11~1
gbm_bo_create_with_modifiers#Base 17.1.0~rc2
gbm_bo_destroy#Base 7.11~1
+ gbm_bo_get_bpp#Base 17.3.0
gbm_bo_get_device#Base 8.1~0
gbm_bo_get_fd#Base 10.2~0
gbm_bo_get_format#Base 8.1~0
## -25,6 +26,7 ##
gbm_device_destroy#Base 7.11~1
gbm_device_get_backend_name#Base 7.11~1
gbm_device_get_fd#Base 7.11~1
+ gbm_device_get_format_modifier_plane_count#Base 17.3.0
gbm_device_is_format_supported#Base 8.1~0
gbm_surface_create#Base 8.1~0
gbm_surface_create_with_modifiers#Base 17.1.0~rc2
EOF'
echo 'gbm-addsymbols.diff' >> debian/patches/series
git add .
git commit -a
ARCH=i386 gbp buildpackage --git-pbuilder --git-dist=sid --arch=i386 --git-force-create --git-upstream-tree=SLOPPY --git-no-pristine-tar --git-ignore-new
# Failed?
# should be same as git reset --hard, debian/rules clean will get ran anyways:
git clean -f -d -i -x
# <fix problem>
git add .
git commit -a
ARCH=i386 gbp buildpackage --git-pbuilder --git-dist=sid --git-arch=i386 --git-force-create --git-upstream-tree=SLOPPY --git-no-pristine-tar --git-ignore-new
# Success?
# Great! The 64bit build should now be trivial (and faster with out gbp):
git clean -f -d -i -x
debuild -us -uc -B
# Let's move the clutter:
mkdir /opt/debs; mv /opt/src/*.deb /opt/debs
# And put some i386 copies in our repo:
cp /opt/debs/*i386.deb /home/user/pbuilder/deps
# Go install them!
sudo dpkg -i /opt/debs/*.deb
sudo aptitude
# Resolve, repeat! Twice should do it!
### OLD:
# Pre 17.2 symbol changes:
debian/libgl1-mesa-glx.symbols
# Top:
d3dadapter9.so.1 libgl1-mesa-glx #MINVER#
D3DAdapter9GetProc#Base 17.1.8
libGL.so.1 libgl1-mesa-glx | libgl1
(arch=!hurd-any)MesaGLInteropGLXExportObject#Base 12.0.4
(arch=!hurd-any)MesaGLInteropGLXQueryDeviceInfo#Base 12.0.4
4) Build Wine Already!
# The generate.py package generator from winehq is less work to modify then the official packages, bit still needs changes for nine
git clone https://github.com/<tut> debian-mesa/
git clone https://github.com/<tut> debian-wine/
git clone https://github.com/wine-compholio/wine-staging.git wine-staging/
git clone https://github.com/wine-compholio/wine-packaging.git wine-packaging/
ARCH=i386 git-pbuilder create
git clone git://source.winehq.org/git/wine.git wine.git/
cd /opt/src/wine.git
# Find staging version and match it to the wine source:
/opt/src/wine-staging/patches/patchinstall.sh --upstream-commit
# Should be on branch master already, so stay there:
git reset --hard 0b1d7ff7655c5aa7ff073f67400bd4401727183f
# OR:
# git checkout -b <name> 0b1d7ff7655c5aa7ff073f67400bd4401727183f
# and use gbp --git-debian-branch=<name>
# Apply ixit patches:
mkdir d3d9; cp ../d3d9/*.patch d3d9/;
cp ../wine-staging/patches/ ./ -R
# Build the winehq packaging and place it:
/opt/src/wine-packaging/generate.py --out . debian-sid-staging
cp /opt/src/wine-packaging/debian-sid-staging/debian debain/ -R
## -1,3 +1,5 ##
+d3dadapter9.so.1 libgl1-mesa-glx #MINVER#
+ D3DAdapter9GetProc#Base 17.1.8
libGL.so.1 libgl1-mesa-glx | libgl1
(arch=!hurd-any)MesaGLInteropGLXExportObject#Base 12.0.4
(arch=!hurd-any)MesaGLInteropGLXQueryDeviceInfo#Base 12.0.4
# Use tabs in things like debian/rules.. you were warned!
nano debian/rules
#Find: "$(CURDIR)/patches/patchinstall.sh" DESTDIR="$(CURDIR)" --all
#Add: patch -p1 < "$(CURDIR)d3d9/staging-helper.patch
# patch -p1 < "$(CURDIR)d3d9/wine-d3d9.patch
# On both:
#Find: $(CONFFLAGS)
#Replace: $(CONFFLAGS) \
--enable-nine
ARCH=i386 gbp buildpackage --git-pbuilder --git-dist=sid --git-arch=i386 --git-force-create --git-upstream-tree=SLOPPY --git-no-pristine-tar --git-ignore-new
wine --check-libs
wine --patches
Good Luck! (You'll need it)
I have a Docker Image which I want to build and when I run the build command on my Windows and Mac Docker it works fine and builds correctly, but if I run the same Dockerfile-Build on a Ubuntu-Server VM with docker I get an error.
The critical part of my Dockerfile is:
[...]
# Dependencies
RUN apt-get update && apt-get install -y apt-utils curl git tar gzip
# Install Go
ENV GO_VERSION 1.8
WORKDIR /tmp
ADD https://storage.googleapis.com/golang/go$GO_VERSION.linux-amd64.tar.gz ./
RUN mv go /usr/local/
[...]
But on the Ubuntu-server VM is fails at the RUN mv go /usr/local/-step
And producing the following error:
Step 10/24 : RUN mv go /usr/local/
---> Running in 6b79a20769eb
mv: cannot stat ‘go’: No such file or directory
And I suppose it does not extract the downloaded tar.gz correctly (but the download works)
Do you guys have any idea?
This is a known issue with 17.06 and patched in 17.06.1. The documented behavior is to download the tgz but not unpack it when pulling from a remote URL. Automatically unpacking the tgz was an unexpected change in behavior in 17.06 that they reverted back to only downloading the tgz in 17.06.1.
Release notes for 17.06 (see the note at the top): https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce
Release notes for 17.06.01: https://github.com/docker/docker-ce/releases/tag/v17.06.1-ce
Issue: https://github.com/moby/moby/issues/33849
PR of Fix: https://github.com/docker/docker-ce/pull/89
Edit, the minimize the number of layers in your image, I'd recommend doing the download, unpack, and cleanup as a single RUN command in your Dockerfile. E.g. here are two different Dockerfiles:
$ cat df.tgz-add
FROM busybox:latest
ENV GO_VERSION 1.8
WORKDIR /tmp
ADD https://storage.googleapis.com/golang/go$GO_VERSION.linux-amd64.tar.gz ./
RUN tar -xzf go$GO_VERSION.linux-amd64.tar.gz \
&& rm go$GO_VERSION.linux-amd64.tar.gz
CMD ls -l .
$ cat df.tgz-curl
FROM busybox:latest
ENV GO_VERSION 1.8
WORKDIR /tmp
RUN wget -O go$GO_VERSION.linux-amd64.tar.gz https://storage.googleapis.com/golang/go$GO_VERSION.linux-amd64.tar.gz \
&& tar -xzf go$GO_VERSION.linux-amd64.tar.gz \
&& rm go$GO_VERSION.linux-amd64.tar.gz
CMD ls -l .
The build output is truncated here...
$ docker build -t test-tgz-add -f df.tgz-add .
...
$ docker build -t test-tgz-curl -f df.tgz-curl .
...
They run identically:
$ docker run -it --rm test-tgz-add
total 4
drwxr-xr-x 11 root root 4096 Aug 31 20:27 go
$ docker run -it --rm test-tgz-curl
total 4
drwxr-xr-x 11 root root 4096 Aug 31 20:29 go
However, doing a single RUN to download, build, and cleanup saves you the 80MB of download from your layer history:
$ docker images | grep test-tgz
test-tgz-curl latest 2776133659af 30 seconds ago 269MB
test-tgz-add latest d625455998ff 2 minutes ago 359MB
Is it possible to use the command git diff --no-index on two remote servers through ssh.
I want to use the output of the command git diff but on a non repository . Between two remote ssh directory .
I've never bridged a diff command across servers, but effectively...you'd just be using diff. git diff uses whatever system-specified diff tool you have available to compare different versions, and since --no-index would allow you to compare across non-versioned files, diff is your best bet.
You can try this:
# Get the working tree from repo 1:
git archive --format=zip --remote=ssh://<user>#<host>/repo1/<repo1 name> <tag or HEAD> > archive1.zip
# Get the working tree from repo 2:
git archive --format=zip --remote=ssh://<user>#<host>/repo2/<repo2 name> <tag or HEAD> > archive2.zip
mkdir tmpdir tmpdir2
(cd tmpdir; git init .)
# Unzip archive1 into tmpdir, make a temporary git repo out of it:
(cd tmpdir; unzip ../archive1.zip ; git add .; git commit -a -m "archive1")
# Move the .git directory to tmpdir2:
mv tmpdir/.git tmpdir2
# Unzip archive2 into tmpdir2, and compare:
(cd tmpdir2; unzip ../archive2.zip ; git diff ) > git-diff.txt
# Cleanup:
rm -rf tmpdir tmpdir2 archive1.zip archive2.zip