On https://gitlab.com
I created a private group called test-kirby-group.
I created a private project in that group called test-project-1
I created another private project in that group called test-sub-project
I instantiated both new projects with a default readme.
In test-kirby-group/test-project-1 , i added a .gitmodules file
[submodule "vendor/submodules/test-sub-project"]
path = submodules/test-sub-project
url = ../../test-group-kirby/test-sub-project.git
In test-kirby-group/test-project-1 , i added a .gitlab-ci.yml file
variables:
GIT_SUBMODULE_STRATEGY: recursive
before_script:
- apk update && apk add git
- git submodule sync --recursive
- git submodule update --init --recursive
test:
script:
- pwd
- ls -al
- ls -al ../
- ls -al ../../
... I then checked the job output ...
Running with gitlab-runner 12.3.0 (a8a019e0)
on docker-auto-scale ed2dce3a
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:e216e233b2581d7f45d2bc2c4dce4f1f293267b29c45bfd929d038a9a67b4058 for ruby:2.5 ...
Running on runner-ed2dce3a-project-15133453-concurrent-0 via runner-ed2dce3a-srm-1572642947-b5456b44...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/test-group-kirby/test-project-1/.git/
Created fresh repository.
From https://gitlab.com/test-group-kirby/test-project-1
* [new ref] refs/pipelines/93160262 -> refs/pipelines/93160262
* [new branch] master -> origin/master
Checking out b624748a as master...
Updating/initializing submodules recursively...
$ apk update && apk add git
/bin/bash: line 89: apk: command not found
$ git submodule sync --recursive
$ git submodule update --init --recursive
$ pwd
/builds/test-group-kirby/test-project-1
$ ls -al
total 48
drwxrwxrwx. 3 root root 4096 Nov 1 21:17 .
drwxrwxrwx. 4 root root 4096 Nov 1 21:17 ..
drwxrwxrwx. 6 root root 4096 Nov 1 21:17 .git
-rw-rw-rw-. 1 root root 274 Nov 1 21:17 .gitlab-ci.yml
-rw-rw-rw-. 1 root root 141 Nov 1 21:17 .gitmodules
-rw-rw-rw-. 1 root root 18 Nov 1 21:17 README.md
$ ls -al ../
total 32
drwxrwxrwx. 4 root root 4096 Nov 1 21:17 .
drwxrwxrwx. 3 root root 4096 Nov 1 21:17 ..
drwxrwxrwx. 3 root root 4096 Nov 1 21:17 test-project-1
drwxrwxrwx. 3 root root 4096 Nov 1 21:17 test-project-1.tmp
$ ls -al ../../
total 24
drwxrwxrwx. 3 root root 4096 Nov 1 21:17 .
drwxr-xr-x. 1 root root 4096 Nov 1 21:17 ..
drwxrwxrwx. 4 root root 4096 Nov 1 21:17 test-group-kirby
Job succeeded
Question:
1. Should I expect to see a directory called vendor?
2. If so, why didn't it clone?
It seems that you may have not added the git submodule correctly.
In test-kirby-group/test-project-1 do the following:
Remove the current .gitmodules which was created manually
Run git submodule add <git#gitlab ...> <path/to/submodule>
You might need to now edit the generated .gitmodules from the above command to use a relative URL (i.e. what you already have url = ../../test-group-kirby/test-sub-project.git), for it to work with GitLab CI
You shouldn't need your current before_script, as long as you are using gitlab-runner v1.10+, you only need the following in your CI script:
variables:
GIT_SUBMODULE_STRATEGY: recursive
See Using Git submodules with GitLab CI for more information.
My use-case is versioning application changes on server because sometimes I need to rollback previous version before update, or before configuration changes.
This is my list of content:
dasper#debian:/usr/share/otrs$
drwxr-xr-x 6 root root 4096 Sep 18 16:56 .
drwxr-xr-x 109 root root 4096 Sep 18 13:13 ..
lrwxrwxrwx 1 root root 21 May 9 11:06 ARCHIVE -> /var/lib/otrs/ARCHIVE
drwxr-xr-x 4 root root 4096 Sep 11 15:53 bin
-rwx------ 1 root root 212450 Sep 18 16:06 .etckeeper
-rw-r--r-- 1 root root 1125 Jan 11 2019 .fetchmailrc.dist
drwx------ 8 root root 4096 Sep 18 16:06 .git
-rw-r--r-- 1 root root 932 Sep 18 15:40 .gitignore
drwxr-xr-x 9 root root 4096 Sep 11 15:53 Kernel
-rw-r--r-- 1 root root 104 Jan 12 2019 RELEASE
drwxr-xr-x 7 root root 4096 Sep 11 15:53 scripts
-rw-r--r-- 1 root root 0 Sep 18 15:36 testfile.txt
lrwxrwxrwx 1 root root 13 May 9 11:06 var -> /var/lib/otrs
I was trying etckeeper to auto commit after package manager make some changes, but git will as defaulte create only empty folder var/ and ARCHIVE/.
I was trying submodules, but then i don't know about changes in symlink folders (git add .) and submodule content can't lead to symlink folder.
My biggest issue is how to store these folders in git and how to clone/checkout in same structure.
Solution was easy & clean, i hope :)
apt-get install etckeeper
Make directory for repository. This is mounted to other disk
mkdir /BACKUP
cd /BACKUP
Initialize git repo & set worktree to root /. Now i can tracking whole system changes, not only /etc
git init
git config --path core.worktree /
My .gitignore inside /BACKUP:
# Ignore everything ~ whitelist
*
# But descend into directories
!*/
# Add /usr/share/otrs/ - main directory
!/usr/share/otrs/**
# Add /var/lib/otrs - symlink # Add /usr/share/otrs/var -> /var/lib/otrs
!/var/lib/otrs/**
...
# Add myself
!/.gitignore
# Add etckeeper staff
!/etc/.etckeeper/**
Add new line at the end of /etc/etckeeper/etckeeper.conf with path to repository:
ETCKEEPER_DIR=/BACKUP
At the end, remove unused repo initialized by etckeeper after installation:
cd /etc
rm -rf .git
rm .gitignore
Now you can make first inital commit: etckeeper commit
I'm getting this rather coo-coo error when using the Vim package manager Vundle's :PluginUpdate command. On further investigation, I've narrowed the problem down to the submodule update operation. Sure enough, if I perform the same git operation as shown in the Vundle log on one of these packages I get:
$ pwd
/home/mastershake/xtian/.vim/bundle/closetag.vim
$ ls -al
total 20K
drwxr-xr-x 4 xtian xtian 4.0K Apr 11 2015 .
drwxr-xr-x 24 xtian xtian 4.0K Oct 24 2015 ..
drwxr-xr-x 8 xtian xtian 4.0K Oct 31 09:05 .git
drwxr-xr-x 2 xtian xtian 4.0K Apr 11 2015 plugin
-rw-r--r-- 1 xtian xtian 1.5K Apr 11 2015 README
$ git submodule update --init --recursive
fatal: /home/mastershake/xtian/Documents.x/Employment.d/Jobs.d: '/home/mastershake/xtian/Documents.x/Employment.d/Jobs.d' is outside repository
On further investigation, I discover anywhere I perform this same git operation I get the same error. I say it's coo-coo, because the error has nothing to do with any individual .git repo itself.
What's git doing, and how do I fix it?
//UPDATE: Added directory contents to make it clearer what's happening.
> git gc --aggressive --prune=now
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66685/66685), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_cO6T53' write error: No space left on device
sigh, ok
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 15G 3.0G 84% /
udev 485M 4.0K 485M 1% /dev
tmpfs 99M 296K 99M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 494M 0 494M 0% /run/shm
cgroup 494M 0 494M 0% /sys/fs/cgroup
doesn't look that bad
ls -lh .git/objects/pack/
total 580M
-r--r--r-- 1 foouser root 12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root 11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root 40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root root 16M Feb 27 08:19 tmp_pack_cO6T53
So, git gc bails out on a tmp pack that's only 16MB big while my disk appears to have 3GB free. What am I missing? How can I get git gc to work more reliably? I've tried without aggressive option and --prune instead of --prune=now as well, same story.
Update
Doing a df -h during the repack action it shows that it is now using all my disk (100% usage). A little while later the repack action fails and it leaves another 14MB file in the .git/objects/pack/ folder. So, to recap, my packs use a total of 580MB. git repack somehow manages to use up 3GB to repack that. I have ~800MB free in the RAM after it's done btw. - maybe it's using so much working memory that it clogs up the swap? I guess my question comes down to: Are there options to make git repack less resource hungry?
versions: git version 1.7.9.5 on Ubuntu 12.04
Update 2
I've updated git to 2.3. Didn't change anything unfortunately.
> git --version
git version 2.3.0
> git repack -Ad && git prune
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (36893/36893), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_N9jyVJ' write error: No space left on device
Update 3
Ok, so I just noticed something that is curious: the .git directory actually uses much more disk space than the 508MB previously reported.
> du -h -d 1 ./.git
8.0K ./.git/info
40K ./.git/hooks
24M ./.git/modules
28K ./.git/refs
4.0K ./.git/branches
140K ./.git/logs
5.0G ./.git/objects
5.0G ./.git
Upon further inspection .git/objects/pack actually uses 4.5GB. The differences lies in hidden temp files I didn't notice before:
ls -lha ./.git/objects/pack/
total 4.5G
drwxr-xr-x 2 foouser root 56K Feb 27 15:40 .
drwxr-xr-x 260 foouser root 4.0K Oct 26 14:24 ..
-r--r--r-- 1 foouser root 12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root 11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr 3 2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root 40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19 2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct 3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct 2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:37 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.idx
-r--r--r-- 1 root root 14M Feb 27 15:29 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:32 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.idx
-r--r--r-- 1 root root 41M Feb 27 15:30 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.pack
-r--r--r-- 1 root root 1.1K Feb 27 15:37 .tmp-7729-pack-051980133b8f0052b66dce418b4d3899de0d1342.idx
(continuing for a *long* while).
Now I'd like to know: Is it safe to just delete those?
So here is what I found out so far: I couldn't find any documentation about these hidden '.tmp-XXXX-pack' in the .git/objects/pack folder. All other threads I can find are about non-hidden files with tmp_ prefix in the same folder. The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well. I can't confirm whether that's still possible in git 2.3.0 (which I've updated to since), but at least the disk space requirement doesn't seem to have changed in this newer version - it still can't complete gc/repack. By deleting these .tmp-files I was able to recover my last 4GB and git still seems to behave fine afterwards - your results may vary though, so please make sure you have a backup before doing this. Finally, even 4GB wasn't enough to repack with gc --agressive. My .git folder is 1.1GB after the cleanup, my entire repository is 1.7GB. So 2x the size of your repository is possibly not enough for git gc, even with the aggressive option (which should save space). So I had to recover more space from elsewhere first.
Here is the command I used to clean up (again, have backups!):
git gc --aggressive --prune=now || rm -f .git/objects/*/tmp_* && rm -f .git/objects/*/.tmp-*
Similar scenario (about 2.3G available), except git gc itself would also fail with fatal: Unable to create '/home/ubuntu/my-app-here/.git/gc.pid.lock': No space left on device
What worked was to git prune first, and then run the gc.
I had an instance of this problem. I was able to free up a considerable amount of disk space, but of course, that didn't solve the problem of what to do about the .tmp-* files. I ran git fsck and the Git repository wasn't damaged in that way.
I did the conventional pack-and-garbage-collect operations
git repack -Ad
git prune
but that didn't remove the .tmp-* files, though it would have ensured that all of the necessary objects were in the standard pack-* files if they needed to be copied from transient files left over from the crashed Git processes in the past.
Eventually I realized that I could safely move the .tmp-* files to a scratch directory, then run git fsck to see if what remained in the .git directory was complete. It turned out that it was, so I deleted the scratch directory and the files it contained. If git fsck had reported problems, I could have moved the .tmp-* files back into the .git directory and researched another solution.
The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well
The way "git repack"(man) created temporary files when it received a signal was prone to deadlocking, which has been corrected with Git 2.39 (Q4 2022).
See commit 9b3fadf (23 Oct 2022), and commit 1934307, commit 9cf10d8, commit a4880b2, commit b639606, commit d3d9c51 (21 Oct 2022) by Jeff King (peff).
(Merged by Taylor Blau -- ttaylorr -- in commit c88895e, 30 Oct 2022)
repack: use tempfiles for signal cleanup
Reported-by: Jan Pokorný
Signed-off-by: Jeff King
When git-repack(man) exits due to a signal, it tries to clean up by calling its remove_temporary_files() function, which walks through the packs dir looking for ".tmp-$$-pack-*" files to delete (where "$$" is the pid of the current process).
The biggest problem here is that remove_temporary_files() is not safe to call in a signal handler.
It uses opendir(), which isn't on the POSIX async-signal-safe list.
The details will be platform-specific, but a likely issue is that it needs to allocate memory; if we receive a signal while inside malloc(), etc, we'll conflict on the allocator lock and deadlock with ourselves.
We can fix this by just cleaning up the files directly, without walking the directory.
We already know the complete list of .tmp-* files that were generated, because we recorded them via populate_pack_exts().
When we find files there, we can use register_tempfile() to record the filenames.
If we receive a signal, then the tempfile API will clean them up for us, and it's async-safe and pretty battle-tested.
And:
repack: drop remove_temporary_files()
Signed-off-by: Jeff King
After we've successfully finished the repack, we call remove_temporary_files(), which looks for and removes any files matching ".tmp-$$-pack-*", where $$ is the pid of the current process.
But this is pointless.
If we make it this far in the process, we've already renamed these tempfiles into place, and there is nothing left to delete.
Nor is there a point in trying to call it to clean up when we aren't successful.
It's not safe for using in a signal handler, and the previous commit already handed that job over to the tempfile API.
It might seem like it would be useful to clean up stray .tmp files left by other invocations of git-repack.
But it won't clean those files; it only matches ones with its pid, and leaves the rest.
Fortunately, those are cleaned up naturally by successive calls to git-repack; we'll consider .tmp-*.pack the same as normal packfiles, so "repack -ad", etc, will roll up their contents and eventually delete them.
So as of two days ago, i had myself a shiny new SVN server. I set it up for WebDAV with no problems. It wasn't until i tried to host multiple projects that i ran into trouble.
At first, i had all of the project folders in one repository. But then i saw the error in my ways, and switched to multiple repositories. Rather than edit the /etc/apache2/sites-available/default file with every new project, i elected to use the svnserve deamon.
So, the problem is, i cn only make commits from Eclipse when the daemon is run as root. If not, I get a Permission Denied Error. Any ideas? Here are the directory stats:
drwxr-xr-x 6 root root 4096 2011-06-07 09:38 .
drwxr-xr-x 4 root root 4096 2011-06-05 00:48 ..
lrwxrwxrwx 1 svn svn 6 2011-06-06 22:35 all -> repos/
drw-rw-r-- 6 svn svn 4096 2011-06-06 13:53 EmpowerView
drw-rw-r-- 6 svn svn 4096 2011-06-06 13:54 GoogleVoicemail
-rw-rw-r-- 2 svn svn 307 2011-06-07 08:47 passwd
drw-rw-r-- 6 svn svn 4096 2011-06-06 13:54 PerlinNoise
drw-rw-r-- 7 svn svn 4096 2011-06-05 02:15 repos
-rw-rw-r-- 3 svn svn 2278 2011-06-07 08:46 svnserve.conf
EDIT
I tried the suggested re-checkout with the daemon not running as root, and still no cigar. Same permission error.
EDIT 2
I also tried running the daemon as the user svn, but no such luck. At one point I screwed up my primary group and no longer had sudo privileges! I had to hack into my own server through a buddy's account
EDIT 3 ANSWER
Well, after a few hours of tinkering, here is the setup that i got to work.
First I deleted any existing checkouts, and killed the deamon.
Then I set all the permissions to 774 for all my repositories (see below)
Next, I ran the daemon with this command sudo -u svn -- svnserve -d -r /storage/svn/
This allowed me to run the deamon as the user 'svn' without entering a password. (svn has no password or shell for security reasons)
Finally, I double checked all my config files, and was able to do checkouts exactly as hoped.
Repositories Directory
drwxr-xr-x 6 root root 4096 2011-06-07 10:34 .
drwxr-xr-x 4 root root 4096 2011-06-05 00:48 ..
lrwxrwxrwx 1 svn svn 6 2011-06-06 22:35 all -> repos/
drwxrwxr-- 6 svn svn 4096 2011-06-06 13:53 EmpowerView
drwxrwxr-- 6 svn svn 4096 2011-06-06 13:54 GoogleVoicemail
-rwxrwxr-- 2 svn svn 307 2011-06-07 08:47 passwd
drwxrwxr-- 6 svn svn 4096 2011-06-06 13:54 PerlinNoise
drwxrwxr-- 7 svn svn 4096 2011-06-05 02:15 repos
-rwxrwxr-- 3 svn svn 2273 2011-06-07 09:50 svnserve.conf
(Note: All of my repositories share common editors and permissions, so in each repositories conf/ folder, i put hardlinks to the passwd and svnserve.conf files above)
Probably the svn daemon has read permission to your repository but not write permission.
First find out under what user the svn daemon runs. Then chown the full repository to that user. That way the daemon can write your commits also.
Weird things happen when you commit, or checkout as root. Try to checkout a given repository not as root, and if that works you should be able to commit to it as a regular user thereafter.
If that still doesn't work, check the permissions on the repository itself. It's possible that the permissions aren't set in order to allow non-root users commit access.