I’ve a Makefile with multiple git repo’s which I need to clone
I use the following which works
clone:
git clone https://github.company.corp/dev-wi/ws-led.git
git clone https://github.company.corp/dev-wi/tools-extension.git
git clone https://github.company.corp/dev-wi/javt-ra.git
While the following code works, I want to do something like this in loop for all the repos on the list
build:
cd ws-led; \
docker build -t ws-led .
cd tools-extension; \
docker build -t tools-extension .
...
For each repo I need to change dir
And run the docker build ,
I want to avoid doing this over and over again ...
I know that I need to extract the string after /dev-wi/ as this is the repo directory which I need to run docker build on.
since I’ve many repo how can I do it easily ?
I try with subset however I have also the git command (in clone) so it doesn't work,any idea?
update
I've created the A new makefile and use only this code (the ws-led and tools-extension is folders in the same level of of the makefile
repos := ws-led tools-extension
.PHONY: all
all: $(patsubst%,%/docker-build.log,$(repos))
%/docker-build.log: %/.git
cd $*; docker build -t $* . >&2 | tee docker-build.log
I got error:
make: Nothing to be done forall'.`
what am I missing here ?
I try to simplify it but removing the git and let say that the folders (repo) existing on the same level of the makefile
UPDATE
Im change the makefile to be under the root
proj
- ws-led
— Dockerfile
-tools-ext
—Dockerfile
-Makefile
I try with the following
all: pre docker-build
.PHONY: pre docker-build
repos := ws-led tools-ext
pre:
$(patsubst %,%docker-build,$(repos))
docker-build:pre
cd $*; docker build -t $* . >&2 | tee docker-build
when I run make I got the following error
ws-leddocker-build ws-leddocker-build
make: ws-leddocker-build: No such file or directory
Any idea?
Looping is generally something you want to avoid. Instead, declare a series of targets for each repo.
repos := ws-led tools-extension javt-ra
.PHONY: all clone
all: $(patsubst %,%/.built,$(repos))
clone: $(patsubst %,%/.git,$(repos))
%/.built: %/.git
cd $*; docker build -t $* .
touch $#
%/.git:
git clone https://github.company.corp/dev-wi/$*.git
The .built flag file is a bit of a wart, and could usefully be replaced with something more useful, like the output from docker build.
all: $(patsubst %,%/docker-build.log,$(repos))
%/docker-build.log: %/.git
cd $*; docker build -t $* . >&2 | tee docker-build.log
The reason we generally try to avoid loops is to allow make to do its primary job properly -- avoid rerunning commands when a target is already up to date. So, for example, if you only changed ws-led, you don't want to force the other two to be rebuilt as well.
Having said that, the $(patsubst ...) is a loop of sorts; it basically loops over repos and creates a small piece of text around each. Without the patsubst we could write
all: ws-led/.built tools-extension/.built javt-ra/.built
which simply says that to make all we need to make those three; and then
%/.built: %/.git
says that for anything matching the pattern, it depends on the same stem with /.git after it. So in an otherwise empty directory, make would find that
to make all, we need to make ws-led/.built, tools-extension/.built, and javt-ra/.built;
to make ws-led/.built, we need to make ws-led/.git;
to make ws-led/.git, we need to
git clone https://github.company.corp/dev-wi/ws-led.git
then once this prerequisite is satisfied,
cd ws-led; docker build -t ws-led .
touch ws-led/.built
to make tools-extension/.built, we need to make tools-extension/.git;
to make tools-extension/.git, we need to
git clone https://github.company.corp/dev-wi/tools-extension.git
... etc etc.
In the future, when make finds that ws-led is newer than ws-led/.built it will build it again; but if it is not, it will conclude that no work needs to be done, etc for the other targets. This is how we avoid building things needlessly; but it obviously requires that the Makefile properly contains a formalization of every relevant dependency. (In this case, you would ideally like for there to be a way to know when the Git upstream has changed and something needs to be pulled by the local Makefile; this currently simply regards everything as done if the local Git clone has not received any updates.)
Related
I am using git in my project at Linux platform. I have plenty of files in a particular directory. I modified some 50 above files in that directory and didn't stage and commit it. I wish to delete all other unmodified files from that directory? Is there a way to do this, using git and Linux Commands?
Not sure why you would want to do this.... but you can:
# Save changes to stash
git stash save
# Remove everything left
rm -rf ./*
# Checkout (restore) all of the changed files
git stash show --stat | grep -v changed | sed -e 's/|.*$//;' | xargs git checkout
# Restore the changes to those files
git stash pop
git reset --hard [HEAD] should work for you repeated
Repeated question How can I discard modified files?
You can also use more simple commands for this purpose:
git clean -Xfd // capital X
git clean -xfd // lower x
It will clean your working directory from the desired files.
Using git clean is what you want. To remove (-x) those files and directories (-d), run:
$ git clean -fdx
If you use the -X option instead of -x, then the files you have told git to ignore will still be kept (e.g., build artifacts). Recent versions of git require either "-f" (force) or "-n" (dry-run) to be specified.
You should run a dry-run first, to show what will happen, but not actually do anything:
$ git clean -ndx
I use this so often, that I have an alias for this (added to your .gitconfig) to check for files that would be deleted when you run git clean. It's also useful to remind me if I've forgotten to "git add" a file that I want to keep.
[alias]
# list files that would be removed via 'clean' (non-destructive)
ifc = clean -ndx
Then, running git ifc (i.e,. "ifc" = "if clean") shows everything that isn't tracked and could be removed, or isn't tracked and should be added.
https://git-scm.com/docs/git-clean
On my project, there are two branches I'm working on (Develop and Release), each of which makes generous use of submodules. The Develop branch uses about twice as many submodules as Release, because it is where we test ideas.
When I switch branches from Develop to Release, the directories of Develop-specific submodules stay where they are, and so they become untracked. This makes things a bit confusing for me, because I do occasionally need to add or remove submodules from Release as well, and the git status message becomes a long list of untracked modules, some of which I want to use and some I don't.
What I would like to do is remove all untracked submodules from my project as soon as I switch from Develop to Release, so that I'm working with a "clean slate", (IE no untracked submodules sitting in my working directory).
I have found several solutions to removing individual submodules one at a time, such as here: How do I remove a submodule?
However solutions such as this assume that the git submodules are in use and being tracked (which they are not), and it is also a pain in the neck to remove them one at a time when I'm working with something like 15-20 submodules.
I have also tried piping linux commands like so:
git ls-files --others --exclude-standard | rm -rf
But the command does not appear to do anything. I have also tried using the same with git rm -rf, to no avail.
Does anyone know if there is an easy to remove all untracked git submodules from a working directory? Any advice anyone can share on this matter would be greatly appreciated. Thank you!
With some advice from helpful folks in the comments section, I determined that there is no obvious, non-tedious solution to this problem. Instead I created a bash script that does the job for me. Here it is, in case anyone else has the same issue:
#!/bin/bash
clear
git ls-files --others --directory --exclude-standard
echo
read -r -p "Are you sure you want to remove these untracked submodules? [y/N] " response
if [[ $response =~ ^([yY][eE][sS]|[yY])$ ]]
then
git ls-files --others --directory --exclude-standard | while read line; do
rm $line/.git &> /dev/null;
done
git clean -d -f
fi
Easy steps for people not super comfortable with Bash scripts:
Step 1: Copy the above script into a file and name it 'cleanUSM'. Save the file to /usr/bin. If you are having trouble saving it or finding /usr/bin, just save it to your current directory, and then use 'sudo mv cleanUSM /usr/bin/cleanUSM' to get it where it needs to go.
Step 2: From within your root directory, run the command 'cleanUSM'
Thanks to everyone who contributed!
I have a custom shell script to make twitter bootstrap from source and then move the files to my node.js app's /lib file:
rm -r bootstrap
make bootstrap
mv -f bootstrap/css/* ../../lib/public/css
mv -f bootstrap/img/* ../../lib/public/img
mv -f bootstrap/js/* ../../lib/public/js
Running this from the shell works just fine using ./make_bootstrap.sh
Now I've created a Makefile for my full app (mainly compiling coffeescript and easy test initialization) and want to have a command that executes this custom shell script to build bootstrap. Here is my makefile
REPORTER = spec
all: build
build:
#./node_modules/coffee-script/bin/coffee \
-c \
-o lib src
bootstrap:
#./src/bootstrap \
./make_bootstrap.sh
clean:
rm -rf lib
mkdir lib
watch:
#./node_modules/coffee-script/bin/coffee \
-o lib \
-cw src
test:
#./node_modules/mocha/bin/mocha \
--reporter $(REPORTER) \
test/*.coffee
.PHONY: build bootstrap clean watch test
with the relevant command being 'make bootstrap'. However when I run make bootstrap from the command line all I get is this error:
make: ./src/bootstrap: Permission denied
make: *** [bootstrap] Error 1
Originally I had assumed that it was a permission error but even setting all permissions on files (chmod 777) results in nothing. Files I have given full permissions at this point include the root Makefile, my custom shell script in the bootstrap folder and the makefile within the bootstrap folder itself.
EDIT:
Based on the comments I have refactored to this
bootstrap:
rm -r src/bootstrap/bootstrap
$(MAKE) -C ./src/bootstrap bootstrap
mv -f src/bootstrap/bootstrap/css/* lib/public/css
mv -f src/bootstrap/bootstrap/img/* lib/public/img
mv -f src/bootstrap/bootstrap/js/* lib/public/js
This duplicates the functionality of the shell script I had before (moving files for my custom project) and still uses the standard makefile that Twitter Bootstrap ships with. Much cleaner... I'm going to live the original answer below so people can see the evolution and refactor.
OLD ANSWER
Ok thank you guys in the comments for pointing my in the right direction. This solution works:
bootstrap:
cd ./src/bootstrap; \
./make_bootstrap.sh
What happens is it executes the change directory (in a sub process so it doesn't affect where I run make from) and then executes the custom script. It seems as if I probably shouldn't be using something like this in a makefile since it feels 'dirty'; perhaps a more clean way to do it would be to invoke the LESS compiler myself and mimic the makefile provided by bootstrap. I'm using this for a tiny personal project though so it does the job.
I am experimenting some linux configuration and I want to track my changes? Of course I don't want to to put my whole OS under version control?
Is there a way (with git, mercurial or any VCS) to track the change without storing the whole OS?
This is what I imagine:
I do a kind of git init -> all hashes of all files are stored, but not the content of the files
I make some changes to my file system -> git detect that the hash of this file has changed
I commit -> the content of the file is stored (or even better the original file and the diff are stored! I know, that is impossible... )
Possible? Impossible? Work-arounds?
EDIT: What I care about is just to minimize the size of the repository and to have a repository containing only my changes. Having all files in my repository is not relevant for me. For example if i push to github I just want it to contain only the files that has changed.
Take a look at etckeeper, it will probably do the job.
What you want is git update-index --info-only or ... --index-info, from the man page: " --info-only is used to register files without placing them in the object database. This is useful for status-only repositories.". --index-info is its industrial-scale cousin.
Do that with the files you want to track, write-tree to write the index structure into the object db, commit-tree that, and update-ref to update a branch.
To get the object name use git hash-objectfilename.
Here is what we do...
su -
cd /etc
echo "*.cache" > .gitignore
git init
chmod 700 .git
cd /etc; git add . && git add -u && git commit -m "Daily Commit"
Then setup crontab:
su -
crontab -e
# Put the following in:
0 3 * * * cd /etc; git add . && git add -u && git commit -m "Daily Commit"
Now you will have a nightly commit of all changes in /etc
If you want to track more than /etc in one repo, then you could simply do it at the root of your filesystem, except add the proper ignore paths to your /.gitignore. I am unclear on the effects of having git within git, so you might want to be extra careful in that case.
I know this question is old, but I thought this might help someone. Inspired by #Jonathon's comment on the How to record concrete modification of specific files question, I have created a shell script that enables you to monitors all the changes done on a specific file, while keeping all the changes history. the script depends on the inotifywait and git packages being installed.
You can find the script here
https://github.com/hisham-hassan/linux-file-monitor
Usage: file-monitor.sh [-f|--file] <absolute-file-path> [-m|--monitor|-h|--history]
file-monitor.sh --help
-f,--file <absolute-file-path> Adding a file to the monitored files List. The <absolute-file-path>
is the absolute file path of the file we need to action.
PLEASE NOTE: Relative file path could cause issues in the script,
please make sure to use the abolute path of the file. also try to
avoid sym links, as it has not been tested.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-m, --monitor Monitoring all the changes on the file. the monitoring will keep
happening as long as the script is running; you may need to run it
in the background.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-h, --history showing the full history of the file.
To exit, press "q"
example: file-monitor.sh -f /absolute/path/to/file/test.txt -h
--uninstall uninstalls the script from the bin direcotry,
and removes the monitoring history.
--install Adds the script to the bin directory, and creates
the directories and files needed for monitoring.
--help Prints this help message.
I'm currently using the usual technique in my Makefile to install individual files:
install:
install -D executable ${BIN_DIR}
But I just ran across a situation where I need to move a whole directory and all files underneath it into place.
Is cp -r the best way or is there a more linux-y/unix-y way to do this?
Yeah, it's hard to think of a more unix-ish way that cp -r, although the -r is a relatively late addition to cp. I can tell you the way we used to do it, and that works neatly across filesystems and such:
Let src be the source directory you want to move, and /path/to/target be an absolute path to the target. Then you can use:
$ tar cf - src | (cd /path/to/target; tar xf -)
My version of install(1) (Debian) has:
-d, --directory
treat all arguments as directory names; create all components of the specified directories
-t, --target-directory=DIRECTORY
copy all SOURCE arguments into DIRECTORY
So if you wanted to use install(1) consistently throughout your Makefile you could do:
install -d destdir
install srcdir/* -t destdir
-t isn't recursive however - if srcdir contains directories, then they won't get copied.
Linking is another viable alternative. That would allow you to keep multiple directories (representing different versions) accessible.