We are currently testing Bitbucket pipelines Beta (love it so far).
However I have a question.
We are uploading our source/ files fine using git-ftp , but on top of that we need to push the full Subdomain patch files (Index.php & .htaccess) they dont very often change, but we have to push them each time fully and not just the changes.
So far we can't get it working using git-ftp.
Are we doing something wrong?
The error message we keep keeping is:
git ftp push --user $Username --passwd $Pwd ftp://dev.iwantaspeaker.com/public_html/
No changed files for dev.iwantaspeaker.com/public_html/. Everything up-to-date.
I have included some of the code below and hope you can help. Thanks.
image: samueldebruyn/debian-git
pipelines:
branches:
develop:
- step:
script:
- echo "Pipeline Init for dev."
- apt-get update
- apt-get -qq install git-ftp
- echo "'_$(git status -uno --porcelain | wc -l)_'"
- git status -uno --porcelain
- echo "Initiating Push site:dev Source."
- git config git-ftp.syncroot Source/
- git ftp push --user $Username --passwd $Pwd ftp://dev.iwantaspeaker.com/public_html/
- echo "Initiating Push site:dev subDomianPatch."
- git config git-ftp.syncroot SubDomainPatches/dev/dev_subdomain_patch/public_html/
- git ftp push --user $Username --passwd $Pwd ftp://dev.iwantaspeaker.com/public_html/
You can use file .git-ftp-include
https://github.com/git-ftp/git-ftp/blob/master/man/git-ftp.1.md#syncing-untracked-files
Related
I'm right now on a stage of automating some parts of our code. The idea is simple, with every commit in our gitlab the script takes Excel from given directory and divides it's worksheets into individual Excel files and saves them into another directory. The problem right now is that I want to merge those files into our branch from Gitlab Runner. I tried editing gitlab-ci file, but I can't get it to work. I get this error:
The request URL returned error: 403
I tried adding personal token and then using it for pushing like this:
variables:
GIT_STRATEGY: clone
build-job:
stage: build
tags:
- data_dict
script:
- echo "Hello!"
- ls
- python3 -V
- pip3 list
test-job1:
stage: test
tags:
- data_dict
script:
- python3 Experiments/ConfAutomation.py
test-job2:
stage: test
tags:
- data_dict
script:
- git show-ref
- git remote -v
- echo "$RUNNER_ACCESS_TOKEN"
- echo "Print runner branch name"
- git config user.email "user#mail.com"
- git config user.name "name"
# - git remote set-url --push origin https://gitlab-ci:"$RUNNER_ACCESS_TOKEN"#gitlab/dir.git
- git add .
- git commit --allow-empty -m "Files from runner to branch"
- git push origin https://gitlab-ci:"$RUNNER_ACCESS_TOKEN"#gitlab/dir.git <branch-name>
deploy-prod:
stage: deploy
tags:
- data_dict
script:
- echo "This job deploys something."
I tried some stuff but can't seem to get it to work. Maybe you had similiar problem or you have some ideas ? Any help would be greatly appreciated.
Try using {$RUNNER_ACCESS_TOKEN} without double quotes.
I have Gitlab CI pipeline which is triggered by bitbucket webhook with current and last commit ids. I also want to re-run pipeline manually whenever the build created Gitlab CI file, triggered by webhook is not working as expected.
I tried RUN-PIPELINE option but shows the error:
The form contains the following error:
No stages/jobs for this pipeline.
Here is the GitLab CI file. Include refers to other project where standard yaml file for the pipeline is kept:
include:
- project: Path/to/project
ref: bb-deployment
file: /bitbucket-deployment.yaml
variables:
TILLER_NAMESPACE: <namespace>
NAMESPACE: testenv
REPO_REF: testenvbranch
LastCommitSHA: <commit sha from webhook>
CurrentCommitSHA: <Current commit she from webhook>
Here is the detailed gitlab-ci file that is provided in other project which has stages:
stages:
- pipeline
- build
variables:
ORG: test
APP_NAME: $CI_PROJECT_NAME
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIIVATE_KEY2" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
Building CI Script:
stage: pipeline
image: python:3.6
only:
refs:
- master
script:
- |
curl https://github.com/org/scripts/branch/install.sh | bash -s latest
source /usr/local/bin/pipeline-variables.sh
git clone git#bitbucket.org:$ORG/$APP_NAME.git
cd $APP_NAME
git checkout $lastCommit
cp -r env old
git checkout $bitbucketCommit
$CMD_DIFF old env
$CMD_BUILD
$CMD_INSTALL updatedReposList.yaml deletedReposList.yaml /tmp/test $NAMESPACE $REPO_REF $ORG $APP_NAME $lastCommit $bitbucketCommit
cat cicd.yaml
mv cicd.yaml ..
artifacts:
paths:
- cicd.yaml
Deplopying Apps:
stage: build
only:
refs:
- master
trigger:
include:
artifact: cicd.yaml
job: Building CI Script
strategy: depend
In the manual trigger, instead of considering the last and current commit she, it should rebuild the application.
Any help will be appreciated.
Thank you for your comment (below), I see you are using the include directive (https://docs.gitlab.com/ce/ci/yaml/#include) in one .gitlab-ci.yml to include a GitLab CI YAML file from another project.
I can duplicate this error (No stages / jobs for this pipeline) by invoking "run pipeline" on project 1 which is configured to include GitLab CI YAML from project 2 when the project 2 GitLab CI YAML is restricted to the master branch but I'm running the project on another branch.
For example, let's say project 1 is called "stackoverflow-test" and its .gitlab-ci.yml is:
include:
- project: atsaloli/test
file: /.gitlab-ci.yml
ref: mybranch
And project 2 is called "test" (in my own namespace, atsaloli) and its .gitlab-ci.yml is:
my_job:
script: echo hello world
image: alpine
only:
refs:
- master
If I select "Run Pipeline" in the GitLab UI in project 1 on a branch other than "master", I then get the error message "No stages / jobs for this pipeline".
That's because there is no job defined for my non-master branch, and then without any job defined, I don't have any stage defined.
I hope that sheds some light on what's going on with your webhook.
I am new to gitlab CI/CD and I'm struggling to figure this out.
All I want to do is when I push to dev branch I want my react app to be built and the folder ./build to be pushed through SSH to my dev server.
Here is what I did so far, including a screenshot of the error message I get.
This is my gitlab-ci.yml
image: node:latest
cache:
paths:
- node_modules/
build_dev:
stage: build
environment: Development
only:
- dev
script:
- ls
- npm install
- npm run build
artifacts:
paths:
- build/
- ecosystem.config.js
deploy_dev:
stage: deploy
environment: Development
only:
- dev
script:
- rsync -r -a -v -e ssh --delete "./build" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}
- rsync -r -a -v -e ssh --delete "./ecosystem.config.js" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}/
- ssh root#dev.teledirectasia.com "cd /var/www/gitlab/${CI_PROJECT_NAME} && pm2 start ecosystem.config.js"
I don't know why I am getting this output with job failed
This is a DNS problem. Your runner cannot resolve the hostname of the GitLab server - gitlab.teledirectgroup.com. Did you set the GitLab hostname if your local workstation's host file manually, or did you set it up in a DNS server as a 'proper' hostname?
If you set up the hostname in a DNS server then the solution may be as simple as adding the DNS server to /etc/resolv.conf on the runner. However, if you just set the GitLab hostname in your workstation's hosts file then you'll need to set it in the runner's /etc/hosts file, too. It's hard to say what the exact solution is without knowing how you set up the GitLab hostname in the first place.
the solution that’s so applied to GitLab?
Use the git clone by ssh, I don’t have a good goal that’s so I can up to push that’s changes over a submodule from runner Shell by GitLab CI. The pipeline ever fails and prints this error.
ERROR PIPELINE JOB
In the local repo as a project the file config contains that line with the URL, more don’t have login with this about the pipeline.
.git/config
Some help or walkthrough of reference to culminate with that challenge in troubleshooting!
This is my code over the file ".gitlab-ci.yml"
variables:
TEST_VAR: "Update Git Submoudel in all Etecnic projects."
job1:
variables: {}
script:
- echo "$TEST_VAR"
job2:
variables: {}
script:
- echo "OK" >> exito.txt
- git add --all
- git commit -m "Update Submodule"
- git push origin HEAD:master
Versions:
GitLab:
gitlab-ce is already the newest version (15.6.0-ce.0).
Runner:
Version: 15.5.1
Git revision: 7178588d
Git branch: 15-5-stable
GO version: go1.18.7
Built: 2022-11-11T09:45:25+0000
OS/Arch: linux/amd64
Thanks so much for your attention.
I'm getting this error after pipeline runs:
Preparing to unpack .../git-ftp_1.3.1-1_all.deb ...
Unpacking git-ftp (1.3.1-1) ...
Setting up libcurl3:amd64 (7.52.1-5+deb9u6) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up curl (7.52.1-5+deb9u6) ...
Setting up git-ftp (1.3.1-1) ...
$ git ftp push --user $FTP_USERNAME --passwd $FTP_PASSWORD sftp://$FTP_HOST
fatal: Remote host not set.
ERROR: Job failed: exit code 1
This is my .yml config:
image: samueldebruyn/debian-git
stage_deploy:
only:
- develop
script:
- apt-get update
- apt-get -qq install git-ftp
- git ftp push --user $FTP_USERNAME --passwd $FTP_PASSWORD sftp://$FTP_HOST
A month ago it worked fine. The values of the variables are correct..
Any ideas?
Before you make git ftp push you have to initialize git ftp, so add the line:
- git ftp init --user $FTP_USERNAME --passwd $FTP_PASSWORD ftp://$FTP_HOST
note: this only work for the first time
or if you want to execute this every commit add the line before git ftp push :
- git reset --hard
I finally solved it pressing the "Clear Runner Caches" button and running it again.
I am using gitlab CI runner to test my code and generating some files. I just want to push the generated files to gitlab repository via CI runner. Is there any way to do that ?
I have resolved this issue by doing this:
Note: If you want to git push to a non protected branch do not set the runner variable as protected
Generate new gitlab access token with api scope: User Settings > Access Tokens
Add a protected CI variable into your project settings with the new token: Your project > Settings > Secret variable using variable name CI_PUSH_TOKEN
Add another protected CI variable with your username using variable name CI_USERNAME
Then you can use this token instead of the default in you gitlab-ci script. for example:
before_script:
- git remote set-url origin https://${CI_USERNAME}:${CI_PUSH_TOKEN}#gitlab.com/${CI_PROJECT_NAME}.git
- git config --global user.email '${GITLAB_USER_EMAIL}'
- git config --global user.name '${GITLAB_USER_ID}'
...
- git checkout -B branch
- # do the file changes here
- git commit -m '[skip ci] commit from CI runner'
- git push --follow-tags origin branch
Generated a SSH Key in gitlab
--> Profile Settings --> SSH Keys --> Generate It
After generating the SSH Key store that in the gitlab variables named SSH
--> Project Settings --> Variables --> Add Variable
In the .gitlab-ci.yml add the below lines.
before_script:
- mkdir -p ~/.ssh
- echo "$SSH" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- ssh-keyscan -H 'Git_Domain' >> ~/.ssh/known_hosts
After that pushed the files to the repository using this below js code.
var child_process = require("child_process");
child_process.execSync("git checkout -B 'Your_Branch'");
child_process.execSync("git remote set-url origin Your_Repository_Git_Url");
child_process.execSync("git config --global user.email 'Your_Email_ID'");
child_process.execSync("git config --global user.name 'Your_User_Name'");
for (var i=0;i<filesToBeAdded.length;i++) {
child_process.execSync("git add "+filesToBeAdded[i]);
}
var ciLog = child_process.execSync("git commit -m '[skip ci]Automated commit for CI'");
var pushLog = child_process.execSync("git push origin Your_Branch");
[skip ci] is most important in commit message. Otherwise it will start a infinity loop of CI process.
You could use of course SSH keys but you could also provide user and password (user with write access) as secret variables and use them.
Example:
before_script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.email 'myuser#mydomain.com'
- git config --global user.name 'MyUser'
You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).
With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)
Example release job:
release:
stage: release
script:
- git branch
- gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
- git push --tags
only:
- master
Another solution using Gitlab API to commit back a file .terraform.lock.hcl in terraform/ directory on $CI_COMMIT_BRANCH with [skip ci] :
script:
- 'STATUS=$(curl -Ss --head --header "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/files/terraform%2F%2Eterraform%2Elock%2Ehcl?ref=$CI_COMMIT_BRANCH" | grep "HTTP/1.1" | cut -d " " -f2)'
- if [[ $STATUS == "404" ]]; then ACTION="create"; else ACTION="update"; fi
- 'curl --request POST --form "branch=$CI_COMMIT_BRANCH" --form "commit_message=[skip ci] terraform.lock.hcl from pipeline" --form "actions[][action]=$ACTION" --form "actions[][file_path]=terraform/.terraform.lock.hcl" --form "actions[][content]=<.terraform.lock.hcl" --header "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/commits"'
The Feature you are looking for is called Artifacts. Artifacts are files which are attached to a build when they are successful.
To enable an Artifact put this in your .gitlab-ci.yml:
artifacts:
paths:
- dir/
- singlefile
This will upload the dir directory and the file singlefile back to GitLab.