In Jenkins i had an analysis job. The job used to checkout and build the merge request sent to the target branch. However, after upgrading the gitlab version from 12.0.1 to 12.1.17 i am unable to checkout source branch.
Below is the groovy script i was using.
#!/usr/bin/env groovy
def call() {
if (env.gitlabMergeRequestId) {
sh "echo '${env.gitlabMergeRequestId}'"
sh "echo 'Merge request detected. Merging...'"
def credentialsId = scm.userRemoteConfigs[0].credentialsId
checkout ([
$class: 'GitSCM',
branches: [[name: "${env.gitlabSourceNamespace}/${env.gitlabSourceBranch}"]],
extensions: [
[$class: 'PruneStaleBranch'],
[$class: 'CleanCheckout'],
[
$class: 'PreBuildMerge',
options: [
fastForwardMode: 'NO_FF',
mergeRemote: env.gitlabTargetNamespace,
mergeTarget: env.gitlabTargetBranch
]
]
],
userRemoteConfigs: [
[
credentialsId: credentialsId,
name: env.gitlabTargetNamespace,
url: env.gitlabTargetRepoHttpURL
],
[
credentialsId: credentialsId,
name: env.gitlabSourceNamespace,
url: env.gitlabSourceRepoHttpURL
]
]
])
} else {
sh "echo 'No merge request detected. Checking out current branch'"
checkout ([
$class: 'GitSCM',
branches: scm.branches,
extensions: [
[$class: 'PruneStaleBranch'],
[$class: 'CleanCheckout']
],
userRemoteConfigs: scm.userRemoteConfigs
])
}
}
I was able to solve it by adding in the branches
branches: [[name: "refs/heads/${env.gitlabSourceBranch}"]]
Related
I've been upgrading our workflow to add an automatic version bump. The problem is that I accidentally added these steps with a typo in the .bumpversion.cfg file and from that moment, the workflow is sure that the releases start at tag 1.0.0. I've created 2 releases with this scenario (1.0.0 and 1.0.1). Once I was on to this, I deleted the two releases (and tags), but now the workflow can't find the latest release tag.
One important piece of info is that the repo is a Node app, so there is already a package.json and such there.
I tried:
bumping manually the versions in my files
bumping with bump2version to the version I expected
bumping with bump2version to the next version
As you'll see, the command is missing the current version. The error in the workflow is:
An error occurred while running semantic-release: Error: Command failed with exit code 2: bump2version --allow-dirty --current-version --new-version 1.0.0 patch
the create release step is:
name: Create new release
on:
workflow_dispatch:
push:
branches:
- main
jobs:
release:
runs-on: ubuntu-latest
if: "github.event_name == 'push' && github.ref == 'refs/heads/main' && !startsWith(github.event.head_commit.message, 'chore')"
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
fetch-depth: 0
token: ${{ secrets.ADMIN_TOKEN }}
- name: setup nodejs
uses: actions/setup-node#v3
with:
node-version: '16'
- name: release using semantic-release
env:
GITHUB_TOKEN: ${{ secrets.ADMIN_TOKEN }}
GIT_AUTHOR_NAME: secrets.automation.dev
GIT_AUTHOR_EMAIL: secrets.automation.dev#il.ibm.com
GIT_COMMITTER_NAME: secrets.automation.dev
GIT_COMMITTER_EMAIL: secrets.automation.dev#il.ibm.com
run: |
sudo apt-get update
sudo apt-get install python
pip install --user bumpversion
npm install #semantic-release/changelog
npm install #semantic-release/exec
npm install #semantic-release/git
npm install #semantic-release/github
npx semantic-release
The .bumpversion.cfg is:
[bumpversion]
current_version = 1.0.40
commit = True
message = Update version {current_version} -> {new_version}
[bumpversion:file:package.json]
search = {current_version}
replace = {new_version}
The .releaserc file is:
{
"debug": true,
"branches": [ "main" ],
"plugins": [
["#semantic-release/commit-analyzer", {
"preset": "angular",
"releaseRules": [
{"type": "release","release": "patch"}
]}],
"#semantic-release/release-notes-generator",
"#semantic-release/changelog",
[
"#semantic-release/exec",
{
"prepareCmd": "bump2version --allow-dirty --current-version ${lastRelease.version} --new-version ${nextRelease.version} patch"
}
],
[
"#semantic-release/git",
{
"message": "chore(release): ${nextRelease.version} [skip ci] release notes\n\n${nextRelease.notes}"
}
],
"#semantic-release/github"
]
}
I used 2 things to fix the issue:
I followed the excellent fix by #alvaropinot from this issue thread. Basically, I had to force-tag the commit I expected to be with the latest tag and push the tags:
git tag -f v1.0.40 356a7b4
git push -f --tags
For #semantic-release/npm to work, I had to rename a secret from "NPM_AUTH_TOKEN" to "NPM_TOKEN".
After that, I revamped my semantic-release workflow to use #semantic-release/npm:
name: Create a new release
on:
workflow_dispatch:
push:
branches:
- main
jobs:
release:
runs-on: Ubuntu-20.04
if: "github.event_name == 'push' && github.ref == 'refs/heads/main' && !startsWith(github.event.head_commit.message, 'chore')"
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
fetch-depth: 0
token: ${{ secrets.ADMIN_TOKEN }}
- name: setup nodejs
uses: actions/setup-node#v3
with:
node-version: '16'
- name: release using semantic-release
env:
GITHUB_TOKEN: ${{ secrets.ADMIN_TOKEN }}
GIT_AUTHOR_NAME: ***
GIT_AUTHOR_EMAIL: ***
GIT_COMMITTER_NAME: ***
GIT_COMMITTER_EMAIL: ***
NPM_TOKEN: ${{ secrets.NPM_AUTH_TOKEN }}
run: |
sudo apt-get update
sudo apt-get install python
pip install --user bumpversion
npm install #semantic-release/changelog
npm install #semantic-release/git
npm install #semantic-release/github
npm install #semantic-release/npm
npx semantic-release
and the .releaserc file now looks like:
{
"debug": true,
"branches": [
"main"
],
"verifyConditions": [
"#semantic-release/changelog",
"#semantic-release/npm",
"#semantic-release/git"
],
"analyzeCommits":[
["#semantic-release/commit-analyzer", {
"preset": "angular",
"releaseRules": [
{"type": "release","release": "patch"}
]}],
],
"generateNotes": [
"#semantic-release/release-notes-generator"
],
"prepare": [
"#semantic-release/changelog",
"#semantic-release/npm",
"#semantic-release/git"
],
"publish": [
[
"#semantic-release/npm",
{
"pkgRoot": "dist"
}
],
{
"path": "#semantic-release/github"
}
]
}
I have a monorepo with Lerna and yarn workspaces. I use Azure Devops to build and publish the application.
Commands
"emis-app:test": "yarn --cwd apps/my-app test", is located on the root package.json
What works
When there is a cache miss or when I don't cache NPM modules,
yarn my-app:test which then trigger yarn --cwd apps/my-app test is successful
What does not work
When the cache is used yarn emis-app:test which then triggers yarn --cwd apps/my-app test does not work and tests are failing.
Here is the output of the cache hit
Resolving key:
- **/yarn.lock, !**/node_modules/**/yarn.lock, !*... [file pattern; matches: 4]
- s/apps/my-app/yarn.lock --> 0E8B2ACAB9CF0A6F80305D0BD6C99FDFA703EE248C33DB254DF57F46CC67B6AF
- s/apps/my-app-1/yarn.lock --> 95AB055F93FBE7A5E118B9C1391F81E1E9885D5ED5F0B6EAAB46985D0619C81D
- s/libs/my-lib/yarn.lock --> C8B48CB9F78F4AAE95941EE10588B139FEE51E2CEDA3313E7FE2B78A32C680B0
- s/yarn.lock --> 31D5354CDC72614EEE3B29335A5F6456576FAEF27417B811967E7DDA9BD91E48
Workspaces
"workspaces": {
"packages": [
"apps/*",
"libs/*"
]
}
Each app is a vue application. Each app contains its own package.json, babel.config, jest.config etc.
jest.config.base extended in each app.
module.exports = {
preset: '#vue/cli-plugin-unit-jest/presets/typescript-and-babel',
transform: {
'vee-validate/dist/rules': 'babel-jest',
'.*\\.(vue)$': 'vue-jest',
'^.+\\.(ts|tsx)$': 'ts-jest',
},
testMatch: [
'**/*.(spec|test).(js|jsx|ts|tsx)',
],
testEnvironmentOptions: {
// Allow test environment to fire onload event
// See https://github.com/jsdom/jsdom/issues/1816#issuecomment-355188615
resources: 'usable',
},
reporters: [
'default',
[
'jest-trx-results-processor',
{
outputFile: './coverage/test-results.trx',
defaultUserName: 'user name to use if automatic detection fails',
},
],
],
moduleFileExtensions: [
'js',
'ts',
'json',
'vue',
],
testURL: 'http://localhost/',
snapshotSerializers: [
'jest-serializer-vue',
],
runner: 'groups',
};
jest.config (my-app)
const baseConfig = require('../../jest.config.base');
const packageJson = require('./package.json');
module.exports = {
...baseConfig,
transformIgnorePatterns: [],
roots: [
'<rootDir>/src',
],
moduleNameMapper: {
'^#/(.*)$': '<rootDir>/src/$1',
'^#libs/registration-lib/(.*)$': '<rootDir>/../../libs/registration-lib/src/$1',
},
name: packageJson.name,
displayName: packageJson.name,
};
Questions
Am I using the cache correctly?
Is it possible to use the caching when working with workspaces
Errors
FAIL #apps/my-app src/ui/views/pages/registration/individual/Individual.vue.spec.js
● Test suite failed to run
Jest encountered an unexpected token
This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.
By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".
Here's what you can do:
• To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
• If you need a custom transformation specify a "transform" option in your config.
• If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.
You'll find more details and examples of these config options in the docs:
https://jestjs.io/docs/en/configuration.html
Details:
SyntaxError: Unexpected token ] in JSON at position 467
at JSON.parse (<anonymous>)
at parse (../../node_modules/tsconfig/src/tsconfig.ts:195:15)
at readFileSync (../../node_modules/tsconfig/src/tsconfig.ts:181:10)
at Object.loadSync (../../node_modules/tsconfig/src/tsconfig.ts:151:18)
at find (../../node_modules/vue-jest/lib/load-typescript-config.js:33:39)
at loadTypescriptConfig (../../node_modules/vue-jest/lib/load-typescript-config.js:73:26)
at compileTypescript (../../node_modules/vue-jest/lib/compilers/typescript-compiler.js:9:20)
at processScript (../../node_modules/vue-jest/lib/process.js:23:12)
at Object.module.exports [as process] (../../node_modules/vue-jest/lib/process.js:42:18)
at ScriptTransformer.transformSource (../../node_modules/#jest/transform/build/ScriptTransformer.js:453:35)
Pipeline YAML
- task: NodeTool#0
inputs:
versionSpec: '15.x'
displayName: 'Install Node.js'
- task: Cache#2
displayName: Cache yarn packages
inputs:
key: '**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock'
path: $(Build.SourcesDirectory)/node_modules
cacheHitVar: CACHE_HIT
- task: Yarn#3
condition: ne(variables['CACHE_HIT'], 'true')
inputs:
customRegistry: 'useFeed'
customFeed: ${{ parameters.packageFeed }}
arguments: --frozen-lockfile
displayName: 'Install NPM dependencies'
- script: yarn my-app:test
displayName: "Test My App"
I am trying to take the chekouts from a different branches,
stage ("git-checkout"){
steps {
checkout([$class: 'GitSCM',
branches: [[name: '**']],
doGenerateSubmoduleConfigurations: false,
extensions: [], submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'Gitlb_creds', url: 'https://gitlab.com']]])
}
I have passed a string parameter as branch in the job configuration where I have multiple branches in git lab repo but it is taking checkouts from only one single branch even after the changing the value.
But I want to trigger the pipeline as per my choice branch either from dev, master or any feature from the git lab repo.
Is there a possible way?
If you want the your BRANCH parameter to fill in during checkout, you have to specify like this:
stage ("git-checkout"){
steps {
checkout([$class: 'GitSCM',
branches: [[name: ${BRANCH}]],
doGenerateSubmoduleConfigurations: false,
extensions: [], submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'Gitlb_creds', url: 'https://gitlab.com']]])
}
I have a long standing declarative pipeline infrastructure
I would like to start putting repeated code into shared libraries
The problem I am facing is calling the git plugin from a shared library function/class. I'm a bit lost as my experience is really only with Jenkins declarative stuff, not the Groovy/Java specifics.
Here is a snippet of the Jenkinsfile, (before using shared library):
pipeline {
agent any
stages {
stage('Prep Workspace') {
steps {
script {
if ((env.BRANCH_NAME == 'staging') || (env.BRANCH_NAME == 'production')) {
BRANCH=env.BRANCH_NAME
} else {
BRANCH='master'
}
}
echo "||------ Get ProjectOne Dependency ------||"
dir('deps/ProjectOne') {
git branch: "${BRANCH}",
changelog: false,
credentialsId: 'jenkinsgit',
poll: false,
url: 'git#github.com:myprivateorg/ProjectOne.git'
}
echo "||------ Get ProjectTwo Dependency ------||"
dir('deps/ProjectTwo') {
git branch: "${BRANCH}",
changelog: false,
credentialsId: 'jenkinsgit',
poll: false,
url: 'git#github.com:myprivateorg/ProjectTwo.git'
}
}
}
}
}
Note the repeated calls to pull down project files from git repos. The goal here, is to move the repeated code to a shared function call.
I've read the following portion in the manual, on how to use git in shared library:
https://www.jenkins.io/doc/book/pipeline/shared-libraries/#accessing-steps
Using the example in the documentation I've created the shared library file
In src/org/test/gitHelper.groovy:
package org.test;
def checkOutFrom(String repo, String branch='master') {
echo "||------ CLONING $repo ------||"
git branch: branch, changelog: false, credentialsId: 'jenkinsgit', poll: false, url: "git#github.com:myprivateorg/$repo.git"
}
return this
Then in the Jenkinsfile:
#Library('jenkins-shared-library') _
pipeline {
agent any
stages {
stage('Prep Workspace') {
steps {
script {
if ((env.BRANCH_NAME == 'staging') || (env.BRANCH_NAME == 'production')) {
BRANCH=env.BRANCH_NAME
} else {
BRANCH='master'
}
def g = new org.test.gitHelper()
g.checkOutFrom('ProjectOne')
g.checkOutFrom('ProjectTwo')
}
}
}
}
}
This loads the class and calls the function fine, but fails when it hits git itself:
groovy.lang.MissingPropertyException: No such property: git for class: java.lang.String
I used g.getClass() to confirm it's of type class org.test.gitHelper and NOT java.lang.String so I'm not sure where it's getting that type from.
Please note I have also tried this way:
vars/pullRepo.groovy
def call(String repo, String branch) {
echo "||------ CLONING $repo ------||"
dir("deps/$repo") {
git branch: branch, changelog: false, credentialsId: 'jenkinsgit', poll: false, url: "git#github.com:myprivateorg/$repo.git"
}
}
Jenkinsfile:
pullRepo('ProjectOne', 'master')
I get the exact same error: groovy.lang.MissingPropertyException: No such property: git for class: java.lang.String
For me, it works to pass the Jenkins context to the shared library like so:
Jenkinsfile:
pullRepo(this, repo, branch)
vars/pullRepo.groovy:
def call(def context, String repo, String branch) {
echo "||------ CLONING $repo ------||"
dir("deps/$repo") {
context.git branch: branch, changelog: false, credentialsId: 'jenkinsgit', poll: false, url: "git#github.com:myprivateorg/$repo.git"
}
}
Note that I'm passing the Jenkins context into the context variable, and calling git as a method of the context. You should also be able to do this by passing the context up to your class.
Looking help on Jenkins, i have written jenkinsfile where in one stage i am running sh git submodule update --init --recursive command my job is getting failed with Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password) error. but i can see on log common-ui-layout directory is present with full access, as per my analysis i found that while hitting the submodule command its not getting authenticated or its not finding the common-ui-layout folder. Im posting my jenkinsfile here, Please provide the fix of this issue.
pipeline {
agent {
label 'agent.com'
}
stages {stage("submodule clone"){
steps
{
checkout(
[
$class: 'GitSCM',
branches: [
[
name: 'master'
]
],
doGenerateSubmoduleConfigurations: false,
extensions: [
[
$class: 'SubmoduleOption',
disableSubmodules: false,
parentCredentials: true,
recursiveSubmodules: true,
reference: '',
trackingSubmodules: false
]
],
submoduleCfg: [],
userRemoteConfigs: [
[
credentialsId: '<***ID****>',
url: 'https://*****gitlab.com/****/common-ui-layout.git'
]
]
]
)
}
}
stage("fetch data"){
steps {
git branch: 'patch-1',
credentialsId: '<***ID****>',
url: 'https://****.gitlab.com/*****/****.git'
sh "pwd"
sh "ls -lat"
}
}
stage ("Installing pre-req"){
steps{
sh '''
yarn install;
yarn global add #angular/cli
'''
}
}
stage('Build app') {
steps {
sh "yarn install";
sh "pwd";
sh 'git submodule update --init --recursive';
//sh "git submodule update --recursive –remote";
sh "yarn run ng build";
println "BUILD NUMBER = $BUILD_NUMBER"
println "Build Success.."
}
}
}
}
Please refer the error snipt here
The above error got fixed after adding the rsa key. after that i got another error when i hit the git submodule update --init --recursive command in pipeline and return the with below error-
Cloning into 'common-ui-layout'...
fatal: could not read Username for 'https://xxx.xxx.com': No such device or address
please refer the snippet here
Please suggest me where i am missing?