The GitVersion step of our CI build now takes 4-5 minutes and we can't really figure out why. Here are some basic logs:
2018-09-25T14:31:02.4222252Z ##[section]Starting: GitVersion
2018-09-25T14:31:02.4227925Z ==============================================================================
2018-09-25T14:31:02.4228168Z Task : GitVersion Task
2018-09-25T14:31:02.4228396Z Description : Easy Semantic Versioning (http://semver.org) for projects using Git
2018-09-25T14:31:02.4228654Z Version : 3.6.5
2018-09-25T14:31:02.4229010Z Author : GitVersion Contributors
2018-09-25T14:31:02.4229302Z Help : See the [documentation](http://gitversion.readthedocs.org/en/latest/) for help
2018-09-25T14:31:02.4229565Z ==============================================================================
2018-09-25T14:31:02.4366228Z Preparing task execution handler.
2018-09-25T14:31:02.6709097Z Executing the powershell script: d:\w\_tasks\GitVersion_e5983830-3f75-11e5-82ed-81492570a08e\3.6.5\GitVersion.ps1
2018-09-25T14:31:03.0124372Z Current Directory: D:\w\_tasks\GitVersion_e5983830-3f75-11e5-82ed-81492570a08e\3.6.5
2018-09-25T14:31:03.0124924Z
2018-09-25T14:31:03.0125184Z
2018-09-25T14:31:03.0130879Z Sources Directory: d:\w\1\s
2018-09-25T14:31:03.0131239Z
2018-09-25T14:31:03.0131454Z
2018-09-25T14:31:03.0230450Z Invoking GitVersion with d:\w\1\s /output buildserver /nofetch
2018-09-25T14:31:03.0231023Z
2018-09-25T14:31:03.0231265Z
2018-09-25T14:31:03.0285337Z D:\w\_tasks\GitVersion_e5983830-3f75-11e5-82ed-81492570a08e\3.6.5\GitVersion.exe d:\w\1\s /output buildserver /nofetch
2018-09-25T14:31:03.1207954Z INFO [09/25/18 14:31:03:10] Working directory: d:\w\1\s
2018-09-25T14:31:03.1208779Z INFO [09/25/18 14:31:03:10] IsDynamicGitRepository: False
2018-09-25T14:31:03.1344488Z INFO [09/25/18 14:31:03:12] Returning Project Root from DotGitDirectory: d:\w\1\s\.git - d:\w\1\s
2018-09-25T14:31:03.1380473Z INFO [09/25/18 14:31:03:12] Running on Windows.
2018-09-25T14:31:03.1438147Z INFO [09/25/18 14:31:03:13] Applicable build agent found: 'VsoAgent'.
2018-09-25T14:31:03.1442989Z INFO [09/25/18 14:31:03:13] Branch from build environment: refs/heads/develop
2018-09-25T14:32:49.2277821Z INFO [09/25/18 14:32:49:22] IsDynamicGitRepository: False
2018-09-25T14:32:49.2281217Z INFO [09/25/18 14:32:49:22] Returning Project Root from DotGitDirectory: d:\w\1\s\.git - d:\w\1\s
2018-09-25T14:32:49.2281501Z INFO [09/25/18 14:32:49:22] Project root is: d:\w\1\s
2018-09-25T14:32:49.2281789Z INFO [09/25/18 14:32:49:22] DotGit directory is: d:\w\1\s\.git
2018-09-25T14:32:49.4925166Z INFO [09/25/18 14:32:49:47] IsDynamicGitRepository: False
2018-09-25T14:32:49.4926214Z INFO [09/25/18 14:32:49:47] Returning Project Root from DotGitDirectory: d:\w\1\s\.git - d:\w\1\s
2018-09-25T14:32:49.5009080Z INFO [09/25/18 14:32:49:49] Begin: Loading version variables from disk cache
2018-09-25T14:32:49.5014606Z INFO [09/25/18 14:32:49:49] Cache file d:\w\1\s\.git\gitversion_cache\EDAA5BA150AAFFAFF7ABE57C2BFD74AA4AA3B217.yml not found.
2018-09-25T14:32:49.5021815Z INFO [09/25/18 14:32:49:49] End: Loading version variables from disk cache (Took: 0.00ms)
2018-09-25T14:32:49.5029581Z INFO [09/25/18 14:32:49:49] IsDynamicGitRepository: False
2018-09-25T14:32:49.5030888Z INFO [09/25/18 14:32:49:49] Returning Project Root from DotGitDirectory: d:\w\1\s\.git - d:\w\1\s
2018-09-25T14:32:49.5151471Z INFO [09/25/18 14:32:49:50] Using latest commit on specified branch
2018-09-25T14:32:50.2769177Z INFO [09/25/18 14:32:50:27] Running against branch: develop (c7dfbbf862e9681bea20563bd4591f004ee74f0d)
2018-09-25T14:32:50.2862740Z INFO [09/25/18 14:32:50:27] Begin: Calculating base versions
2018-09-25T14:32:50.5260304Z INFO [09/25/18 14:32:50:52] Fallback base version: 0.1.0 with commit count source ed0e73ed52b1f271e3914267b510b5dd230eedce
2018-09-25T14:33:02.7492807Z INFO [09/25/18 14:33:02:74] Git tag 'v0.5.0-unstable.3243': 0.5.0-unstable.3243 with commit count source 63ee9c744ab7eeec6ea313f9630087b7c4cde4f1
2018-09-25T14:33:02.7607508Z INFO [09/25/18 14:33:02:75] Git tag 'v0.5.0-unstable.3240': 0.5.0-unstable.3240 with commit count source d71d3d708b84252ad53f256000279142e9288687
2018-09-25T14:33:02.7698133Z INFO [09/25/18 14:33:02:75] Git tag 'v0.5.0-ie11-clipboard.9+3256': 0.5.0-ie11-clipboard.9+3256 with commit count source 438fd61e9a4ee18abd55f956764a92595097be45
... for about 3000 lines ...
2018-09-25T14:33:54.1517668Z INFO [09/25/18 14:33:54:14] Git tag 'v0.1.0-unstable.1736': 0.1.0-unstable.1736 with commit count source 1469b1d1b38e7113ff765b3d591d42529f03f03a
2018-09-25T14:33:54.1734453Z INFO [09/25/18 14:33:54:16] Git tag 'v0.1.0-unstable.1735': 0.1.0-unstable.1735 with commit count source 5375858e279e253fc4578af8b0be29d5b4269e3e
2018-09-25T14:33:54.2005144Z INFO [09/25/18 14:33:54:19] Git tag 'v0.1.0-unstable.1734': 0.1.0-unstable.1734 with commit count source 63b6ad54121d9e4d06ddac4e80eda3f9f5709763
2018-09-25T14:34:09.7603293Z INFO [09/25/18 14:34:09:75] Merge target tagged 'v0.5.0-unstable.3243': 0.5.0-unstable.3243 with commit count source 63ee9c744ab7eeec6ea313f9630087b7c4cde4f1
2018-09-25T14:34:09.7706672Z INFO [09/25/18 14:34:09:75] Merge target tagged 'v0.5.0-unstable.3240': 0.5.0-unstable.3240 with commit count source d71d3d708b84252ad53f256000279142e9288687
2018-09-25T14:34:09.7805193Z INFO [09/25/18 14:34:09:77] Merge target tagged 'v0.5.0-ie11-clipboard.9+3256': 0.5.0-ie11-clipboard.9+3256 with commit count source 438fd61e9a4ee18abd55f956764a92595097be45
... for about 4000 lines like these ...
2018-09-25T14:35:13.8552137Z INFO [09/25/18 14:35:13:85] Merge target tagged 'v0.1.0-unstable.1736': 0.1.0-unstable.1736 with commit count source 1469b1d1b38e7113ff765b3d591d42529f03f03a
2018-09-25T14:35:13.8851950Z INFO [09/25/18 14:35:13:88] Merge target tagged 'v0.1.0-unstable.1735': 0.1.0-unstable.1735 with commit count source 5375858e279e253fc4578af8b0be29d5b4269e3e
2018-09-25T14:35:13.9069833Z INFO [09/25/18 14:35:13:89] Merge target tagged 'v0.1.0-unstable.1734': 0.1.0-unstable.1734 with commit count source 63b6ad54121d9e4d06ddac4e80eda3f9f5709763
2018-09-25T14:35:14.1701044Z INFO [09/25/18 14:35:14:16] Merge message 'Merge remote-tracking branch 'origin/release/v0.4.4' into develop': 0.4.4 with commit count source 081fe710a28ef0698520cc71c04ab4dd84028cbb
2018-09-25T14:35:14.1845666Z INFO [09/25/18 14:35:14:18] Merge message 'Merge branch 'release/v0.4.4' into develop': 0.4.4 with commit count source 46869edcecde68f5dc6d48721f0f4bb33b966932
2018-09-25T14:35:14.2046186Z INFO [09/25/18 14:35:14:19] Merge message 'Merge branch 'release/v0.4.4' of https://Company.visualstudio.com/Project/_git/GitRepo into release/v0.4.4': 0.4.4 with commit count source 407eea8e4dc3e779bc29359c39366e92f74db3dc
2018-09-25T14:35:14.2204322Z INFO [09/25/18 14:35:14:21] Merge message 'Merge branch 'hotfix/0.4.3'': 0.4.3 with commit count source 7e520eefc12c7aae61576fefba9aecb733259fdd
2018-09-25T14:35:14.2360008Z INFO [09/25/18 14:35:14:22] Merge message 'Merge branch 'hotfix/0.4.3' into develop': 0.4.3 with commit count source aa1bf7038192e1e43c5922997e9c7deee32365bb
2018-09-25T14:35:14.2505000Z INFO [09/25/18 14:35:14:24] Merge message 'Merge remote-tracking branch 'origin/release/v0.4.1' into develop': 0.4.1 with commit count source a00395b8c59b7bc0f1b4f6b2e15a69d779f00be5
2018-09-25T14:35:14.2754392Z INFO [09/25/18 14:35:14:27] Found multiple base versions which will produce the same SemVer (0.5.0), taking oldest source for commit counting (Merge message 'Merge branch 'hotfix/0.4.3' into develop')
2018-09-25T14:35:14.2754999Z INFO [09/25/18 14:35:14:27] Base version used: Merge message 'Merge remote-tracking branch 'origin/release/v0.4.1' into develop': 0.4.1 with commit count source aa1bf7038192e1e43c5922997e9c7deee32365bb
2018-09-25T14:35:14.2755712Z INFO [09/25/18 14:35:14:27] End: Calculating base versions (Took: 144,000.01ms)
2018-09-25T14:35:15.0181063Z INFO [09/25/18 14:35:15:00] 3280 commits found between aa1bf7038192e1e43c5922997e9c7deee32365bb and c7dfbbf862e9681bea20563bd4591f004ee74f0d
2018-09-25T14:35:15.0446700Z INFO [09/25/18 14:35:15:03] Begin: Creating dictionary
2018-09-25T14:35:15.0478209Z INFO [09/25/18 14:35:15:03] End: Creating dictionary (Took: 0.00ms)
2018-09-25T14:35:15.0503865Z INFO [09/25/18 14:35:15:03] Begin: Storing version variables to cache file d:\w\1\s\.git\gitversion_cache\EDAA5BA150AAFFAFF7ABE57C2BFD74AA4AA3B217.yml
2018-09-25T14:35:15.1043234Z INFO [09/25/18 14:35:15:10] End: Storing version variables to cache file d:\w\1\s\.git\gitversion_cache\EDAA5BA150AAFFAFF7ABE57C2BFD74AA4AA3B217.yml (Took: 62.50ms)
2018-09-25T14:35:15.1109747Z INFO [09/25/18 14:35:15:10] Applicable build agent found: 'VsoAgent'.
2018-09-25T14:35:15.1119265Z Executing GenerateSetVersionMessage for 'VsoAgent'.
2018-09-25T14:35:15.2560540Z Executing GenerateBuildLogOutput for 'VsoAgent'.
2018-09-25T14:35:15.2633144Z ##[section]Async Command Start: Update Build Number
2018-09-25T14:35:15.4482707Z Update build number to 0.5.0-unstable.3280 for build 6560
2018-09-25T14:35:15.4483044Z ##[section]Async Command End: Update Build Number
2018-09-25T14:35:15.4484260Z ##[section]Finishing: GitVersion
This is a VSTS build. We are using what we think is GitFlow. Feature branches are sometimes deleted, sometimes not. Maybe there is some cleanup to do here.
Any ideas as to why this is taking so long to execute?
Are you using mainline and seldom create tags to set versions?
If so, try to set a version every now and then with a tag.
Otherwise gitversion will go through a lot of history to calculate the version.
https://gitversion.readthedocs.io/en/latest/reference/mainline-development/#mainline-development
Related
I'm attempting to setup GitLab dependency scanning for a repository in my self-hosted GitLab server. I have included the job template and the test stage since I have overridden the stage clause. The job starts, but it fails soon after. When I set the variable SECURE_LOG_LEVEL to debug I see the following output.
$ /analyzer run
Using java version 'adoptopenjdk-17.0.2+8'
[INFO] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/cmd/gemnasium-maven/main.go:55] ▶ GitLab gemnasium-maven analyzer v3.11.1
[DEBU] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/finder/finder.go:64] ▶ inspect directory: .
[DEBU] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/finder/finder.go:96] ▶ skip ignored directory: .git
[DEBU] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/finder/detect.go:84] ▶ Selecting gradle for maven because this is the first match
[INFO] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/finder/finder.go:116] ▶ Detected supported dependency files in '.'. Dependency files detected in this directory will be processed. Dependency files in other directories will be skipped.
[DEBU] [gemnasium-maven] [2023-01-28T15:21:00Z] [/go/src/app/cmd/gemnasium-maven/main.go:234] ▶ Exporting dependencies for /path/to/my/app/build.gradle
[DEBU] [gemnasium-maven] [2023-01-28T15:21:05Z] [/go/src/app/builder/gradle/gradle.go:85] ▶ /path/to/my/app/gradlew --init-script /gemnasium-gradle-plugin-init.gradle gemnasiumDumpDependencies
Downloading https://services.gradle.org/distributions/gradle-7.1.1-bin.zip
..........10%...........20%...........30%..........40%...........50%...........60%..........70%...........80%...........90%...........100%
Welcome to Gradle 7.1.1!
Here are the highlights of this release:
- Faster incremental Java compilation
- Easier source set configuration in the Kotlin DSL
For more details see https://docs.gradle.org/7.1.1/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
FAILURE: Build failed with an exception.
* Where:
Initialization script '/gemnasium-gradle-plugin-init.gradle'
* What went wrong:
Could not compile initialization script '/gemnasium-gradle-plugin-init.gradle'.
> startup failed:
> General error during conversion: Unsupported class file major version 61
java.lang.IllegalArgumentException: Unsupported class file major version 61
at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:189)
at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:170)
at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:156)
at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:277)
...
How can I resolve this issue? I am using GitLab server v15.7.5.
I found a StackOverflow thread that is very similar to the issue I'm having.
It appears the version of ASM bundled with GitLab gemnasium-maven analyzer v3.11 does not support class files version 61 in Java v17.0.2+8.
I was able to get this working by downgrading GitLab gemnasium-maven analyzer to v2.31.0 and have filed a support request with GitLab to notify them of the issue.
To downgrade the dependency scanner, add the following block of code to your CICD template
.ds-analyzer:
variables:
DS_MAJOR_VERSION: 2
I am trying to deploy some of my jar libraries through JitPack. So far I am still testing things out, thus version codes are dev-SNAPSHOT or master-SNAPSHOT for the libraries.
For most libraries this seems to work well (at least as far as fetching the artifacts goes); however, one library had a failed build for master-SNAPSHOT.
The corresponding build, master-36ef0715cd-1, reports failure. The last lines of the log read:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.995 s
[INFO] Finished at: 2021-01-23T06:52:54Z
[INFO] ------------------------------------------------------------------------
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Found module: org.traffxml:traff-source-android:0.0.1-SNAPSHOT
Build tool exit code: 0
Looking for artifacts...
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Looking for pom.xml in build directory and ~/.m2
Found artifact: org.traffxml:traff-source-android:0.0.1-SNAPSHOT
ERROR: Time-out getting container status
Error building
So, apparently, the jar got built but fetching container status failed after that. This could well be an issue with JitPack’s infrastructure: over the last couple hours, it has taken several retries to get the artifacts to build, and I do not see any difference between this library and the others which built successfully.
How can I retry the failed build, or otherwise fix this (other than by going the crude way of pushing a new commit to my repo)?
Eventually I had to commit more changes, which eventually solved this for the moment.
On the long run, one thing to try:
Go to https://jitpack.io/
In the box, enter your group and artifact ID, and click Look up.
In the list of availale builds, click the Branches tab.
Look for the corresponding -SNAPSHOT build for your branch, and click Get it.
No guarantee this works, but it did trigger a new build after pushing a new commit. Feedback appreciated.
I am writing partitioned data (Parquet file) to AWS S3 using Apache Spark (3.0) from my local machine without having Hadoop installed in my machine. I was getting FileNotFoundException while writing to S3 when I have lot of files to write to around 50 partitions(partitionBy = date).
Then I have come across new S3A committer, So I tried to configure "partitioned" committer instead. But still I could see that Spark uses ParquetOutputCommitter instead of PartitionedStagingCommitter when the file format is "parquet". And still I am getting FileNotFoundException when I have lot of data to write.
My Configuration:
sparkSession.conf().set("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", 2);
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.name", "partitioned");
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.magic.enabled ", false);
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.conflict-mode", "append");
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.unique-filenames", true);
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads", true);
sparkSession.conf().set("spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a", "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory");
sparkSession.conf().set("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol");
sparkSession.conf().set("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter");
sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.tmp.path", "tmp/staging");
What am I doing incorrect? Could someone please help?
Note: I have created a JIRA in Spark for the same but no help till now: SPARK-31072
==============================================================
I tried the answer from (#Rajadayalan). But its still using FileOutputFormatter. I tried downgrading the spark version to 2.4.5 without any luck.
20/04/06 12:44:52 INFO ParquetFileFormat: Using user defined output committer for Parquet: org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
20/04/06 12:44:52 WARN AbstractS3ACommitterFactory: **Using standard FileOutputCommitter to commit work**. This is slow and potentially unsafe.
20/04/06 12:44:52 INFO FileOutputCommitter: File Output Committer Algorithm version is 2
20/04/06 12:44:52 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/04/06 12:44:52 INFO AbstractS3ACommitterFactory: Using Commmitter FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_20200406124452_0000}; taskId=attempt_20200406124452_0000_m_000000_0, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter#61deb03f}; outputPath=s3a://******/observation, workPath=s3a://******/observation/_temporary/0/_temporary/attempt_20200406124452_0000_m_000000_0, algorithmVersion=2, skipCleanup=false, ignoreCleanupFailures=false} for s3a://********/observation
20/04/06 12:44:53 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 81.077046 ms
20/04/06 12:44:54 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 31.993775 ms
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 9.967359 ms
Note: I don't have Spark installed in my local. So gave spark-hadoop-cloud_2.11 as compile time dependency
My build.gradle looks as follows:
compile group: 'org.apache.spark', name: 'spark-hadoop-cloud_2.11', version: '2.4.2.3.1.3.0-79'
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.10.0'
// https://mvnrepository.com/artifact/org.apache.parquet/parquet-column
compile group: 'org.apache.parquet', name: 'parquet-column', version: '1.10.1'
// https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop
compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
compile group: 'org.apache.parquet', name: 'parquet-avro', version: '1.10.1'
// https://mvnrepository.com/artifact/org.apache.spark/spark-sketch
compile group: 'org.apache.spark', name: 'spark-sketch_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-catalyst
compile group: 'org.apache.spark', name: 'spark-catalyst_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-tags
compile group: 'org.apache.spark', name: 'spark-tags_2.11', version: '2.4.5'
compile group: 'org.apache.spark', name: 'spark-avro_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.spark/spark-hive
compile group: 'org.apache.spark', name: 'spark-hive_2.11', version: '2.4.5'
// https://mvnrepository.com/artifact/org.apache.xbean/xbean-asm6-shaded
compile group: 'org.apache.xbean', name: 'xbean-asm7-shaded', version: '4.15'
compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.2.1'
// compile group: 'org.apache.hadoop', name: 'hadoop-s3guard', version: '3.2.1'
compile group: 'org.apache.hadoop', name: 'hadoop-aws', version: '3.2.1'
compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.2.1'
compile group: 'com.amazonaws', name: 'aws-java-sdk-bundle', version: '1.11.271'
Had the same issue, the solution from
How To Get Local Spark on AWS to Write to S3 worked to load PartitionedStagingCommitter. you also have to download spark-hadoop-cloud jar from as mentioned in the solution.
I also use spark 3.0 and this version of jar worked https://repo.hortonworks.com/content/repositories/releases/org/apache/spark/spark-hadoop-cloud_2.11/2.4.2.3.1.3.0-79/
Settings in my spark-defaults.conf
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.fs.s3a.committer.name partitioned
spark.hadoop.fs.s3a.committer.magic.enabled false
spark.hadoop.fs.s3a.commiter.staging.conflict-mode append
spark.hadoop.fs.s3a.committer.staging.unique-filenames true
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads true
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a
org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
spark.sql.sources.commitProtocolClass
org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class
org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
I got this working with a small change from what #Rajadayalan suggested. Apart from sparkSession.config().set() as in the initial question I have added the option() param in the df while writing parquet files
df.distinct()
.withColumn("date", date_format(col(EFFECTIVE_PERIOD_START), "yyyy-MM-dd"))
.repartition(col("date"))
.write()
.format(fileFormat)
.partitionBy("date")
.mode(SaveMode.Append)
.option("fs.s3a.committer.name", "partitioned")
.option("fs.s3a.committer.staging.conflict-mode", "append")
.option("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol")
.option("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter")
.option("compression", compressionCodecName.name().toLowerCase())
.save(DOWNLOADS_NON_COMPACT_PATH);
This makes the difference and the following stacktrace depicts that its using PartitionedStagingCommitter
Also I could also see that _SUCCESS file is a JSON instead of empty touch file (_SUCCESS) in S3.
20/04/06 14:27:26 INFO ParquetFileFormat: Using user defined output committer for Parquet: org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
20/04/06 14:27:26 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
20/04/06 14:27:26 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/04/06 14:27:26 INFO AbstractS3ACommitterFactory: Using committer partitioned to output data to s3a://************/observation
20/04/06 14:27:26 INFO AbstractS3ACommitterFactory: Using Commmitter PartitionedStagingCommitter{StagingCommitter{AbstractS3ACommitter{role=Task committer attempt_20200406142726_0000_m_000000_0, name=partitioned, outputPath=s3a://*********/observation, workPath=file:/tmp/hadoop-**********/s3a/local-1586197641397/_temporary/0/_temporary/attempt_20200406142726_0000_m_000000_0}, conflictResolution=APPEND, wrappedCommitter=FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_20200406142726_0000}; taskId=attempt_20200406142726_0000_m_000000_0, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter#4494e88a}; outputPath=file:/Users/**********/Downloads/SparkParquetSample/tmp/staging/**********/local-1586197641397/staging-uploads, workPath=null, algorithmVersion=1, skipCleanup=false, ignoreCleanupFailures=false}}} for s3a://parquet-uuid-test/device-metric-observation6
20/04/06 14:27:27 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 14:27:27 INFO CodeGenerator: Code generated in 52.744811 ms
20/04/06 14:27:27 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 14:27:27 INFO CodeGenerator: Code generated in 48.78277 ms
I am trying to figure out the simple Mainline developement mode.
My observation is the following.
Each direct commit on the master branch is incrementing the patch version.
When I create a feature branch, and then when I make commits on this feature branch, the patch is not increasing. I am wondering why? Only the PreReleaseTag is increasing. Example featureBranchName.1, featureBranchName.2 and so on for each commit. Patch does not change.
Why does not the patch version increase when commits are made to feature branch? Does a feature branch always imply a prerelease?
Is there a way to increase the patch on commits to feature branch?
My GitVersion.yml file is below. The -diag output follows, if that helps.
next-version: 0.1.0
mode: MainLine
assembly-versioning-scheme: MajorMinorPatch
assembly-file-versioning-scheme: MajorMinorPatchTag
increment: Inherit
continuous-delivery-fallback-tag: ci
tag-prefix: '[vV]'
major-version-bump-message: '\+semver:\s?(breaking|major)'
minor-version-bump-message: '\+semver:\s?(feature|minor)'
patch-version-bump-message: '\+semver:\s?(fix|patch)'
no-bump-message: '\+semver:\s?(none|skip)'
legacy-semver-padding: 4
build-metadata-padding: 4
commits-since-version-source-padding: 4
commit-message-incrementing: Enabled
commit-date-format: 'yyyy-MM-dd'
branches:
master:
tag: ''
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
regex: ^master
tracks-release-branches: false
is-release-branch: false
release:
mode: ContinuousDelivery
tag: beta
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
regex: ^releases?[/-]
tracks-release-branches: false
is-release-branch: true
pre-release-weight: 1000
feature:
tag: useBranchName
increment: Inherit
prevent-increment-of-merged-branch-version: false
track-merge-target: false
regex: ^features?[/-]
tracks-release-branches: false
is-release-branch: false
pull-request:
mode: ContinuousDelivery
tag: PullRequest
increment: Inherit
prevent-increment-of-merged-branch-version: false
tag-number-pattern: '[/-](?<number>\d+)[-/]'
track-merge-target: false
regex: ^(pull|pull\-requests|pr)[/-]
tracks-release-branches: false
is-release-branch: false
hotfix:
mode: ContinuousDelivery
tag: beta
increment: Patch
prevent-increment-of-merged-branch-version: false
track-merge-target: false
regex: ^hotfix(es)?[/-]
tracks-release-branches: false
is-release-branch: false
support:
mode: ContinuousDelivery
tag: ''
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
regex: ^support[/-]
tracks-release-branches: false
is-release-branch: false
develop:
mode: ContinuousDeployment
tag: unstable
increment: Minor
prevent-increment-of-merged-branch-version: false
track-merge-target: true
regex: ^dev(elop)?(ment)?$
tracks-release-branches: true
is-release-branch: false
ignore:
sha: []
merge-message-formats: {}
INFO [01/26/20 22:13:23:28] Dumping commit graph:
INFO [01/26/20 22:13:23:45] * c7eeed9 21 minutes ago (HEAD -> feature/fet2)
* 97227bd 46 minutes ago (master)
* 26f8b7b 55 minutes ago
* 59ba070 60 minutes ago
* 948b86f 84 minutes ago (origin/master, origin/HEAD)
* b45349d 87 minutes ago
* 7e48b14 2 hours ago
* 4da28e2 2 hours ago
* 0b0eabd 2 hours ago
* 8106caa 2 hours ago
* 013cbca 3 hours ago
* 5f26e19 3 hours ago (tag: 1.0.0)
* 052b643 3 hours ago
* 0f0f4cf 6 hours ago (tag: v0.1.0)
INFO [01/26/20 22:13:23:45] Working directory: D:\Trials\Git\GitVer\MainLineDevT1
INFO [01/26/20 22:13:23:45] IsDynamicGitRepository: False
INFO [01/26/20 22:13:23:48] Returning Project Root from DotGitDirectory: D:\Trials\Git\GitVer\MainLineDevT1\.git\ - D:\Trials\Git\GitVer\MainLineDevT1\
INFO [01/26/20 22:13:23:48] Running on Windows.
INFO [01/26/20 22:13:23:48] Project root is: D:\Trials\Git\GitVer\MainLineDevT1\
INFO [01/26/20 22:13:23:48] DotGit directory is: D:\Trials\Git\GitVer\MainLineDevT1\.git
INFO [01/26/20 22:13:23:69] Using latest commit on specified branch
INFO [01/26/20 22:13:23:71] Begin: Attempting to inherit branch configuration from parent branch
INFO [01/26/20 22:13:23:72] Begin: Finding branch source of 'feature/fet2'
INFO [01/26/20 22:13:23:75] Begin: Finding merge base between 'feature/fet2' and 'master'.
INFO [01/26/20 22:13:23:77] Found merge base of 97227bd0043bf85078313ea20b0b03e8e7c7263b
INFO [01/26/20 22:13:23:78] Merge base of feature/fet2' and 'master is 97227bd0043bf85078313ea20b0b03e8e7c7263b
INFO [01/26/20 22:13:23:78] End: Finding merge base between 'feature/fet2' and 'master'. (Took: 30.87ms)
INFO [01/26/20 22:13:23:79] End: Finding branch source of 'feature/fet2' (Took: 67.29ms)
INFO [01/26/20 22:13:23:79] Begin: Getting branches containing the commit '97227bd0043bf85078313ea20b0b03e8e7c7263b'.
INFO [01/26/20 22:13:23:79] Trying to find direct branches.
INFO [01/26/20 22:13:23:80] Direct branch found: 'master'.
INFO [01/26/20 22:13:23:80] End: Getting branches containing the commit '97227bd0043bf85078313ea20b0b03e8e7c7263b'. (Took: 3.24ms)
INFO [01/26/20 22:13:23:80] Found possible parent branches: master
INFO [01/26/20 22:13:23:80] End: Attempting to inherit branch configuration from parent branch (Took: 87.76ms)
INFO [01/26/20 22:13:23:81] Running against branch: feature/fet2 (c7eeed9e3c0cd0379804fab4c7fa137b533a5c64)
INFO [01/26/20 22:13:23:81] Begin: Calculating base versions
INFO [01/26/20 22:13:23:82] Fallback base version: 0.1.0 with commit count source 0f0f4cfc7313bfb24f542f33eb14486708ce97a1 (Incremented: None)
INFO [01/26/20 22:13:23:92] NextVersion in GitVersion configuration file: 0.1.0 with commit count source External Source (Incremented: None)
INFO [01/26/20 22:13:23:94] Git tag '1.0.0': 1.0.0 with commit count source 5f26e1972c25ee131706aaf0ab91f6bb42a41c49 (Incremented: 1.1.0)
INFO [01/26/20 22:13:23:94] Git tag 'v0.1.0': 0.1.0 with commit count source 0f0f4cfc7313bfb24f542f33eb14486708ce97a1 (Incremented: 0.2.0)
INFO [01/26/20 22:13:23:99] Found multiple base versions which will produce the same SemVer (1.1.0), taking oldest source for commit counting (Git tag '1.0.0')
INFO [01/26/20 22:13:23:99] Base version used: Git tag '1.0.0': 1.0.0 with commit count source 5f26e1972c25ee131706aaf0ab91f6bb42a41c49 (Incremented: 1.1.0)
INFO [01/26/20 22:13:23:99] End: Calculating base versions (Took: 180.31ms)
INFO [01/26/20 22:13:23:99] Begin: Using mainline development mode to calculate current version
INFO [01/26/20 22:13:24:01] Found possible mainline branches: master
INFO [01/26/20 22:13:24:01] Mainline for current branch is master
INFO [01/26/20 22:13:24:02] Found branch merge point; choosing 97227bd0043bf85078313ea20b0b03e8e7c7263b as effective mainline tip
INFO [01/26/20 22:13:24:02] Current branch (feature/fet2) was branch from 97227bd0043bf85078313ea20b0b03e8e7c7263b
INFO [01/26/20 22:13:24:02] Direct commit on master 013cbca52c56952a5deecdf0dcc1f0ea9fd7b9e3 incremented base versions Patch, now 1.0.1
INFO [01/26/20 22:13:24:02] Direct commit on master 8106caa0e355d696899876bb6f96533f8421cee6 incremented base versions Patch, now 1.0.2
INFO [01/26/20 22:13:24:02] Direct commit on master 0b0eabdf3e9be9f93b0c777c909b866c7f1b6f24 incremented base versions Patch, now 1.0.3
INFO [01/26/20 22:13:24:04] Direct commit on master 4da28e2a2c7cf785458d94717ea54e3cda4f17c8 incremented base versions Patch, now 1.0.4
INFO [01/26/20 22:13:24:04] Direct commit on master 7e48b14da1fdb71747174502d4f113b8b0aa7671 incremented base versions Patch, now 1.0.5
INFO [01/26/20 22:13:24:04] Direct commit on master b45349dbc02e9dadd235669c69d1469bae6776d2 incremented base versions Patch, now 1.0.6
INFO [01/26/20 22:13:24:04] Direct commit on master 948b86fb2d2d1c4bed0033b6670efbd036222f53 incremented base versions Minor, now 1.1.0
INFO [01/26/20 22:13:24:04] Direct commit on master 59ba070ad8a2ada5c298aef4ca0f2a364e89248f incremented base versions Patch, now 1.1.1
INFO [01/26/20 22:13:24:04] Direct commit on master 26f8b7b06062da0a0811d3c9a3b4ff52d4b00065 incremented base versions Patch, now 1.1.2
INFO [01/26/20 22:13:24:04] Direct commit on master 97227bd0043bf85078313ea20b0b03e8e7c7263b incremented base versions Patch, now 1.1.3
INFO [01/26/20 22:13:24:04] 1 commits found between 97227bd0043bf85078313ea20b0b03e8e7c7263b and c7eeed9e3c0cd0379804fab4c7fa137b533a5c64
INFO [01/26/20 22:13:24:05] Performing Patch increment for current branch
INFO [01/26/20 22:13:24:05] End: Using mainline development mode to calculate current version (Took: 52.54ms)
INFO [01/26/20 22:13:24:05] Using branch name to calculate version tag
INFO [01/26/20 22:13:24:05] Begin: Getting version tags from branch 'refs/heads/feature/fet2'.
INFO [01/26/20 22:13:24:06] End: Getting version tags from branch 'refs/heads/feature/fet2'. (Took: 8.20ms)
You didn't set a mode for your feature branch configuration, so it is inheriting the global configuration of Mainline. In Mainline delivery mode, versioning is calculated based on merges to the branch (usually, Mainline is only used for the master branch).
Try setting mode: ContinuousDeployment to achieve the desired result.
I have one private build agent in VSTS which I configured back in March this year, now I wanted to deploy a second one, but I'm having issue with NPM test task on the new build agent. It's using the same Agents package published by Microsoft, so I don't believe the issue should be there. I have install the same version of Node.js (6.10.1) and the same NPM version (5.10.10) on both agents, but the new one has issue where the task npm test executes in the build definition, but does not produce any result and neither is anything shown in the VSTS log files/console, it just hangs until timeout period is reached (80 minutes), or until I cancel it manually. When I run it locally on the agent it works perfectly fine, I also tried running it in the pipeline via the CMD task, not the NPM and it also works, so I know there is no issue with the server/connectivity/etc.
If someone experienced a similar issue, I would really welcome some insights into this problem, because as I've mentioned, same pipeline works on the first agent, so the configuration of the tasks, pipeline, code/folder structure cannot be the issue here.
Log file from the VSTS agent follows, manual cancel at the end after 7 minutes,
because normally it takes around 20 seconds (XXXXX is the path to project):
[2017-11-27 14:24:29Z INFO StepsRunner] Processing step: DisplayName='Execute React Unit Tests', ContinueOnError=False, Enabled=True
[2017-11-27 14:24:29Z INFO ExpressionManager] Evaluating: succeeded()
[2017-11-27 14:24:29Z INFO ExpressionManager] Expanded: True
[2017-11-27 14:24:29Z INFO ExpressionManager] Result: True
[2017-11-27 14:24:29Z INFO StepsRunner] Starting the step.
[2017-11-27 14:24:29Z INFO TaskManager] Loading task definition 'C:\VSTSAgent\_work\_tasks\Npm_fe47e961-9fa8-4106-8639-368c022d43ad\1.0.8\task.json'.
[2017-11-27 14:24:29Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.IJobExtension'
[2017-11-27 14:24:29Z INFO BuildJobExtension] Build JobExtension resolving path use source provide: XXXXX
[2017-11-27 14:24:29Z INFO BuildJobExtension] The Default Path Root of Build JobExtension is build.sourcesDirectory: C:\VSTSAgent\_work\7\s
[2017-11-27 14:24:29Z INFO BuildJobExtension] After prefix Default Path Root provide by JobExtension: C:\VSTSAgent\_work\7\s\XXXXX
[2017-11-27 14:24:29Z INFO BuildJobExtension] Return absolute path after prefix DefaultPathRoot: C:\VSTSAgent\_work\7\s\XXXXX
[2017-11-27 14:24:29Z INFO TaskRunner] Build JobExtension resolved a rooted path:: C:\VSTSAgent\_work\7\s\XXXXX
[2017-11-27 14:24:29Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.IJobExtension'
[2017-11-27 14:24:29Z INFO BuildJobExtension] Build JobExtension resolving path use source provide:
[2017-11-27 14:24:29Z INFO BuildJobExtension] The Default Path Root of Build JobExtension is build.sourcesDirectory: C:\VSTSAgent\_work\7\s
[2017-11-27 14:24:29Z INFO BuildJobExtension] After prefix Default Path Root provide by JobExtension: C:\VSTSAgent\_work\7\s
[2017-11-27 14:24:29Z INFO BuildJobExtension] Return absolute path after prefix DefaultPathRoot: C:\VSTSAgent\_work\7\s
[2017-11-27 14:24:29Z INFO TaskRunner] Build JobExtension resolved a rooted path:: C:\VSTSAgent\_work\7\s
[2017-11-27 14:24:29Z INFO NodeHandler] Inspect node_modules folder, make sure vsts-task-lib doesn't overwrite String.startsWith/endsWith.
[2017-11-27 14:24:29Z INFO NodeHandler] This task has already been scanned and corrected, no more operation needed.
[2017-11-27 14:24:29Z INFO HostContext] Well known directory 'Bin': 'C:\VSTSAgent\bin'
[2017-11-27 14:24:29Z INFO HostContext] Well known directory 'Root': 'C:\VSTSAgent'
[2017-11-27 14:24:29Z INFO HostContext] Well known directory 'Externals': 'C:\VSTSAgent\externals'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Starting process:
[2017-11-27 14:24:29Z INFO ProcessInvoker] File name: 'C:\VSTSAgent\externals\node\bin\node.exe'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Arguments: '"C:\VSTSAgent\_work\_tasks\Npm_fe47e961-9fa8-4106-8639-368c022d43ad\1.0.8\npm.js"'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Working directory: 'C:\VSTSAgent\_work\7\s'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Require exit code zero: 'True'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Encoding web name: utf-8 ; code page: '65001'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Force kill process on cancellation: 'False'
[2017-11-27 14:24:29Z INFO ProcessInvoker] Process started with process id 4124, waiting for process exit.
[2017-11-27 14:24:29Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.
[2017-11-27 14:24:30Z INFO JobServerQueue] Try to upload 1 log files or attachments, success rate: 1/1.
[2017-11-27 14:24:32Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.
[2017-11-27 14:24:33Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.
[2017-11-27 14:24:34Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.
[2017-11-27 14:24:34Z INFO JobServerQueue] Try to append 1 batches web console lines, success rate: 1/1.
[2017-11-27 14:31:34Z INFO Worker] Cancellation/Shutdown message received.