I am trying to use threads inside my own plugin. I need to deploy 5 files into Artifactory simultaneously to save time.
baseDir.eachFile {
def Thread thread = new Thread(){
public void run(){
def InputStream fileAsInputStream = new File (it.getAbsolutePath()).newInputStream()
def File fileRepoPath = new File (baseDirInRepo, it.getName())
if(!deployRepoMetadata(item.repoPath.repoKey, fileRepoPath, fileAsInputStream)) {
throw new Exception ("Can't deploy " + fileRepoPath + " for package " + item.name)
}
}
}
thread.start()
}
def boolean deployRepoMetadata(String repoKey, File deployPath, InputStream binary) {
def String targetRepoKey = repoKey
def String targetPath = deployPath
def RepoPath deployRepoPath = RepoPathFactory.create(targetRepoKey, targetPath)
repositories.deploy(deployRepoPath, binary)
}
But I've got the following error:
User _system_ is not permitted to deploy
How can I deal with this error?
Related
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def build_log = new URL (Build_Log_URL).getText()
def found_errors = null
for(knownError in knownErrorListbyLine) {
if (build_log.contains(knownError)) {
found_errors = build_log.readLines().findAll{ it.contains(knownError) }
for(error in found_errors) {
println "FOUND ERROR: " + error
}
}
}
I wrote this code to find listed errors in a string, but it takes about 20 seconds.
How can I improve the performance? I would love to learn from this.
Thanks a lot!
list.txt contains a string per line:
Step ... was FAILED
[ERROR] Pod-domainrouter call failed
#type":"ErrorExtender
[postDeploymentSteps] ... does not exist.
etc...
And build logs is where I need to find these errors.
Try this:
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def build_log = new URL (Build_Log_URL)
def found_errors = null
for(knownError in knownErrorListbyLine) {
build_log.eachLine{
if ( it.contains(knownError) ) {
println "FOUND ERROR: " + error
}
}
}
This might be even more performant:
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def build_log = new URL (Build_Log_URL)
def found_errors = null
build_log.eachLine{
for(knownError in knownErrorListbyLine) {
if ( it.contains(knownError) ) {
println "FOUND ERROR: " + error
}
}
}
Attempt using the last one relying on string eachLine instead.
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def build_log = new URL (Build_Log_URL).getText()
def found_errors = null
build_log.eachLine{
for(knownError in knownErrorListbyLine) {
if ( it.contains(knownError) ) {
println "FOUND ERROR: " + error
}
}
}
Try to move build_log.readLines() to the variable outside of the loop.
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def build_log = new URL (Build_Log_URL).getText()
def found_errors = null
def buildLogByLine = build_log.readLines()
for(knownError in knownErrorListbyLine) {
if (build_log.contains(knownError)) {
found_errors = buildLogByLine.findAll{ it.contains(knownError) }
for(error in found_errors) {
println "FOUND ERROR: " + error
}
}
}
Update: using multiple threads
Note: this may help in case errorList size is large enough. And also if the matching errors distributed evenly.
def sublists = knownErrorListbyLine.collate(x)
// int x - the sublist size,
// depends on the knownErrorListbyLine size, set the value to get e. g. 4 sublists (threads).
// Also do not use more than 2 threads per CPU. Start from 1 thread per CPU.
def logsWithErrors = []// list for store results per thread
def lock = new Object()
def threads = sublists.collect { errorSublist ->
Thread.start {
def logs = build_log.readLines()
errorSublist.findAll { build_log.contains(it) }.each { error ->
def results = logs.findAll { it.contains(error) }
synchronized(lock) {
logsWithErrors << results
}
}
}
}
threads*.join() // wait for all threads to finish
logsWithErrors.flatten().each {
println "FOUND ERROR: $it"
}
Also, as was suggested earlier by other user, try to measure the logs download time, it could be the bottleneck:
def errorList = readFile WORKSPACE + "/list.txt"
def knownErrorListbyLine = errorList.readLines()
def start = Calendar.getInstance().timeInMillis
def build_log = new URL(Build_Log_URL).getText()
def end = Calendar.getInstance().timeInMillis
println "Logs download time: ${(end-start)/1000} ms"
def found_errors = null
I try to create a zip file within a shared library for jenkins pipeline with the help of AntBuilder. A simple zip file can be created, but as soon as I try to use a block it does not work. I don't get any error message or Exceptions. How can I debug a Pipepeline-Script? Or how can I solve the issue?
My Code looks as follow (step zipFolder doesn't work, step zipFolder2 does work)
Jenkinsfile
#Library('utils') _
pipeline {
agent any
stages {
stage('Build') {
steps {
zipFolder( )
zipFolder2( )
}
}
}
}
shared library:
vars/zipFolder.groovy:
import com.example.Utilities
def call(){
new Utilities(this).zip(pwd())
}
vars/zipFolder2.groovy
import com.example.Utilities
def call(){
new Utilities(this).zip2(pwd())
}
src/com/example/Utilities.groovy
package com.example
import groovy.util.AntBuilder
import org.apache.tools.ant.DefaultLogger
class Utilities implements Serializable {
def steps
def byteStream
Utilities(steps) {this.steps = steps}
def zip(baseDir){
def ant = setupAnt(baseDir)
def destFolder = "${baseDir}/Dist"
def destfile = "${destFolder}/test.zip"
ant.delete dir: destFolder
ant.mkdir(dir: destFolder)
ant.zip(destfile: destfile, whenempty: 'create', excludes: destfile) {
zipfileset (dir: "${baseDir}/install", includes: "test.txt", erroronmissingdir: false)
}
steps.echo "Zip1"
steps.echo "Ant-Result: " + byteStream.toString()
}
def zip2(baseDir){
def ant = setupAnt(baseDir)
def destFolder = "${baseDir}/Dist2"
def destfile = "${destFolder}/test.zip"
ant.delete dir: destFolder
ant.mkdir(dir: destFolder)
ant.zip(destfile: destfile, whenempty: 'create', excludes: destfile, basedir: baseDir)
steps.echo "Zip2"
steps.echo "Ant-Result: " + byteStream.toString()
}
def setupAnt(baseDir){
def ant = new AntBuilder()
byteStream = new ByteArrayOutputStream()
def printStream = new PrintStream( byteStream )
def project = ant.project
project.buildListeners.each {
if ( it instanceof DefaultLogger ) {
it.setMessageOutputLevel(org.apache.tools.ant.Project.MSG_DEBUG)
it.setOutputPrintStream printStream
it.setErrorPrintStream printStream
}
}
ant
}
}
AntBuilder is not Serializable. What we did is encapsulate the call to AntBuilder inside an extra groovy script which is created on the fly and called by calling the groovy binary like:
writeFile file:createZip.groovy text:”””
def antBuilder = new AntBuilder()
...
“””
sh ‘groovy createZip.groovy’
You could also consider using the Jenkins Pipeline zip step. See https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#zip-create-zip-file
With the hint of Joerg S I solved the issue with the following Code:
src/com/example/Utilities.groovy
package com.example
import org.apache.tools.ant.DefaultLogger
package com.example
import org.apache.tools.ant.DefaultLogger
class Utilities implements Serializable {
def steps
def byteStream
Utilities(steps) {this.steps = steps}
def zip(baseDir, Closure callback ){
def ant = setupAnt()
def destFolder = "${baseDir}/Dist"
def destfile = "${destFolder}/test.zip"
ant.delete dir: destFolder
ant.mkdir(dir: destFolder)
ant.zip(destfile: destfile, whenempty: 'create', excludes: destfile, callback)
steps.echo "Ant-Result: " + byteStream.toString()
}
def setupAnt(){
def ant = new AntBuilder()
byteStream = new ByteArrayOutputStream()
def printStream = new PrintStream( byteStream )
def project = ant.project
project.buildListeners.each {
if ( it instanceof DefaultLogger ) {
it.setMessageOutputLevel(org.apache.tools.ant.Project.MSG_DEBUG)
it.setOutputPrintStream printStream
it.setErrorPrintStream printStream
}
}
ant
}
}
vars/zipFolder.groovy
import com.example.Utilities
def call(Closure callback){
new Utilities(this).zip(pwd(), callback)
}
Jenkinsfile
#Library('utils') _
pipeline {
agent any
stages {
stage('Build') {
steps {
cleanWs()
writeFile file: 'test.txt', text: 'Sample Text'
writeFile file: 'test2.txt', text: 'Sample Text 2'
writeFile file: 'src/test3.txt', text: 'Sample Text 3'
zipIt()
}
}
}
}
#NonCPS
def zipIt(){
zipFolder( ) {
delegate.zipfileset (dir: "${pwd()}", includes: "test.txt", excludes: "src/**" , filemode: "0755", prefix: "bin/examples", erroronmissingdir: false)
delegate.zipfileset (dir: "${pwd()}/src", includes: "*.txt", erroronmissingdir: false)
}
}
Important is the use of the annotation #NonCPS and delegate.zipfileset instead of simply using zipfileset.
Edit: This solution does not work on a slave. This code is executed at the master server, even if the build is run on a slave.
I need to call a method inside Jar from groovy script inside SoapUI project.
Due to the lack of administrative access I can't place that jar in "../bin/ext" folder in SaopUI intall directory.
So the only option left is load the jar in runtime and call the method. Very simple approach.
I tried the below approach.
this.class.classLoader.rootLoader.addURL(new URL("file:///H://Foo-2//Foo.jar"));
def cls = Class.forName("test").newInstance();
cls.add()
This is not working as rootLoader is null.
second approach .
def classLoader = ClassLoader.systemClassLoader
def newClassLoader = new URLClassLoader([new File("file:///H://Foo-2//Foo.jar")
.toString().toURL()] as URL[], classLoader)
def cls = Class.forName("test").newInstance();
this is not working too , it's giving me ClassNotFoundException.
I spent a day in this. even change the class name to lower case after seeing this thread.
Edit 1
I tried this too. and change my code like this.
def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )
def classpathHacker = new com.eviware.soapui.support.ClasspathHacker ()
log.info "utils=" + groovyUtils
mystring = "file://H://Baz.jar"
com.eviware.soapui.support.ClasspathHacker.addURL( new URL(mystring) )
def cls = new bar() // how to call the static method add of `bar` class ?
log.info cls
My Jar code is too simple. Here it is
public class bar {
public static void main(String[] args) {
add();
}
public static void add(){
String path = "H:" + File.separator + "Groovy" + File.separator + "hi.txt";
File f = new File(path);
f.getParentFile().mkdirs();
try {
f.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
}
What all the other option I have? What i am doing wrong?
Solved. Here is the final Groovy script.
def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )
def classpathHacker = new com.eviware.soapui.support.ClasspathHacker ()
path = groovyUtils.getProjectPath()
myfile = new java.io.File(path + "/Baz.jar")
mystring = "file://" + path + "/Baz.jar"
log.info "myfile=" + myfile
classpathHacker.addFile( myfile )
com.eviware.soapui.support.ClasspathHacker.addFile( myfile )
com.eviware.soapui.support.ClasspathHacker.addURL( new URL(mystring) )
//import Baz
def instance = this.class.classLoader.loadClass( 'bar', true, false )?.newInstance()
instance.add();
Do you know about com.eviware.soapui.support.ClasspathHacker?
Maybe that is away to go about it, if you really cannot put it to the /ext folder.
Reference:
https://community.smartbear.com/t5/SoapUI-Open-Source/Soapui-is-not-loading-external-jar-file-location-added-to/td-p/7619
def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )
def classpathHacker = new com.eviware.soapui.support.ClasspathHacker ()
log.info "utils=" + groovyUtils
path = groovyUtils.getProjectPath()
myfile = new java.io.File(path + "/ojdbc14.jar")
mystring = "file://" + path + "/ojdbc14.jar"
log.info "myfile=" + myfile
classpathHacker.addFile( myfile )
com.eviware.soapui.support.ClasspathHacker.addFile( myfile )
com.eviware.soapui.support.ClasspathHacker.addURL( new URL(mystring) )
import groovy.sql.*
def sql = groovy.sql.Sql.newInstance( db, userid, password, 'oracle.jdbc.driver.OracleDriver' )
I want to test the following class:
public class DBSync {
public dbNotify( String path ) {
if (!path) {
return
}
def pathIndex = path.lastIndexOf(File.separator)
if (pathIndex > 0) {
def folder = path[0..pathIndex - 1]
def fileName = path[pathIndex + 1..path.length() - 1]
println "Syncing from directory $folder for filename $fileName"
if (fileName.contains(EXCLUDE_FILE_PATTERN)) {
println "Filename is $EXCLUDE_FILE_PATTERN skipping db write "
return
}
writeToDB(folder, fileName)
}
}
public writeToDB ( folder, file ) {
// inserting to database
}
}
The test class is:
public class TestDBSync {
#Test
public void test() {
def dbSyncContext = new MockFor(DBSync)
def file = "file.zip"
def folder = "data"
def path = folder + File.separator + file
def called = false
// Expect at least one call
mockDBSyncContext.demand.writeToDB(1..1) { String folderargs, String fileargs -> called = true }
mockDBSyncContext.demand.dbNodify(1..1) { String pathargs -> return }
// Obtaining a usuable mock instance
def mockDBSyncProxy = mockDBSyncContext.proxyInstance()
// Fake calling the method
mockDBSyncContext.use {
mockDBSyncProxy.dbNotify(path )
}
// Verify invoked at least once?
mockDBSyncContext.verify(mockDBSyncProxy)
}
}
The test is failing and I am getting the following error:
junit.framework.AssertionFailedError: No call to 'dbNotify' expected
at this point. Still 1 call(s) to 'writeToDB' expected.
I would like to update the contents of text file located inside a zipfile.
I cannot find out how to do this, and the code below is not working properly.
May thanks for any help!!
import java.util.zip.ZipFile
import java.util.zip.ZipEntry
import java.util.zip.ZipOutputStream
String zipFileFullPath = "C:/path/to/myzipfile/test.zip"
ZipFile zipFile = new ZipFile(zipFileFullPath)
ZipEntry entry = zipFile.getEntry ( "someFile.txt" )
if(entry){
InputStream input = zipFile.getInputStream(entry)
BufferedReader br = new BufferedReader(new InputStreamReader(input, "UTF-8"))
String s = null
StringBuffer sb = new StringBuffer()
while ((s=br.readLine())!=null){
sb.append(s)
}
sb.append("adding some text..")
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipFileFullPath))
out.putNextEntry(new ZipEntry("someFile.txt"));
int length
InputStream fin = new ByteArrayInputStream(sb.toString().getBytes("UTF8"))
while((length = fin.read(sb)) > 0)
{
out.write(sb, 0, length)
}
out.closeEntry()
}
Just some slight modifications to #Opal's answer, I've just:
used groovy methods where possible
packaged in a method
Groovy Snippet
void updateZipEntry(String zipFile, String zipEntry, String newContent){
def zin = new ZipFile(zipFile)
def tmp = File.createTempFile("temp_${System.nanoTime()}", '.zip')
tmp.withOutputStream { os ->
def zos = new ZipOutputStream(os)
zin.entries().each { entry ->
def isReplaced = entry.name == zipEntry
zos.putNextEntry(isReplaced ? new ZipEntry(zipEntry) : entry)
zos << (isReplaced ? newContent.getBytes('UTF8') : zin.getInputStream(entry).bytes )
zos.closeEntry()
}
zos.close()
}
zin.close()
assert new File(zipFile).delete()
tmp.renameTo(zipFile)
}
Usage
updateZipEntry('/tmp/file.zip', 'META-INF/web.xml', '<foobar>new content!</foobar>')
What exactly isn't working? Is there any exception thrown?
As far as I know it's not possible to modify a zip file in situ. The following script rewrites the file and if desired entry is processed - modifies it.
import java.util.zip.*
def zipIn = new File('lol.zip')
def zip = new ZipFile(zipIn)
def zipTemp = File.createTempFile('out', 'zip')
zipTemp.deleteOnExit()
def zos = new ZipOutputStream(new FileOutputStream(zipTemp))
def toModify = 'lol.txt'
for(e in zip.entries()) {
if(!e.name.equalsIgnoreCase(toModify)) {
zos.putNextEntry(e)
zos << zip.getInputStream(e).bytes
} else {
zos.putNextEntry(new ZipEntry(toModify))
zos << 'lollol\n'.bytes
}
zos.closeEntry()
}
zos.close()
zipIn.delete()
zipTemp.renameTo(zipIn)
UPDATE
I wasn't right. It's possible to modify zip file in situ, but Your solution will omit other files that were zipped. The output file will contain only one single file - the file You wanted to modify. I also suppose that You file was corrupted because of not invoking close() on out.
Below is You script slightly modified (more groovier):
import java.util.zip.*
def zipFileFullPath = 'lol.zip'
def zipFile = new ZipFile(zipFileFullPath)
def entry = zipFile.getEntry('lol.txt')
if(entry) {
def input = zipFile.getInputStream(entry)
def br = new BufferedReader(new InputStreamReader(input, 'UTF-8'))
def sb = new StringBuffer()
sb << br.text
sb << 'adding some text..'
def out = new ZipOutputStream(new FileOutputStream(zipFileFullPath))
out.putNextEntry(new ZipEntry('lol.txt'))
out << sb.toString().getBytes('UTF8')
out.closeEntry()
out.close()
}