I'm trying to create a Substfile rule that will expand a key to the transformed contents of another file. I'm not clear on the setup here to ensure the source file is registered as a dependency.
Logically I want something like:
out = env.Substfile( 'file.in', SUBST_DICT = {
'%SOME_CONTENT%': transform( readfile('depends.txt') ),
}
I'm using a combination of Action and Command to do what I want. I ended up not using Substfile, though it could probably be chained to the command.
This RawStringIt action loads a text file and emits a C++ encoded raw string for the content.
def RawStringIt(varName):
def Impl(target, source, env):
content = source[0].get_text_contents()
with open(target[0].get_path(), 'w') as target_file:
target_file.write( "std::string {} = R\"~~~~({})~~~~\";".format(varName,content))
return 0
return Action(Impl, "creating C++ Raw String $TARGET from $SOURCE" )
base_leaf = env.Command( 'include/runner/base.leaf.hpp', '../share/base.leaf', RawStringIt("dataBaseLeaf") )
Related
Artifactory Version: 5.8.4
In Artifactory, files are stored in the internal database via file's checksum (SHA1) and for retrieval purposes, SHA-256 is useful (for verifying if file is intact).
Read this first: https://www.jfrog.com/confluence/display/RTF/Checksum-Based+Storage
Let's say there are 2 Jenkins jobs, which creates few artifacts/file (rpm/jar/etc). In my case, I'll take a simple .txt file which stores date in MM/DD/YYYY format and some other jobA/B specific build result files (jars/rpms etc).
If we focus only on the text file (as I mentioned above), then:
Jenkins_jobA > generates jobA.date_mm_dd_yy.txt
Jenkins_jobA > generates jobB.date_mm_dd_yy.txt
Jenkins jobA and jobB run multiple times per day in no given run order. Sometime jobA runs first and sometime jobB.
As the content of the file are mostly same for both jobs (per day), the SHA-1 value on jobA's .txt file and jobB.txt file will be same i.e. in Artifactory, both files will be stored in the first 2 character based directry folder structure (as per the Check-sum based storage mechanism).
Basically running sha1sum and sha256sum on both files in Linux, would return the exact same output.
Over the time, these artifacts (.txt, etc) gets promoted from one repository to another (promotion process i.e. from snapshot -> stage -> release repo) so my current logic written in Groovy is to find the URI of the artifact sitting behind a "VIRTUAL" repository (which contains a set of physical local repositories in some order) is listed below:
// Groovy code
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
jsonSlurper = new JsonSlurper()
// The following function will take artifact.SHA_256 as it's parameter to find URI of the artifact
def checkSumBasedSearch(artifactSha) {
virt_repo = "jar-repo" // this virtual may have many physical repos release/stage/snapshot for jar(maven) or it can be a YUM repo for (rpm) or generic repo for (.txt file)
// Note: Virtual repos don't span different repo types (i.e. a virtual repository in Artifactory for "Maven" artifacts (jar/war/etc) can NOT see YUM/PyPi/Generic physical Repos).
// Run aqlCmd on Linux, requires "...", "..", "..." for every distinctive words / characters in the cmd line.
checkSum_URL = artifactoryURL + "/api/search/checksum?sha256="
aqlCmd = ["curl", "-u", username + ":" + password, "${checkSum_URL}" + artifactSha + "&repos=" + virt_repo]
}
def procedure = aqlCmd.execute()
def standardOut = new StringBuilder(), standardErr = new StringBuilder()
procedure.waitForProcessOutput(standardOut, standardErr)
// Fail early
if (! standardErr ) {
println "\n\n-- checkSumBasedSearch() - standardErr exists ---\n" + standardErr +"\n\n-- Exiting with error 12!!!\n\n"
System.exit(12)
}
def obj = jsonSlurper.parseText(standardOut.toString())
def results = obj.results
def uri = results[0].uri // This would work, if a file's sha-1 /256 is always different or sits in different repo at least.
return uri
// to get the URL, I can use:
//aqlCmd = ["curl", "-u", username + ":" + password, "${uri}"]
//def procedure = aqlCmd.execute()
//def obj = jsonSlurper.parseText(standardOut.toString())
//def url = obj.downloadUri
//return url
//aqlCmd = [ "curl", "-u", username + ":" + password, "${url}", "-o", somedirectory + "/" + variableContainingSomeArtifactFilenameThatIWant ]
//
// def procedure = aqlCmd.execute()
//def standardOut = new StringBuilder(), standardErr = new StringBuilder()
//procedure.waitForProcessOutput(standardOut, standardErr)
// Now, I'll get the artifact downloaded in some Directory as some Filename.
}
My concern is, as both files (even though different name -or file-<versioned-timestamp>.txt) have same content in them and generated multiple times per day, how can I get a specific versioned file downloaded for jobA or jobB?
In Artifactory, the SHA_256 property for all files containing same content will be same!! (Artifactory will use SHA-1 for storing these files efficiently to save space, new uploads will be just minimal database level transactions transparent to the user).
Questions:
Will the above logic return jobA's file or jobB's .txt file or any Job's .txt file which uploaded it's file first or latest/acc. to LastModified -aka- last upload time?
How can I get jobA's .txt file and jobB's .txt file downloaded for a given timestamp?
Do I need to add more properties during my rest Api call?
If I was just concerned for the file content, then it doesn't matter much (sha-1/256 dependent) whether it's coming from JobA .txt or job's .txt file, but in a complex case, one may have file name containing meaningful info that they'd like to know to find which file was download (A / B)!
You can use AQL (Artifactory Query Langueage)
https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language
curl -u<username>:<password> -XPOST https://repo.jfrog.io/artifactory/api/search/aql -H "Content-Type: text/plain" -T ./search
The content of the file named search is:
items.find(
{
"artifact.module.build.name":{"$eq":"<build name>"},
"artifact.sha1":"<sha1>"
}
)
The above logic (in the original question) will return one of them arbitrary, since you are taking the first result returned and there is no guarantee on the order.
Since your text file contains the timestamp in the name, then you can add the name to the aql given above, it will also filter by the name.
AQL search API is more flexible than the checksum search, use it and customise your query according to the parameters you need.
So, I ended up doing this instead of just returning [0]th element from array in every case.
// Do NOT return [0] first element as yet as Artifactory uses SHA-1/256 so return [Nth].uri where artifact's full name matches with the sha256
// def uri = results[0].uri
def nThIndex=0
def foundFlag = 'false'
for (r in results) {
println "> " + r.uri + " < " + r.uri.toString() + " artifact: " + artFullName
if ( r.uri.toString().contains(artFullName) ) {
foundFlag = 'true'
println "- OK - Found artifact: " + artFullName + " at results[" + nThIndex + "] index."
break; // i.e. a match for the artifact name with SHA-256 we want - has been found.
} else {
nThIndex++;
}
}
if ( foundFlag == 'true' ) {
def uri = results[nThIndex].uri
return uri
} else {
// Fail early if results were found based on SHA256 but not for the artifact but for some other filename with same SHA256
if (! standardErr ) {
println "\n\n\n\n-- [Cool] -- checkSum_Search() - SHA-256 unwanted situation occurred !!! -- results Array was set with some values BUT it didn't contain the artifact (" + artFullName + ") that we were looking for \n\n\n-- !!! Artifact NOT FOUND in the results array during checkSum_Search()---\n\n\n-- Exiting with error 17!!!\n\n\n\n"
System.exit(17) // Nooka
}
}
How can I use groovy to keep the content of SQL Queries in SoapUI in sync with an external editor?
The solution might look like this:
Groovy script to export all the queries of a SoapUI TestSuite or
TestStep into SQL file(s).
Edit and save SQL with external editor.
Groovy script to update the queries in SoapUI based on changed
files.
Initial issues:
How do I access the query of a teststep? It is not there as a
porperty, is it?
Is there a way to run steps 1 and 3 on a project file (XML) instead
from within a test itself (as setUp/tearDown scripts)?
Motivation
SQL Query input fields of JDBC test steps
very small and
does not provide any code formatting like indenting, re-wrapping, or upper-casing of SQL keywords (there is just syntax highlighting).
This is IMHO very cumbersome when writing a query that contains more than a couple of where clauses or even joins.
Side note: If somebody could point me to some functionality (builtin, plugin?) to format the SQL code directly in SoapUI (not pro!), I would gladly pass on groovy scripts.
These are my groovy-scripts that I have implemented in the form of Groovy test steps (they might also be setUp/tearDown scripts, but I prefer separate test steps that are clearly visible and where I can easily toggle their activity):
Export all the JDBC/SQL queries of a SoapUI test case into SQL file(s). This might be the final step of the testcase.
def testCase = testRunner.testCase
def testSuite = testCase.testSuite
def project = testSuite.project
// Remove .xml extension from filename
String pathToProjectDir = project.path.replaceAll(~/\.\w+$/, '')
File projectDir = new File(pathToProjectDir)
File suiteDir = new File(projectDir, testSuite.name)
File caseDir = new File(suiteDir, testCase.name)
caseDir.mkdirs()
assert caseDir.exists()
assert caseDir.isDirectory()
log.info "Exporting to '${caseDir}'."
testCase.getTestStepsOfType(com.eviware.soapui.impl.wsdl.teststeps.JdbcRequestTestStep)
.each{testStep ->
String filename = "${testStep.name}.sql"
File file = new File(caseDir, filename)
file.text = testStep.query
log.info "'${filename}' written."
}
log.info "Files written."
Edit and save SQL with external editor.
Update the queries in SoapUI based on changed files. Missing or empty files are ignored in order not to break too much in the existing project.
def testCase = testRunner.testCase
def testSuite = testCase.testSuite
def project = testSuite.project
// Remove .xml extension from filename
String pathToProjectDir = project.path.replaceAll(~/\.\w+$/, '')
File projectDir = new File(pathToProjectDir)
File suiteDir = new File(projectDir, testSuite.name)
File caseDir = new File(suiteDir, testCase.name)
assert caseDir.exists()
assert caseDir.isDirectory()
log.info "Importing from '${caseDir}'."
testCase.getTestStepsOfType(com.eviware.soapui.impl.wsdl.teststeps.JdbcRequestTestStep)
.each{testStep ->
String filename = "${testStep.name}.sql"
File file = new File(caseDir, filename)
if (file.exists()) {
if (file.text) {
testStep.query = file.text
log.info "'${filename}'"
} else {
log.warn "Ignoring '${filename}'"
}
} else {
log.warn "'${filename}' does not exist."
}
}
log.info "Files imported."
The files are placed in the following directory structure:
Starting from the SoapUI project file path/to/project.xml itself, a tree is (created and) populated with one SQL file per JDBC test step:
path/to/project/${name of TestSuite}/${name of TestCase}/${name of TestStep}.sql
-Is there any way to pass Global Properties from external file or something?
-I don't want to navigate Preference-Global Properties and changing the values.
-Is there any alternative way to do this.
Thanks,
Arivazhagan
You can parse external file in groovy script Step, for example *.csv file with values into the local groovy variables, and then set value in Properties of Test Suite or Test Case or Global Properties too.
Example of parsing *.csv file:
def testDataSet = []
def index = testRunner.testCase.getPropertyValue("index")
int indx = index.toInteger()
def fileName = "phoneNumbers.csv"
//read from file
new File(fileName).eachLine { line -> testDataSet.add( line.split(";") ) }
log.info( "Read " + testDataSet.size() + " test values from " + fileName )
//convert value to properties
def testDataLine = testDataSet[indx]
phoneNumber = testDataLine[0].value as String
log.info phoneNumber
Example of set property:
testRunner.testCase.setPropertyValue("phoneNumber", phoneNumber)
indx++
String indexString = Integer.toString(indx)
testRunner.testCase.setPropertyValue("index", indexString)
Example of set Global Preporty:
globalProperty = com.eviware.soapui.SoapUI.globalProperties.getPropertyValue( "MyProp" )
more info provide here https://www.soapui.org/scripting-properties/tips-tricks.html
Global properties are stored in soapui-settings.xml in the user.dir. If you change it, it will be recognized by readyApi/soapUI.
I need to copy a file in Groovy and saw some ways to achieve it on the web:
1
new AntBuilder().copy( file:"$sourceFile.canonicalPath",
tofile:"$destFile.canonicalPath")
2
command = ["sh", "-c", "cp src/*.txt dst/"]
Runtime.getRuntime().exec((String[]) command.toArray())
3
destination.withDataOutputStream { os->
source.withDataInputStream { is->
os << is
}
}
4
import java.nio.file.Files
import java.nio.file.Paths
Files.copy(Paths.get(a), Paths.get(b))
The 4th way seems cleanest to me as I am not sure how good is it to use AntBuilder and how heavy it is, I saw some people reporting issues with Groovy version change.
2nd way is OS dependent, 3rd might not be efficient.
Is there something in Groovy to just copy files like in the 4th statement or should I just use Java for it?
If you have Java 7, I would definitely go with
Path source = ...
Path target = ...
Files.copy(source, target)
With the java.nio.file.Path class, it can work with symbolic and hard links. From java.nio.file.Files:
This class consists exclusively of static methods that operate on
files, directories, or other types of files. In most cases, the
methods defined here will delegate to the associated file system
provider to perform the file operations.
Just as references:
Copy files from one folder to another with Groovy
http://groovyconsole.appspot.com/view.groovy?id=8001
My second option would be the ant task with AntBuilder.
If you are doing this in code, just use something like:
new File('copy.bin').bytes = new File('orig.bin').bytes
If this is for build-related code, this would also work, or use the Ant builder.
Note, if you are sure the files are textual you can use .text rather than .bytes.
If it is a text file, I would go with:
def src = new File('src.txt')
def dst = new File('dst.txt')
dst << src.text
I prefer this way:
def file = new File("old.file")
def newFile = new File("new.file")
Files.copy(file.toPath(), newFile.toPath())
To append to existing file :
def src = new File('src.txt')
def dest = new File('dest.txt')
dest << src.text
To overwrite if file exists :
def src = new File('src.txt')
def dest = new File('dest.txt')
dest.write(src.text)
I'm using AntBuilder for such tasks. It's simple, consistent, 'battle-proven' and fun.
2nd approach is too OS-specific (Linux-only in your case)
3rd it too low-level and it eats up more resources. It's useful if you need to transform the file on the way: change encoding for example
4th looks overcomplicated to me... NIO package is relatively new in JDK.
In the end of the day, I'd go for 1st option. There you can switch from copy to scp task, without re-developing the script almost from scratch
This is the way using platform independent groovy script. If anyone has questions please ask in the comments.
def file = new File("java/jcifs-1.3.18.jar")
this.class.classLoader.rootLoader.addURL(file.toURI().toURL())
def auth_server = Class.forName("jcifs.smb.NtlmPasswordAuthentication").newInstance("domain", "username", "password")
def auth_local = Class.forName("jcifs.smb.NtlmPasswordAuthentication").newInstance(null, "local_username", "local_password")
def source_url = args[0]
def dest_url = args[1]
def auth = auth_server
//prepare source file
if(!source_url.startsWith("\\\\"))
{
source_url = "\\\\localhost\\"+ source_url.substring(0, 1) + "\$" + source_url.substring(1, source_url.length());
auth = auth_local
}
source_url = "smb:"+source_url.replace("\\","/");
println("Copying from Source -> " + source_url);
println("Connecting to Source..");
def source = Class.forName("jcifs.smb.SmbFile").newInstance(source_url,auth)
println(source.canRead());
// Reset the authentication to default
auth = auth_server
//prepare destination file
if(!dest_url.startsWith("\\\\"))
{
dest_url = "\\\\localhost\\"+ dest_url.substring(0, 1) + "\$" +dest_url.substring(2, dest_url.length());
auth = auth_local
}
def dest = null
dest_url = "smb:"+dest_url.replace("\\","/");
println("Copying To Destination-> " + dest_url);
println("Connecting to Destination..");
dest = Class.forName("jcifs.smb.SmbFile").newInstance(dest_url,auth)
println(dest.canWrite());
if (dest.exists()){
println("Destination folder already exists");
}
source.copyTo(dest);
For copying files in Jenkins Groovy
For Linux:
try {
echo 'Copying the files to the required location'
sh '''cd /install/opt/
cp /install/opt/ssl.ks /var/local/system/'''
echo 'File is copied successfully'
}
catch(Exception e) {
error 'Copying file was unsuccessful'
}
**For Windows:**
try {
echo 'Copying the files to the required location'
bat '''#echo off
copy C:\\Program Files\\install\\opt\\ssl.ks C:\\ProgramData\\install\\opt'''
echo 'File is copied successfully'
}
catch(Exception e) {
error 'Copying file was unsuccessful'
}
At the moment I'm using some magic to get the current git revision into my scons builds.. I just grab the version a stick it into CPPDEFINES.
It works quite nicely ... until the version changes and scons wants to rebuild everything, rather than just the files that have changed - becasue the define that all files use has changed.
Ideally I'd generate a file using a custom builder called git_version.cpp and
just have a function in there that returns the right tag. That way only that one file would be rebuilt.
Now I'm sure I've seen a tutorial showing exactly how to do this .. but I can't seem to track it down. And I find the custom builder stuff a little odd in scons...
So any pointers would be appreciated...
Anyway just for reference this is what I'm currently doing:
# Lets get the version from git
# first get the base version
git_sha = subprocess.Popen(["git","rev-parse","--short=10","HEAD"], stdout=subprocess.PIPE ).communicate()[0].strip()
p1 = subprocess.Popen(["git", "status"], stdout=subprocess.PIPE )
p2 = subprocess.Popen(["grep", "Changed but not updated\\|Changes to be committed"], stdin=p1.stdout,stdout=subprocess.PIPE)
result = p2.communicate()[0].strip()
if result!="":
git_sha += "[MOD]"
print "Building version %s"%git_sha
env = Environment()
env.Append( CPPDEFINES={'GITSHAMOD':'"\\"%s\\""'%git_sha} )
You don't need a custom Builder since this is just one file. You can use a function (attached to the target version file as an Action) to generate your version file. In the example code below, I've already computed the version and put it into an environment variable. You could do the same, or you could put your code that makes git calls in the version_action function.
version_build_template="""/*
* This file is automatically generated by the build process
* DO NOT EDIT!
*/
const char VERSION_STRING[] = "%s";
const char* getVersionString() { return VERSION_STRING; }
"""
def version_action(target, source, env):
"""
Generate the version file with the current version in it
"""
contents = version_build_template % (env['VERSION'].toString())
fd = open(target[0].path, 'w')
fd.write(contents)
fd.close()
return 0
build_version = env.Command('version.build.cpp', [], Action(version_action))
env.AlwaysBuild(build_version)