I have an issue when creating a performance script, which has related to reading data from csv data config.
My script has a structure below:
setup Thread
Create csv Thread. After view dashboard, using Json extractor to get list of data and put it to csv file
Create csv file - After this thread, i will have a lot of csv file base on number of center. For example: 4 files with different names
String[] attempt = (vars.get("ListAttemptId_ALL")).split(",");
int length = attempt.length;
String dir = props.get("UserFilePath").toString();
String center = vars.get("Center");
File csvFile = new File(dir, center + ".csv");
if(!csvFile.exists()){
FileWriter fstream = new FileWriter(csvFile);
BufferedWriter out = new BufferedWriter(fstream);
for(int i = 1; i <= length; i++){
out.write(attempt[i-1]);
out.write(System.getProperty("line.separator"));
}
out.close();
fstream.close();
}
Next thread gets the name of the file and using CSV file to loop over
all line
String center = vars.get("Center");
String fileName = center + ".csv";
props.put("path_${__threadNum}", String.valueOf(fileName));
Because i have alot of threads will run the same file, so i just check __threadNum of to find the name of the file this thread need to use.
I'm using loop Controller to go over CSV file, run to the end of the file will stop thread. Here is inside this loop
CSV data Set config:
Filename: ${__property(UserFilePath)}\\${__P(path_${__threadNum})}
where ${__property(UserFilePath)} = path of the folder and
${__P(path_${__threadNum})} is name of the csv file were extracted
My issue is this code is not stable, sometimes threads can read file normally, sometimes it's show error that file does not exist (actually it did) so it's hard to chase that where issue from. Can anyone suggest a solution to my issue? Or suggest any idea better than my solution to read csv file in thread group?
I have answer for this issue:
- I add all data AttemptId, Center to one file csv and read from beginning to and end. Using If controller to verify data before action.
This statement can be problematic:
props.put("path_${__threadNum}", String.valueOf(fileName));
as per JSR223 Sampler documentation
JMeter processes function and variable references before passing the script field to the interpreter, so the references will only be resolved once. Variable and function references in script files will be passed verbatim to the interpreter, which is likely to cause a syntax error. In order to use runtime variables, please use the appropriate props
methods, e.g.
props.get("START.HMS");
props.put("PROP1","1234");
So I would recommend replacing ${__threadNum} with ctx.getThreadNum() where ctx is a shorthand for JMeterContext class
According to Execution Order chapter of JMeter Documentation:
0. Configuration elements
1. Pre-Processors
2. Timers
3. Sampler
4. Post-Processors (unless SampleResult is null)
5. Assertions (unless SampleResult is null)
6. Listeners (unless SampleResult is null)
your CSV Data Set Config is executed at the first place, before any other scripting test elements. So the times when it "works" IMO are being caused by "false positive" situation as JMeter Properties are global and "live" while JMeter (and underlying JVM) is running. When you launch JMeter next time the properties will be null and your CSV Data Set Config will fail. So my expectation is that you should consider using __CSVRead() function instead which is evaluated in the runtime exactly in the place where it's being called. Check out Apache JMeter Functions - An Introduction article to learn more about JMeter Functions concept.
Related
I am in need of processing several thousands small log files.
I opted for Databricks to handle this problem, because it has good parallel computing capacities and interacts nicely with the Azure Blob storage account where the files are hosted.
After some researching, I always retrieve the same snippet of code (in PySpark).
# Getting your list of files with custom function
list_of_files = get_my_files()
# Create a path_rdd and use a custom udf to parse it
path_rdd = sc.parallelize(list_of_files)
content = path_rdd.map(parse_udf).collect()
Is there a better any method to do this? Would you opt for a flatmap if the logfiles are in a CSV format?
Thank you!
My current solution is:
content = sc.wholeTextFiles('/mnt/container/foo/*/*/', numPartitions=XX)
parsed_content = content.flatMap(custom_parser).collect()
I read all the content of the files as a string and keep their filenames.
I then pass this to my parsing function "custom_parser" using a flatMap, "where custom_parser" is defined as
def custom_parser(*argv):
file, content = argv
# Apply magic
return parsed_content_
I am currently finishing with a .collect() action, but I will alter this to save the output directly.
I am getting a value from HTTP request which I am writing it into a CSV file, each and every time when the program is executed, the new values are overwritten and not appended to the CSV. I would like to append the values instead of overwriting. I am using Regex and XPath extractor to get the values from the HTTP requests and writing it an CSV file.
new File('/Users/ddd/testgui/queueId1.csv').newWriter().withWriter { w ->
w << vars.get('queueid')
}
So this works for me, on groovysh 2.5.3 :
new File('/Users/ddd/testgui/queueId1.csv').newWriter(true).withWriter { w ->
w << vars.get('queueid')
}
The true in the newWriter is for append == true.
You can do just:
new File('/Users/ddd/testgui/queueId1.csv') << vars.get('queueid')
Be aware that your code is going to work fine only when you have 1 thread, if there will be more - you may suffer from a race condition when 2 threads will be simultaneously writing into a file.
If you're going to execute this code with > 1 virtual user I would rather recommend going for Sample Variables functionality.
If you add the next line to user.properties file:
sample_variables=queueid
and restart JMeter to pick the property up next time you run your test the .jtl results file will have an extra column with queueid variable value for each thread/request.
If you want to store it into a separate file - go for Flexible File Writer
I am novice in groovy scripting however trying to get some repetitive work done by automating a generation of an XML file creation.
So the question is:
I have an excel data source step in ready api and i would like to parametrize all values so they are passed to groovy script.
At present if i hard code the values in script, XML is formed as per schema however I need to iterate through all data in excel. so my test suite currently contains the following structure:
Data source
Groovy Script
Data Source Loop
I would like my data read each row at a time which will build XML elements as needed. So far I have the following code in script:
//package test
import groovy.xml.MarkupBuilder
/**
* A Simple Example that builds an XML document.
*/
class Test
{
static main(args)
{
def fileWriter = new FileWriter("c:/test.xml")
def fileBuilder = new MarkupBuilder(fileWriter)
fileBuilder.Header
{
date ('')
Item
{
SKU('')
Description('')
Amt('')
Qty('')
}
}
fileWriter.close()
}
}
Looking forward to hearing from experts on this.
Thanks
You don't need a groovy script if you are using Ready API for your task
1) Data Source step :- in Ready API when this step is executed it will populate all the data from excel into Ready API
2) You have to create various properties to use those values. for example :- Description variable will store value from Column1, Example
3) Since you have values stored in step 2 and lets assume your datasource step name is DS, so in your xml you can use variable like ${#DS#Description}
4) finally you will point the Data Source Loop to the second step and not data source loop. So this way everytime description will have a new value and your xml will be run with new values each time
Example
So I would try ask over in this thread IronPython - Run an Excel Macro but I don't have enough reputation.
So roughly following the code given in the link I created some code which would save a file to a specific location, then open a workbook that exists there, calling the macro's within it, which would perform a small amount of manipulation on the data which I downloaded to .xls, to make it more presentable.
Now I've isolated the problem to this particular part of the code (below).
Spotfire normally is not that informative but it gives me very little to go on here. It seems to be something .NET related but that's about all I can tell.
The Error Message
Traceback (most recent call last): File
"Spotfire.Dxp.Application.ScriptSupport", line unknown, in
ExecuteForDebugging File "", line unknown, in
StandardError: Exception has been thrown by the target of an
invocation.
The Script
from Spotfire.Dxp.Data.Export import DataWriterTypeIdentifiers
from System.IO import File, Directory
import clr
clr.AddReference("Microsoft.Office.Interop.Excel")
import Microsoft.Office.Interop.Excel as Excel
excel = Excel.ApplicationClass()
excel.Visible = True
excel.DisplayAlerts = False
workbook = ex.Workbooks.Open('myfilelocation')
excel.Run('OpenUp')
excel.Run('ActiveWorkbook')
excel.Run('DoStuff')
excel.Quit()
Right, so I'm answering my own question here but I hope it helps somebody. So the above code, as far as I'm aware was perfectly fine but didn't play well with the way my spotfire environment is configured. However, after going for a much more macro heavy approach I was able to find a solution.
On the spotfire end I've two input fields, one which gives the file path, the other the file name. Then there's a button to run the below script. These are joined together in the script but are crucially separate, as the file name needs to be inputted into a separate file, to be called by the main macro so that it can find the file location of the file.
Fundamentally I created three xml's to achieve this solution. The first was the main one which contained all of the main vba stuff. This would look into a folder in another xml which this script populates to find the save location of the file specified in an input field in spotfire. Finding the most recent file using a custom made function, it would run a simple series of conditional colour operations on the cell values in question.
This script populates two xml files and tells the main macro to run, the code in that macro automatically closing excel after it is done.
The third xml is for a series of colour rules, so user can custom define them depending on their dashboard. This gives it some flexibility for customisation. Not necessary but a potential request by some user so I decided to beat them to the punch.
Anyway, code is below.
from Spotfire.Dxp.Data.Export import DataWriterTypeIdentifiers
from System.IO import File, Directory
import clr
clr.AddReference("Microsoft.Office.Interop.Excel")
import Microsoft.Office.Interop.Excel as Excel
from Spotfire.Dxp.Data.Export import *
from Spotfire.Dxp.Application.Visuals import *
from System.IO import *
from System.Diagnostics import Process
# Input field which takes the name of the file you want to save
name = Document.Properties['NameOfDocument']
# Document property that takes the path
location = Document.Properties['FileLocation']
# just to debug to make sure it parses correctly. Declaring this in the script
# parameters section will mean that the escape characters of "\" will render in a
# unusable way whereas doing it here doesn't. Took me a long time to figure that out.
print(location)
# Gives the file the correct extension.
# Couldn't risk leaving it up to the user.
newname = name + ".xls"
#Join the two strings together into a single file path
finalProduct = location + "\\" + newname
#initialises the writer and filtering schema
writer = Document.Data.CreateDataWriter(DataWriterTypeIdentifiers.ExcelXlsDataWriter)
filtering = Document.ActiveFilteringSelectionReference.GetSelection(table).AsIndexSet()
# Writes to file
stream = File.OpenWrite(finalProduct)
# Here I created a seperate xls which would hold the file path. This
# file path would then be used by the invoked macro to find the correct folder.
names = []
for col in table.Columns:
names.append(col.Name)
writer.Write(stream, table, filtering, names)
stream.Close()
# Location of the macro. As this will be stored centrally
# it will not change so it's okay to hardcode it in.
runMacro = "File location\macro name.xls"
# uses System.Diagnostics to run the macro I declared. This will look in the folder I
# declared above in the second xls, auto run a function in vba to find the most
# up to date file
p = Process()
p.StartInfo.FileName = runMacro
p.Start()
Long story short: To run excel macro's from spotfire one solution is to use the system.dianostics method I use above and simply have your macro set to auto run.
I'm looking for way how to get random row from file in JMeter.
I would be appreciate for any suggestions.
Not sure regarding groovy, maybe there is an easier way, but for instance you can do it with Beanshell Sampler using the following code:
import org.apache.commons.io.FileUtils; //necessary import
List lines = FileUtils.readLines(new File("/path/to/your/file")); // read file into lines array
int random = new Random().nextInt(lines.size()); // get random line number
String randomLine = lines.get(random); // get random line
vars.put("randomLine", randomLine); // store the random line into ${randomLine} variable
substitute /path/to/your/file with relative or absolute path to the file you want the random line from
you will be able to access the random line as ${randomLine} where required
See How to Use BeanShell: JMeter's Favorite Built-in Component guide for more information on using JMeter and Java APIs from Beanshell test elements in your JMeter test
N.B. The above code is valid groovy code as well
This should be possible using a bean shell controller. Here you will need to do bit of programming to achieve the desired behaviour.
There is an easy way to achieve the desired outcome.
Introduce a ransom timer into your thread group.
The timer will ensure the randomness of the threads.
Configure a CSV confit element to read data from a file.
Ensure the file is shared with all the threads.
you can get a random var
random__Random(1,9)
//syntax: __Random(start,end)
Then you can pull out
File fileInstance = new File("path/yourfile.txt");
readerInstance = new IndexedFileReader(fileInstance);
lines = readerInstance.readLines(randomIdx, randomIdx+1);
readLines takes a start and an end line number..
However if you are using anything other then very small files, with low iterations I really suggest you read the entire file into a buffer, and use the random var to pull lines from that
You can also use the Random CSV Data Set Config Plugin Jmeter plugin by Blazemeter. It has all the randomising a regualr user will want on top of the conventional JMeters CSV Data Set Config.