JMeter: Get list of column headers and their index - groovy

I am looking for a solution to get list of column headers and their indexes.
Task: pass column headers from a CSV file and their indexes to a variable for further use in HTTP request. Idea is to create a mapping which would indicate which column to use for the given header, as following example:
name
position
company
John
manager
Alphabet
Smith
intern
JP Morgan
So the request would contain:
{"name":0},{"position":1},{"company":2}
I use multiple files where number of columns can be anything from 3 to 50 (or more), so there's no max size. I thought about following approach:
Read all headers and split by coma into a list / collection
Loop through them and use index of the list items as the index I need:
Headers: |header_1 | header_2 | ... | heade_n |
Request: {"header_1":0},{"header_2":1}...{"header_n":n-1}
Question: How to iterate through all columns while size of the file is unknown?
I found this answer where the OP has a fixed number of columns, and solution uses every value for a separate request, but I have to send only one request with the list of headers and indexes.
P.S. I am new to JMeter and Groovy, so didnt have enough time to make a full scale research. So if the answer will explain how to use pass this variable to the request would be appreciated.

There are no "headers" or "columns" in CSV files, there is "first line" and entries delimited by comma.
So if you file looks like:
name,position,company
John,manager,Alphabet
Smith,intern,JP Morgan
You can generate the desired request body like:
def payload = []
def firstLine = new File('test.csv').readLines().get(0)
def entries = firstLine.split(',').size() - 1
0.upto(entries, { index ->
payload.add([('header_' + (index + 1)): index])
})
vars.put('payload', new groovy.json.JsonBuilder(payload).toString())
and refer generated value as ${payload} where required
Demo:
More information:
Reading a File in Groovy
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It

You can use Groovy to get the names of column headers.
File file = new File("<path-to-file-name>.csv")
String header_1, header_2, header_3
file.withReader {reader ->
def lstColumns = reader.readLine().split(",")
header_1 = lstColumns[0]
header_2 = lstColumns[1]
header_3 = lstColumns[2]
}

I need to get the value for given index so the request would be like {"name":0},{"position":1},{"company":2}, i.e. not just "header_{0}"; and with your heads-up and those links, came up with following:
def payload = []
def firstLine = new File('test.csv').readLines().get(0)
def entries = firstLine.split(',')
entries.eachWithIndex{value, index ->
log.info("'${value}': ${index}")
payload.add("{'${value}': ${index}}")
}
vars.put('payload', new groovy.json.JsonBuilder(payload).toString())
//log.info('payload : ' + new groovy.json.JsonBuilder(payload).toString())

Related

Extract numbers only from specific lines within a txt file with certain keywords in Python

I want to extract numbers only from lines in a txt file that have a certain keyword and add them up then compare them, and then print the highest total number and the lowest total number. How should I go about this?
I want to print the highest and the lowest valid total numbers
I managed to extract lines with "valid" keyword in them, but now I want to get numbers from this lines, and then add the numbers up of each line, and then compare these numbers with other lines that have the same keyword and print the highest and the lowest valid numbers.
my code so far
#get file object reference to the file
file = open("shelfs.txt", "r")
#read content of file to string
data = file.read()
#close file<br>
closefile = file.close()
#get number of occurrences of the substring in the string
totalshelfs = data.count("SHELF")
totalvalid = data.count("VALID")
totalinvalid = data.count("INVALID")
print('Number of total shelfs :', totalshelfs)
print('Number of valid valid books :', totalvalid)
print('Number of invalid books :', totalinvalid)
txt file
HEADER|<br>
SHELF|2200019605568|<br>
BOOK|20200120000000|4810.1|20210402|VALID|<br>
SHELF|1591024987400|<br>
BOOK|20200215000000|29310.0|20210401|VALID|<br>
SHELF|1300001188124|<br>
BOOK|20200229000000|11519.0|20210401|VALID|<br>
SHELF|1300001188124|<br>
BOOK|20200329001234|115.0|20210331|INVALID|<br>
SHELF|1300001188124|<br>
BOOK|2020032904567|1144.0|20210401|INVALID|<br>
FOOTER|
What you need is to use the pandas library.
https://pandas.pydata.org/
You can read a csv file like this:
data = pd.read_csv('shelfs.txt', sep='|')
it returns a DataFrame object that can easily select or sort your data. It will use the first row as header, then you can select a specific column like a dictionnary:
header = data['HEADER']
header is a Series object.
To select columns you can do:
shelfs = data.loc[:,data['HEADER']=='SHELF']
to select only the row where the header is 'SHELF'.
I'm just not sure how pandas will handle the fact that you only have 1 header but 2 or 5 columns.
Maybe you should try to create one header per colmun in your csv, and add separators to make each row the same size first.
Edit (No External libraries or change in the txt file):
# Split by row
data = data.split('<br>\n')
# Split by col
data = [d.split('|') for d in data]
# Fill empty cells
n_cols = max([len(d) for d in data])
for i in range(len(data)):
while len(data[i])<n_cols:
data[i].append('')
# List VALID rows
valid_rows = [d for d in data if d[4]=='VALID']
# Get sum min and max
valid_sum = [d[1]+d[2]+d[3] for d in valid_rows]
valid_minimum = min(valid_sum)
valid_maximum = max(valid_sum)
It's maybe not exactly what you want to do but it solves a part of your problem. I didn't test the code.

How can I loop through a string list of multiple elements and append the string to another? (Python)

I am fairly new to Python.
I am trying to append elements of one string to another. The idea is to append different tags to a url. This url is to contain a list (or tuple) of multiple urls that I can access further down in the code. This code is for web scraping purposes.
I have been able to achieve as follows (i.e. this is my current code chunk):
def Commodoties(url, headings):
a = ''
for h in range(0, len(headings)):
print(url, '/', headings[h], sep='')
url = url + headings[h]
return url, url
headings = ['currencies', 'commodities']
url = 'https://tradingeconomics.com/'
test = Commodoties(url, headings)
print(type(test))
Ideally I want to achieve an output along these lines:
['https://tradingeconomics.com/currencies', 'https://tradingeconomics.com/commodities']
Currently I have achieved the following output:
('https://tradingeconomics.com/currenciescommodities', 'https://tradingeconomics.com/currenciescommodities')
I have implemented different methods in the Commodoties function, but I have not been able to achieve my desired output.
Any help would be greatly appreciated thanks!!
#aj96, try this code to confirm that's what you want format:
def Commodoties(url, headings):
result = []
for h in headings:
# tmp = url + h
result.append(url + h) # simplify
print(result) # can comment out;
return result
>>> headings = ['currencies', 'commodities']
>>> url = 'https://tradingeconomics.com/'
>>> test = Commodoties(url, headings)
['https://tradingeconomics.com/currencies']
['https://tradingeconomics.com/currencies', 'https://tradingeconomics.com/commodities']

How do I iterate through two lists in API call Python 3

I have two files containing information that I need to input in the same script. One containing ID's, one on each line, and the other list containing parameters on their own individual lines as well. It should be made known that this list contains over 4000 lines each. Other API calls have been successful but this one is a bit harder to figure out.
The way this is intended to work is that the script reads line 1 from the ID file, insert that ID where %s is in the url. This will complete the url necessary for the API call. Then I need the parameters which are on the same lines matching their respective network ID's in the ID file, placed in %s in the payload section.
I got it to this point and what is happening now is when an ID is picked in the ID list, the URL becomes complete and does what it is supposed to. However when the script starts reading the contents file, it is iterating over and over until all parameters for ALL networks are complete and this is applied for just that one network, which is not supposed to happen, then it moves onto the next network ID and does the same thing.
I posted a sample visual to give you an idea of what the output is. I know there must be a way to have them read one line at a time, run the script, and iterate over to the next line in sequence, and do this until both of the entire lists are complete.
Python is not my strongest area so any assistance is greatly appreciated.
The files are .txt files and properly formatted. These data have been tested using postman and have been successful in our other API calls as well, so we can eliminate a couple of factors.
with open('TZ.txt') as file1, open ('TZContents.txt') as file2:
array1 = file1.readlines()
file = file2.readlines()
for line in array1:
url = 'https://dashboard.meraki.com/api/v0/networks/%s' %line.rstrip("\n")
for line2 in file:
payload = '%s' % line2.rstrip("\n")
headers = {'X-Cisco-Meraki-API-Key': 'API Key','Content-Type': "application/json"}
response = requests.request('PUT', url, headers = headers, data = payload, allow_redirects=True, timeout = 10)
print(response.text)
Output Example Below:
{"id":"1111", "type":"wireless","name":"Network A}
{"id":"1111", "type":"wireless","name":"Network B}
{"id":"1111", "type":"wireless","name":"Network C}
{"errors":["Name has already been taken"]}
{"errors":["Name has already been taken"]}
{"errors":["Name has already been taken"]}
{"errors":["Name has already been taken"]}
{"errors":["Name has already been taken"]}
{"id":"2222", "type":"appliance","name":"Network A}
{"id":"2222", "type":"appliance","name":"Network B}
{"id":"2222", "type":"appliance","name":"Network C}
Should be this:
{"id":"1111", "type":"wireless","name":"Network A}
{"id":"2222", "type":"appliance","name":"Network B}
{"id":"3333", "type":"combined","name":"Network C}
I read your description and I guess that the two files contain exactly the same number of lines. Is that correct?
In the present code a nested for iterations is used, resulting in redudant output.
You might use a same index to locate the same line in either file.
A modified code might be
with open('TZ.txt') as file1, open ('TZContents.txt') as file2:
ids = file1.readlines()
params = file2.readlines()
n_lines = len(ids)
for line_num in list(range(n_lines)):
url = 'https://dashboard.meraki.com/api/v0/networks/%s' %ids[line_num].rstrip("\n")
payload = '%s' % params[line_num].rstrip("\n")
headers = {'X-Cisco-Meraki-API-Key': 'API Key','Content-Type': "application/json"}
response = requests.request('PUT', url, headers = headers, data = payload, allow_redirects=True, timeout = 10)
print(response.text)

How to perform a check with a csv file

I want to know if there is a better way that iterating through a csv when performing a check. Virtually I am using SOAP UI (free version) to test a web service based on a search.
What I want to do is look at a response from a particular search request (the step name of the SOAP Request is 'Search Request') and look for all instances of test found in between xml tags <TestID> for both within <IFInformation> and <OFInformation> (this will be in a groovy script step).
def groovyUtils = new com.eviware.soapui.support.GroovyUtils(context)
import groovy.xml.XmlUtil
def response = messageExchange.response.responseContent
def xml = new XmlParser().parseText( response )
def IF = xml.'soap:Body'
.IF*
.TestId.text()
def OF = xml.'soap:Body'
.OF*
.TestId.text()
Now what I want to do is for each instance of the 'DepartureAirportId', I want to check that the ID is within a CSV file. There are two columns within the csv file (let's call it Search.csv) and both columns contain many rows. If the flight is found within any row within the first column, add a count +1 for the variable 'Test1', else if found in second column in csv, add count +1 for variable 'Test2'. If not found within any, add count +1 for variable 'NotFound'
I don't know if iterating through a csv is the best outcome or output all the data from the csv into an array list and iterate it through there but I want to know how this can be done and the best way for my own learning experience?
don't know about your algorithm, but the easiest way to iterate through simple csv file in groovy by line and splitting each line with separator:
new File("/1.csv").splitEachLine(","){line->
println " ${ line[0] } ${ line[1] } "
}
http://docs.groovy-lang.org/latest/html/groovy-jdk/java/io/File.html#splitEachLine(java.lang.String,%20groovy.lang.Closure)
You might want to use CSV Validator.
Format.of(String regex)
It should do the trick - just provide the literal you're looking for as a rule for first column and check if it throws an exception or not.

convert data string to list

I'm having some troubles processing some input.
I am reading data from a log file and store the different values according to the name.
So my input string consists of ip, name, time and a data value.
A log line looks like this and it has \t spacing:
134.51.239.54 Steven 2015-01-01 06:09:01 5423
I'm reading in the values using this code:
loglines = file.splitlines()
data_fields = loglines[0] # IP NAME DATE DATA
for loglines in loglines[1:]:
items = loglines.split("\t")
ip = items[0]
name = items[1]
date = items[2]
data = items[3]
This works quite well but I need to extract all names to a list but I haven't found a functioning solution.
When i use print name i get:
Steven
Max
Paul
I do need a list of the names like this:
['Steven', 'Max', 'Paul',...]
There is probably a simple solution and i haven't figured it out yet, but can anybody help?
Thanks
Just create an empty list and add the names as you loop through the file.
Also note that if that file is very large, file.splitlines() is probably not the best idea, as it reads the entire file into memory -- and then you basically copy all of that by doing loglines[1:]. Better use the file object itself as an iterator. And don't use file as a variable name, as it shadows the type.
with open("some_file.log") as the_file:
data_fields = next(the_file) # consumes first line
all_the_names = [] # this will hold the names
for line in the_file: # loops over the rest
items = line.split("\t")
ip, name, date, data = items # you can put all this in one line
all_the_names.append(name) # add the name to the list of names
Alternatively, you could use zip and map to put it all into one expression (using that loglines data), but you rather shouldn't do that... zip(*map(lambda s: s.split('\t'), loglines[1:]))[1]

Resources