I am new to Gatling, I am trying to do the performance testing for couple of rest calls. In my scenario I need to extract a value from json response of the 1st call and add those values to the list after looping for few times. Again after looping for few times and adding the values into the list, I want to reuse each value in my next rest call by iterating over the values in the list. Can anyone please suggest on how to implement this. I tried something as below,
var datasetIdList = List.empty[String]
val datasetidsFeeder = datasetIdList.map(datasetId => Map("datasetId" -> datasetId)).iterator
def createData() = {
repeat(20){
feed("").exec(http("create dataset").post("/create/data").header("content-type", "application/json")
.body(StringBody("""{"name":"name"}"""))
.asJson.check(jsonPath("$.id").saveAs("userId"))))
.exec(session => { var usrid = session("userId").as[String].trim
datasetIdList:+= usrid session})
}}
def upload()= feed(datasetidsFeeder).exec(http("file upload").post("/compute-metaservice/datasets/${datasetId}/uploadFile")
.formUpload("File","./src/test/resources/data/File.csv")
.header("content-type","multipart/form-data")
.check(status is 200))
val scn = scenario("create data and upload").exec(createData()).exec(upload())
setUp(scn.inject(atOnceUsers(1))).protocols(httpConf)
}
I am seeing an exception that ListFeeder is empty when trying to run above script. Can someone please help
Updated Code:
class ParallelcallsSimulation extends Simulation{
var idNumbers = (1 to 50).iterator
val customFeeder = Iterator.continually(Map(
"name" -> ("test_gatling_"+ idNumbers.next())
))
val httpConf = http.baseUrl("http://localhost:8080")
.header("Authorization","Bearer 6a4aee03-9172-4e31-a784-39dea65e9063")
def createDatasetsAndUpload() = {
repeat(3) {
//create dataset
feed(customFeeder).exec(http("create data").post("/create/data").header("content-type", "application/json")
.body(StringBody("""{ "name": "${name}","description": "create data and upload file"}"""))
.asJson.check(jsonPath("$.id").saveAs("userId")))
.exec(session => {
val name = session("name").asOption[String]
println(name.getOrElse("COULD NOT FIND NAME"))
val userId = session("userId").as[String].trim
println("%%%%% User ID ====>"+userId)
val datasetIdList = session("datasetIdList").asOption[List[_]].getOrElse(Nil)
session.set("datasetIdList", userId :: datasetIdList)
})
}
}
// File Upload
def fileUpload() = foreach("${datasetIdList}","datasetId"){
exec(http("file upload").post("/uploadFile")
.formUpload("File","./src/test/resources/data/File.csv")
.header("content-type","multipart/form-data")
.check(status is 200))
}
def getDataSetId() = foreach("${datasetIdList}","datasetId"){
exec(http("get datasetId")
.get("/get/data/${datasetId}")
.header("content-type","application/json")
.asJson.check(jsonPath("$.dlp.dlp_job_status").optional
.saveAs("dlpJobStatus")).check(status is 200)
).exec(session => {
val datastId = session("datasetId").asOption[String]
println("request for datasetId >>>>>>>>"+datastId.getOrElse("datasetId not found"))
val jobStatus = session("dlpJobStatus").asOption[String]
println("JOB STATUS:::>>>>>>>>>>"+jobStatus.getOrElse("Dlp Job Status not Found"))
println("Time: >>>>>>"+System.currentTimeMillis())
session
}).pause(10)
}
val scn1 = scenario("create multiple datasets and upload").exec(createDatasetsAndUpload()).exec(fileUpload())
val scn2 = scenario("get datasetId").pause(100).exec(getDataSetId())
setUp(scn1.inject(atOnceUsers(1)),scn2.inject(atOnceUsers(1))).protocols(httpConf)
}
I see below error when I try to execute above script
[ERROR] i.g.c.s.LoopBlock$ - Condition evaluation crashed with message 'No attribute named 'datasetIdList' is defined', exiting loop
var datasetIdList = List.empty[String] defines a mutable variable pointing to a immutable list.
val datasetidsFeeder = datasetIdList.map(datasetId => Map("datasetId" -> datasetId)).iterator uses the immutable list. Further changes to datasetIdList is irrelevant to datasetidsFeeder.
Mutating a global variable with your virtual user is usually not a good idea.
You can save the value into the user's session instead.
In the exec block, you can write:
val userId = session("userId").as[String].trim
val datasetIdList = session("datasetIdList").asOption[List[_]].getOrElse(Nil)
session.set("datasetIdList", userId :: datasetIdList)
Then you can use foreach to iterate them all without using a feeder at all.
foreach("${datasetIdList}", "datasetId") {
exec(http("file upload")
...
}
You should put more work in your question.
Your code is not syntax-highlighted, and is formatted poorly.
You said "I am seeing an exception that ListFeeder is empty" but the words "ListFeeder" are not seen anywhere.
You should post the error message so that it's easier to see what went wrong.
In the documentation linked, there is a Warning. Quoted below:
Session instances are immutable!
Why is that so? Because Sessions are messages that are dealt with in a multi-threaded concurrent way, so immutability is the best way to deal with state without relying on synchronization and blocking.
A very common pitfall is to forget that set and setAll actually return new instances.
This is why the code in the updated question doesn't update the list.
session => {
...
session.set("datasetIdList", userId :: datasetIdList)
println("%%%% List =====>>>" + datasetIdList.toString())
session
}
The updated session is simply discarded. And the original session is returned in the anonymous function.
My query checks while inserting if a record is duplicate
def insertValues(tableName:String, model:User):Insert = {
QueryBuilder.insertInto(tableName).value("bucket",model.profile.internalProfileDetails.get.bucketId)
....
.ifNotExists();
}
I am saving a duplicate entry and expect that Cassandra will return an error. Instead I am getting the existing record back. Shouldn't Insert return an error?
def save(user:User):Future[Option[User]] = Future {
saveDataToDatabase(user)
}
def saveDataToDatabase(data:M):Option[M] = {
println("inserting in table "+tablename+" with partition key "+partitionKeyColumns +" and values "+data)
val insertQuery = insertValues(tablename,data)
println("insert query is "+insertQuery)
try {
val resultSet = session.execute(insertQuery) //execute can take a Statement. Insert is derived from Statement so I can use Insert.
println("resultset after insert: " + resultSet)
Some(data)
}catch {
case e:Exception => {
println("cassandra exception "+e)
None
}
}
}
The table schema is
users (
bucket int,
email text,
authprovider text,
firstname text,
lastname text,
confirmed boolean,
hasher text,
id uuid,
password text,
salt text,
PRIMARY KEY ((bucket, email), authprovider, firstname, lastname)
In my test case, I expect return value to be None but I am getting Some(user)
"UsersRepository" should {
"not save a new user if the user already exist in the system" in {
val insertUserStatement =
s"""
| INSERT INTO users (bucket,email,firstname,lastname,authprovider,password,confirmed,id,hasher,salt) VALUES
| (1,'${testEnv.email}','fn','ln','${testEnv.loginInfo.providerID}','somePassword',false,${testEnv.mockHelperMethods.getUniqueID()},'someHasher','someSalt')
""".stripMargin
testCassandra.executeScripts(new CqlStatements(insertUserStatement))
val userKeys = UserKeys(1, testEnv.email ,testEnv.loginInfo, "fn", "ln")
val cassandraConnectionService = CassandraConnectionManagementService()
val (cassandraSession,cluster) = cassandraConnectionService.connectWithCassandra()
cassandraConnectionService.initKeySpace(cassandraSession,"mykeyspace")
val userRepository = new UsersRepository(testEnv.mockHelperMethods,cassandraSession,"users")
val resultCheckUser = await[Option[User]](userRepository.findOne(userKeys))(Timeout(Duration(5000,"millis")))
val user = User(UUID.fromString("11111111-1111-1111-1111-111111111111"),
UserProfile(
Some(InternalUserProfile(LoginInfo("credentials","test#test.com"),1,false,Some(PasswordInfo("someHasher","somePassword",None)))),
ExternalUserProfile("test#test.com","fn","ln",None)))
println(s"found initial user result ${resultCheckUser}")
resultCheckUser mustBe Some(user)
println(s"user already exists. Will try to add duplicate ")
println(s"adding user with user ${user}")
val resultAddUser = await[Option[User]](userRepository.save(user))(Timeout(Duration(5000,"millis")))
resultAddUser mustBe None
}
}
Output of test execution
insert query is INSERT INTO users (bucket,email,authprovider,firstname,lastname,confirmed,id,password,hasher,salt) VALUES (1,'test#test.com','credentials','fn','ln',false,11111111-1111-1111-1111-111111111111,'somePassword','someHasher','') IF NOT EXISTS;
[info] c.g.n.e.c.Cassandra - INFO [Native-Transport-Requests-1] 2019-06-07 06:13:57,659 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
[info] c.g.n.e.c.Cassandra - INFO [HANDSHAKE-localhost/127.0.0.1] 2019-06-07 06:13:57,683 OutboundTcpConnection.java:560 - Handshaking version with localhost/127.0.0.1
resultset after insert: ResultSet[ exhausted: false, Columns[[applied](boolean), bucket(int), email(varchar), authprovider(varchar), firstname(varchar), lastname(varchar), confirmed(boolean), hasher(varchar), id(uuid), password(varchar), salt(varchar)]]
running afterEach statements
afterEach: cassandra state is STARTED
[debug] c.g.n.e.c.t.TestCassandra - Stop TestCassandra 3.11.1
Some(User(11111111-1111-1111-1111-111111111111,UserProfile(Some(InternalUserProfile(LoginInfo(credentials,test#test.com),1,false,Some(PasswordInfo(someHasher,somePassword,None)))),ExternalUserProfile(test#test.com,fn,ln,None)))) was not equal to None
ScalaTestFailureLocation: UnitSpecs.RepositorySpecs.UsersRepositorySpecs at (UsersRepositorySpecs.scala:362)
Expected :None
Actual :Some(User(11111111-1111-1111-1111-111111111111,UserProfile(Some(InternalUserProfile(LoginInfo(credentials,test#test.com),1,false,Some(PasswordInfo(someHasher,somePassword,None)))),ExternalUserProfile(test#test.com,fn,ln,None))))
executeQuery returns ResultSet which has wasApplied method. This method returns true if the insert operation was done, otherwise it returns false. If a record is duplicate, wasApplied is false.
try {
val resultSet = session.execute(insertQuery) //execute can take a Statement. Insert is derived from Statement so I can use Insert.
println("resultset after insert: " + resultSet)
if(resultSet.wasApplied()){
Some(data)
} else {
None
}
}catch {
case e:Exception => {
println("cassandra exception "+e)
None
}
}
I want to insert data to my table in Cassandra and then return value from column "user_id" instead of full ResultSet. Here it is snippet of my code:
def create(user: User): Future[UUID] = {
insert
.value(_.id, user.id)
.value(_.email, user.email)
.value(_.name, user.name)
.consistencyLevel_=(ConsistencyLevel.ALL)
.future()
.map(r => fromRow(r.one()).id)
}
def fromRow(r: Row): User = {
User(id(r), email(r), name(r))
}
So future() returns Future[ResultSet]. After that I try to retrieve Row from ResultSet, modify it to User and get id eventually. Despite the fact that data were saved to my table I got
ResultSet[ exhausted: true, Columns[]]
columns of the ResultSet are empty and consequently r.one() returned null.
I haven't found any examples for my purpose. So, can phantom-dsl do something like Quill?
val q = quote {
query[Product].insert(lift(Product(0L, "My Product", 1011L))).returning(_.id)
}
So in more recent versions of phantom that create method is automatically generated. More details here. The fromRow method is also automatically generated so you don't need to type it manually.
Long story short, this is what you could use:
def create(user: User): Future[UUID] = {
store(user)
.consistencyLevel_=(ConsistencyLevel.ALL)
.future()
.map(_ => user.id)
}
I'm using Spark 2.0.2. I have a DataFrame that has an alias on it, and I'd like to be able to retrieve that. A simplified example of why I'd want that is below.
def check(ds: DataFrame) = {
assert(ds.count > 0, s"${df.getAlias} has zero rows!")
}
The above code of course fails because DataFrame has no getAlias function. Is there a way to do this?
You can try something like this but I wouldn't go so far to claim it is supported:
Spark < 2.1:
import org.apache.spark.sql.catalyst.plans.logical.SubqueryAlias
import org.apache.spark.sql.Dataset
def getAlias(ds: Dataset[_]) = ds.queryExecution.analyzed match {
case SubqueryAlias(alias, _) => Some(alias)
case _ => None
}
Spark 2.1+:
def getAlias(ds: Dataset[_]) = ds.queryExecution.analyzed match {
case SubqueryAlias(alias, _, _) => Some(alias)
case _ => None
}
Example usage:
val plain = Seq((1, "foo")).toDF
getAlias(plain)
Option[String] = None
val aliased = plain.alias("a dataset")
getAlias(aliased)
Option[String] = Some(a dataset)
Disclaimer: as stated above, this code relies on undocumented APIs subject to change. It works as of Spark 2.3.
After much digging into mostly undocumented Spark methods, here is the full code to pull the list of fields, along with the table alias for a dataframe in PySpark:
def schema_from_plan(df):
plan = df._jdf.queryExecution().analyzed()
all_fields = _schema_from_plan(plan)
iterator = plan.output().iterator()
output_fields = {}
while iterator.hasNext():
field = iterator.next()
queryfield = all_fields.get(field.exprId().id(),{})
if not queryfield=={}:
tablealias = queryfield["tablealias"]
else:
tablealias = ""
output_fields[field.exprId().id()] = {
"tablealias": tablealias,
"dataType": field.dataType().typeName(),
"name": field.name()
}
return list(output_fields.values())
def _schema_from_plan(root,tablealias=None,fields={}):
iterator = root.children().iterator()
while iterator.hasNext():
node = iterator.next()
nodeClass = node.getClass().getSimpleName()
if (nodeClass=="SubqueryAlias"):
# get the alias and process the subnodes with this alias
_schema_from_plan(node,node.alias(),fields)
else:
if tablealias:
# add all the fields, along with the unique IDs, and a new tablealias field
iterator = node.output().iterator()
while iterator.hasNext():
field = iterator.next()
fields[field.exprId().id()] = {
"tablealias": tablealias,
"dataType": field.dataType().typeName(),
"name": field.name()
}
_schema_from_plan(node,tablealias,fields)
return fields
# example: fields = schema_from_plan(df)
For Java:
As #veinhorn mentioned, it is also possible to get the alias in Java. Here is a utility method example:
public static <T> Optional<String> getAlias(Dataset<T> dataset){
final LogicalPlan analyzed = dataset.queryExecution().analyzed();
if(analyzed instanceof SubqueryAlias) {
SubqueryAlias subqueryAlias = (SubqueryAlias) analyzed;
return Optional.of(subqueryAlias.alias());
}
return Optional.empty();
}
Suppose i have 2 records in data base
1) 2007-12-10 10:35:31.000
2) 2008-12-10 10:35:31.000
FirstOrDefault() method will give me the first record match in sequence like 2007-12-10 10:35:31.000 but i need the latest one which is 2008-12-10 10:35:31.000
if ((from value in _names where value != null select value.ExpiryDate < now).Any())
{
return _names.FirstOrDefault();
}
You can use:
return _names.LastOrDefault();
However, your if just sends another unnecessary query (and it is a wrong query too). If you don't have any record, LastOrDefault and FirstOrDefault will return null. You can use something like this to improve the code:
var name = _names.LastOrDefault();
if(name != null)
{
return name;
}
// other code here
If you really want to use FirstOrDefault, you should order descending, like:
var name = _names.Where(n => n.ExpiryDate < now).OrderByDescending(n => n.ExpiryDate).FirstOrDefault();