I'm trying to create multiple chains and execute them in sequence using gatling. According to the documentation, this should work since 1.3 and I'm using 2.0 but it still doesn't work for me. I get an error not found value exex.
package basic
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class BenchmarkSimulation extends Simulation {
val httpConf = http
.baseURL("http://127.0.0.1:9001")
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.doNotTrackHeader("1")
.acceptLanguageHeader("en-US,en;q=0.5")
.acceptEncodingHeader("gzip, deflate")
.userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
val helloworld = exec(http("write-hello-world").get("/hello"))
val datetimenow = exec(http("write-datetime-now").get("/now"))
val scn = scenario("My Scenario").exec(helloworld, datetimenow)
setUp(scn.inject(ramp(1000 users) over (10 seconds))).protocols(httpConf)
}
The exact errors are something like,
ne-api/gatling/user-files/simulations/basic/BenchmarkSimulation.scala:17: not found: value exec
00:09:39.660 [ERROR] i.g.a.ZincCompiler$ - val helloworld = exec(http("write-hello-world").get("/hello"))
00:09:39.661 [ERROR] i.g.a.ZincCompiler$ - ^
00:09:39.663 [ERROR] i.g.a.ZincCompiler$ -
00:09:39.663 [ERROR] i.g.a.ZincCompiler$ - val datetimenow = exec(http("write-datetime-now").get("/now"))
00:09:39.663 [ERROR] i.g.a.ZincCompiler$ - ^
00:09:40.671 [ERROR] i.g.a.ZincCompiler$ - two errors found
Exception in thread "main" Compilation failed
at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:105)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:98)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:98)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:98)
at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:155)
at sbt.compiler.AggressiveCompile$$anonfun$3.compileScala$1(AggressiveCompile.scala:97)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:138)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:86)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:30)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:28)
at sbt.inc.Incremental$.cycle(Incremental.scala:73)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:33)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:32)
at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:41)
at sbt.inc.Incremental$.compile(Incremental.scala:32)
at sbt.inc.IncrementalCompile$.apply(Compile.scala:25)
at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:146)
at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:70)
at com.typesafe.zinc.Compiler.compile(Compiler.scala:161)
at com.typesafe.zinc.Compiler.compile(Compiler.scala:142)
at io.gatling.app.ZincCompiler$.main(ZincCompiler.scala:111)
at io.gatling.app.ZincCompiler.main(ZincCompiler.scala)
Exception in thread "main" org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
at io.gatling.app.ZincCompilerLauncher$.apply(ZincCompilerLauncher.scala:54)
at io.gatling.app.SimulationClassLoader$.fromSourcesDirectory(SimulationClassLoader.scala:32)
at io.gatling.app.Gatling$$anonfun$15.apply(Gatling.scala:171)
at io.gatling.app.Gatling$$anonfun$15.apply(Gatling.scala:171)
at scala.Option.getOrElse(Option.scala:120)
at io.gatling.app.Gatling.start(Gatling.scala:171)
at io.gatling.app.Gatling$.fromMap(Gatling.scala:59)
at io.gatling.app.Gatling$.runGatling(Gatling.scala:80)
at io.gatling.app.Gatling$.main(Gatling.scala:54)
at io.gatling.app.Gatling.main(Gatling.scala)
I think you're mixing Gatling 1 and Gatling 2 libraries in your classpath.
Also, "ramp(1000 users) over (10 seconds))" is Gatling 1 syntax, it should be "rampUsers(1000) over (10 seconds)".
And datetimenow scenario will begin execution only after helloworld scenario is complete.
No. helloworld and datetimenow are not scenarios but chains (scenario parts). You're just merging then (chaing actually) into one scenario. If you want to wait for all users to complete helloworld before starting datetimenow, you have to write 2 different simulations are launch them sequentially.
Related
I have a requirements of pretty custom non-trivial synchronization which can be implemented with a fair ReentrantLock and Phaser. It does not seem to be possible (without a non-trivial customization) to implement on fs2 and cats.effect.
Since it's required to wrap all blocking operation into a Blocker here is the code:
private val l: ReentrantLock = new ReentrantLock(true)
private val c: Condition = l.newCondition
private val b: Blocker = //...
//F is declared on the class level
def lockedMutex(conditionPredicate: Int => Boolean): F[Unit] = blocker.blockOn {
Sync[F].delay(l.lock()).bracket(_ => Sync[F].delay{
while(!conditionPredicate(2)){
c.await()
}
})(_ => Sync[F].delay(l.unlock()))
}
QUESTION:
Is it guaranteed that the code containing c.await() will be executed in the same Thread which acquires/releases the ReentrantLock?
This is a crucial part since if it's not IllegalMonitorStateException will be thrown.
You really do not need to worry about threads when using something like cats-effect, rather you can describe your problem on a higher level.
This should get the same behavior you want, it will be running high-priority jobs until there isn't more to then pick low-priority jobs. After finishing a low-priority job each fiber will first check if there are more high-priority jobs before trying to pick again a low-priority one:
import cats.effect.Async
import cats.effect.std.Queue
import cats.effect.syntax.all._
import cats.syntax.all._
import scala.concurrent.ExecutionContext
object HighLowPriorityRunner {
final case class Config[F[_]](
highPriorityJobs: Queue[F, F[Unit]],
lowPriorityJobs: Queue[F, F[Unit]],
customEC: Option[ExecutionContext]
)
def apply[F[_]](config: Config[F])
(implicit F: Async[F]): F[Unit] = {
val processOneJob =
config.highPriorityJobs.tryTake.flatMap {
case Some(hpJob) => hpJob
case None => config.lowPriorityJobs.tryTake.flatMap {
case Some(lpJob) => lpJob
case None => F.unit
}
}
val loop: F[Unit] = processOneJob.start.foreverM
config.customEC.fold(ifEmpty = loop)(ec => loop.evalOn(ec))
}
}
You can use the customEC to provide your own ExecutionContext to control the number of real threads that are running your fibers under the hood.
The code can be used like this:
import cats.effect.{Async, IO, IOApp, Resource}
import cats.effect.std.Queue
import cats.effect.syntax.all._
import cats.syntax.all._
import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
object Main extends IOApp.Simple {
override final val run: IO[Unit] =
Resource.make(IO(Executors.newFixedThreadPool(2)))(ec => IO.blocking(ec.shutdown())).use { ec =>
Program[IO](ExecutionContext.fromExecutor(ec))
}
}
object Program {
private def createJob[F[_]](id: Int)(implicit F: Async[F]): F[Unit] =
F.delay(println(s"Starting job ${id} on thread ${Thread.currentThread.getName}")) *>
F.delay(Thread.sleep(1.second.toMillis)) *> // Blocks the Fiber! - Only for testing, use F.sleep on real code.
F.delay(println(s"Finished job ${id}!"))
def apply[F[_]](customEC: ExecutionContext)(implicit F: Async[F]): F[Unit] = for {
highPriorityJobs <- Queue.unbounded[F, F[Unit]]
lowPriorityJobs <- Queue.unbounded[F, F[Unit]]
runnerFiber <- HighLowPriorityRunner(HighLowPriorityRunner.Config(
highPriorityJobs,
lowPriorityJobs,
Some(customEC)
)).start
_ <- List.range(0, 10).traverse_(id => highPriorityJobs.offer(createJob(id)))
_ <- List.range(10, 15).traverse_(id => lowPriorityJobs.offer(createJob(id)))
_ <- F.sleep(5.seconds)
_ <- List.range(15, 20).traverse_(id => highPriorityJobs.offer(createJob(id)))
_ <- runnerFiber.join.void
} yield ()
}
Which should produce an output like this:
Starting job 0 on thread pool-1-thread-1
Starting job 1 on thread pool-1-thread-2
Finished job 0!
Finished job 1!
Starting job 2 on thread pool-1-thread-1
Starting job 3 on thread pool-1-thread-2
Finished job 2!
Finished job 3!
Starting job 4 on thread pool-1-thread-1
Starting job 5 on thread pool-1-thread-2
Finished job 4!
Finished job 5!
Starting job 6 on thread pool-1-thread-1
Starting job 7 on thread pool-1-thread-2
Finished job 6!
Finished job 7!
Starting job 8 on thread pool-1-thread-1
Starting job 9 on thread pool-1-thread-2
Finished job 8!
Finished job 9!
Starting job 10 on thread pool-1-thread-1
Starting job 11 on thread pool-1-thread-2
Finished job 10!
Finished job 11!
Starting job 15 on thread pool-1-thread-1
Starting job 16 on thread pool-1-thread-2
Finished job 15!
Finished job 16!
Starting job 17 on thread pool-1-thread-1
Starting job 18 on thread pool-1-thread-2
Finished job 17!
Finished job 18!
Starting job 19 on thread pool-1-thread-1
Starting job 12 on thread pool-1-thread-2
Finished job 19!
Starting job 13 on thread pool-1-thread-1
Finished job 12!
Starting job 14 on thread pool-1-thread-2
Finished job 13!
Finished job 14!
Thanks to Gavin Bisesi (#Daenyth) for refining my original idea into this!
Full code available here.
Executing the above in Spark 3.0.2 produces
Exception in thread "main" java.lang.AssertionError: assertion failed: Found duplicate rewrite attributes.
It was working in Spark 2.4.3.
SELECT
COALESCE(view_1_alias.name, view_2.name) AS name,
COALESCE(view_1_alias.id, view_2.id) AS id,
COALESCE(view_1_alias.second_id, view_2.second_id) AS second_id,
COALESCE(view_1_alias.local_timestamp, view_2.local_timestamp) AS local_timestamp,
COALESCE(view_1_alias.utc_timestamp, view_2.utc_timestamp) AS utc_timestamp,
view_1_alias.alias_1_column_1,
view_1_alias.alias_1_column_2,
view_1_alias.alias_1_column_3,
view_1_alias.alias_1_column_4,
view_1_alias.alias_1_column_5,
view_1_alias.alias_1_column_6,
view_1_alias.alias_1_column_7,
view_2.alias_2_coumn_1
FROM
view_1 view_1_alias FULL OUTER JOIN view_2
ON
view_1_alias.name = view_2.name AND
view_1_alias.id = view_2.id AND
view_1_alias.second_id = view_2.second_id AND
view_1_alias.local_timestamp = view_2.local_timestamp;
The header for view_1:
| name |id|second_id|local_timestamp|utc_timestamp|alias_1_column_1|alias_1_column_2 | alias_1_column_3|alias_1_column_4|alias_1_column_5|alias_1_column_6|alias_1_column_7 |
The header for the view_2:
| name | id |second_id |local_timestamp |utc_timestamp|alias_2_coumn_1|
Intrestingly we have several of these queries, but only one specifically gives issues.
I saw this question but not sure how to translate it to spark sql.
I am trying to use distutils.dir_util.copy_tree() to copy a whole directory tree to a new location. However, the call aborted because there were some broken symlinks in the source directory:
[ERROR] * src = 'src_dir/some_broken_symlink'
[ERROR] * dst = 'dst_dir/the_file_pointed_to_by_the_symlink'
[ERROR] * preserve_mode = 1, preserve_times = 1, update = 0, link = None, verbose = 1
[ERROR] * dry_run = 0
[ERROR] *
[ERROR] * def copy_file(src, dst, preserve_mode=1, preserve_times=1, update=0,
[ERROR] * link=None, verbose=1, dry_run=0):
[ERROR] * """Copy a file 'src' to 'dst'. If 'dst' is a directory, then 'src' is
[ERROR] * copied there with the same name; otherwise, it must be a filename. (If
[ERROR] * the file exists, it will be ruthlessly clobbered.) If 'preserve_mode'
[ERROR] * is true (the default), the file's mode (type and permission bits, or
[ERROR] * whatever is analogous on the current platform) is copied. If
[ERROR] * 'preserve_times' is true (the default), the last-modified and
[ERROR] * last-access times are copied as well. If 'update' is true, 'src' will
[ERROR] * only be copied if 'dst' does not exist, or if 'dst' does exist but is
[ERROR] * older than 'src'.
[ERROR] *
[ERROR] * 'link' allows you to make hard links (os.link) or symbolic links
[ERROR] * (os.symlink) instead of copying: set it to "hard" or "sym"; if it is
[ERROR] * None (the default), files are copied. Don't set 'link' on systems that
[ERROR] * don't support it: 'copy_file()' doesn't check if hard or symbolic
[ERROR] * linking is available. If hardlink fails, falls back to
[ERROR] * _copy_file_contents().
[ERROR] *
[ERROR] * Under Mac OS, uses the native file copy function in macostools; on
[ERROR] * other systems, uses '_copy_file_contents()' to copy file contents.
[ERROR] *
[ERROR] * Return a tuple (dest_name, copied): 'dest_name' is the actual name of
[ERROR] * the output file, and 'copied' is true if the file was copied (or would
[ERROR] * have been copied, if 'dry_run' true).
[ERROR] * """
[ERROR] * # XXX if the destination file already exists, we clobber it if
[ERROR] * # copying, but blow up if linking. Hmmm. And I don't know what
[ERROR] * # macostools.copyfile() does. Should definitely be consistent, and
[ERROR] * # should probably blow up if destination exists and we would be
[ERROR] * # changing it (ie. it's not already a hard/soft link to src OR
[ERROR] * # (not update) and (src newer than dst).
[ERROR] *
[ERROR] * from distutils.dep_util import newer
[ERROR] * from stat import ST_ATIME, ST_MTIME, ST_MODE, S_IMODE
[ERROR] *
[ERROR] * if not os.path.isfile(src):
[ERROR] * raise DistutilsFileError(
[ERROR] * > "can't copy '%s': doesn't exist or not a regular file" % src)
[ERROR] * E distutils.errors.DistutilsFileError: can't copy 'src_dir/some_broken_symlink': doesn't exist or not a regular file
[ERROR] *
[ERROR] * /usr/software/pkgs/Python-3.7.1/lib/python3.7/distutils/file_util.py:105: DistutilsFileError
I don't know if there is way to let distutils.dir_util.copy_tree() ignore these errors and behave like the shell command cp -r, so that it doesn't get aborted on broken symlinks from the source directory?
The code for the part is:
from distutils.dir_util import copy_tree
copy_tree(src_dir, dst_dir)
preserve_symlinks=1 fixed it.
Original code (not working with broken symlinks in src_dir):
from distutils.dir_util import copy_tree
copy_tree(src_dir, dst_dir)
New code (working with broken symlinks in src_dir):
from distutils.dir_util import copy_tree
copy_tree(src_dir, dst_dir, preserve_symlinks=1)
I am trying to unit test Kafka Stream with the MockedStreams library.
To test a simple example, I just want to convert string into upper case.
I try the code below :
import com.madewithtea.mockedstreams.MockedStreams
import org.apache.kafka.common.serialization.Serdes
import org.scalatest.{Matchers, WordSpec}
class mockedStreamsSpec extends WordSpec with Matchers {
val input = Seq(("x", "foo"), ("y", "bar"))
val exp = Seq(("x", "FOO"), ("y", "BAR"))
"Put in upper case " should {
"always return value in upper case" in {
MockedStreams()
.topology {
builder =>
builder.stream[String, String]("topic-in")
.mapValues[String](_.toUpperCase())
.to("topic-out")
}
.input("topic-in", Serdes.String(), Serdes.String(), input)
.output("topic-out", Serdes.String(), Serdes.String(), exp.size) shouldEqual exp
}
}
}
I get an error avout java.lang.String :
[info] mockedStreamsSpec:
[info] Put in upper case
[info] - should always return value in upper case *** FAILED ***
[info] java.lang.ClassCastException: [B cannot be cast to java.lang.String
[info] at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40)
[info] at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
[info] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
[info] at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
[info] at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
[info] at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
[info] at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
[info] at org.apache.kafka.test.ProcessorTopologyTestDriver.process(ProcessorTopologyTestDriver.java:276)
[info] at org.apache.kafka.test.ProcessorTopologyTestDriver.process(ProcessorTopologyTestDriver.java:315)
[info] at com.madewithtea.mockedstreams.MockedStreams$Builder.$anonfun$produce$1(MockedStreams.scala:110)
It seems weird, because Scala String and Java String should be identical.
I've never used this MockedStreams library but the exception "[B cannot be cast to java.lang.String" means a byte array is being cast to a String. I suggest you try using the variant of builder.stream() method (inside the topology) that lets you explicitly specify the Serdes to use, as it seams the ones given to the input() method are not being used. I guess those are only used to serialize the test data onto the stream.
[![enter image description here][1]][1]I want to create streams using springXD. According to basic definition of streams, I have:
Source : http
Transform script : I written one groovy script which I stored at
/xd/modules/processor/script/transform.groovy
Sink : cassandra
I want to store the unstructured json data, which I posted from http to cassandra table.
I run springXD on single-node mode, and then run xd-shell.
For stream creation, I use:
stream create --name test2 --definition "http --port=9000 | transform --
script=insert_transform.groovy |cassandra --contactPoints=127.0.0.1 --
keyspace=db1 --ingestQuery='insert into table1 (emp_id,emp_name,amount,time)
values (?,?,?,?)'" –deploy
I got message:
Created and deployed new stream 'test2'
After that when I am posting data through http it gives following errors:
500 INTERNAL_SERVER_ERROR
On the xd-singlenode, following error log appears:
Caused by: java.io.FileNotFoundException: class path resource [insert_transform.groovy] cannot be opened because it does not exist.
Which version of Spring XD? I just tested it with 1.3.0.RELEASE and it worked fine...
xd:>stream create foo --definition "time | transform --script=test.groovy | log" --deploy
.
$ cat xd/modules/processor/scripts/test.groovy
'Time = ' + payload
.
2016-01-11T09:45:19-0500 1.3.0.RELEASE INFO xdbus.foo.1-1 sink.foo - Time = 2016-01-11 09:45:19
2016-01-11T09:45:20-0500 1.3.0.RELEASE INFO xdbus.foo.1-1 sink.foo - Time = 2016-01-11 09:45:20
2016-01-11T09:45:21-0500 1.3.0.RELEASE INFO xdbus.foo.1-1 sink.foo - Time = 2016-01-11 09:45:21