Where, in a scala application, is the best place to store a Spark broadcast variable, so that it can be referenced elsewhere in the app?
val broadcast:Broadcast = ...
It does not appear to be possible to save it in an Object, because:
an object cannot have an uninitialized variable, so it cannot be
passed a reference to the broadcast variable by calling a method on the object to set the reference.
an Object cannot setup the broadcast variable
itself, as an object has no constructor into which to pass a
reference to the SparkContext.
Thanks
Using a var rather than a val is likely the solution.
The following is one approach. In the example, the broadcast variable is being used to hold a cache.
Object Cache {
private var cache:Broadcast;
// This method must be called by client to initialize the cache
def init(sc:SparkContext) = {
cache = sc.broadcast(loadCache)
}
def getCache() = {
// check that variable is initialized
if cache!=null {
Some(cache)
}
else
{
None
}
}
private def loadCache():List[String] = {
// load data from DB
}
}
Related
I'm passing an object from a parent component to a child component via Inputs(). I know because I'm passing an object and not a primitive type it's passing a reference to the object. Thus when the object changes in the parent I see it being reflected in the child.
What is the optimal way to pass an object via Inputs() that does NOT update the child component when it changes in the parent?
You need to have two properties that you use. One that you modify and a second that is passed to the child component but is a clone of the original.
#Component({..})
export class MyComponent implements OnInit {
// the original object value
public value: any;
// a value used by child components
public forChild: any;
public OnInit() {
// use deconstruction to make a copy
this.forChild = {...this.value};
// use assign to make a copy
this.forChild = Object.assign({}, this.value);
// use JSON to make a deep copy
this.forChild = JSON.parse(JSON.stringify(this.value));
}
}
As you mentioned:
When we pass objects using #Input(), it would be passed as a reference,
and When we pass primitive types, it would be passed as a value.
So i think one solution is to convert your object to string, and then pass it using #Input(). Later you can decode this stringifyed object to a object if you need.
var method = 'serviceName.MethodName'
I Just want to call it like
serviceName.methodName(function(output callback){
});
Is there any approach to call it.thanks
There are two methods that I can think of now.
JS eval
You can use the javascript eval function to convert any string into code snippet like below. Although eval is a quick solution but should not be used unless you dont have any other option by your side.
var method = 'UserService.getData';
eval(method)();
Factory pattern
Use a below pattern to get the service
You would need to define the services in such a manner that you can access them using a pattern.
var Services = {
// UserService and AccountsService are again objects having some callable functions.
UserService : {getData: function(){}, getAge: function(){}},
AccountsService : {getData: function(){}, getAge: function(){}},
// getService is the heart of the code which will get you the required service depending on the string paramter you pass.
getService : function(serviceName){
var service = '';
switch(serviceName){
case 'User':
service = this.UserService;
break;
case 'Accounts':
service = this.AccountsService;
break;
}
return service;
}
}
You can use get the required service with below code
Services.getService('User')
I'm not aware of any way you can resolve the serviceName part of that string to an object, without using eval. So obviously you need to be extremely careful.
Perhaps:
if (method.match(/^[a-zA-Z0-9_]+\.[a-zA-Z0-9_]+$/) {
var servicePart = eval(method.split('.')[0]);
var methodPart = method.split('.')[1];
servicePart[methodPart](...)
}
There are two separate problems in your question:
How to access object property by property name (string)?
How to access object by it's name (string)?
Regarding the first problem - it is easy to access object property by string using the following notation:
const myObject = {
myProp: 1,
};
console.log(myObject['myProp']);
And regarding the second problem - it depends on what serviceName is:
if it is a property of some other object, then use someObject['serviceName']['MethodName']
if it is a local variable, consider using a Map (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) to associate strings with objects;
I have an issue with the following Groovy script/
nestedView("JS Pipelines") {
views {
build_pipelines.each {
def build_pipeline = it
buildPipelineView(build_pipeline.build_name + " JS Pipeline") {
selectedJob(build_pipeline.start_job)
}
}
}
}
It requires many calls to the buildPipelineView method, which sits on the views object (code), based on the contents of build_pipelines.
When the call to buildPipelineView is wrapped in each {} call, the method resolves to another buildPipelineView method that is defined in a much higher context. When each block is removed, the method call resolves properly. How can the buildPipelineView resolve properly within each call?
If in doubt store the outer reference, which will be visible to the closure and call this method explicit on that object:
nestedView("JS Pipelines") {
views {
final outer = it
build_pipelines.each { build_pipeline ->
outer.buildPipelineView(build_pipeline.build_name + " JS Pipeline") {
selectedJob(build_pipeline.start_job)
}
}
}
}
In fact, the ContextHelper.groovy of job-dsl-plugin has changed the delegation strategy of the views closure, exactly the brackets of views.
In your case, only the build_pipelines was wrapped in views closure, so you did change the delegation strategy of each closure, but buildPipelineView closure still kept default strategy.
You can use getDelegate to give back the delegation strategy to buildPipelineView closure.
nestedView("JS Pipelines") {
views {
build_pipelines.each {
def build_pipeline = it
def d = getDelegate()
d.buildPipelineView(build_pipeline.build_name + " JS Pipeline") {
selectedJob(build_pipeline.start_job)
}
}
}
}
In nested closures, each one has an owner which is its immediately containing closure. (See Delegation strategy in the Groovy docs.) For the closure you pass to each, its owner is the closure passed to views.
A Groovy closure's default delegation strategy is Closure.OWNER_FIRST, which means when you access a property or calls a method without explicitly mentioning the receiver [like buildPipelineView(...)], Groovy will try to access the property/method on the owner, and if there is none, it'll access the property/method on the delegate.
The straightforward solution is to explicitly provide the receiver, e.g. this.buildPipelineView(...).
I have an Actor and some other object:
object Config {
val readValueFromConfig() = { //....}
}
class MyActor extends Actor {
val confValue = Config.readValueFromConfig()
val initValue = Future {
val a = confValue // sometimes it's null
val a = Config.readValueFromConfig() //always works well
}
//..........
}
The code above is a very simplified version of what I actually have. The odd thing is that sometimes val a = confValue returns null, whereas if I replace it with val a = Config.readValueFromConfig() then it always works well.
I wonder, is this due to the fact that the only way to interact with an actor is sending it a message? Therefore, since val confValue is not a local variable, I must either use val a = Config.readValueFromConfig() (a different object, not an actor) or val a = self ! GetConfigValue and read the result afterwards?
val readValueFromConfig() = { //....}
This gives me a compile error. I assume you mean without parentheses?
val readValueFromConfig = { //....}
Same logic with different timing gives different result = a race condition.
val confValue = Config.readValueFromConfig() is always executed during construction of MyActor objects (because it's a field of MyActor). Sometimes this is returning null.
val a = Config.readValueFromConfig() //always works well is always executed later - after MyActor is constructed, when the Future initValue is executed by it's Executor. It seems this never returns null.
Possible causes:
Could be explained away if the body of readValueFromConfig was dependent upon another
parallel/async operation having completed. Any chance you're reading the config asynchronously? Given the name of this method, it probably just reads synchronously from a file - meaning this is not the cause.
Singleton objects are not threadsafe?? I compiled your code. Here's the decompilation of your singleton object java class:
public final class Config
{
public static String readValueFromConfig()
{
return Config..MODULE$.readValueFromConfig();
}
}
public final class Config$
{
public static final MODULE$;
private final String readValueFromConfig;
static
{
new ();
}
public String readValueFromConfig()
{
return this.readValueFromConfig;
}
private Config$()
{
MODULE$ = this;
this.readValueFromConfig = // ... your logic here;
}
}
Mmmkay... Unless I'm mistaken, that ain't thread-safe.
IF two threads are accessing readValueFromConfig (say Thread1 accesses it first), then inside method private Config$(), MODULE$ is unsafely published before this.readValueFromConfig is set (reference to this prematurely escapes the constructor). Thread2 which is right behind can read MODULE$.readValueFromConfig before it is set. Highly likely to be a problem if '... your logic here' is slow and blocks the thread - which is precisely what synchronous I/O does.
Moral of story: avoid stateful singleton objects from Actors (or any Threads at all, including Executors) OR make them become thread-safe through very careful coding style. Work-Around: change to a def, which internally caches the value in a private val.
I wonder, is this due to the fact that the only way to interact with an actor is sending it a message? Therefore, since val confValue is not a local variable, I must either use val a = Config.readValueFromConfig() (a different object, not an actor)
Just because it's not an actor, doesn't mean it's necessarily safe. It probably isn't.
or val a = self ! GetConfigValue and read the result afterwards?
That's almost right. You mean self ? GetConfigValue, I think - that will return a Future, which you can then map over. ! doesn't return anything.
You cannot read from an actor's variables directly inside a Future because (in general) that Future could be running on any thread, on any processor core, and you don't have any memory barrier there to force the CPU caches to reload the value from main memory.
A SoapUI project can run random script upon load.
Load Script is invoked with log and project variables.
In my shared lib I have method - addAsserts() that traverses the whole project and adds schema compliance assertions to SOAP test steps. In my Load Script I call shared method
addAsserts(this)
passing 'this' as a parameter and set closure.delegate to it inside addAsserts method to make 'project' variable accessible within the closure scope
addAsserts method is defined in sharedUtil.groovy:
static def addAsserts(that){
def closure={
project.testSuites.each { testSuiteName, testSuiteObject ->
testSuiteObject.testCases.each { testCaseName, testCaseObject ->
testCaseObject.testSteps.each { testStepName, testStepObject ->
if ("class com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep" == testStepObject.getClass().toString() ) {
log.info "adding 'Schema Compliance' assertion to ${testSuiteName}/${testCaseName}/${testStepName}"
testStepObject.addAssertion('Schema Compliance')
}
}
}
}
}//closure
closure.delegate=that // <--- i would like NOT to pass 'that' as parameter
// but rather detect in runtime with some kind of
// getCallerInstance() method
return closure.call()
}
QUESTION:
Is it possible to detect caller instance in runtime with some kind of getCallerInstance() method ?
No, I don't believe this is possible. Wasn't in Java either (you can find out the name/method of the calling class using some horrible stacktrace hacking, but not the instance of the class itself)
Edit...
It might be possible with a Category (but I am not experienced with SoapUI, so I don't know if this technique would fit)
Say we have a class Example defined like so:
class Example {
String name
}
We can then write a class very similar to your example code, which in this case will set the delegate of the closure, and the closure will print out the name property of the delegate (as we have set the resolve strategy to DELEGATE_ONLY)
class AssetAddingCategory {
static def addAsserts( that ) {
def closure = {
"Name of object: $name"
}
closure.delegate = that
closure.resolveStrategy = Closure.DELEGATE_ONLY
closure.call()
}
}
Later on in our code, it is then possible to do:
def tim = new Example( name:'tim' )
use( AssetAddingCategory ) {
println tim.addAsserts()
}
And this will print out
Name of object: tim