In our poc, we have a cache in PARTIONED MODE, with 2 backups, and we started 3 nodes. 100 entries were loaded into cache and we did below steps to retrive it.
public void perform () throws GridException {
final GridCache<Long, Entity> cache= g.cache("cache");
GridProjection proj= g.forCache("cache");
Collection< Collection<Entity>> list= proj .compute().broadcast(
new GridCallable< Collection<Entity>>() {
#Override public Collection<Entity> call() throws Exception {
Collection<Entity> values= cache.primaryValues();
System.out.println("List size on each Node: "+ values.size());
// console from each node shows 28,38,34 respectively, which is correct
return values;
}
}).get();
for (Collection<Entity> e: list){
System.out.println("list size when arrives on main Node :"+ e.size());
//console shows 28 for three times, which is not correct
}
}
I assume that primaryValues() is to take value of each element returned by primaryEntrySet() out and put into a Collection. I also tried to use primaryEntrySet and it works without such problem.
The way GridGain serializes cache collections is by reference which may not be very intuitive. I have filed a Jira issue with Apache Ignite project (which is the next version of GridGain open source edition): https://issues.apache.org/jira/browse/IGNITE-38
In the mean time, please try the following from your GridCallable, which should work:
return new ArrayList(cache.primaryValues());
Related
I'm working on a blazor server-side project and I have a component that gets passed a model (pickedWeek) as a parameter. I can use the model fine in-line with the html, but OnInitializedAsync always thinks that the model is null.
I have passed native types in as parameters, from the Page into a component, this way without an issue. I use a NullWeek as a default parameter, so the number getting used in OnInitializedAsync only ever appears to be from the NullWeek. In case this is related, there is a sibling component that is returning the Week model to the Page through an .InvokeAsync call, where StateHasChanged() is being called after the update. It appears that the new Week is getting updated on the problem component, but that OnInitializeAsync() either doesn't see it, or just never fires again- which maybe is my problem, but I didn't think it worked that way.
For instance, the below code will always show "FAILURE" but it will show the correct Week.Number. Code below:
<div>#pickedWeek.Number</div>
#if(dataFromService != null)
{
<div>SUCCESS</div>
}
else
{
<div>FAILURE</div>
}
#code{
[Parameter]
public Week pickedWeek { get; set; }
protected IEnumerable<AnotherModel> dataFromService { get; set; }
protected override async Task OnInitializedAsync()
{
if (pickedWeek.Number > 0)
{
dataFromService = await _injectedService.MakeACall(pickedWeek.Id);
}
}
}
#robsta has this correct in the comments, you can use OnParametersSet for this. Then, you will run into another issue, in that each rerender will set your parameters again and generate another call to your service. I've gotten around this by using a flag field along with the the OnParametersSet method. Give this a shot and report back.
private bool firstRender = true;
protected override async Task OnParametersSetAsync()
{
if (pickedWeek.Number > 0 && firstRender)
{
dataFromService = await _injectedService.MakeACall(pickedWeek.Id);
firstRender = false;
// MAYBE call this if it doesn't work without
StateHasChanged();
}
}
Another alternative is to use the OnAfterRender override, which supplies a firstRender bool in the the method signature, and you can do similar logic. I tend to prefer the first way though, as this second way allows it to render, THEN sets the value of your list, THEN causes another rerender, which seems like more chatter than is needed to me. However if your task is long running, use this second version and build up a loading message to display while the list is null, and another to display if the service call fails. "FAILURE" is a bit misleading as you have it as it's being displayed before the call completes.
I've also found that a call to await Task.Delay(1); placed before your service call can be useful in that it breaks the UI thread loose from the service call awaiter and allows your app to render in a loading state until the data comes back.
I am broadcasting a hashmap and returning a map from the below method
public static Map<Object1, Object2> lkpBC (JavaSparkContext ctx, String FilePath) {
Broadcast<Map<Object1, Object2>> CodeBC = null;
Map<Object1, Object2> codePairMap=null;
try {
Map<Object1, Object2> CodepairMap = LookupUtil.loadLookup(ctx, FilePath);
CodeBC = ctx.broadcast(codePairMap);
codePairMap= CodeBC.value();
} catch (Exception e) {
LOG.error("Error while broadcasting ", e);
}
return codePairMap;
}
and passing the map to the below method
public static JavaRDD<Object3> fetchDetails(
JavaSparkContext ctx,
JavaRDD<Object3> CleanFileRDD,
String FilePath,
Map<Object1, Object2> BcMap
) {
JavaRDD<Object3r> assignCd = CleanFileRDD.map(row -> {
object3 FileData = null;
try {
FileData = row;
if (BCMap.containsKey("some key")) {......}
} catch (Exception e) {
LOG.error("Error in Map function ", e);
}
return some object;
});
return assignCd;
}
in the local mode it works fine without any issues but when i run this on a spark standalone cluster(1 master 3 slaves) on EC2 this doesn't fetch any values nor throws an error. All the objects you see in the methods are serialized. Does it matter if i am calling these methods from a main class or any other different class?
PS: We use Kyro serializer in the spark conf
I think what's going on is you are not accessing the broadcast variable inside the closure of your map function. I think you are directly accessing the underlying BcMap (or BCMap, not sure if they are supposed to be different).
Line if (BCMap.containsKey("some key")) isn't accessing the broadcast variable CodeBC. Since it seems the type of BCMap is Map, not Broadcast.
To access the broadcast variable you would call CodeBC.value.containsKey.
Spark is designed in a functional way, it doesn't "do" anything to the underlying map, it makes a copy of it, broadcasts the copy, and wraps that copy in a Broadcast type.
I don't know what LookupUtil.loadLookup does, but I guess if the file doesn't exist or is empty does it return an empty map?
Here is an example of how you would do it in Scala:
val bcMap = ctx.broadcast(LookupUtil.loadLookup(ctx, FilePath))
cleanFileRDD.map(row =>
if (bcMap.value.containsKey("some key") ...
else ...)
I think you will solve your situation by following the wise words of a friend of mine "first solve all the obvious issues, then the harder issues seem to solve themselves". In your case they are:
Using mutable variables that get initialised to null
Using try catches that log errors but don't re-throw them. Just let exceptions bubble up.
Prematurely splitting things out into lots of different methods before you have it working as just one method.
And just because something works locally doesn't mean it will work when distributed. There are a lot of differences between running something locally and across a cluster, like: a) Data locality b) Serialization c) Closure capture d) Number of threads e) execution order ... etc
I have a data driven CodedUI test method named myTestMethod, which uses XML for supplying input data.
For each run on a data set, CodedUI reports something like this in the Test Explorer:
Test Passed - myTestMethod (Data Row 0)
Test Passed - myTestMethod (Data Row 1)
Test Failed - myTestMethod (Data Row 2) <error details>
Test Failed - myTestMethod (Data Row 3) <error details>
I was wondering is there is a way to set the test name to something more identifiable (probably from the input data set itself).
Seems like CodedUI uses TestContext.TestName for this reporting purpose, but, it is a readonly property. Is there a way to set it somewhere else, somehow?
Please help.
Thanks,
Harit
Ok i understand you now. I actually have a class where i write general functions. One of them is for save the test results as i want.
I use data driven based in XML. then my Employe1 and Employe2 are different runs of the same test_method.
Just locate [TestCleanup()] and call here a function to save a log has you need.
Log can be saved in csv format, plain text separated by ; for example with StreamWriter.
namespace NAME_SPACE
{
[CodedUITest]
public class Program_Something_BlaBla
{
Stopwatch stopWatch = new Stopwatch();
[TestMethod(), Timeout(999999999)]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", PATH_XML, "DATOS", DataAccessMethod.Sequential)]
public void Program_Something_BlaBla_Method()
{
string employe = TestContext.DataRow["EMPLOYE"].ToString();
try
{
//Test actions
...
{
catch (Exception g)
{
...
return;
}
}
#region Additional test attributes
// You can use the following additional attributes as you write your tests:
////Use TestInitialize to run code before running each test
[TestInitialize()]
public void MyTestInitialize()
{
stopWatch.Start();
...
}
//Use TestCleanup to run code after each test has run
[TestCleanup()]
public void MyTestCleanup()
{
stopWatch.Stop();
...
Common.EndTest(employe);
}
#endregion
Hope it helps,
I don't want to say it's impossible, but the Test Explorer in populated at compile time and the data would be pulled in at run time.
I don't think this would be possible. The only way for you is to read the generated "TRX" file ( as it is xml) and replace the values by reading your csvs. Might need to do a tool but the overhead is huge.
I have a big doubt. The problem is Out of Memory Exception in my class. But it seems something strange here. I have class in a dll.
public class MyClass : IDisposible
{
List<ClassA> a_classLists = new .....// new instance.
List<ClassB> b_classLists = new .....// new instance.
public string Method1(int IDValue)
{
// do here some web service call and get some XML data from it.
// parse the xml.
// Iterate through a for loop and add each node value to a_classLists
// Usually contains 10 or 15 items
Method2(); // from here calling another method
FinalSaveToDB(); // finally save the data to DB
return "";
}
private void Method2()
{
// do here some web service call and get some XML data from it.
// Iterate through a forloop.
// parse the xml. [large xml data. ie, image in binary format]
// For each loop add image binary data and other xml to b_classLists
// Usually it contains 50 or 60 such large lists.
}
private void FinalSaveToDB()
{
// using sqlbulkcopy, i am saving the data in the 2 lists to 2 different
// tables in the DB.
// Tab lock is mentioned in sqlbulk class.
// Actually 2 sqlbulkcopy class for 2 lists.
// Only 1 sql connection opens, then do the sqlbulkcopy. [there is no dataset or datareader]
// sqlconnection closes. I am using "using" clause for sqlconnection, bulkcopy etc
// these all are working fine.
}
private void Dispose()
{
// here nulling everything
// proxy null
// all the lists null....
}
}
This is the class I am instantiating 1000 times using reactive framework's Observable.Start
method as shown below...
private IObservable<string> SendEmpDetails(Employee emp)
{
using (MyClass p = new MyClass())
{
return Observable.Start(() => p.Method1(emp.ID), Scheduler.ThreadPool);
}
// here I hope it will call the Dispose and release all objects in the class.
}
// This EmployeeLists contains 1000 employee objects
EmployeeLists.ToObservable().Select(x => SendEmpDetails(x).Select(y => new { emp = x, retval = y }))
.Merge(10)
.ObserveOn(Scheduler.CurrentThread)
.Subscribe(x =>
{
SendStatus(x.retval.Item1, x.retval);
});
Even though, why i am getting out of memory exception ??? After starting the app, when it
process the 200th (or above) MyClass object, it throws error.
I forgot to mention 1 more thing, I am using VS 2010 and C# 4.0 (win7, 64 bit OS).
I need to log each activity. [ie, i need to understand the each and every process the app has gone through]. SO i declared a class [MyClass] level private string variable and assign each process details like "called this method", "got 5 records from this web service" etc.
logdata = Environment.Newline() + "This method has completed";
So the error is thrown here saying out of memory with some evalution failed.
So I turned off the string evaluation check box from Options in VS.
Again, there is no use.
So I changed the string to StringBuilder and tried to append the activity string each time.
Still no use. I dont understand what is the problem in it.
Is this because all the threads are working parallel, do they exchange the MyClass resources ??? Why the objects are not released ???
Please help me in this matter.
I'm trying to use JDT SearchEngine to find references to a given object. But I'm getting a "NullPointerException" while invoking the "search" method of org.eclipse.jdt.core.search.SearchEngine.
Following is the error trace:
java.lang.NullPointerException at
org.eclipse.jdt.internal.core.search.BasicSearchEngine.findMatches(BasicSearchEngine.java:214)
at
org.eclipse.jdt.internal.core.search.BasicSearchEngine.search(BasicSearchEngine.java:515)
at
org.eclipse.jdt.core.search.SearchEngine.search(SearchEngine.java:582)
And following is the method I'm using to perform search:
private static void search(String elementName) { //elementName -> a method Name
try {
SearchPattern pattern = SearchPattern.createPattern(elementName, IJavaSearchConstants.METHOD,
IJavaSearchConstants.REFERENCES, SearchPattern.R_PATTERN_MATCH);
IJavaSearchScope scope = SearchEngine.createWorkspaceScope();
SearchRequestor requestor = new SearchRequestor() {
#Override
public void acceptSearchMatch(SearchMatch match) {
System.out.println("Element - " + match.getElement());
}
};
SearchEngine searchEngine = new SearchEngine();
SearchParticipant[] searchParticipants = new SearchParticipant[] { SearchEngine
.getDefaultSearchParticipant() };
searchEngine.search(pattern, searchParticipants, scope, requestor, null);
} catch (Exception e) {
e.printStackTrace();
}
}
Refer the "Variables" window of the following snapshot to check the values of the arguments passing to the "searchEngine.search()":
I think the the issue is because of the value of "scope" [Highlighted in 'BLACK' above].
Which means "SearchEngine.createWorkspaceScope()" doesn't return expected values in this case.
NOTE: Please note that this is a part of my program which runs as a stand-alone java program (not an eclipse plugin) using JDT APIs to parse a given source code (using JDT-AST).
Isn't it possible to use JDT SearchEngine in such case (non eclipse plugin program), or is this issue due to some other reason?
Really appreciate your answer on this.
No. You cannot use the search engine without openning a workspace. The reason is that the SearchEngine relies on the eclipse filesystem abstraction (IResource, IFile, IFolder, etc.). This is only available when the workspace is open.