Here is the deal, i have a website that is required to search from multiple webservices, then join all the results returned from webservices and display them mixed. I've done the code for search a single place at one
WsPesquisa pesq = new WsPesquisa();
IEnumerable<Objecto> Resultados = pesq.PesquisaObjecto("URL TO SEARCH", "TEXT TO SEARCH");
now i need to use threads to search in multiple places at once but having doubts how to do so.
Can someone please provide a thread sample to call multiple times the code i've used above and then join the results from all threads in a single List of Objectos?
Thanks in advance
One way to do this is to use a standard LINQ query, and use PLINQ to parallelise it.
Assuming you have your query stored in query, a list of the websites you want to search stored in variable called sites, and you have a method SearchSite(string query, string site) that runs the search against a single site, the following should do the trick:
var searchResults = from site in sites.AsParallel()
select SearchSite(query, site);
var resultList = new List<object>();
foreach (var searchResult in searchResults)
{
// process result
resultList.Add(searchResult);
}
This assumes the search query is the same for each site, so:
AsParallel() indicates that you want your LINQ query to be run in parallel
select SearchSite(query, site) - takes your query and runs the SearchSite method on it
PLINQ takes care of waiting for all the results to come in, so you can just process them in a regular for loop
Related
Need to search multiple product codes using using SolrSearchQueryTemplate so that we can get all products in a single solr call
I am able to search one product code at a time using SolrSearchQueryTemplate but not sure how shall modify solr query so that i can pass multiple product codes in solr request
please find below implementation i have done to search single product code using SolrSearchQueryTemplate and it works fine
$productType=testProductType
$ftsQueryBuilder=multiFieldFreeTextQueryBuilder
INSERT_UPDATE SolrSearchQueryTemplate;name[unique=true];ftsQueryBuilder[default=$ftsQueryBuilder];showFacets;restrictFieldsInResponse;indexedType(Identifier)[default=$productType]
;TESTTEMPLATE;;false;true;
INSERT_UPDATE SolrSearchQueryProperty;ftsQuery;indexedProperty(name)[unique=true];searchQueryTemplate(name)[unique=true]
;true;code;TESTTEMPLATE
final PageableData pageableData = createPageableData(0, getSearchPageSize(), null, ShowMode.Page);
final SearchStateData searchState = new SearchStateData();
final SearchQueryData searchQueryData = new SearchQueryData();
searchQueryData.setValue(XSSFilterUtil.filter(prodIdentifier.get(0)));
searchQueryData.setSearchQueryContext(SearchQueryContext.THIRDPARTYINTEGRATION);
searchState.setQuery(searchQueryData);
pageableData.setFlow(MyConstants.THIRDPARTYINTEGRATION);
final ProductSearchPageData<SearchStateData, de.hybris.platform.commercefacades.product.data.ProductData> searchPageData = testCloudProductSearchFacade
.getProductByCodeMafIdBrand(searchState, pageableData, MyConstants.THIRDPARTYINTEGRATION);
List<ProductData> prodFromSolr = searchPageData.getResults();
let me know what modifications are required to be done in this so that i can pass multiple product codes in request
Note:
searchQueryData.setValue(XSSFilterUtil.filter(prodIdentifier.get(0)));
setValue method accepts single parameter of type String
is it possible to send multiple codes separated by some token which SolrSearchQueryTemplate will automatically consider as multiple search terms
I find myself doing a lot of queries to fetch just the first couple of items of a big set, e.g. to show the three most recent news articles or blog posts on the homepage of a website.
As long as this query only involves predefined or custom Parts, I can do something like this:
public IEnumerable<ContentItem> GetTopArticles(int amount)
{
var cultureRecord = _cultureManager.GetCultureByName(_orchardServices.WorkContext.CurrentCulture);
var articles = _orchardServices.ContentManager.Query().ForType("Article")
.Where<LocalizationPartRecord>(lpr => lpr.CultureId == cultureRecord.Id)
.OrderBy<CommonPartRecord>(cpr => cpr.PublishedUtc)
.Slice(0, amount);
return articles;
}
I'm assuming this will more or less be the same as a SELECT TOP [amount] ... in SQL and will have good performance on a large number of records.
However, sometimes I use Migrations or Import to create Content Types from an external source and want to conditionally check a field from the generic Part. In this case I don't have a Part or PartRecord class that I can pass as a parameter to the ContentQuery methods and if I want to do a conditional check on any of the fields I currently do something like this:
public IEnumerable<ContentItem> GetTopArticles(int amount)
{
var articles = _orchardServices.ContentManager.Query().ForType("Article")
.OrderBy<CommonPartRecord>(cpr => cpr.PublishedUtc)
.List()
.Where(a => a.Content.Article.IsFeatured.Value == true)
.Take(amount);
return articles;
}
This is really wasteful and causes large overhead on big sets but I really, REALLY, do not want to delve into the database to figure out Orchard's inner workings and construct long and complex HQL queries every time I want to do something like this.
Is there any way to rewrite the second query with IContentQuery methods without incurring a large performance hit?
I'm working on something similar (being able to query model data with a dynamic name). Sadly, I haven't found anything that makes it easy.
The method I've found that works is to do plain SQL queries against the database. Check out this module for syntax on that if you do later find yourself willing to delve into the database.
I have a use case where I have to remove a subset of entities stored in couchbase, e.g. removing all entities with keys starting with "pii_".
I am using NodeJS SDK but there is only one remove method which takes one key at a time: http://docs.couchbase.com/sdk-api/couchbase-node-client-2.0.0/Bucket.html#remove
In some cases thousands of entities need to be deleted and it takes very long time if I delete them one by one especially because I don't keep list of keys in my application.
I agree with the #ThinkFloyd when he saying: Delete on server should be delete on server, rather than requiring three steps like get data from server, iterate over it on client side and finally for each record fire delete on the server again.
In this regards, I think old fashioned RDBMS were better all you need to do is 'DELETE * from database where something=something'.
Fortunately, there is something similar to SQL is available in CouchBase called N1QL (pronounced nickle). I am not aware about JavaScript (and other language syntax) but this is how I did it in python.
Query to be used: DELETE from <bucketname> b where META(b).id LIKE "%"
layer_name_prefix = cb_layer_key + "|" + "%"
query = ""
try:
query = N1QLQuery('DELETE from `test-feature` b where META(b).id LIKE $1', layer_name_prefix)
cb.n1ql_query(query).execute()
except CouchbaseError, e:
logger.exception(e)
To achieve the same thing: alternate query could be as below if you are storing 'type' and/or other meta data like 'parent_id'.
DELETE from <bucket_name> where type='Feature' and parent_id=8;
But I prefer to use first version of the query as it operates on key, and I believe Couchbase must have some internal indexes to operate/query faster on key (and other metadata).
The best way to accomplish this is to create a Couchbase view by key and then range query over that view via your NodeJS code, making deletes on the results.
http://docs.couchbase.com/admin/admin/Views/views-querySample.html
http://docs.couchbase.com/couchbase-manual-2.0/#couchbase-views-writing-querying-selection-partial
http://docs.couchbase.com/sdk-api/couchbase-node-client-2.0.8/ViewQuery.html
For example, your Couchbase view could look like the following:
function(doc, meta) {
emit(meta.id, null);
}
Then in your NodeJS code, you could have something that looks like this:
var couchbase = require('couchbase');
var ViewQuery = couchbase.ViewQuery;
var query = ViewQuery.from('designdoc', 'by_id');
query.range("pii_", "pii_" + "\u0000", false);
var myBucket = myCluster.openBucket();
myBucket.query(query, function(err, results) {
for(i in results) {
// Delete code in here
}
});
Of course your Couchbase design document and view will be named differently than the example that I gave, but the important part is the ViewQuery.range function that was used.
All document ids prefixed with pii_ would be returned, in which case you can loop over them and start deleting.
Best,
I am scraping an 90K record database using JSON-RPC and I am trying to put in some basic error checking. I want to start by scraping the database twice using two different settings and adding a prefix to the second scrape. This way I can check to ensure that the two settings are not producing different records (due to dropped updates, etc). I wanted to implement the comparison using a view which compares each document from the first scrape with it's twin produced by the second scrape and then emit the names of records with a difference between them.
However, I cannot quite figure out how to pull in another doc in the view, everything I have read only discusses external docs using the emit() function, which is too late to permit me to compare it. In the example below, the lookup() function would grab the referenced document.
Is this just not possible?
function(doc) {
if(doc._id.slice(0,1)!=='$' && doc._id.slice(0,1)!== "_"){
var otherDoc = lookup('$test" + doc._id);
if(otherDoc){
var keys = doc.value.keys();
var same = true;
keys.forEach(function(key) {
if ((key.slice(0,1) !== '_') && (key.slice(0,1) !=='$') && (key!=='expires')) {
if (!Object.equal(otherDoc[key], doc[key])) {
same = false;
}
}
});
if(!same){
emit(doc._id, 1);
}
}
}
}
Context
You are correct that this is not possible in CouchDB. The whole point of the map function is that it must be idempotent, otherwise you lose all the other nice benefits of a pre-calculated index.
This is why you cannot access external resources in the map function, whether they be other records or the clock. Any time you run a map you must always get the same result if you put the same record into it. Since there are no relationships between records in CouchDB, you cannot promise that this is possible.
Solution
However, you can still achieve your end goal, just be different means. Some possibilities...
Assuming there is some meaningful numeric value in each doc, you could use a view to take the sum of all those values and group them by which import you did ({key: <batch id>, value: <meaningful number>}). Then compare the two numbers in your client or the browser to see if they match.
A brute force approach would be to use a view to pair the docs that should match. Each doc is on a different row, but they're grouped by a common field. Then iterate through the entire index comparing the pairs. This would certainly be the quickest to code and doesn't depend on your application or data.
Implement a validation function to enforce a schema on your data. Just be warned that this will reduce your write throughput since each written record will be piped out of Erlang and into the JS engine. Also, this is only applicable if you're worried about properly formed records instead of their precise content, which might not be the case.
Instead of your different batch jobs creating different docs, have them place them into the same doc. The structure might look like this: { "_id": "something meaningful", "batch_one": { ..data.. }, "batch_two": { ..data.. } } Then your validation function could compare them or you could create a view that indexes all the docs that don't match. All depends on where in your pipeline you want to do the error checking and correction.
Personally I like the last option better, but only if you don't plan to use the database as is in production. Ie., you wouldn't want to carry around all that extra data in each record.
Hope that helps.
Cheers.
I am migrating an application from mySQL to couchDB. (Okay, Please dont pass judgements on this).
There is a function with signature
getUserBy($column, $value)
Now you can see that in case of SQL it is a trivial job to construct a query and fire it.
However as far as couchDB is concerned I am supposed to write views with map functions
Currently I have many views such as
get_user_by_name
get_user_by_email
and so on. Can anyone suggest a better and yet scalable way of doing this ?
Sure! One of my favorite views, for its power, is by_field. It's a pretty simple map function.
function(doc) {
// by_field: map function
// A single view for every field in every document!
var field, key;
for (field in doc) {
key = [field, doc[field]];
emit(key, 1);
}
}
Suppose your documents have a .name field for their name, and .email for their email address.
To get users by name (ex. "Alice" and "Bob"):
GET /db/_design/example/_view/by_field?include_docs=true&key=["name","Alice"]
GET /db/_design/example/_view/by_field?include_docs=true&key=["name","Bob"]
To get users by email, from the same view:
GET /db/_design/example/_view/by_field?include_docs=true&key=["email","alice#gmail.com"]
GET /db/_design/example/_view/by_field?include_docs=true&key=["name","bob#gmail.com"]
The reason I like to emit 1 is so you can write reduce functions later to use sum() to easily add up the documents that match your query.