Entity Framework fuzzy search on list of strings - string

EntityFrameworkCore 3.1.14
So i'm looking to implement a search where you can add multiple different elements to a search which will pull the relevant data from the DB (including lists of search params).
I'm fairly new to EF, but couldn't find anything on this..
The requirement is this:
if the list is empty - don't filter (there might be other search params)
if the list contains some strings, do a fuzzy search - i.e. string.Contains(string2)
it works in code - e.g.
var list = new List<string> { "aaa", "bbb" };
var a = new List<string> { "a" };
var c = new List<string> { "c" };
var ab = list.Any(listString => a.Any(aString => listString.Contains(aString))); //returns true
var ac = list.Any(listString => c.Any(cString => listString.Contains(cString))); //returns false
However, it doesn't work in entityframework.. i have tried using the below (only 1 of the lines in the WHERE at a time - result of the search is at the end of the row)
var data = context.Meals
.Where(x =>
(!descriptions.Any() || x.Ingredients.Any(db => descriptions.Any(input => input.Contains(db.Description)))) --FAIL
(!descriptions.Any() || x.Ingredients.Any(db => descriptions.Any(input => db.Description.Contains(input)))) --FAIL
(!descriptions.Any() || descriptions.Any(input => x.Ingredients.Any(db => db.Description.Contains(input)))) --FAIL
(!descriptions.Any() || descriptions.Any(input => x.Ingredients.Any(db => input.Contains(db.Description)))) --FAIL
(!descriptions.Any() || x.Ingredients.Any(db => db.Description.Contains(descriptions.First()))) --PASS - but restricted to using .First()
(!descriptions.Any() || x.Ingredients.Any(db => descriptions.Contains(db.Description))) --PASS - but no fuzzy logic
).ToList();
ERROR MESSAGE:
The LINQ expression ... could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().

i found a solution - which is to loop through the list of strings outside EF and for each element, add a new where clause to the EF object..
However, this fix seems more like a workaround, than a proper implementation for using fuzzy search on 2 sets of strings
var data = context.Meals;
if(descriptions.Any()){
descriptions.ForEach(input => data = data.Where(x => x.Any(db => db.Ingredient.Contains(input))))
}
return data.ToList();

Related

Using Node.js iterating array of object and if value matched then insert all related values in different array

INPUT
[
{"Id":1,"text":"Welcome","question":"san","translation":"willkommen."},
{"Id":1,"text":"Welcome","question":"se","translation":"bienvenida"},
{"Id":1,"text":"Welcome","question":"fr","translation":"propriétaires"},
{"Id":1,"text":"ajax","question":"san","translation":"ommen."},
{"Id":1,"text":"ajax","question":"se","translation":"bienve"},
{"Id":1,"text":"ajax","question":"fr","translation":"propires"}
]
if question = san then all "san" objects will be inserted in array like and so on-
san:[{"text":"Welcome","question":"san","translation":"willkommen.},
{"text":"ajax","question":"san","translation":"ommen."},
se:[{"text":"Welcome","question":"se","translation":"bienvenida.},
{"text":"ajax","question":"se","translation":"bienve."},
fr:[{"text":"Welcome","question":"fr","translation":"propriétaires.},
{"text":"ajax","question":"fr","translation":"propires."},
Question is how do i check if question=san then make one array and insert all san values in it and so on without hardcoding the question property values.
Tried looping things but how to match without hardcoding because in future question attribute can change .
question="san" will be all together in an array "se" will be all together in an array and so on.
New to this not know much about nodejs.
Tried something like this but not coming as required way
fs.readFile('./data.json', 'utf8', function (err,data) {
data = JSON.parse(data);
var array = [];
for(var i = 0; i < data.length; i++) {
var lang = data[i].language;
for(var j= 0; j< data.length; j++) {
if(lang == data[j].language){
array.push(data[j].language);
array.push(data[j].translation);
array.push(data[j].text);
}
}
}
output Required
san:[{"text":"Welcome","question":"san","translation":"willkommen.},
{"text":"ajax","question":"san","translation":"ommen."},
se:[{"text":"Welcome","question":"se","translation":"bienvenida.},
{"text":"ajax","question":"se","translation":"bienve."},
fr:[{"text":"Welcome","question":"fr","translation":"propriétaires.},
{"text":"ajax","question":"fr","translation":"propires."},
I recommend you to use ES6 functions instead of for. You can separate the different processes and make the code more modular and declarative. This way you can change easily the desired output since your code is made by little pieces.
const data = [
{"Id":1,"text":"hi all present ","language":"sde","translation":"Hernjd ndjjsjdj"},
{"Id":1,"text":"hi all present","language":"ses","translation":"dfks kdfk kdfk"},
{"Id":1,"text":"hi all present","language":"sfr","translation":"bsh kkoweofeo"},
{"Id":1,"text":"hi all present","language":"szh","translation":"kdijo keow"},
{"Id":1,"text":"activated","language":"sde","translation":"Konto eid ke"},
{"Id":1,"text":"activated","language":"ses","translation":"La cueweffewfefwef."},
{"Id":1,"text":"activated","language":"sfr","translation":"Cowefrwef"},
{"Id":1,"text":"activated","language":"szh","translation":"fhewjhfwh"},
{"Id":1,"text":"completed","language":"sde","translation":"Ihr fwejiewf"},
{"Id":1,"text":"completed","language":"ses","translation":"Ya hfuwifrw"},
{"Id":1,"text":"completed","language":"sfr","translation":"Votrkwfwe"},
{"Id":1,"text":"completed","language":"szh","translation":"dmksfkwkf"},
{"Id":1,"text":"ACTION","language":"sde","translation":"AKTION"},
{"Id":1,"text":"ACTION","language":"ses","translation":"ACCIONES"},
{"Id":1,"text":"ACTION","language":"fr","translation":"ACTION"}];
// Define the properties that we want to filter for each element
const filterProperties = (item) => ({
text:item.text,
language: item.language,
translation:item.translation
})
// Given a type of languages ('sde'), filter the data in function of this value
const getItemsByLanguage = (language) => {
return data.filter((item) => item.language === language)
}
const onlyUnique = (value, index, self) => {
return self.indexOf(value) === index;
}
// Get the unique values of languages: ['sde', 'ses', 'sfr', ...]
const uniqueLanguages = data.map((item) => item.language).filter(onlyUnique)
// Get all found items for a language ('sde') and get the desired format (returns array of objects)
const resultArray = uniqueLanguages.map((language) => (
{[language]: getItemsByLanguage(language).map(filterProperties)}
))
// Convert the array of objects to single object
const result = Object.assign({}, ...resultArray)
console.log(result)
const data = [
{"Id":1,"text":"hi all present ","language":"sde","translation":"Hernjd ndjjs
jdj"},
{"Id":1,"text":"hi all present","language":"ses","translation":"dfks kdfk
kdfk"},
{"Id":1,"text":"hi all present","language":"sfr","translation":"bsh kkowe
ofeo"},
{"Id":1,"text":"hi all present","language":"szh","translation":"kdijo keow"},
{"Id":1,"text":"activated","language":"sde","translation":"Konto eid ke"},
{"Id":1,"text":"activated","language":"ses","translation":"La cueweffewfef
wef."},
{"Id":1,"text":"activated","language":"sfr","translation":"Cowefrwef"},
{"Id":1,"text":"activated","language":"szh","translation":"fhewjhfwh"},
{"Id":1,"text":"completed","language":"sde","translation":"Ihr fwejiewf"},
{"Id":1,"text":"completed","language":"ses","translation":"Ya hfuwifrw"},
{"Id":1,"text":"completed","language":"sfr","translation":"Votrkwfwe"},
{"Id":1,"text":"completed","language":"szh","translation":"dmksfkwkf"},
{"Id":1,"text":"ACTION","language":"sde","translation":"AKTION"},
{"Id":1,"text":"ACTION","language":"ses","translation":"ACCIONES"},
{"Id":1,"text":"ACTION","language":"fr","translation":"ACTION"}];
// Define the properties that we want to filter for each element
const filterProperties = (data) => ({
text:data.text,
question: data.question,
translation:data.translation
})
// Given a type of question ('san'), filter the data in function of this value
const getQuestions = (question) => {
return data.filter((item) => item.question === question)
}
const onlyUnique = (value, index, self) => {
return self.indexOf(value) === index;
}
// Get the unique values of questions: ['san', 'se', 'fr']
const uniqueQuestions = data.map((item) => item.question).filter(onlyUnique)
// Get all found values for a question and get the desired format (returns
array of objects)
const resultArray = uniqueQuestions.map((question) => (
{[question]: getQuestions(question).map(filterProperties)}
))
// Convert the array of objects to single object
const result = Object.assign({}, ...resultArray)
console.log(result)

Cakephp 3 - Remove empty post from pagination

I'm working on multi language website.
In the English version of the website all works properly, because I entered translations.
$blogPosts = $this->BlogPosts->find('all')->where(['BlogPosts.active' => 1]);
$this->set('blogPosts',$this->paginate($blogPosts));
If I changed the language of website I would like to remove all the posts whose translation of the title is not entered.
I tried this, but it does not work:
$blogPosts = $this->BlogPosts->find('all')->where(['BlogPosts.active' => 1,'BlogPosts.title IS NOT' => null]);
$this->set('blogPosts',$this->paginate($blogPosts));
Still printed posts without titles.
How to solve this problem?
Try this conditions:
public function index()
{
$this->paginate['conditions'] = ['BlogPosts.active' => true , 'BlogPosts.title IS NOT' => null, 'BlogPosts.title !=' => ' '];
$this->paginate['order'] = ['BlogPosts.id' => 'desc'];
$this->paginate['limit'] = 4;
$blogPosts = $this->paginate($this->BlogPosts);
$this->set(compact('blogPosts'));
$this->set('_serialize', ['blogPosts']);
}
Query:
SELECT *
FROM blogPosts BlogPosts
WHERE (
BlogPosts.active = 1
AND BlogPosts.title IS NOT NULL
AND BlogPosts.title != ''
)
ORDER BY BlogPosts.id desc
LIMIT 4 OFFSET 0
I think faced with simmilar issue before. The pagination didn't worked as I excepted when I set order, limit together with custom where condition.
Try this:
$this->paginate['order'] = ['BlogPosts.id' => 'desc'];
$this->paginate['limit'] = 4;
$query = $this->BlogPost->find()
->select(['id', 'title', 'active'])
->where(['BlogPost.active' => true, 'BlogPost.title IS NOT' => null, 'BlogPost.id !=' => ''])
$data = $this->paginate($query);
$this->set('BlogPost', $data);

Elastic Search-Search string having spaces and special characters in it using C#

I am looking for ElasticSearch nest query which will provide exact match on string having spaces in it using C#.
for example - I want to search for a word like 'XYZ Company Solutions'. I tried querystring query but it gives me all the records irrespective of search result. Also i read on the post and found that we have to add some mappings for the field. I tried 'Not_Analyzed' analyzer on the field but still it does not worked.
Here is my code of C#
var indexDefinition = new RootObjectMapping
{
Properties = new Dictionary<PropertyNameMarker, IElasticType>(),
Name = elastic_newindexname
};
var notAnalyzedField = new StringMapping
{
Index = FieldIndexOption.NotAnalyzed
};
indexDefinition.Properties.Add("Name", notAnalyzedField);
objElasticClient.DeleteIndex(d => d.Index(elastic_newindexname));
var reindex = objElasticClient.Reindex<dynamic>(r => r.FromIndex(elastic_oldindexname).ToIndex(elastic_newindexname).Query(q => q.MatchAll()).Scroll("10s").CreateIndex(i => i.AddMapping<dynamic>(m => m.InitializeUsing(indexDefinition))));
ReindexObserver<dynamic> o = new ReindexObserver<dynamic>(onError: e => { });
reindex.Subscribe(o);**
**ISearchResponse<dynamic> ivals = objElasticClient.Search<dynamic>(s => s.Index(elastic_newindexname).AllTypes().Query(q => q.Term("Name","XYZ Company Solutions")));** //this gives 0 records
**ISearchResponse<dynamic> ivals1 = objElasticClient.Search<dynamic>(s => s.Index(elastic_newindexname).AllTypes().Query(q => q.Term(u => u.OnField("Name").Value("XYZ Company Solutions"))));** //this gives 0 records
**ISearchResponse<dynamic> ivals = objElasticClient.Search<dynamic>(s => s.Index(elastic_newindexname).AllTypes().Query(#"Name = 'XYZ Company Solutions'"));** //this gives all records having fields value starting with "XYZ"
If anyone have complete example or steps in C# then can you please share with me?
Have you tried the match_phrase query?
The query DSL the request is the following:
"query": {
"match_phrase": {
"title": "XYZ Company Solutions"
}
}
In C# try the following:
_client.Search<T>(s => s
.Index(IndexName)
.Types(typeof (T))
.Query(q => q.MatchPhrase(m => m
.OnField(f => f.Name)
.Query("XYZ Company Solutions"))));
Check the official documentation for more information:
http://www.elastic.co/guide/en/elasticsearch/guide/master/phrase-matching.html#phrase-matching
Please refer the below code ,I think this will meet your requirements.
Here I have created and mapped index with dynamic template and then did the XDCR.
Now all string fields will be not_analysed.
IIndicesOperationResponse result = null;
if (!objElasticClient.IndexExists(elastic_indexname).Exists)
{
result = objElasticClient.CreateIndex(elastic_indexname, c => c.AddMapping<dynamic>(m => m.Type("_default_").DynamicTemplates(t => t
.Add(f => f.Name("string_fields").Match("*").MatchMappingType("string").Mapping(ma => ma
.String(s => s.Index(FieldIndexOption.NotAnalyzed)))))));
}
Thanks
Mukesh Raghuwanshi
It looks like you just need to refresh the new index following the re-indexing operation.
Using your code example (and your first term query), I was seeing the same result -- 0 hits.
Adding the following Refresh call after the reindex.Subscribe() call results in a single hit being returned:
objElasticClient.Refresh(new RefreshRequest() { });

ServiceStack Ormlite SqlExpressionVisitor null check in Where extension

I would like to write a method for querying table with one method by null cheking parameters using SqlExpressionVisitor of Ormlite
Here is my method :
public static List<UserChatsDTO> GetUserChats(int startRow, int rowCount, DateTime? startDate, DateTime? endDate, string operatorName, short? rating, string visitorName)
{
using (IDbConnection db = DbFactory.OpenDbConnection())
{
SqlExpressionVisitor<UserChatsDTO> ev = OrmLiteConfig.DialectProvider.ExpressionVisitor<UserChatsDTO>();
ev.Where(q =>
(startDate.HasValue && q.Dated >= startDate) &&
(endDate.HasValue && q.Dated <= endDate) &&
(!string.IsNullOrEmpty(operatorName) && q.TakenByUser.Contains(operatorName)) &&
(rating.HasValue && q.Rating == (short)rating) &&
(!string.IsNullOrEmpty(visitorName) && q.VisitorName.Contains(visitorName)));
//ev.OrderBy();
ev.Limit(startRow, rowCount);
return db.Select<UserChatsDTO>(ev);
}
}
But Object reference not set to an instance of an object. NullReferenceException is thrown when i call ev.Where part.
Is there a bug here or i am missing something ?
Thank you.
You can actually build up the ExpressionVisitor inside the Select method like so:
var chats = db.Select<UserChatsDTO>(q => q
.Where(x => startDate.HasValue && x.Date >= startDate)
.Where(x => endDate.HasValue && x.Date <= endDate)
.Where(x => string.IsNullOrEmpty(operatorName) || x.TakeByUser.Contains(operatorName))
.Where(x => rating.HasValue && x.Rating == (short)rating)
.Where(x => string.IsNullOrEmpty(visitorName) || x.VisitorName.Contains(visitorName)
.Limit(startRow, rowCount));
I know this question is 7 months old but I had a similar issue & this was the first question that came up when I searched. I wanted to add my working solution since Master Morality's didn't work for me.
Originally, I tried syntax roughly similar to mustafasturan's first attempt. I got the same NullReferenceException he did. Master Morality's answer did not help either...
I'm trying to build a search function that performs LIKE searches rather than exact match. There are multiple criteria on the request object which may or may not be null (for simplicity's sake we'll use an example with 2 criteria). Following Master Morality's suggestion, I tried this:
var searchResults = db.Select<Item>(q => q
.Where(x => string.IsNullOrWhiteSpace(request.Criteria1) || x.Criteria1.Contains(request.Criteria1))
.Where(x => string.IsNullOrWhiteSpace(request.Criteria2) || x.Criteria2.Contains(request.Criteria2))
);
I still got a NullReferenceException. Seems like the && and || operators are not using short-circuit evaluation inside the lambda expression.
What I finally had to go with is this:
SqlExpressionVisitor<Item> ev = new ServiceStack.OrmLite.MySql.MySqlExpressionVisitor<Item>();
if (!String.IsNullOrWhiteSpace(request.Criteria1)) {
ev.Where(q => q.Criteria1.Contains(request.Criteria1));
}
if (!String.IsNullOrWhiteSpace(request.Criteria2)) {
ev.Where(q => q.Criteria2.Contains(request.Criteria2));
}
searchResults = db.Select<Item>(ev);
It doesn't feel very elegant, but it's the only thing I could find that works.

Performance issue : How to execute Two lambda expression at once?. "Contains" and "Any" operator used

Sample code
var Ids = _db.Projects.Where(Project=>Project.Title!="test23rdoct")
.Select (pro => pro.Id);
Expression<Func<Company, bool>> masterExpression =
Company => Company.Participants.Any(part => ids.Contains(part.Project.Id));
IQueryable<Object> queryEntity = _db.Companies.Where(masterExpression)
The above query executing twice. Storing ids in the server(sometime ids are more than 50k count). It causes performance issues. Could anybody suggest how to combine these two queries and execute at once?
How about:
var queryEntity = _db.Companies.Where(c => c.Partipants.Any(p => p.Project.Title != "test23rdoct"));
EDIT:
With the complex query, you could also split that:
Func<Project, bool> projectFilter = Project => ((Compare(Convert(Project.Title), "a") > 0) AndAlso ((Convert(Project.Title) != "test23rdoct") AndAlso
(Project.Participants.Any(Participant => (Compare(Convert(Participant.ParticipantRole.Name), "Finance") > 0)) AndAlso
(Project.Participants.Any(Participant => (Convert(Participant.Person.FirstName) != "test1")) AndAlso
Project.Participants.Any(Participant => (Compare(Convert(Participant.Company.Name), "test") > 0))))));
And then do:
var queryEntity = _db.Companies.Where(c => c.Partipants.Any(p => projectFilter(p.Project));
Would something like this using Join suit your needs?
Expression<Func<Company, bool>> masterExpression =
Company => Company.Participants.Join (Ids, p => p.Project.ID, id => id, (p, id) => p).Any();
IQueryable<Object> queryEntity = _db.Companies.Where(masterExpression);
I got this solution for avoiding execution of Lambda twice. To achieve this I used these extension methods Invoke() and AsExpandable(). Its available in Linqkit dll.
Expression<Func<Company, bool>> masterExpression = Company => Company.Participants.Any(part => masterLamba.Invoke(part.Project));
queryEntity = _db.Companies.AsExpandable().Where(masterExpression);

Resources