I have an object that looks like this:
let queries = [
{
name: "checkUsers",
query: 'select * from users where inactive = 1'
},
{
name: "checkSubscriptions",
query: 'select * from subscriptions where invalid = 1'
}
]
I am making an AWS Lambda function that will iterate these queries, and if any of them returns a value, I will send an email.
I have come up with this pseudo code:
for (let prop in queries) {
const result = await mysqlConnector.runQuery(prop.query).catch(async error => {
// handle error in query
});
if (result.length < 0){
// send email
}
}
return;
I am wondering is this ideal approach? I need to iterate all the object queries.
I don't see anything wrong with what you are trying to achieve but there are few changes you could do
Try to use Promise.all if you can. This will speed up the overall process as things will execute in parallel. It will depend on number of queries as well.
Try leverage executing multiple statements in one query. This way you will make one call and then you can add the logic to identify. Check here
From the docs:
You can also chain multiple where() methods to create more specific queries (logical AND).
How can I perform an OR query?
Example:
Give me all documents where the field status is open OR upcoming
Give me all documents where the field status == open OR createdAt <= <somedatetime>
OR isn't supported as it's hard for the server to scale it (requires keeping state to dedup). The work around is to issue 2 queries, one for each condition, and dedup on the client.
Edit (Nov 2019):
Cloud Firestore now supports IN queries which are a limited type of OR query.
For the example above you could do:
// Get all documents in 'foo' where status is open or upcmoming
db.collection('foo').where('status','in',['open','upcoming']).get()
However it's still not possible to do a general OR condition involving multiple fields.
With the recent addition of IN queries, Firestore supports "up to 10 equality clauses on the same field with a logical OR"
A possible solution to (1) would be:
documents.where('status', 'in', ['open', 'upcoming']);
See Firebase Guides: Query Operators | in and array-contains-any
suggest to give value for status as well.
ex.
{ name: "a", statusValue = 10, status = 'open' }
{ name: "b", statusValue = 20, status = 'upcoming'}
{ name: "c", statusValue = 30, status = 'close'}
you can query by ref.where('statusValue', '<=', 20) then both 'a' and 'b' will found.
this can save your query cost and performance.
btw, it is not fix all case.
I would have no "status" field, but status related fields, updating them to true or false based on request, like
{ name: "a", status_open: true, status_upcoming: false, status_closed: false}
However, check Firebase Cloud Functions. You could have a function listening status changes, updating status related properties like
{ name: "a", status: "open", status_open: true, status_upcoming: false, status_closed: false}
one or the other, your query could be just
...where('status_open','==',true)...
Hope it helps.
This doesn't solve all cases, but for "enum" fields, you can emulate an "OR" query by making a separate boolean field for each enum-value, then adding a where("enum_<value>", "==", false) for every value that isn't part of the "OR" clause you want.
For example, consider your first desired query:
Give me all documents where the field status is open OR upcoming
You can accomplish this by splitting the status: string field into multiple boolean fields, one for each enum-value:
status_open: bool
status_upcoming: bool
status_suspended: bool
status_closed: bool
To perform your "where status is open or upcoming" query, you then do this:
where("status_suspended", "==", false).where("status_closed", "==", false)
How does this work? Well, because it's an enum, you know one of the values must have true assigned. So if you can determine that all of the other values don't match for a given entry, then by deduction it must match one of the values you originally were looking for.
See also
in/not-in/array-contains-in: https://firebase.google.com/docs/firestore/query-data/queries#in_and_array-contains-any
!=: https://firebase.googleblog.com/2020/09/cloud-firestore-not-equal-queries.html
I don't like everyone saying it's not possible.
it is if you create another "hacky" field in the model to build a composite...
for instance, create an array for each document that has all logical or elements
then query for .where("field", arrayContains: [...]
you can bind two Observables using the rxjs merge operator.
Here you have an example.
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/observable/merge';
...
getCombinatedStatus(): Observable<any> {
return Observable.merge(this.db.collection('foo', ref => ref.where('status','==','open')).valueChanges(),
this.db.collection('foo', ref => ref.where('status','==','upcoming')).valueChanges());
}
Then you can subscribe to the new Observable updates using the above method:
getCombinatedStatus.subscribe(results => console.log(results);
I hope this can help you, greetings from Chile!!
We have the same problem just now, luckily the only possible values for ours are A,B,C,D (4) so we have to query for things like A||B, A||C, A||B||C, D, etc
As of like a few months ago firebase supports a new query array-contains so what we do is make an array and we pre-process the OR values to the array
if (a) {
array addObject:#"a"
}
if (b) {
array addObject:#"b"
}
if (a||b) {
array addObject:#"a||b"
}
etc
And we do this for all 4! values or however many combos there are.
THEN we can simply check the query [document arrayContains:#"a||c"] or whatever type of condition we need.
So if something only qualified for conditional A of our 4 conditionals (A,B,C,D) then its array would contain the following literal strings: #["A", "A||B", "A||C", "A||D", "A||B||C", "A||B||D", "A||C||D", "A||B||C||D"]
Then for any of those OR combinations we can just search array-contains on whatever we may want (e.g. "A||C")
Note: This is only a reasonable approach if you have a few number of possible values to compare OR with.
More info on Array-contains here, since it's newish to firebase docs
If you have a limited number of fields, definitely create new fields with true and false like in the example above. However, if you don't know what the fields are until runtime, you have to just combine queries.
Here is a tags OR example...
// the ids of students in class
const students = [studentID1, studentID2,...];
// get all docs where student.studentID1 = true
const results = this.afs.collection('classes',
ref => ref.where(`students.${students[0]}`, '==', true)
).valueChanges({ idField: 'id' }).pipe(
switchMap((r: any) => {
// get all docs where student.studentID2...studentIDX = true
const docs = students.slice(1).map(
(student: any) => this.afs.collection('classes',
ref => ref.where(`students.${student}`, '==', true)
).valueChanges({ idField: 'id' })
);
return combineLatest(docs).pipe(
// combine results by reducing array
map((a: any[]) => {
const g: [] = a.reduce(
(acc: any[], cur: any) => acc.concat(cur)
).concat(r);
// filter out duplicates by 'id' field
return g.filter(
(b: any, n: number, a: any[]) => a.findIndex(
(v: any) => v.id === b.id) === n
);
}),
);
})
);
Unfortunately there is no other way to combine more than 10 items (use array-contains-any if < 10 items).
There is also no other way to avoid duplicate reads, as you don't know the ID fields that will be matched by the search. Luckily, Firebase has good caching.
For those of you that like promises...
const p = await results.pipe(take(1)).toPromise();
For more info on this, see this article I wrote.
J
OR isn't supported
But if you need that you can do It in your code
Ex : if i want query products where (Size Equal Xl OR XXL : AND Gender is Male)
productsCollectionRef
//1* first get query where can firestore handle it
.whereEqualTo("gender", "Male")
.addSnapshotListener((queryDocumentSnapshots, e) -> {
if (queryDocumentSnapshots == null)
return;
List<Product> productList = new ArrayList<>();
for (DocumentSnapshot snapshot : queryDocumentSnapshots.getDocuments()) {
Product product = snapshot.toObject(Product.class);
//2* then check your query OR Condition because firestore just support AND Condition
if (product.getSize().equals("XL") || product.getSize().equals("XXL"))
productList.add(product);
}
liveData.setValue(productList);
});
For Flutter dart language use this:
db.collection("projects").where("status", whereIn: ["public", "unlisted", "secret"]);
actually I found #Dan McGrath answer working here is a rewriting of his answer:
private void query() {
FirebaseFirestore db = FirebaseFirestore.getInstance();
db.collection("STATUS")
.whereIn("status", Arrays.asList("open", "upcoming")) // you can add up to 10 different values like : Arrays.asList("open", "upcoming", "Pending", "In Progress", ...)
.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#Nullable QuerySnapshot queryDocumentSnapshots, #Nullable FirebaseFirestoreException e) {
for (DocumentSnapshot documentSnapshot : queryDocumentSnapshots) {
// I assume you have a model class called MyStatus
MyStatus status= documentSnapshot.toObject(MyStatus.class);
if (status!= null) {
//do somthing...!
}
}
}
});
}
I would like to implement a DynamoDB Scan with the following logic:
Scanning -> Filtering(boolean true or false) -> Limiting(for pagination)
However, I have only been able to implement a Scan with this logic:
Scanning -> Limiting(for pagination) -> Filtering(boolean true or false)
How can I achieve this?
Below is an example I have written that implements the second Scan logic:
var parameters = {
TableName: this.tableName,
Limit: queryStatement.limit
};
if ('role' in queryStatement) {
parameters.FilterExpression = '#role = :role';
parameters.ExpressionAttributeNames = {
'#role': 'role'
};
parameters.ExpressionAttributeValues = {
':role': queryStatement.role
};
}
if ('startKey' in queryStatement) {
parameters.ExclusiveStartKey = { id: queryStatement.startKey};
}
this.documentClient.scan(parameters, (errorResult, result) => {
if (errorResult) {
errorResult._status = 500;
return reject(errorResult);
}
return resolve(result);
});
This codes works like second one.
Scanning -> Limiting -> Filtering
The DynamoDB LIMIT works as mentioned below (i.e. second approach in your post) by design. As it works by design, there is no solution for this.
LastEvaluatedKey should be used to get the data on subsequent scans.
Scanning -> Limiting(for pagination) -> Filtering(boolean true or false)
In a request, set the Limit parameter to the number of items that you
want DynamoDB to process before returning results.
In a response, DynamoDB returns all the matching results within the
scope of the Limit value. For example, if you issue a Query or a Scan
request with a Limit value of 6 and without a filter expression,
DynamoDB returns the first six items in the table that match the
specified key conditions in the request (or just the first six items
in the case of a Scan with no filter). If you also supply a
FilterExpression value, DynamoDB will return the items in the first
six that also match the filter requirements (the number of results
returned will be less than or equal to 6).
For either a Query or Scan operation, DynamoDB might return a
LastEvaluatedKey value if the operation did not return all matching
items in the table. To get the full count of items that match, take
the LastEvaluatedKey value from the previous request and use it as the
ExclusiveStartKey value in the next request. Repeat this until
DynamoDB no longer returns a LastEvaluatedKey value.
Use --max-items=2 instead of --limit=2, max-items will do limit after filtering.
Sample query with max-items:
aws dynamodb query --table-name=limitTest --key-condition-expression="gsikey=:hash AND gsisort>=:sort" --expression-attribute-values '{ ":hash":{"S":"1"}, ":sort":{"S":"1"}, ":levels":{"N":"10"}}' --filter-expression="levels >= :levels" --scan-index-forward --max-items=2 --projection-expression "levels,key1" --index-name=gsikey-gsisort-index
Sample query with limit:
aws dynamodb query --table-name=limitTest --key-condition-expression="gsikey=:hash AND gsisort>=:sort" --expression-attribute-values '{ ":hash":{"S":"1"}, ":sort":{"S":"1"}, ":levels":{"N":"10"}}' --filter-expression="levels >= :levels" --scan-index-forward --limit=2 --projection-expression "levels,key1" --index-name=gsikey-gsisort-index
If there is just one field that is of interest for the pagination you could create an index with that field as a key. Then you do not need to recurse for the number of items in the limit.
limts add now in dynamodb
var params = {
TableName: "message",
IndexName: "thread_id-timestamp-index",
KeyConditionExpression: "#mid = :mid",
ExpressionAttributeNames: {
"#mid": "thread_id"
},
ExpressionAttributeValues: {
":mid": payload.thread_id
},
Limit: (3 , 2 ,3),
LastEvaluatedKey: 1,
ScanIndexForward: false
};
req.dynamo.query(params, function (err, data) {
console.log(err, data);
})
Limit defines the number of items/records evaluated using only the KeyCondition(If Present) before applying any filters. To solve this problem, as pointed out in earlier solutions, one approach could be to use a GSI, where the filtering condition is part of your GSI key. However, this is fairly restrictive as it's not practical to introduce a new GSI for every access pattern which requires pagination. A more realistic approach could be to query track the Count in the query response and keep querying & appending the next page of results until the aggregated Count satisfies the client-defined Limit. Keep in mind, that you would need to have some custom logic to build the LastEvaluatedKey, which would be required to fetch the subsequent result-page. In Go, this can be achieved in the following way.
func PaginateWithFilters(ctx context.Context, keyCondition string, filteringCondition int, cursor *Cursor) ([]*Records, error) {
var collectiveResult []map[string]types.AttributeValue
var records []*Records
expr, err := buildFilterQueryExpression(keyCondition, filteringCondition)
if err != nil {
return nil, err
}
queryInput := &dynamodb.QueryInput{
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
KeyConditionExpression: expr.KeyCondition(),
FilterExpression: expr.Filter(),
TableName: aws.String(yourTableName),
Limit: aws.Int32(cursor.PageLimit),
}
if cursor.LastEvaluatedKey != nil {
queryInput.ExclusiveStartKey = cursor.LastEvaluatedKey
}
paginator := dynamodb.NewQueryPaginator(dbClient, queryInput)
for {
if !paginator.HasMorePages() {
fmt.Println("no more records in the partition")
cursor.LastEvaluatedKey = nil
break
}
singlePage, err := paginator.NextPage(ctx)
if err != nil {
return nil, err
}
pendingItems := int(cursor.PageLimit) - len(collectiveResult)
if int(singlePage.Count) >= pendingItems {
collectiveResult = append(collectiveResult, singlePage.Items[:pendingItems]...)
cursor.LastEvaluatedKey = buildExclusiveStartKey(singlePage.Items[pendingItems-1])
break
}
collectiveResult = append(collectiveResult, singlePage.Items...)
}
err = attributevalue.UnmarshalListOfMaps(collectiveResult, &records)
if err != nil {
return nil, err
}
return records, nil
}
This Medium article discusses pagination with DynamoDB in a bit more depth and includes code snippets in Go to paginate query responses with filters.
I have a Json like this:
"datos_personales":
{
"nombre":"Dionel",
"apellido":"Delgado",
"fechanacimiento":"1990-12-31T04:00:00.000Z",
"lugarNacimiento":"Venezuela, Maracaibo",
"edad":25,
"genero":"Masculino",
"cedula":"21076754",
"direccion":"San Carlos",
"telefonofijo":"0262-6871111",
"telefonomovil":"0262-6871111"
},
"datos_emergencia":
{
"nombre1":"Jeaynie",
"apellido1":"Valbuena",
"telefono1":"0262-6871111",
"telefono2":"0262-6871111",
"parentesco1":"Madre",
"nombre2":"Diones",
"apellido2":"Delgado",
"telefono3":"0262-6871111",
"telefono4":"0262-6871111",
"parentesco2":"Padre"
},
"datos_sociales":
{
"civil":"Soltero",
"estudios4":true,
"ocupacion":"Programador Web",
"hijos":"No"
},
"datos_medicotratante":
{
"nombre":"Naikelin",
"apellido":"Ruiz",
"telefono1":"0262-6871111",
"telefono2":"0262-6871111",
"especialidad":"Pediatría",
"sangre":"AB",
"rh":"Negativo",
"seguro":"No"
}
`
I need to query these kinds of documents with nodeJS using "cedula" to find them
{"datos_personales.cedula":21076754} << in this case
I tried using this but it doesn't fetch the document
var ced = 21076754;
db.getCollection('users').find({"datos_personales.cedula":ced});
any suggestions?
"cedula":"21076754",
Your result is a string (See the quotes).
Change your query to:
var ced = "21076754";
db.getCollection('users').find({"datos_personales.cedula":ced});
Another solution to the problem is -
find() query in MongoDB returns a cursor while findOne() query returns a JSON Object which matches the argument passed in the query. You can retrieve the desired record by using the query below -:
db.getCollection('users').findOne({"datos_personales.cedula":ced});
I have the following Object Structure:
[
{
name: "someThing"
,securities: [ {"id": "2241926"} ]
}
]
I want to be able to return all objects in the outer array, that has at least one child secuirty with an id that starts with a value. I have tried a few things and keep running up short. On the mongoo console, this query works:
db.caseSummary.find({"securities.id": /^224.*/i})
We are using ES6, so please apologies for the generator syntax:
const q = require("q");
const startsWith = function(term){
return new RegExp('^' + term + '.*', 'i')
}
const find = function*(collection, criteria){
const command = q.nbind(collection.find, collection),
myCriteria = criteria || {},
results = yield command(myCriteria),
toArray = q.nbind(results.toArray, results) ;
return yield toArray()
}
const searchBySecurity = function*(mongo, term) {
const collection = mongo.collection("caseSummary"),
criteria = {"securities.id": startsWith(term) };
return yield find(collection, criteria);
}
so searchBySecurity(this.mongo, '224') It returns an empty array, it should return the same results as the mongo console. I guess I am missing a pretty basic concept in translating my search criteria or invoking the find writing this in node from raw mongo console query.
Edit 1: to be clear I want to return all parent objects, which have subarrays that contain a value that starts with the term passed in...
Edit 2:
I changed the criteria to be:
criteria = { "securities": {
"$elemMatch": {
"id": "901774109" //common.startsWith(term)
}
}
};
Still the same results.
Edit 3:
Using nodejs - mongodb "version": "1.4.38"
Edit 4:
this ended up not being an issue