I am new to dynamodb.
I want to increment the Sort Key
If the id=0 the next id=1 and so on,
If the user(Partition key), id(Sort Key) add items the next add items the id increment 1.
The code use on PutItem with dynamodb.
Is possible to do that?
I did not want use the UUID( unique Key)
Most situations don't need an auto-incrementing attribute and DynamoDB doesn't provide this feature out of the box. This is considered to be an anti-pattern in distributed systems.
But, see How to autoincrement in DynamoDB if you really need to.
I understand that you may need this number because it is a legal obligation to have incremental invoice numbers for example.
One way would be to create a table to store your number sequences.
Add fields like:
{
name: "invoices",
prefix: "INV",
numberOfDigits: 5,
leasedValue: 1,
appliedValue: 1,
lastUpdatedTime: '2022-08-05'
},
{
name: "deliveryNotes",
prefix: "DN",
numberOfDigits: 5,
leasedValue: 1,
appliedValue: 1,
lastUpdatedTime: '2022-08-05'
}
You need 2 values (a lease and an applied value), to make sure you never skip a beat, even when things go wrong.
That check-lease-apply-release/rollback logic looks as follows:
async function useSequence(name: string, cb: async (uniqueNumber: string) => void) {
// 1. GET THE SEQUENCE FROM DATABASE
const sequence = await getSequence("invoices");
this.validateSequence(sequence);
// 2. INCREASE THE LEASED VALUE
const oldValue = sequence.appliedValue;
const leasedValue = oldValue + 1;
sequence.leasedValue = leasedValue;
await saveSequence(sequence);
try {
// 3. CREATE AND SAVE YOUR DOCUMENT
await cb(format(leasedValue));
// 4. INCREASE THE APPLIED VALUE
sequence.appliedValue++;
await saveSequence(sequence);
} catch(err) {
// 4B. ROLLBACK WHEN THINGS ARE BROKEN
console.err(err)
try {
const sequence = await getSequence(name);
sequence.leasedValue--;
this.validateSequence(sequence);
await saveSequence(sequence);
} catch (err2) {
console.error(err2);
}
throw err;
}
}
function validateSequence(sequence) {
// A CLEAN STATE, MEANS THAT THE NUMBERS ARE IN SYNC
if (sequence.leasedValue !== sequence.appliedValue) {
throw new Error("sequence is broken.");
}
}
Then, whenever you need a unique number you can use the above function to work in a protected scope, where the number will be rollbacked when something goes wrong.
const details = ...;
await useSequence("invoice", async (uniqueNumber) => {
const invoiceData = {...details, id: uniqueNumber};
const invoice = await this.createInvoice(invoiceData);
await this.saveInvoice(invoice);
})
Can it scale? Can it run on multiple instances? No, it can't. It never will be, because in most countries it's just not legal to do so. You're not allowed to send out invoice 6 before invoice 5 or to cancel invoice 5 after you've send invoice 6.
The only exception being, if you have multiple sequences. e.g. in some cases you're allowed to have a sequence per customer, or a sequence per payment system, ... Hence, you want them in your database.
i have an issue with unique constraint on one of my fields.
I'm adding records to database to be able to check by tests is my code working as expected.
One of table field is unique number that is provided from outside (it's not related to some other table in the same database), i need to generate this unique number for each test, but i met with unique constraint issue.
I have following function:
export const findMinUniqueUserId = async (): Promise<number> => {
const subscriptions = await prisma.$queryRaw<Subscription[]>(`
SELECT "userId"
FROM public."Subscriptions"
ORDER BY "userId" DESC
LIMIT 1
`);
const firstFreeUserId = (subscriptions[0]?.userId || 0) + 1;
return firstFreeUserId;
};
that returns the first minimum free "userId" field.
I have also the following tests:
describe("Test 1", () => {
it("should do something", async () => {
const draftSub = {
userId: await findMinUniqueUserId()
...some other fields
}
await prisma.subscription.create({
data: draftSub
})
...some other test stuff
})
})
And the second one:
describe("Test 2", () => {
it("should do something", async () => {
const draftSub = {
userId: await findMinUniqueUserId()
...some other fields
}
await prisma.subscription.create({
data: draftSub
})
...some other test stuff
})
})
Sometimes i'm getting an error:
Unique constraint failed on the fields: (`userId`)
I've heard that each of test suit (describe block) works on seperate worker thread, i was trying to prepare some kind of singleton class, that can helps me but i think each instance of class is creating in separete worker thread, so generated userId is not unique.
This is what i was trying with singleton class:
export class UserIdManager {
private static instance: UserIdManager
private static userIdShiftBeforeDatabaseCall = 0
private static minFreeUserIdAfterDatabaseCall = 0
private constructor() {
return;
}
private static async init() {
this.minFreeUserIdAfterDatabaseCall = await findMinUniqueUserId();
}
public static async reserveMinFreeUserId() {
let minFreeUserId = UserIdManager.userIdShiftBeforeDatabaseCall;
UserIdManager.userIdShiftBeforeDatabaseCall++;
if (!UserIdManager.instance) {
UserIdManager.instance = new UserIdManager();
await this.init();
}
minFreeUserId += UserIdManager.minFreeUserIdAfterDatabaseCall;
return minFreeUserId;
}
}
But i realize that it doesn't help me with multithreading. I've used this, but with the same result:
....
const draftSub = {
userId: await UserIdManager.reserveMinFreeUserId()
...some other fields
}
....
So, the question is how to generate unique number for each test. When i pass --runInBand option to jest everything is working correctly, but it takes much more time.
What you are using is the typical MAX()+1 method of assigning unique values. Unfortunately this is a virtual guarantee you will get duplicate values for your unique value. This is a result of the Multi-Version Concurrency Control (MVCC) nature of Postgres. In a MVCC database the actions taken by one session cannot be seen by another session until the first session commits. Thus when multiple sessions access max()+1 they each get the same result. The first one to commit succeeds, the second fails. The solution to this is creating a sequence and let Postgres assign the unique value, it will not assign the same value twice regardless how many sessions access the sequence concurrently. The cost however being your values will contain gaps - accept it, get over it, and move on. You can have the sequence generated by defining your userid as a generated identity (Postgres10 or later) or as serial for older versions.
create table subscriptions ( id generated always as identity ...) -- for versions Postgres 10 or later
or
create table subscriptions ( id serial ...) -- for versions prior to Postgers 10
With either of those in place get rid of your findMinUniqueUserId function. You may also want to look into insert...returning... functionality
I use firebase on node.js .
My given structure should look like this:
{
...
batch-1:
id-1(suppose):
name:...
phone:...
id-2:
...
id-3:
...
batch-2:
...
batch-3:
...
...
batch-n:
...
}
How can I get an id-1 object by its identifier in such an architecture?
Does the database have to go around all the batches?
Is there a better solution?
The main task: Create a batch with many objects that will have SHORT and a UNIQUE identifier and optimally receive data by this identifier
To search for a particular ID that is a child of a list of unknown IDs, you need to use orderByChild(). In your use case, you are looking for a particular ID in a list of batch IDs. If you used orderByChild() on this list, you would get back results for each and every batch ID, even if it didn't have the ID you wanted. This is because even null (non-existant) values are included (and sorted at the start) in the results. To get the data of the desired ID, you would get the data for the last result of the query, which if it existed, would be sorted to the end of the list. Note that if the desired ID doesn't exist, the last result (if there are any results) would have a null value. To return only the last result of the query, you would use limitToLast(1).
Putting this all together, gives the following code:
let idToFind = "unique-id-1";
let batchesRef = firebase.database().ref(); // parent key of "batch-1", "batch-2", etc.
// assumed to be the database root here
batchesRef.orderByChild(idToFind).limitToLast(1).once('value')
.then((querySnapshot) => {
if (!querySnapshot.numChildren()) { // handle rare no-results case
throw new Error('expected at least one result');
}
let dataSnapshot;
querySnapshot.forEach((snap) => dataSnapshot = snap); // get the snapshot we want out of the query's results list
if (!dataSnapshot.exists()) { // value may be null, meaning idToFind doesn't exist
throw new Error(`Entry ${idToFind} not found.`);
}
// do what you want with dataSnapshot
console.log(`Entry ${idToFind}'s data is:`, dataSnapshot.val());
})
.catch((error) => {
console.log("Unexpected error:", error);
})
For small data sets, the above code will work just fine. But if the list of batches starts growing quite large, you may wish to build an index that maps a particular ID to the batch ID that contains it.
Here is my method which allows you to search by id or to search by key value such as email uniqueemail
// gets primary key
const getSnapshotValKey = snapshot => (Object.keys(snapshot).length > 0 ? Object.keys(snapshot)[0] : null)
const getUser = async ({ id, key, value }) => {
let user = null
const ref = id ? '/users/' + id : 'users'
const userRef = admin.database().ref(ref)
const valueRef = id ? userRef : await userRef.orderByChild(key).equalTo(value)
const snapshot = await valueRef.once('value')
const val = snapshot.val()
if (val) {
const key = id || getSnapshotValKey(val)
user = {
id: key,
...(id ? val : val[key]),
}
}
return user
}
From the docs:
You can also chain multiple where() methods to create more specific queries (logical AND).
How can I perform an OR query?
Example:
Give me all documents where the field status is open OR upcoming
Give me all documents where the field status == open OR createdAt <= <somedatetime>
OR isn't supported as it's hard for the server to scale it (requires keeping state to dedup). The work around is to issue 2 queries, one for each condition, and dedup on the client.
Edit (Nov 2019):
Cloud Firestore now supports IN queries which are a limited type of OR query.
For the example above you could do:
// Get all documents in 'foo' where status is open or upcmoming
db.collection('foo').where('status','in',['open','upcoming']).get()
However it's still not possible to do a general OR condition involving multiple fields.
With the recent addition of IN queries, Firestore supports "up to 10 equality clauses on the same field with a logical OR"
A possible solution to (1) would be:
documents.where('status', 'in', ['open', 'upcoming']);
See Firebase Guides: Query Operators | in and array-contains-any
suggest to give value for status as well.
ex.
{ name: "a", statusValue = 10, status = 'open' }
{ name: "b", statusValue = 20, status = 'upcoming'}
{ name: "c", statusValue = 30, status = 'close'}
you can query by ref.where('statusValue', '<=', 20) then both 'a' and 'b' will found.
this can save your query cost and performance.
btw, it is not fix all case.
I would have no "status" field, but status related fields, updating them to true or false based on request, like
{ name: "a", status_open: true, status_upcoming: false, status_closed: false}
However, check Firebase Cloud Functions. You could have a function listening status changes, updating status related properties like
{ name: "a", status: "open", status_open: true, status_upcoming: false, status_closed: false}
one or the other, your query could be just
...where('status_open','==',true)...
Hope it helps.
This doesn't solve all cases, but for "enum" fields, you can emulate an "OR" query by making a separate boolean field for each enum-value, then adding a where("enum_<value>", "==", false) for every value that isn't part of the "OR" clause you want.
For example, consider your first desired query:
Give me all documents where the field status is open OR upcoming
You can accomplish this by splitting the status: string field into multiple boolean fields, one for each enum-value:
status_open: bool
status_upcoming: bool
status_suspended: bool
status_closed: bool
To perform your "where status is open or upcoming" query, you then do this:
where("status_suspended", "==", false).where("status_closed", "==", false)
How does this work? Well, because it's an enum, you know one of the values must have true assigned. So if you can determine that all of the other values don't match for a given entry, then by deduction it must match one of the values you originally were looking for.
See also
in/not-in/array-contains-in: https://firebase.google.com/docs/firestore/query-data/queries#in_and_array-contains-any
!=: https://firebase.googleblog.com/2020/09/cloud-firestore-not-equal-queries.html
I don't like everyone saying it's not possible.
it is if you create another "hacky" field in the model to build a composite...
for instance, create an array for each document that has all logical or elements
then query for .where("field", arrayContains: [...]
you can bind two Observables using the rxjs merge operator.
Here you have an example.
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/observable/merge';
...
getCombinatedStatus(): Observable<any> {
return Observable.merge(this.db.collection('foo', ref => ref.where('status','==','open')).valueChanges(),
this.db.collection('foo', ref => ref.where('status','==','upcoming')).valueChanges());
}
Then you can subscribe to the new Observable updates using the above method:
getCombinatedStatus.subscribe(results => console.log(results);
I hope this can help you, greetings from Chile!!
We have the same problem just now, luckily the only possible values for ours are A,B,C,D (4) so we have to query for things like A||B, A||C, A||B||C, D, etc
As of like a few months ago firebase supports a new query array-contains so what we do is make an array and we pre-process the OR values to the array
if (a) {
array addObject:#"a"
}
if (b) {
array addObject:#"b"
}
if (a||b) {
array addObject:#"a||b"
}
etc
And we do this for all 4! values or however many combos there are.
THEN we can simply check the query [document arrayContains:#"a||c"] or whatever type of condition we need.
So if something only qualified for conditional A of our 4 conditionals (A,B,C,D) then its array would contain the following literal strings: #["A", "A||B", "A||C", "A||D", "A||B||C", "A||B||D", "A||C||D", "A||B||C||D"]
Then for any of those OR combinations we can just search array-contains on whatever we may want (e.g. "A||C")
Note: This is only a reasonable approach if you have a few number of possible values to compare OR with.
More info on Array-contains here, since it's newish to firebase docs
If you have a limited number of fields, definitely create new fields with true and false like in the example above. However, if you don't know what the fields are until runtime, you have to just combine queries.
Here is a tags OR example...
// the ids of students in class
const students = [studentID1, studentID2,...];
// get all docs where student.studentID1 = true
const results = this.afs.collection('classes',
ref => ref.where(`students.${students[0]}`, '==', true)
).valueChanges({ idField: 'id' }).pipe(
switchMap((r: any) => {
// get all docs where student.studentID2...studentIDX = true
const docs = students.slice(1).map(
(student: any) => this.afs.collection('classes',
ref => ref.where(`students.${student}`, '==', true)
).valueChanges({ idField: 'id' })
);
return combineLatest(docs).pipe(
// combine results by reducing array
map((a: any[]) => {
const g: [] = a.reduce(
(acc: any[], cur: any) => acc.concat(cur)
).concat(r);
// filter out duplicates by 'id' field
return g.filter(
(b: any, n: number, a: any[]) => a.findIndex(
(v: any) => v.id === b.id) === n
);
}),
);
})
);
Unfortunately there is no other way to combine more than 10 items (use array-contains-any if < 10 items).
There is also no other way to avoid duplicate reads, as you don't know the ID fields that will be matched by the search. Luckily, Firebase has good caching.
For those of you that like promises...
const p = await results.pipe(take(1)).toPromise();
For more info on this, see this article I wrote.
J
OR isn't supported
But if you need that you can do It in your code
Ex : if i want query products where (Size Equal Xl OR XXL : AND Gender is Male)
productsCollectionRef
//1* first get query where can firestore handle it
.whereEqualTo("gender", "Male")
.addSnapshotListener((queryDocumentSnapshots, e) -> {
if (queryDocumentSnapshots == null)
return;
List<Product> productList = new ArrayList<>();
for (DocumentSnapshot snapshot : queryDocumentSnapshots.getDocuments()) {
Product product = snapshot.toObject(Product.class);
//2* then check your query OR Condition because firestore just support AND Condition
if (product.getSize().equals("XL") || product.getSize().equals("XXL"))
productList.add(product);
}
liveData.setValue(productList);
});
For Flutter dart language use this:
db.collection("projects").where("status", whereIn: ["public", "unlisted", "secret"]);
actually I found #Dan McGrath answer working here is a rewriting of his answer:
private void query() {
FirebaseFirestore db = FirebaseFirestore.getInstance();
db.collection("STATUS")
.whereIn("status", Arrays.asList("open", "upcoming")) // you can add up to 10 different values like : Arrays.asList("open", "upcoming", "Pending", "In Progress", ...)
.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#Nullable QuerySnapshot queryDocumentSnapshots, #Nullable FirebaseFirestoreException e) {
for (DocumentSnapshot documentSnapshot : queryDocumentSnapshots) {
// I assume you have a model class called MyStatus
MyStatus status= documentSnapshot.toObject(MyStatus.class);
if (status!= null) {
//do somthing...!
}
}
}
});
}
I have the following node js code that should list all items from a DynamoDB table,
import * as dynamoDbLib from "../../libs/dynamodb-lib";
import { success, failure } from "../../libs/response-lib";
export async function main(event, context) {
const params = {
TableName: "brands",
KeyConditionExpression: "brandId = :brandId",
ExpressionAttributeValues: {
":brandId": ''
}
};
try {
const result = await dynamoDbLib.call("query", params);
return success(result.Items);
} catch (e) {
console.log(e);
return failure({ status: false });
}
}
The id is in uuid format which when inserted from my node js was imported by using:
import uuid from "uuid";
then inserted to the table like:
brandId: uuid.v1()
Now when I query the items in the table I can only get only one record if and only if I hard coded the uuid of a record in the expression attribute value (either the KeyConditions or KeyConditionExpression parameter must be specified). So I thought about adding a regular expression to match all the uuids, my regular expression was copied from some solutions on the web but it didn't work, it was like the following:
[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}
and
\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b
and I have tried other examples but non of them worked, is it right to add a regular expression to get all the items, and if so what is the right regex for it?
Use a Scan operation to get all items in a table.
From the AWS Developer Guide:
The scan method reads every item in the table and returns all the data in the table. You can provide an optional filter_expression, so that only the items matching your criteria are returned. However, the filter is applied only after the entire table has been scanned.