I have been using Meilisearch for a couple of months and have recently upgraded to 0.26.0. For some reason, when I am today trying to create an index using the node package, nothing seems to happen.
I can successfully use the createIndex method like below:
client.createIndex("movies")
and the return value for the method shows the task:
{
uid: 26858,
indexUid: 'movies',
status: 'enqueued',
type: 'indexCreation',
enqueuedAt: '2022-04-08T15:15:06.325108519Z'
}
However, when I look up this task it seems that it has not been started:
{
uid: 26858,
indexUid: 'movies',
status: 'enqueued',
type: 'indexCreation',
details: { primaryKey: null },
duration: null,
enqueuedAt: '2022-04-08T15:15:06.325108519Z',
startedAt: null,
finishedAt: null
}
And indeed I can't find the index using the getIndexes method.
Strangely I created an index without issue just a few days ago.
Any idea what the issue might be or how I could debug this?
Most of Meilisearch's asynchronous operations belong to a category called "tasks". Create an index is one of them.
Depending on the queue size and server processing power it may take a while to process recently created tasks.
You can find more information on section asynchronous operation of documentation.
Related
I am trying to update in a collection and there are 1400+ offices are there and after checking and running the query I am updating in a collection document and update in the subcollection with few details after querying but sometimes i am getting this error
10 ABORTED: Too much contention on these documents. Please try again.
and i am simply using batch for writing in the doc here is my code for updation in the collection.
batch.set(
rootCollections.x.doc(getISO8601Date())
.collection(subCollection.y)
.doc(change.after.get('Id')),
{
officeId: change.after.get('Id'),
office: change.after.get('office'),
status: change.after.get('status'),
creationTimestamp:
change.after.get('creationTimestamp') ||
change.after.get('createTimestamp') ||
change.after.get('timestamp'),
activeUsers: [...new Set(phoneNumbers)].length,
confirmedUers: activityCheckinSubscription.docs.length,
uniqueActivities: [...new Set(activities)].length,
payments: 0,
contact: [
`${change.after.data().creator.phoneNumber},${
change.after.data().attachment['First Contact'].value
}`,
],
},
{ merge: true },
);
batch.set(
rootCollections.x.doc(getISO8601Date()),
{
Added: admin.firestore.FieldValue.increment(1),
},
{ merge: true },
);
PromiseArray.push(batch.commit());
await Promise.all(PromiseArray);
It seems that you are facing the same issue from this similar case here, where there were thousands of records in the database being updated. As clarified there, you have limitation of how much writes you can perform in a document in one second - more details here - and even though Firestore sometimes might hang on with the faster writes, it will fail at some point.
As this is hard coded and a limit that it's imposed by Firestore, what you can try is the solution explained in this similar case here, that it's either change to Realtime Database, where the limit is not the number of writes, but the size of the data or in case of the usage of a counter or some other data aggregation in Firestore, to use a distributed counter solution, that you can get more details here.
To summarize, there isn't much you can do unless of workaround it with this solution, as this is a limitation documented of Firestore.
I'm trying to make a messaging system that writes each message to a mongo entry. I'd like the message entry to reflect the user that sends the message, and the actual message content. This is the message schema:
const MessageSchema = new Schema({
id: {
type: String,
required: true
},
messages: {
type: Array,
required: true
},
date: {
type: Date,
default: Date.now
}
});
And this is where I either create a new entry, or append to an existing one:
Message.findOne({ id: chatId }).then(message => {
if(message){
Message.update.push({ messages: { 'name': user.name, 'message': user.message } })
} else {
const newMessage = new Message(
{ id: chatId },
{ push: { messages: { 'name': user.name, 'message': user.message } } }
)
newMessage
.save()
.catch(err => console.log(err))
}
})
I'd like the end result to look something like this:
id: '12345'
messages: [
{name: 'David', message: 'message from David'},
{name: 'Jason', message: 'message from Jason'},
etc.
]
Is something like this possible, and if so, any suggestions on how to get this to work?
This questions contains lots of topics (in my mind at least). I really want to try to break this questions to its core components:
Design
As David noted (first comment) there is a design problem here - an ever-growing array as a sub document is not ideal (please refer to this blog post for more details).
On the over hand - when we imagine how a separate collection of messages will looks like, it will be something like this:
_id: ObjectId('...') // how do I identify the message
channel_id: 'cn247f9' // the message belong to a private chat or a group
user_id: 1234 // which user posted this message
message: 'hello or something' // the message itself
Which is also not that great because we are repeating the channel and user ids as a function of time. This is why the bucket pattern is used
So... what is the "best" approach here?
Concept
The most relevant question right now is - "which features and loads this chat is suppose to support?". I mean, many chats are only support messages display without any further complexity (like searching inside a message). Keeping that in mind, there is a chance that we store in our database an information that is practically irrelevant.
This is (almost) like storing a binary data (such an image) inside our db. we can do this, but with no actual good reason. So, if we are not going to support a full-text search inside our messages, there is no point to store the messages inside our db.. at all
But.. what if we want to support a full-text search? well - who said that we need to give this task to our database? we can easily download messages (using pagination) and make the search operation on the client side itself (while keyword not found, download previous page and search it), taking the loads out of our database!
So.. it seems like that messages are not ideal for storage in database in terms of size, functionality and loads (you may consider this conclusion as a shocking one)
ReDesign
Using a hybrid approach where messages are stored in a separated collection with pagination (the bucket pattern supports this as described here)
Store messages outside your database (since your are using Node.js you may consider using chunk store), keeping only a reference to them in the database itself
Set your page with a size relevant to your application needs and also with calculated fields (for instances: number of current messages in page) to ease database loads as much as possible
Schema
channels:
_id: ObjectId
pageIndex: Int32
isLastPage: Boolean
// The number of items here should not exceed page size
// when it does - a new document will be created with incremental pageIndex value
// suggestion: update previous page isLastPage field to ease querying of next page
messages:
[
{ userId: ObjectID, link: string, timestamp: Date }
]
messagesCount: Int32
Final Conclusion
I know - it seems like a complete overkill for such a "simple" question, but - Dawid Esterhuizen convinced me that designing your database to support your future loads from the very beginning is crucial and always better than simplifying db design too much
The bottom line is that the question "which features and loads this chat is suppose to support?" is still need to be answered if you intend to desgin your db efficiently (e.g. to find the Goldilocks zone where your design suits your application needs in the most optimal way)
We are facing a timeout issue with our mongo updates. Our collection currently contains around 300 thousand documents. When we try to update a record via the UI, the server times out and the UI is stuck in limbo.
Lead.updateOne({
_id: body.CandidateID
}, {
$set: {
ingestionStatus: 'SUBMITTED',
program: body.program,
participant: body.participant,
promotion: body.promotion,
addressMeta: body.addressMeta,
CreatedByID: body.CreatedByID,
entryPerson: body.entryPerson,
lastEnteredOn: body.lastEnteredOn,
zipcode: body.zipcode,
state: body.state,
readableAddress: body.readableAddress,
promotionId: body.promotionId,
programId: body.programId,
phone1: body.phone1,
personId: body.personId,
lastName: body.lastName,
hasSignature: body.hasSignature,
firstName: body.firstName,
city: body.city,
email: body.email,
addressVerified: body.addressVerified,
address: body.address,
accountId: body.accountId
}
This is how we update a single record. We are using mlab and Heroku in our stack. Looking for advice on how to speed this up considerably.
Thank you.
If your indexes are fine then you could try rebuilding indexes on this collection.
collection indexes from the mango command line:
For example, rebuild the lead collection indexes from the mongo command line:
db.lead.reIndex();
Reference:
https://docs.mongodb.com/v3.2/tutorial/manage-indexes/
https://docs.mongodb.com/manual/reference/command/repairDatabase/
if you are not using this then try this one
Index builds can block write operations on your database, so you don’t want to build indexes in the foreground on large tables during peak usage. You can use the background creation of indexes by specifying background: true when creating.
db.collection.createIndex({ a:1 }, { background: true })
This will ultimately take longer to complete, but it will not block operations and will have less of an impact on performance.
1) Shard Lead collection by id as shard key.
2) Check if the memory taken by mongodb due to index is less than the memory of the mongoDb server.
Have you tried what this answer suggests? Namely, updating with no write-concern?
I have a process that triggers a number of requests that in turn trigger off a number of webhooks. I know the process is complete when I've received all of my webhooks. My model looks something like this.
{
name: 'name',
status: 'PENDING',
children: [{
name: 'child1',
status: 'PENDING'
}, {
name: 'child2',
status: 'PENDING'
}]
}
The idea is as the webhooks come in, I update the subdoc to COMPLETE. At the end of each update I check if the others are complete and if they are, I set status='COMPLETE' on the parent. It looks like one invocation of the service is marking it as COMPLETE after another invocation has determined it was still PENDING but before that second invocation has saved. When the second one saves, it overwrites COMPLETE with PENDING on the parent.
Here is the code from my schema:
methods: {
doUpdate: function(data) {
var child = this.children.id(data.childId);
child.status = 'COMPLETE';
if (_.every(this.children.status, 'COMPLETE'))
this.status = 'COMPLETE';
return this.save();
}
}
I think you can solve your issue when you just modify and save your child objects instead of saving / overwriting the whole document each time.
To do that you can use the positional $ in your update statement.
Essentially it would look something like this:
db.yourChildrenCollection.update(
{ _id: this._id, 'children.name': childName },
{ $set: { 'children.$.status' : 'COMPLETE' } }
)
You have to modify the variable names as I do not know your collection name etc. but I think you get the point.
The solution is to first find and update the status in one operation, using a variant of mongo's findAndModify method, and then check if the other children have completed. I was trying to do two updates in one operation. By breaking it up into two steps, I know that when I check the statuses of the other children, all other invocations will have the most recent state of the document and not get a false reading while another invocation is waiting for save() to complete.
I'm using node.js and the api key for auth. All my get actions work, and I've been able to post a new project, but new tasks always return 'server error'. Here's the object I'm sending to the /tasks endpoint:
data: {
name: 'Figure this out',
notes: '',
assignee: null,
completed: false,
assignee_status: 'later',
completed_at: null,
due_on: null,
tags: [],
parent: null,
followers: [ { id: 5622986387066 }, { id: 5622895459066 } ],
projects: [ 6156399090409 ],
workspace: 1707039411020
}
Any ideas? I've tried passing those ID values a variety of ways, and I've tried creating a more simple task, always fails with a 'server error' response.
Seems like it's the "parent": null that's causing the unhelpful Server Error, which definitely seems like a bug on our side - I've filed it and will hopefully get time to look into it soon. Trim that out, and it gives you actual error messages.
Just to save you some time: you can't set completed_at or tags, and followers should be just an array of integers ("followers": [ 5622986387066, 5622895459066 ]).
You can set completed: [true/false], and the completed_at will be set to the time at which it was marked complete. Not being able to attach tags to a task is a known issue, and one we're hoping to rectify.
Additionally, it's just a little annoying that the format of a response doesn't map 1-1 to the format for posting/updating. We're hoping to do a pass on the overall design of the API to unify those parts a bit more.