I am integrating elastic search with my Nest application. positing my workflow and question below.
flow:
front-end updating one entity, so in back-end updates the primary storage MYSQL and cache storage elastic search.
Immediately front-end hit the get request to get the updated entity, so the back-end fetches the entity from elastic-search and sends a response to the front-end.
Now the issue is sometimes elastic search giving not updated data due to index not refreshed.
Solutions:
We can use the refresh:'wait-for' config in the update API.
We can trigger the refresh API to refresh the indices
The question is, after the update API, if we trigger the refresh API without waiting for the result. in meantime, if the front-end requested the data what will happen, the elastic search will wait for the refresh indices operation completion or will it serve old data?
As you already know Elasticsearch provides near real time searches, and if you are making a search call, as soon as document is indexed to Elasticsearch, it will not be available unless refresh happened on the index.
Now, As you mentioned, you can solve the issue by using the wait_for with your index/update operation, only downside is that its resource intensive and if you do it frequently on a large index, it can cause severe performance issues in your cluster.
Your second option will not work, as you can still query the Elasticsearch from FE before refresh finishes and you will get obsolete data.
Related
Let's say, hypothetically, I am working on a website which provides live score updates for sporting fixtures.
A script checks an external API for updates every few seconds. If there is a new update, the information is saved to a database, and then pushed out to the user.
When a new user accesses the website, a script queries the database and populates the page with all the information ingested so far.
I am using socket.io to push live updates. However, when someone is accessing the page for the first time, I have a couple of options:
I could use the existing socket.io infrastructure to populate the page
I could request the information when routing the user, pass it into res.render() as an argument and render the data using, for example, Pug.
In this circumstance, my instinct would be to utilise the existing socket.io infrastructure; purely because it would save me writing additional code. However, I am curious to know whether there are any other reasons for, or against, using either approach. For example, would it be more performant to render the data, initially, using one approach or the other?
Hey guys I'm a complete beginner in term of developing web application using node.js, react and so on.
So far I decided to use graphql and especially subscriptions to implement the communication between my server and client side.
I wan't to build one server using graphql to provide the same data for every client side access. Because every client needs to see the exact same data set as everyone else.
And i want to try to cache the last fetched data from the Database until there is a reason to access the database again.
For example:
When client one requests a list of books i want the server to cache the data.
So that when client two does the same request it just gets the data that was last fetched
from the database.
Its on my part to make sure the internal cache will be erased when the database table changed. But i don't want to request the Database every single time just to get the same answer 20 times in a row.
Is there a good way to cache the data inside the server ?
Frontend is React and Backend in Node.js.
Currently storing unregistered user carts in MongoDB using sessions (express-session and connect-mongo)
Now in the frontend, I'm making an API call every time a page loads to keep the mini cart on the top right updated. So basically every navbar rerender I'm making a call to the backend and querying to session in the database.
Is this the correct way of doing things or will this cause some stress on the backend since its been queried so much?
I'm still new to API call and database query costs etc. I just want this to cause a bottleneck.
It depends on if it's stateless or not. If it is we need to make continuous requests to the API, otherwise we can skip this step.
I've been working on a Xamarin.Forms application in Visual Studio using Azure for the backend for a while now, and I've come across a really strange issue.
Please note, that I am following the methods mentioned in this blog
For some strange reason the PullAsync() method seems to have some bizarre problems. Any data that I create and sync will only be pulled by PullAsync() from that solution. What I mean by that is that if I create another solution that accesses the exact same backend, it can perform it's own create/sync data, but will not bring over the data generated by the other solution, even though they both seem to have the exact same access. This appears to be some kind of a security feature/issue, but I can't quite make sense of it.
Has anyone else encountered this at all? Was there a work-around at all? This could potentially cause problems down the road if I were to ever want to create another solution that accesses the same system/data for whatever reason.
For some strange reason the PullAsync() method seems to have some bizarre problems. Any data that I create and sync will only be pulled by PullAsync() from that solution.
According to your provided tutorial, I found that the related PullAsync is using Incremental Sync.
await coffeeTable.PullAsync("allCoffees", coffeeTable.CreateQuery());
Incremental Sync:
the first parameter to the pull operation is a query name that is used only on the client. If you use a non-null query name, the Azure Mobile SDK performs an incremental sync. Each time a pull operation returns a set of results, the latest updatedAt timestamp from that result set is stored in the SDK local system tables. Subsequent pull operations retrieve only records after that timestamp.
Here is my test, you could refer to it for a better understanding of Incremental Sync:
Client : await todoTable.PullAsync("todoItems-02", todoTable.CreateQuery());
The client SDK would check if there has a record with the id equals deltaToken|{table-name}|{query-id} from the __config table of your SQLite local store.
If there has no record, then the SDK would send a request as following for pulling your records:
https://{your-mobileapp-name}.azurewebsites.net/tables/TodoItem?$filter=(updatedAt%20ge%20datetimeoffset'1970-01-01T00%3A00%3A00.0000000%2B00%3A00')&$orderby=updatedAt&$skip=0&$top=50&__includeDeleted=true
Note: the $filter would be set as (updatedAt ge datetimeoffset'1970-01-01T00:00:00.0000000+00:00')
While there has a record, then the SDK would pick up the value as the latest updatedAt timestamp and send the request as follows:
https://{your-mobileapp-name}.azurewebsites.net/tables/TodoItem?$filter=(updatedAt%20ge%20datetimeoffset'2017-06-26T02%3A44%3A25.3940000%2B00%3A00')&$orderby=updatedAt&$skip=0&$top=50&__includeDeleted=true
Per my understanding, if you handle the same logical query with the same query id (non-null) in different mobile client, you need to make sure the local db is newly created by each client. Also, if you want to opt out of incremental sync, pass null as the query ID. In this case, all records are retrieved on every call to PullAsync, which is potentially inefficient. For more details, you could refer to How offline synchronization works.
Additionally, you could leverage fiddler for capturing the network traces when you invoke the PullAsync, in order to troubleshoot your issue.
What is the recommended method of getting CategoryIds. I understand Foursquare provides this list: https://developer.foursquare.com/categorytree. My question is should I just use this list and hard-code the values or fetch the Ids on first opening of the app and caching these results?
From the venues/categories API documentation:
When designing client applications, please download this list only once per session, but also avoid caching this data for longer than a week to avoid stale information.
So fetch on app launch and cache for the current session to insure the hierarchy is always up-to-date.