I have two unix timestamps, a start time (startDate) and an end time (endDate). Using the moment function below gives me the very start/end of the day:
search = {
"end": { "$lte": moment(endDate).startOf('day').unix() },
"start": {"$gte": moment(startDate).startOf('day').unix() }
};
I then pass my search into my buildfire function:
buildfire.publicData.search(search ,'routes', (err, res) => {})
My res is an array of objects, each has a data property with a start and end property inside of that:
res = [
{
data: {
end: 1503554370297,
start: 1503554368711
}
}
]
All results are returning, nothing is being filtered.
Here is the docs datastore search https://github.com/BuildFire/sdk/wiki/How-to-use-Datastore#buildfiredatastoresearchoptions-tag-optional-callback
I believe you are not sending the filter correctly.
You are sending:
search = {
"end": { "$lte": moment(endDate).startOf('day').unix() },
"start": {"$gte": moment(startDate).startOf('day').unix() }
};
buildfire.publicData.search(search ,'routes', (err, res) => {})
And it should be more like
var options = {
"filter":{
"$json.end": { "$lte": moment(endDate).startOf('day').unix() },
"$json.start": {"$gte": moment(startDate).startOf('day').unix() }
}
};
buildfire.publicData.search(search ,'routes', (err, res) => {})
Basically since your filter property in your options objectis undefined it returns everything
Related
I use the Javascript client of elastic search to index and search my data and I have a problem to use the scroll method.
I can't set the right index but I know I have the right technique because in the DevTools console of Kibana I get the right result.
In DevTools I have the following queries that work (I execute the first request which is a search and then the second request which is a scroll by providing the scrollId returned by the first request):
GET my_index-*/_search?scroll=1m
{
"query": {
"bool": {
"filter": [{
"match": {
"field":"data"
}
},{
"range": {
"#timestamp": {
"gte": "2020-05-08T10:00:00.100Z",
"lte": "2022-05-11T10:00:00.200Z",
"format": "strict_date_optional_time"
}
}
}]
}
},
"_source": {
"includes": ["some_field1", "some_field2"]
},
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
],
"size": 100
}
GET _search/scroll
{
"scroll": "1m",
"scroll_id":"DnF1ZXJ5VG ... NtUHdsQQ=="
}
These two requests return the expected result.
My backend is in Typescript and uses NodeJs with Express.
The first request, the search, works normally and returns me the right result. It is this code :
some import
...
const client = new Client({
node: configElasticClient.node
})
export const searchRouter = express.Router();
searchRouter.get('/search', async (req, res) => {
const response = await client.search<SearchResponse<HitResult>>({
index: "my_index-*",
scroll: "1m",
body: {
query: {
"bool": {
"filter": [{
"match": {
"field_to_search":"data_to_search"
}
},{
"range": {
"#timestamp": {
"gte": "2020-05-08T10:00:00.100Z",
"lte": "2022-05-11T10:00:00.200Z",
"format": "strict_date_optional_time"
}
}
}]
}
},
_source: ["some_field1", "some_field2"],
sort: [
{
"#timestamp": {
order: "desc"
}
}
],
size: 100
}
});
console.log("result: ", response);
res.json(response);
});
Then, when I make the scroll I should change my index to: "_search/scroll". But no matter what I try, when I try to access my route using scroll I get a "Cannot GET (the route)" error.
Here is the code of this second route (I put in comment the other way tried):
searchRouter.get('/search/scrollnext/:scrollId', async (req, res) => {
const response = await client.scroll<SearchResponse<HitResult>>({
scroll_id: req.params.scrollId,
scroll: "1m",
//index: "_search/scroll", wherever I put this field I get an error because the scroll method has no index field defined
});
//I also try this way :
//const response = await client.scroll<SearchResponse<HitResult>>({
// method: 'GET',
// body: {
// scroll_id: req.params.scrollId,
// scroll: "1m",
//}});
console.log("result: ", response);
res.json(response);
});
I get my scroll_id as a parameter from the first request which I fill in my second request in the scroll function.
I think that my problem comes from the scroll index which must be wrong. Does anyone know how to fix the index used by the scroll method?
Any help would be greatly appreciated :)
Finally find the answer, the syntax was good and it wasn't an index problem, I had to add an async management layer in my request like this:
searchRouter.get('/path/to/route/:scrollId', (asyncHandler(async (req, res) => {
const response = await client.scroll<YourDataType>({
scroll_id: req.params.scrollId,
scroll: "1m",
});
res.json(response);
})));
with asyncHandler looking something like this:
export function asyncHandler(fct: some_interface_to_define_request_caracteristic): RequestHandler {
return (req, res, next) => {
//Here add a try/catch layer
}
}
I'm trying to get all documents from my database collection "posts" but I'm getting an empty array instead.
The strange thing is that I'm able to get all documents from another collection called "users" that has the same structure and using the exact same code.
I've spent days looking for an answer but I haven't been able to find the solution.
This is the request:
const db = admin.firestore();
exports.getAllPosts = (req, res) => {
db.collection('posts')
.orderBy('createdAt', 'desc')
.get()
.then(snapshot => {
let posts = [];
snapshot.forEach((doc) => {
posts.push({
id: doc.id,
body: doc.data().body,
author: doc.data().author,
createdAt: doc.data().timestamp,
voteScore: doc.data().voteScore
});
});
return res.json(posts);
})
.catch(err => {
console.error(err);
res.status(500).json({ error: err.code });
});
}
And this is the response:
[]
This is what my current collection looks like:
Posts collection screenshot
This the response that I get when I return "snapshot":
{
"_query": {
"_firestore": {
"_settings": {
"projectId": "readable-bf7a6",
"firebaseVersion": "9.6.0",
"libName": "gccl",
"libVersion": "4.10.0 fire/9.6.0"
},
"_settingsFrozen": true,
"_serializer": {
"allowUndefined": false
},
"_projectId": "readable-bf7a6",
"registeredListenersCount": 0,
"bulkWritersCount": 0,
"_backoffSettings": {
"initialDelayMs": 100,
"maxDelayMs": 60000,
"backoffFactor": 1.3
},
"_clientPool": {
"concurrentOperationLimit": 100,
"maxIdleClients": 1,
"activeClients": {},
"failedClients": {},
"terminated": false,
"terminateDeferred": {
"promise": {}
}
}
},
"_queryOptions": {
"parentPath": {
"segments": []
},
"collectionId": "posts",
"converter": {},
"allDescendants": false,
"fieldFilters": [],
"fieldOrders": [
{
"field": {
"segments": [
"createdAt"
]
},
"direction": "DESCENDING"
}
],
"kindless": false
},
"_serializer": {
"allowUndefined": false
},
"_allowUndefined": false
},
"_readTime": {
"_seconds": 1622395245,
"_nanoseconds": 513743000
},
"_size": 0,
"_materializedDocs": null,
"_materializedChanges": null
}
Notice how the request for the collection "users" works successfully:
const db = admin.firestore();
exports.getAllUsers = (req, res) => {
db.collection('users')
.orderBy('createdAt', 'desc')
.get()
.then(snapshot => {
let users = [];
snapshot.forEach((doc) => {
let users = [];
snapshot.forEach((doc) => {
users.push({
id: doc.data().userId,
email: doc.data().email,
handle: doc.data().handle
});
});
return res.json(users);
})
.catch(err => {
console.error(err);
res.status(500).json({ error: err.code });
});
}
And the response:
[
{
"id": "EPoHBxhQFUXbcL3TCVx1LdUG2nO2",
"email": "ruben#gmail.com"
},
{
"id": "RqEa3dEq8TSDcZYeolXafju67rB2",
"email": "user10#gmail.com"
},
{
"id": "dxveb4n2iMQej5Q14uprsKRxFp23",
"email": "user4#gmail.com",
"handle": "user4"
},
{
"id": "YQPzBPcsqlVZk9iJEuZTHKUNuVG2",
"email": "user2#gmail.com",
"handle": "user2"
},
{
"id": "CZ05BJxi3TUOpIrmBaz539OWlbC3",
"email": "user#gmail.com",
"handle": "user"
},
{
"id": "t0t83BVwt4gVgJkDv7HL1r1MaKr1",
"email": "userJose2#gmail.com",
"handle": "Jose"
}
]
This is what the users collection looks like in Firebase:
Users collection screenshot
Why is one collection failing when the other works fine and I'm using the same code? What am I missing here?
Thanks in advance and I hope I've made it as clear as possible. Please let me know if you need me to provide anything else.
Very simple my friend, your posts documents can't be ordered like this:
.orderBy('createdAt', 'desc')
Because the post documents does not have the createdAt property, but they have a timestamp property, you should use that property to order your posts like this:
.orderBy('timestamp', 'desc')
I hope that helps 👍
I am not answering directly to #Ruben Garcia Bri, but for future firebase developers who may run into the problem of getting empty documents, I also ran into the same problem, but I solved it by adding a field to the particular document I am trying to retrieve.
Sometimes the cause is because the documents you are trying to get have no field in them.
I mean that a document must have a field before the server can recognize it as an existing document.
So if you run into this problem, consider adding a field to that document before you can successfully retrieve it.
I have a document that has an array field called telephone which can have multiple extension array objects that can have multiple objects in it. So its an array inside an array. I am listening to the db using changeStream. If I change telephone[0].extension[0].valueBoolean = true where telephone[0].extension[0].url == "https//google.com",
I get the whole telephone array back in change.updateDescription.updatedFields NOT just telephone[0].extension[0]
updatedFields
{
"telephone": [{
"use": "offline",
"extension": [
{
"url": "https//gmail.com",
"valueDateTime": "2021-01-12T06:31:48.000Z"
}, {
"url": "https//yahoo.com",
"valueDateTime": "1700-01-01T00:00:00.000Z"
}, {
"url": "https//google.com",
"TimeLastModified": "2021-02-23T11:06:06.000Z",
"valueBoolean": false
}],
"value": "+123456789",
"system": "phone"
}, {
"use": "internet",
"extension": [
{
"url": "https//gmail.com",
"valueDateTime": "2021-01-12T06:31:48.000Z"
}, {
"url": "https//yahoo.com",
"valueDateTime": "1700-01-01T00:00:00.000Z"
}, {
"url": "https//google.com",
"TimeLastModified": "2021-02-23T11:06:06.000Z",
"valueBoolean": false
}],
"value": "+123456799",
"system": "phone"
}]
}
Here's what i have so far
MongoClient.connect(CONNECTION_STRING, {
useUnifiedTopology: true,
})
.then((client) => {
console.log("Connected successfully to server");
dbConnected = true;
}
// specify db and collections
const db = client.db(DB_NAME);
const myCollection = db.collection(COLLECTION_NAME);
const options = { fullDocument: "updateLookup" };
const changeStream = myCollection.watch(options);
// start listening to changes
changeStream.on("change", async (change) => {
// console.log("CHANGE!");
// console.log(JSON.stringify(change));
// check operationType
try {
if (
change.operationType == "insert" ||
change.operationType == "update" ||
change.operationType == "replace"
) {
const updatedFields = change.updateDescription.updatedFields
console.log("updatedFields", JSON.stringify(change.updateDescription.updatedFields));
}
} catch (e) {
console.log(e);
}
});
})
.catch((e) => {
console.log(`Error: ${e}`);
});
How do I see what exact element in a nested array changed with changeStream ?
Unfortunately it seems that this is currently not supported - there's an open Jira-ticket that is related to your problem, see https://jira.mongodb.org/browse/SERVER-41559 for further details.
I want to replace all array's elements in 'prices' filed as below:
{
"name": "My customer name"
"taxCode":123456
"prices":
[
{
"name": "Chocolate",
"unitPrice": 10
},
{
"name": "Cookie",
"unitPrice": 9
}
]
}
The JSON that uses to change 'prices' is:
{
"prices":
[
{
"name": "Chocolate1",
"unitPrice": 10
},
{
"name": "Candy",
"unitPrice": 5
}
]
}
And here is my code to replace the 'prices' array
router.route('/:obj/:id')
.put((req, res) => {
const PObj = require('../models/customer');
PObj.findById(req.params.id, (err, doc) => {
if (err) {
console.log('Lookup error: ' + err);
res.status(500).send('Error');
} else if (doc) {
doc.update({$set: req.body}, (err, task) => {
res.status(200).json(task);
}); } else {
res.status(404).send('Something is wrong');
}
});
});
After code executed is done but without any changes in Mongo DB. Please help me to correct my code. Thank!
If your req.body prints that prices array then it has to be req.body.prices, also rather than fetching the document & updating it - Which is a two- way process, You can try this :
router.route("/:obj/:id").put((req, res) => {
const PObj = require("../models/customer");
PObj.findByIdAndUpdate(
req.params.id, /** this 'req.params.id' has to be `_id` value of doc in string format */
/** internally mongoose will send this as { $set: { prices: req.body.prices }} which will replace `prices` array will new array,
* Just in case if you wanted to push new values, have to manually do { $push: { prices: req.body.prices }} each object */
{ prices: req.body.prices },
{ new: true }, /** returns updated doc, this option is not needed if you don't need doc - by default it returns old doc */
(err, doc) => {
if (err) {
console.log("Lookup error: " + err);
res.status(500).send("Error");
} else if (doc) {
res.status(200).json(task);
} else { /** `doc` value will be null if no doc is not found for given id */
res.status(404).send("Something is wrong");
}
}
);
});
Ref : .findByIdAndUpdate()
This is the array that i get from my database, let's call it product_list
{
"seller": "5cee0e69f67e171ac8ef14c7",
"products": [...]
},
{
"seller": "5d1c36910aec8934cefdda8e",
"products": [...]
}
And i want to send the same array but transforming the seller field into the complete seller object. In the given example would be these ones.
{
"_id": "5cee0e69f67e171ac8ef14c7",
"name": "Will"
},
{
"_id": "5d1c36910aec8934cefdda8e",
"name" : "Jess"
}
So my desired output would be
{
"seller": {
"_id": "5cee0e69f67e171ac8ef14c7",
"name": "Will"
},
"products": [...]
},
{
"seller": {
"_id": "5d1c36910aec8934cefdda8e",
"name": "Jess"
},
"products": [...]
}
I tried the following
product_list.forEach(element => {
User.findById(element.seller, function(err, user){
element.seller = user;
};
});
What i'm trying is to take all the seller ids from the product_list object, convert it to the complete object with User.findById, and return the updated product_list. But the problem is that i'm not to good with asynchronous code, and since the mongoose calls are callbacks when i return the object with res.json(product_list), the mongoose query is not finished and i receive the object without the modification.
I tried promises and awaits but im not getting any result.
I hope you understand my explanation, and thank you very much.
You can try this approach, wrap User.findById and you can use the async / await pattern.
// Wrapper function for User.findById. We could use utils.promisify for this.
function getUserById(id) {
return new Promise((resolve, reject) => {
User.findById(id, (err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
})
});
}
async function getSalesDetails(productList) {
for(let product of productList) {
let userDetails = await getUserById(product.seller);
product.seller = userDetails;
}
return productList;
}
async function testGetSalesDetails() {
let productList = [{
"seller": "5cee0e69f67e171ac8ef14c7",
"products": [1,2,3]
},{
"seller": "5d1c36910aec8934cefdda8e",
"products": [4,5,6]
}];
let productListWithSeller = await getSalesDetails(productList);
console.info("Product list (with seller): ", productListWithSeller);
}
testGetSalesDetails();
You could also use utils.promisify to generate the getUserById function (it's very handy!), for example:
const getUserById = util.promisify(User.findById);