i am a newbie to elasticsearch and was trying to index some json objects. My json object looks like this:
{
"name": "ABC",
"age":22,
"education": "XYZ",
"metadata": {
asset1:{
fingerprintID: "dnadklwdkl",
image: "URL"
}
}
}
Here, the metadata is given by the user and can be anything, null, an array, object of different things. other field like name, age, education are consistent but not metadata.so, i am having trouble indexing it. i was following this link: https://www.compose.com/articles/getting-started-with-elasticsearch-and-node/ and creating index for the data and pushing to the elastic server running locally. I have around 1 million json data like above fetched from a database and have no idea how the metadata will be. On creating the index and running the service, i get this error:
{ error:
{ root_cause: [ [Object] ],
type: 'mapper_parsing_exception',
reason: 'failed to parse',
caused_by:
{ type: 'illegal_argument_exception',
reason:
'Limit of total fields [1000] has been exceeded while adding new fields [2]' } },
some of received json data from database have been pushed to the server successfully, but most of them are not. How do i handle this issue? Can someone help?
According to the error message, it seems like some of your JSON objects have a lot of fields. By default, Elasticsearch allows only 1000 total fields within an object to prevent mapping explosion. If you know what you are doing, you can simply increase this limitation by doing:
PUT your_index/_settings
{
"index.mapping.total_fields.limit": 9999
}
Related
I have a Nodejs app with Mongodb. Now I want to user Elasticsearch to replicate data from mongo to Elasticsearch. I'm using npm package "elasticsearch". For example for collection "Posts" I have like this:
items: [
{
_id: '111111111111',
title: 'test1',
status: true,
},
{
_id: '22222222',
title: 'test2',
status: 0,
},
{
_id: '333333333',
title: 'test1',
status: {published: trye},
}
]
As you can see, My data is unstructured and Elasticsearch shows me error while I'm adding these items. I want a trick to turn off Elasticsearch Restriction and allow me to add these data. I can't made changes on my data its huge.
Any solution?
It gives you errors because your status field is a boolean first, a number next, and an object at the end -> mapping conflicts. If you don't want to change the data I assume you don't expect to search over the fields that show conflicts (how could you query consistently over a field that could be anything?). Then, my best recommendation is to store the conflictual fields without indexing them. That means you will see them in the documents, but you won't be able to query them or aggregate over them. To disable indexing you set their mapping type as an Object and set the enabled mapping property to false (see Docs).
If you want to be able to query or aggregate over everything, you must do the extra effort of preprocessing your data consistently.
What I am trying to do is to get the data from the firebase realtime database through a nodejs application and from there allow the frontend to make requests directly to my node application and not to the firebase. I made the request to the firebase database, just like the code below but it returns a null value and when the frontend reads this value it generates an error. I already checked in the firebase database and there is no null value. The firebase returns all data as it should, but it always returns this "null", just like the image below.
{
"data": [
null,
{
"data": "12/04/2016",
"subtitulo": "test...",
"titulo": "test"
},
{
"data": "05/05/2019",
"subtitulo": "test...",
"titulo": "test"
},
{
"data": "02/01/2020",
"subtitle": "title",
"title": "test"
}
]
}
Node Js code:
db.ref('myreference/here/').once('value', (snapshot) => {
data = snapshot.val();
return res.json({
data,
});
});
Error generated by React Js when I request the api:
While the Firebase Realtime Database doesn't natively store arrays, it's APIs and SDKs try to handle arrays in the source data gracefully. When you store an array, they convert it to a JSON map with sequential, numeric keys ("0", "1", etc). And then you read such a structure back from the database, they convert it to an array.
That last action is what you see in your code: since your keys look like array indexes, the client converts the data to an array. And since key "0" is missing, it inserts a null element in the array for that.
The easiest way to circumvent this array coercion is by add a prefix to your keys, like:
"Avisos": {
"key1": { ... },
"key2": { ... },
"key2": { ... },
}
That will ensure the Firebase API/SDK doesn't coerce the data into an array.
In general, I recommend not using sequential numeric keys (Kato gives a good overview in Best Practices: Arrays in Firebase), but instead use the keys generated by Firebase's push() method.
I am sending JSON from a NodeJS API to an Angular app to be displayed as part of an Angular component. The returned data is simple Mongo id and name string. The data is present in debugger and Postman like so:
{ "httpAllNames": [
{
"_id": "5d5c54315be61d26c0b2afb8",
"campaignTitle": "Make America Zombie Free Again"
},
{
"_id": "5d5c54735be61d26c0b2afba",
"campaignTitle": "Zmobie Free 2"
},
{
"_id": "5d5d3fb280dead0604fe6f8c",
"campaignTitle": "Universal Basic Income For All"
},
{
"_id": "5d5eeaee3278d24b10093988",
"campaignTitle": "Remove All Zombies from the US"
} ]}
I pass the data from my Campaign Service to the component without any trouble. The code in question for accessing it within the browseCampaigns.component is as follows:
browsingCampaignNames: httpAllNames[];
campaignCountDisplayed: number = 0;
onLoadCampaigns() {
this.browsCampServ.fetchCampaignsForBrowsing().subscribe(camps => {
this.browsingCampaignNames = camps;
});
this.campaignCountDisplayed = this.browsingCampaignNames.length;
}
What I'm expecting is to have an array of as many items as are within the httpAllNames object, however, Angular is treating that as a single array, with the desired array nested within.
I guess what I'm trying to do is 'unwrap' the outer layer so that my browsingCampaignNames property is able to access it.
I've tried adjusting the output of the API by removing the status code and unwrapping it from a generic httpResponse object. I've also tried this.browsingCampaignNames = camps[0]; and this.browsingCampaignNames = camps['httpAllNames']; as though to try to access the data by index, even though those methods are 'hacky.'
Thank you in advance.
I'm trying to learn GraphQL (& node.js & MongoDB etc.). I cant get this simple nested query to return results :
query getLocationByPerson {
People {
firstName
lastName
service {
location
}
}
}
The result I get is :
{
"data": {
"People": [
{
"firstName": "John",
"lastName": "DOE",
"service": null
}
]
},
"errors": [
{
"message": "Cannot return null for non-nullable field Service.location.",
"locations": [
{
"line": 6,
"column": 7
}
],
"path": [
"People",
0,
"service",
"location"
]
}
]
}
All the code is available here : https://github.com/fabricezerrouki/graphql-playground
Can anyone have a look and help me to figure out what I'm doing wrong?
Thanks.
The problem lies in the way you're setting the service attribute when you create and update a person. For nested types, you want to pass an object containing the subtypes. Right now you're passing it just a string, so when you try to set Service equal to args.serviceId in your createPerson resolver, it doesn't exist, and therefore sets the value of that field to null in MongoDB. Re-factor like so:
In index.js
type Mutation {
createPerson(Xid: String!, firstName: String */ ... */ service: { id: ID!, name: String!, location: String! }): People!
}
In /resolvers/Mutation.js
const Mutations = {
createPerson: async (parent, args) => {
const newPerson = new People({
Xid: args.Xid,
firstName: args.firstName
/* and so on... */
Service: args.service.id
});
}
}
Also, you may run into problems in the future given the way your saving the Service object to MongoDB as a Capital "S" Service, whereas your variable in the queries is lowercase "s" service. You'll have to remember any time you search the database to now directly destructure the the args.service into your Model.find({service}) method. Kind of a niche problem, but I've ran into that exact problem myself, so figured I would mention it.
Edited in response to OP comment
Yes it is possible to do an SQL join-link operation to return the Service object with the Person query, but you'll need to to that in the query resolver and just store the Service id during the mutation. In that case I guess the reason it's null is your calling args.serviceId but the createPerson() parameter is simply "service" so try changing Mutation.js to Service: args.service.
As for the actual join operation, that's a bit more complicated. I would recommend first returning the serviceId string to make sure that works correctly. Then you'll need to delve into the MongoDB aggregation pipeline. More specifically, the $lookup pipeline operator performs a function similar to an SQL join. I end up banging my head against the wall every time I have to deal with aggregation pipelines. It's a pretty advanced subject with a confusing syntax IMO. But there are plenty of tutorials and examples out there for doing exactly what your wanting.
I'm building a rest api that allows users to submit and retrieve data produced by surveys: questions are not mandatory and each "submit" could be different from each other. Each submit is a json with data and a "survey id":
{
id: abc123,
surveyid: 123,
array: [],
object: {}
...
}
I have to store this data and allow retrieving and querying.
First approach: going without schema and putting everything in a single collection: it works, but each json field is treated as a "String" and making queries on numeric values is problematic.
Second approach: get questions datatypes for each survey, make/save a mongoose schema on a json file and then keep updated this file.
Something like this, where entry "schema : {}" represent a mongoose schema used for inserting and querying/retrieving data.
[
{
"surveyid" : "123",
"schema" : {
"name": "string",
"username" : "string",
"value" : "number",
"start": "date",
"bool" : "boolean",
...
}
},
{ ... }
]
Hoping this is clear, I've some questions:
Right now I've a single collection for all "submits" and everything is treated as a string. Can I use a mongoose schema, without other modifications, in order to specify that some fields are numeric (or date or whatever)? Is it allowed or is it even a good idea?
Are there any disadvantage using an external json file? Mongoose schemas are loaded at run time when requested or does the service need to be restart when this file is updated?
How to store data with a "schema" that could change often ?
I hope it's clear!
Thank you!