Where to add index buffer size in Elasticsearch 6x - node.js

I want to increase the buffer size for elasticsearch in template settings, but where should I add this setting. Where should I write indices.memory.index_buffer_size:? As I understand it should be defined for the index, actually I have one index.
const indexTemplateSettings = {
number_of_shards: 2,
number_of_replicas : 0,
refresh_interval: '3600s',
'indices.memory.index_buffer_size': '30%',
// 'store.throttle.max_bytes_per_sec' : '100mb',
max_result_window: 1000000000,
'index.routing.allocation.enable': 'all',
}
export const init = async types => {
try {
let client = createClient()
const templateSettings = {
index_patterns : ['*'],
settings: indexTemplateSettings,
mappings : types.reduce((p, type) => ({
...p,
[type] : {
numeric_detection: true,
_source : {enabled : true},
dynamic_templates: [
{
all: {
match: '*',
mapping: {
copy_to: searchField,
},
},
},
],
properties: {
_searchField: {
type: 'text',
},
},
},
}), {}),
}
await client.indices.putTemplate({
name: 'default',
body: templateSettings,
},(error, response) => {
logger.silly('Pushing of index template completed', response)
})
} catch (e) {
logger.error(e)
}
}

This should go to the elasticsearch.yml configuration file. Quoting from the relevant docs:
The following settings are static and must be configured on every data node in the cluster
Default is 10% of the heap. That is a node wide setting and not on a per index base.
On a side note: This should be 512MB per shard as the maximum as described in the docs.

Related

Node.Js Elasticsearch Responding Blank _source

I have created one sample index using elasticsearch and node.js with below code setup.
const { Client } = require('#elastic/elasticsearch');
const { ELASTIC_SEARCH } = require('../config');
// Elastic Search Cloud Client Setup
const elasticClient = new Client({
cloud: { id: ELASTIC_SEARCH.CLOUDID },
auth: {
apiKey: ELASTIC_SEARCH.API_KEY
}
});
async function prepareIndex() {
const merchantIndexExists = await elasticClient.indices.exists({ index: 'index2' });
if (merchantIndexExists) return;
await elasticClient.indices.create({
index: 'index2',
body: {
mappings: {
dynamic: 'strict',
properties: {
company_name: { type: 'text' },
company_email: { type: 'keyword' },
name: { type: 'text' },
price: { type: 'scaled_float', scaling_factor: 10 },
created_date: { type: 'date' },
is_delete: { type: 'boolean', doc_values: false },
merchant: { type: 'keyword', index: 'true' }
}
}
}
});
}
After index creation i have added document with below code:
const { company_name, company_email, price } = req.body;
const response = await elasticClient.index({
index: 'index2',
document: {
company_email,
company_name,
price
}
});
Now when I'm calling search API from my kibana cloud console it's returning the exact search results with all the fileds. like
But when I'm hitting same search query via code in postman it's returning blank _source. Here is the search query with postman response
const response = await elasticClient.search({
index: 'index2',
query: {
match_all: {}
}
});
Can anyone please help me out of this?
When using the Node.js ElasticSearch client, you have to wrap the query into a body property and pass it to the search.
const response = await elasticClient.search({
index: 'index2',
body: {
query: {
match_all: {}
}
}
});

Elastic search sort error - search_phase_execution_exception

I am facing an issue with sorting values in an elastic search query. I am doing a simple search with sort but getting the following error. The query works without a sort parameter.
Elastic search client version: Version 7.6.1(Using this version because I am using opensearch)
search_phase_execution_exception: [illegal_argument_exception] Reason:
Text fields are not optimised for operations that require per-document
field data like aggregations and sorting, so these operations are
disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [subtype] in order to load field
data by uninverting the inverted index. Note that this can use
significant memory.
Code Sample:
const {Client} = require('#elastic/elasticsearch') // Version 7.6.1
var connectionString = 'https://admin:admin#localhost:9200'
const client = new Client({
node: connectionString,
ssl: {
rejectUnauthorized: false
}
})
client.info()
.then(async response => {
console.log('success', response.statusCode)
var query = {
"query": {
"match": {
"revhostname": {
"query": "ten.tsacmoc.ac.1dsh.631-651-14-37-c",
},
},
},
"sort": [
{
"revhostname": {"order": "asc"},
"subtype": {"order": "asc"},
"value": {"order": "asc"},
}
],
};
var response = await client.search({
index: 'r7',
body: query,
});
console.log("Search results:", JSON.stringify(response));
})
.catch(error => {
console.error('error', JSON.stringify(error))
})
Mapping:
{
"properties": {
"revhostname": {
"type" : "keyword"
},
"value": {
"type" : "keyword"
},
"subtype": {
"type" : "keyword"
},
"timestamp": {
"type" : "long"
},
"ip": {
"type" : "ip"
}
}
}
I tried adding fielddata=true in mapping but the issue was not solved. Your help is much appreciated.
Thank you.
As you mentioned mapping in a comment, your revhostname field is defined as text and keyword both type of field and Elasticsearch dont allow sorting on text type of field.
If your mapping is still same as you mentioned in comment then you need to use the field name like revhostname.keyword which will resolved issue.
const {Client} = require('#elastic/elasticsearch') // Version 7.6.1
var connectionString = 'https://admin:admin#localhost:9200'
const client = new Client({
node: connectionString,
ssl: {
rejectUnauthorized: false
}
})
client.info()
.then(async response => {
console.log('success', response.statusCode)
var query = {
"query": {
"match": {
"revhostname": {
"query": "ten.tsacmoc.ac.1dsh.631-651-14-37-c",
},
},
},
"sort": [
{
"revhostname.keyword": {"order": "asc"},
"subtype.keyword": {"order": "asc"},
"value.keyword": {"order": "asc"},
}
],
};
var response = await client.search({
index: 'r7',
body: query,
});
console.log("Search results:", JSON.stringify(response));
})
.catch(error => {
console.error('error', JSON.stringify(error))
})

How to grab field value during a MongooseModel.bulkWrite operation?

Context:
I am trying to upsert in bulk an array of data, with an additional computed field: 'status'.
Status should be either :
- 'New' for newly inserted docs;
- 'Removed' for docs present in DB, but inexistent in incoming dataset;
- a percentage explaining the evolution for the field price, comparing the value in DB to the one in incoming dataset.
Implementations:
data.model.ts
import { Document, model, Model, models, Schema } from 'mongoose';
import { IPertinentData } from './site.model';
const dataSchema: Schema = new Schema({
sourceId: { type: String, required: true },
name: { type: String, required: true },
price: { type: Number, required: true },
reference: { type: String, required: true },
lastModified: { type: Date, required: true },
status: { type: Schema.Types.Mixed, required: true }
});
export interface IData extends IPertinentData, Document {}
export const Data: Model<IData> = models.Data || model<IData>('Data', dataSchema);
data.service.ts
import { Data, IPertinentData } from '../models';
export class DataService {
static async test() {
// await Data.deleteMany({});
const data = [
{
sourceId: 'Y',
reference: `y0`,
name: 'y0',
price: 30
},
{
sourceId: 'Y',
reference: 'y1',
name: 'y1',
price: 30
}
];
return Data.bulkWrite(
data.map(function(d) {
let status = '';
// #ts-ignore
console.log('price', this);
// #ts-ignore
if (!this.price) status = 'New';
// #ts-ignore
else if (this.price !== d.price) {
// #ts-ignore
status = (d.price - this.price) / this.price;
}
return {
updateOne: {
filter: { sourceId: d.sourceId, reference: d.reference },
update: {
$set: {
// Set percentage value when current price is greater/lower than new price
// Set status to nothing when new and current prices match
status,
name: d.name,
price: d.price
},
$currentDate: {
lastModified: true
}
},
upsert: true
}
};
}
)
);
}
}
... then in my backend controller, i just call it with some route :
try {
const results = await DataService.test();
return new HttpResponseOK(results);
} catch (error) {
return new HttpResponseInternalServerError(error);
}
Problem:
I've tried lot of implementation syntaxes, but all failed either because of type casting, and unsupported syntax like the $ symbol, and restrictions due to the aggregation...
I feel like the above solution might be closest to a working scenario but i'm missing a way to grab the value of the price field BEFORE the actual computation of status and the replacement with updated value.
Here the value of this is undefined while it is supposed to point to current document.
Questions:
Am i using correct Mongoose way for a bulk update ?
if yes, how to get the field value ?
Environment:
NodeJS 13.x
Mongoose 5.8.1
MongoDB 4.2.1
EUREKA !
Finally found a working syntax, pfeeeew...
...
return Data.bulkWrite(
data.map(d => ({
updateOne: {
filter: { sourceId: d.sourceId, reference: d.reference },
update: [
{
$set: {
lastModified: Date.now(),
name: d.name,
status: {
$switch: {
branches: [
// Set status to 'New' for newly inserted docs
{
case: { $eq: [{ $type: '$price' }, 'missing'] },
then: 'New'
},
// Set percentage value when current price is greater/lower than new price
{
case: { $ne: ['$price', d.price] },
then: {
$divide: [{ $subtract: [d.price, '$price'] }, '$price']
}
}
],
// Set status to nothing when new and current prices match
default: ''
}
}
}
},
{
$set: { price: d.price }
}
],
upsert: true
}
}))
);
...
Explanations:
Several problems were blocking me :
the '$field_value_to_check' instead of this.field with undefined 'this' ...
the syntax with $ symbol seems to work only within an aggregation update, using update: [] even if there is only one single $set inside ...
the first condition used for the inserted docs in the upsert process needs to check for the existence of the field price. Only the syntax with BSON $type worked...
Hope it helps other devs in same scenario.

Unable to fetch list in react relay

I am following schema same as mentioned here
I want to fetch all users so I updated my schema like this
var Root = new GraphQLObjectType({
name: 'Root',
fields: () => ({
user: {
type: userType,
resolve: (rootValue, _) => {
return getUser(rootValue)
}
},
post: {
type: postType,
args: {
...connectionArgs,
postID: {type: GraphQLString}
},
resolve: (rootValue, args) => {
return getPost(args.postID).then(function(data){
return data[0];
}).then(null,function(err){
return err;
});
}
},
users:{
type: new GraphQLList(userType),
resolve: (root) =>getUsers(),
},
})
});
And in database.js
export function getUsers(params) {
console.log("getUsers",params)
return new Promise((resolve, reject) => {
User.find({}).exec({}, function(err, users) {
if (err) {
resolve({})
} else {
resolve(users)
}
});
})
}
I am getting results in /graphql as
{
users {
id,
fullName
}
}
and results as
{
"data": {
"users": [
{
"id": "VXNlcjo1Nzk4NWQxNmIwYWYxYWY2MTc3MGJlNTA=",
"fullName": "Akshay"
},
{
"id": "VXNlcjo1Nzk4YTRkNTBjMWJlZTg1MzFmN2IzMzI=",
"fullName": "jitendra"
},
{
"id": "VXNlcjo1NzliNjcyMmRlNjRlZTI2MTFkMWEyMTk=",
"fullName": "akshay1"
},
{
"id": "VXNlcjo1NzliNjgwMDc4YTYwMTZjMTM0ZmMxZWM=",
"fullName": "Akshay2"
},
{
"id": "VXNlcjo1NzlmMTNkYjMzNTNkODQ0MmJjOWQzZDU=",
"fullName": "test"
}
]
}
}
but If I try to fetch this in view as
export default Relay.createContainer(UserList, {
fragments: {
userslist: () => Relay.QL`
fragment on User #relay(plural: true) {
fullName,
local{
email
},
images{
full
},
currentPostCount,
isPremium,
}
`,
},
});
I am getting error Minified exception occurred; use the non-minified dev environment for the full error message and additional helpful warnings.
Please tell me what I am missing .
I tried a lot with and without #relay(plural: true).
Also tried to update schema with arguments as
users:{
type: new GraphQLList(userType),
args: {
names: {
type: GraphQLString,
},
...connectionArgs,
},
resolve: (root, {names}) =>connectionFromArray(getUsers(names)),
},
but I got error Cannot read property 'after' of undefined in implementing react-relay
Thanks in Advance.
Relay currently only supports three types of root fields (see facebook/relay#112):
Root field without arguments, returning a single node:
e.g. { user { id } } returning {"id": "123"}
Root field with one argument, returning a single node:
e.g. { post(id: "456") { id } } returning {"id": "456"}
Root field with one array argument returning an array of nodes with the same size as the argument array (also known as "a plural identifying root field"):
e.g. { users(ids: ["123", "321"]) { id } } returning [{"id": "123"}, {"id": "321"}]
A workaround is to create a root field (often called viewer) returning a node that has those fields. When nested inside the Viewer (or any other node), fields are allowed to have any return type, including a list or connection. When you've wrapped the fields in this object in your GraphQL server, you can query them like this:
{
viewer {
users {
id,
fullName,
}
}
}
The Viewer type is a node type, and since there will just be one instance of it, its id should be a constant. You can use the globalIdField helper to define the id field, and add any other fields you want to query with Relay:
const viewerType = new GraphQLObjectType({
name: 'Viewer',
interfaces: [nodeInterface],
fields: {
id: globalIdField('Viewer', () => 'VIEWER_ID'),
users:{
type: new GraphQLList(userType),
resolve: (viewer) => getUsers(),
},
},
});
On the client you'll need to change the root query in your route to { viewer } and define the fragment on Viewer:
export default Relay.createContainer(UserList, {
fragments: {
viewer: () => Relay.QL`
fragment on Viewer {
users {
fullName,
local {
email,
},
images {
full,
},
currentPostCount,
isPremium,
}
}
`,
},
});

Not Getting Search value in Sencha Touch using searchfield

I want to display predictive text in search field, value for predictive text which comes from server. Here is my code so far:
View:
Ext.define('MyApp.view.AutoSearch', {
extend: 'Ext.dataview.List',
alias : 'widget.mainPanel',
config: {
store : 'AutoSearchStore',
itemTpl: '<div class="myWord">'+
'<div>Word is --<b>{name}</b>--- after search!!!</div>' +
'</div>',
emptyText: '<div class="myWord">No Matching Words</div>',
items: [
{
xtype: 'toolbar',
docked: 'top',
items: [
{
xtype: 'searchfield',
placeHolder: 'Search...',
itemId: 'searchBox'
}
]
}
]
}
});
Store:
Ext.define('MyApp.store.AutoSearchStore',{
extend: 'Ext.data.Store',
config:
{
model: 'MyApp.model.AutoSearchModel',
autoLoad:true,
id:'Contacts',
proxy:
{
type: 'ajax',
url: 'http://alucio.com.np/trunk/dev/sillydic/admin/api/word/categories/SDSILLYTOKEN/650773253e7f157a93c53d47a866204dedc7c363',
reader:
{
rootProperty:''
}
}
}
});
Model:
Ext.define('MyApp.model.AutoSearchModel', {
extend: 'Ext.data.Model',
requires: ['MyApp.model.AutoSearchModelMenu'],
config: {
fields: [
{name:'data', mapping: 'data'},
{name: 'name'},
],
},
});
and
Ext.define('MyApp.model.AutoSearchModelMenu', {
extend: 'Ext.data.Model',
config: {
fields: [
'name',
],
belongsTo: "MyApp.model.AutoSearchModel"
}
});
Controller:
Ext.define('MyApp.controller.SearchAutoComplete', {
extend : 'Ext.app.Controller',
config: {
profile: Ext.os.deviceType.toLowerCase(),
stores : ['MyApp.store.AutoSearchStore'],
models : ['MyApp.model.AutoSearchModel'],
refs: {
myContainer: 'mainPanel'
},
control: {
'mainPanel': {
activate: 'onActivate'
},
'mainPanel searchfield[itemId=searchBox]' : {
clearicontap : 'onClearSearch',
keyup: 'onSearchKeyUp'
}
}
},
onActivate: function() {
console.log('Main container is active--Search');
},
onSearchKeyUp: function(searchField) {
queryString = searchField.getValue();
console.log(this,'Please search by: ' + queryString);
var store = Ext.getStore('AutoSearchStore');
store.clearFilter();
if(queryString){
var thisRegEx = new RegExp(queryString, "i");
store.filterBy(function(record) {
if (thisRegEx.test(record.get('name'))) {
return true;
};
return false;
});
}
},
onClearSearch: function() {
console.log('Clear icon is tapped');
var store = Ext.getStore('AutoSearchStore');
store.clearFilter();
},
init: function() {
console.log('Controller initialized for SearchAutoComplete');
}
});
Json Data Looks Like:
"data":[
{
"name":"paint",
"author":"admin",
"word_id":"1",
"category":"Business",
"is_favourite":"yesStar"
},
{
"name":"abacus",
"author":"admin",
"word_id":"2",
"category":"Education",
"is_favourite":"yesStar"
},
{
"name":"abate",
"author":"admin",
"word_id":"3",
"category":"Education",
"is_favourite":"noStar"
},
{
"name":"testing adsf",
"author":"admin",
"word_id":"7",
"category":"Education",
"is_favourite":"noStar"
},
{
"name":"sprite",
"author":"admin",
"word_id":"6",
"category":"Business",
"is_favourite":"noStar"
},
{
"name":"newword",
"author":"admin",
"word_id":"8",
"category":"Architecture",
"is_favourite":"noStar"
}
]
})
If I type "A", then it displays No Matching Words, but I have words from "A" on json coming from server. How to solve this problem?
Any idea!
Code Sources Link
I don't know why you are using two models but just one thing you need to specify in AutoSearchStore :
reader:
{
rootProperty:'data'
}
instead of
reader:
{
rootProperty:''
}
to get the expected results in the list.
Hope this will be helpful :)

Resources