I'm building a mapping app with Node.js. We have about 40,000 polygons that display on the map so I'm trying to improve performance by merging them where possible. Turf.js has a merge function that seems like the ticket. I haven't been able to get it to work though.
Here is the code where I'm trying to use turf in my controller.
var mongoose = require('mongoose');
var bodyParser = require('body-parser');
var turf = require('turf');
var fs = require('fs');
exports.show = function(req, res) {
res.render('map', {
title: 'Map'
});
};
exports.postSearch = function(req, res) {
// Bunch of query stuff goes into array below, (omitted for post)
mongoose.model('Claim').find({
$and:array
}, function(err, polygons){
// fs.writeFile('poly.txt', polygons);
var test = turf.merge(polygons);
res.json(test);
});
};
I put that fs.writeFile in there to get a sample of the geojson returned from mongodb. This is what I get:
{
properties: {
TTLTPCD: 'CL',
RNHCTRS: 389,
TNRTPCD: 'C'
},
geometry: {
coordinates: [
[Object]
],
type: 'Polygon'
},
type: 'Feature',
_id: '56d2a2061c601e547e1099ee'
}, {
properties: {
TTLTPCD: 'CS',
RNHCTRS: 261,
TNRTPCD: 'C'
},
geometry: {
coordinates: [
[Object]
],
type: 'Polygon'
},
type: 'Feature',
_id: '56d2a2071c601e547e109d37'
},
// this repeats a couple hundred times on my test query.
I get a stack trace but it doesn't make sense to me:
PROJECT_FOLDER/node_modules/turf/node_modules/turf-merge/index.js:55
var merged = clone(polygons.features[0]),
^
TypeError: Cannot read property '0' of undefined
at Object.module.exports [as merge] (/media/ng/DATA/LighthouseLabs/ClaimMaps/nodeMaps/MapEmGems/node_modules/turf/node_modules/turf-merge/index.js:55:39)
at Query.<anonymous> (/media/ng/DATA/LighthouseLabs/ClaimMaps/nodeMaps/MapEmGems/controllers/map.js:75:14)
at /media/ng/DATA/LighthouseLabs/ClaimMaps/nodeMaps/MapEmGems/node_modules/kareem/index.js:177:19
at /media/ng/DATA/LighthouseLabs/ClaimMaps/nodeMaps/MapEmGems/node_modules/kareem/index.js:109:16
at doNTCallback0 (node.js:430:9)
at process._tickCallback (node.js:359:13)
Turf seems to be looking for a features key in the geojson, but there is none. Does anyone have a solution for this?
###################### Edit after ghybs's answer
OK, ghybs sort of solved the geojson formatting issue. I changed the last bit of the controller to this:
mongoose.model('Claim').find({
$and:array
}, function(err, polygons){
polygons = {
type: "FeatureCollection",
features: [polygons] };
fs.writeFile('poly.txt', polygons);
var test = turf.merge(polygons);
fs.writeFile('test.txt', test);
res.json(test);
});
};
I put a second fs.writeFile to see what turf.merge was returning. The original geojson was not an array so I added the []'s. No more turf error. My map can't understand the output though.
Now poly.txt gives me this:
[object Object]
and test.txt contains this:
{ properties: null,
geometry: undefined,
type: 'Feature',
_id: '56d2a2061c601e547e1099ee',
__v: undefined },{ properties: null,
geometry: undefined,
type: 'Feature',
_id: '56d2a2071c601e547e109d37',
__v: undefined },
// This repeats 336 times
So I'm one step closer, I think. Any ideas? The map worked fine with the original data that was in poly.txt. I'm trying to get the same thing but merged.
The polygons object from your fs.writeFile() debug is not GeoJSON compliant. Turf expects a GeoJSON compliant FeatureCollection, which must have a features member.
See also the doc on Turf merge method, it gives you a code sample of what a GeoJSON compliant FeatureCollection should look like.
So it looks like you should simply wrap your polygons object in a FeatureCollection to make it compliant:
polygons = {
type: "FeatureCollection",
features: polygons // assuming polygons is an array
};
EDIT following the question edit
If your initial polygons really outputs "[Object]" in the geometry coordinates, rather than an array of coordinates, turf will not be able to understand your polygons geometries.
But if you say it was working before trying to merge, the issue would probably be something else.
How do you know your polygons are not an array? Are you sure doing [polygons] is the right solution?
For the next issues, please open a different question.
Related
What I was hoping to do was store an array of objects using RedisJSON very simply and then query that array.
I have something similar to this:
const data = [
{
_id: '63e7d1d85ad7e2f69df8ed6e',
artist: {
genre: 'rock',
},
},
{
_id: '63e7d1d85ad7e2f69df8ed6f',
artist: {
genre: 'metal',
},
},
{
_id: '63e7d1d85ad7e2f69df8ed6g',
artist: {
genre: 'rock',
},
},
]
then I can easily store and retrieve this:
await redisClient.json.set(cacheKey, '$', data)
await redisClient.json.get(cacheKey)
works great. but now I want to also query this data, I've tried creating an index as below:
await redisClient.ft.create(
`idx:gigs`,
{
'$.[0].artist.genre': {
type: SchemaFieldTypes.TEXT,
AS: 'genre',
},
},
{
ON: 'JSON',
PREFIX: 'GIGS',
}
)
and when I try and search this index what I expect is it to return the 2 documents with the correct search filter, but instead it always returns the entire array:
const searchResult = await redisClient.ft.search(`idx:gigs`, '#genre:(rock)')
produces:
{
total: 1,
documents: [
{ id: 'cacheKey', value: [Array] }
]
}
I can't quite work out at which level I'm getting this wrong, but any help would be greatly appreciated.
Is it possible to store an array of objects and then search the nested objects for nested values with RedisJSON?
The Search capability in Redis stack treats each key containing a JSON document as a separate search index entry. I think what you are doing is perhaps storing your whole array of documents in a single Redis key, which means any matches will return the document at that key which contains all of your data.
I would suggest that you store each object in your data array as its own key in Redis. Make sure that these will be indexed by using the GIGS prefix in the key name, for example GIGS:63e7d1d85ad7e2f69df8ed6e and GIGS:63e7d1d85ad7e2f69df8ed6f.
You'd want to change your index definition to account for each document being an object too so it would look something like this:
await redisClient.ft.create(
`idx:gigs`,
{
'$.artist.genre': {
type: SchemaFieldTypes.TEXT,
AS: 'genre',
},
},
{
ON: 'JSON',
PREFIX: 'GIGS:',
}
)
Note I also updated your PREFIX to be GIGS: not GIGS - this isn't strictly necessary, but does stop your index from accidentally looking at other keys in Redis whose name begins GIGS<whatever other characters>.
I'm working on an API using MongoDB and the Node.js driver for mongoDB. One of the routes takes in coordinates and calls an aggregation pipeline that returns results sorted by closeness and a number of other factors. The code for the first step of the pipeline is:
{$geoNear:{
near:{
type: "Point",
coordinates: [lat, long]
},
spherical:true,
distanceField:"distance"
}},
This was working fine until all of the sudden, it wasn't. Now it works for some coordinates and for others it gives the error MongoError: invalid argument in geo near query: type. It works fine with {lat:0, long:0}, fine with {lat:40.7411, long:-73.9897}, fine with {lat:46.68758191106798, long:2.8106467331441767}, but gives the error with {lat:37.785834, long:-122.406417} and {lat:47.643628614308675, long:-122.34560056016635}. The only pattern I can find is that it hates coordinates on the West Coast. I've verified that the 2d-sphere index is still there on the data. I've tried setting spherical to false. I've even tried deleting all the data, and still get the same behavior even with an empty collection. I'm truly mystified as to why the argument to geonear would be valid for some points and not for others.
Edit: I've now commented out everything except for the following code and I'm still getting the same behavior where some coordinates work and others give MongoError: invalid argument in geo near query: type
router.post("/", asyncHandler(async (req, res, next) => {
const {lat, long} = req.body;
const errors = [];
if(typeof(lat) !== "number" || Math.abs(lat) > 90) errors.push("Lat is invalid");
if(typeof(long) !== "number" || Math.abs(long) > 180) errors.push("Long is invalid");
if(errors.length) return next(errors);
const feed = await req.app.locals.db.collection("posts").aggregate(
[
{$geoNear:{
near:{
type: "Point",
coordinates: [lat, long]
},
spherical:true,
distanceField:"distance"
}},
]
);
const feedArr = await feed.toArray();
res.json(feedArr);
}));
See GeoJSON Objects:
If specifying latitude and longitude coordinates, list the longitude first and then latitude
You did coordinates: [lat, long]
I have a problem with logging out the contents of an array inside an object. The actual object looks like this
var stuff = { accepted: [ 'item1', 'item2' ],
rejected: [],
response: 'Foo',
envelope: { from: 'The sender', to: ['new item1', 'new item2'] },
messageId: 'xxxxxxxxxxxxx' } }
The console.log shows the items of the first array fine but the second array is being output as [Object].
{ accepted: [ 'item1', 'item2' ],
rejected: [],
response: 'Foo',
envelope: { from: 'The sender', to: [Object] },
messageId: 'xxxxxxxxxxxxx' } }
What is happening here and how can I get the items of the second array to show when I console.log. Thanks for any help!
UPDATE
Sorry, I forgot to add that I am working exclusively in Node.js so it's a server side log that needs to display the object exactly as it's received from a callback with a straight console.log, ie. no further stringify-ing.
I also just tried creating another random object with a similar structure like this.
var objText = {
fun: {
stuff: 'Some stuff',
list: ['this', 'it', 'takes']
}
};
The console.log for the above is:
{ fun: { stuff: 'Some stuff', list: [ 'this', 'it', 'takes' ] } }
This appears to be the same structure to me and yet the console.log works fine so it seems to be perfectly possible in Node to log arrays content even when it's embedded inside and an object inside an outer object.
It looks like this is an old topic, anyway
I've faced the same issue, embedded array printed as [Array].
It is because of console.log in the node.js uses util.inspect for print, the default depth is 2.
So, to print something which is deeper than 2 followings can be used:
const util = require('util')
console.log(util.inspect(errors, true, 10))
This is the default way for some browser and implementations of showing too complex or deep objects/arrays with console.log. An alternative is to use JSON.stringify with console.log:
var stuff = {
accepted: ['item1', 'item2'],
rejected: [],
response: 'Foo',
envelope: {
from: 'The sender',
to: ['new item1', 'new item2']
},
messageId: 'xxxxxxxxxxxxx'
}
console.log(JSON.stringify(stuff, null, 4));
EDIT:
Another alternative is to use console.dir in case you have a too complex or recursive object, see https://stackoverflow.com/a/27534731/6051261
Try it with: console.log(JSON.stringify(variable))
If you like Chrome devtools, that folds your json objects and let you observe a lot of things, you can use the --inspect flag:
node --inspect index.js
The console will then give you an URL and you just have to copy paste in Google Chrome to enjoy Google Chrome console.
More information on this link
I am new to node.js and newer to Sails.js framework.
I am currently trying to work with my database, I don't understand all the things with Sails.js but I manage to do what I want step by step. (I am used to some PHP MVC frameworks so it is not too difficult to understand the structure.)
Here I am trying to get a row from my database, using 2 JOIN clause. I managed to do this using SQL and the Model.query() function, but I would like to do this in a "cleaner" way.
So I have 3 tables in my database: meta, lang and meta_lang. It's quite simple and a picture being better than words, here are some screenshots.
meta
lang
meta_lang
What I want to do is to get the row in meta_table that match with 'default' meta and 'en' lang (for example).
Here are Meta and Lang models (I created them with sails generate model command and edited them with what I needed):
Meta
module.exports = {
attributes: {
code : { type: 'string' },
metaLangs:{
collection: 'MetaLang',
via: 'meta'
}
}
};
Lang
module.exports = {
attributes: {
code : { type: 'string' },
metaLangs:{
collection: 'MetaLang',
via: 'lang'
}
}
};
And here is my MetaLang model, with 3 functions I created to test several methods. The first function, findCurrent, works perfectly, but as you can see I had to write SQL. That is what I want to avoid if it is possible, I find it more clean (and I would like to use Sails.js tools as often as I can).
module.exports = {
tableName: 'meta_lang',
attributes: {
title : { type: 'string' },
description : { type: 'text' },
keywords : { type: 'string' },
meta:{
model:'Meta',
columnName: 'meta_id'
},
lang:{
model:'Lang',
columnName: 'lang_id'
}
},
findCurrent: function (metaCode, langCode) {
var query = 'SELECT ml.* FROM meta_lang ml INNER JOIN meta m ON m.id = ml.meta_id INNER JOIN lang l ON l.id = ml.lang_id WHERE m.code = ? AND l.code = ?';
MetaLang.query(query, [metaCode, langCode], function(err, metaLang) {
console.log('findCurrent');
if (err) return console.log(err);
console.log(metaLang);
// OK this works exactly as I want (I would have prefered a 'findOne' result, only 1 object instead of an array with 1 object in it, but I can do with it.)
});
},
findCurrentTest: function (metaCode, langCode) {
Meta.findByCode(metaCode).populate('metaLangs').exec(function(err, metaLang) {
console.log('findCurrentTest');
if (err) return console.log(err);
console.log(metaLang);
// I get what I expected (though not what I want): my meta + all metaLangs related to meta with code "default".
// What I want is to get ONE metaLang related to meta with code "default" AND lang with code "en".
});
},
findCurrentOthertest: function (metaCode, langCode) {
MetaLang.find().populate('meta', {where: {code:metaCode}}).populate('lang', {where: {code:langCode}}).exec(function(err, metaLang) {
console.log('findCurrentOthertest');
if (err) return console.log(err);
console.log(metaLang);
// Doesn't work as I wanted: it gets ALL the metaLang rows.
});
}
};
I also tried to first get my Meta by code, then my Lang by code, and MetaLang using Meta.id and Lang.id . But I would like to avoid 3 queries when I can have only one.
What I'm looking for would be something like MetaLang.find({meta.code:"default", lang.code:"en"}).
Hope you've got all needed details, just comment and ask for more if you don't.
Do you know what populate is for ? its for including the whole associated object when loading it from the database. Its practically the join you are trying to do, if all you need is row retrieval than quering the table without populate will make both functions you built work.
To me it looks like you are re-writing how Sails did the association. Id suggest giving the Associations docs another read in Sails documentation: Associations. As depending on your case you are just trying a one-to-many association with each table, you could avoid a middle table in my guess, but to decide better id need to understand your use-case.
When I saw the mySQL code it seemed to me you are still thinking in MySQL and PHP which takes time to convert from :) forcing the joins and middle tables yourself, redoing a lot of the stuff sails automated for you. I redone your example on 'disk' adapter and it worked perfectly. The whole point of WaterlineORM is to abstract the layer of going down to SQL unless absolutely necessary. Here is what I would do for your example, first without SQL just on a disk adapter id create the models :
// Lang.js
attributes: {
id :{ type: "Integer" , autoIncrement : true, primaryKey: true },
code :"string"
}
you see what i did redundantly here ? I did not really need the Id part as Sails does it for me. Just an example.
// Meta.js
attributes: {
code :"string"
}
better :) ?
// MetaLang.js
attributes:
{
title : "string",
desc : "string",
meta_id :
{
model : "meta",
},
lang_id :
{
model : "lang",
}
}
Now after simply creating the same values as your example i run sails console type :
MetaLang.find({meta_id : 1 ,lang_id:2}).exec(function(er,res){
console.log(res);
});
Output >>>
sails> [ { meta_id: 1,
lang_id: 2,
title: 'My blog',
id: 2 } ]
Now if you want to display what is meta with id 1 and what is lang with id 2, we use populate, but the referencing for join/search is just as simple as this.
sails> Meta_lang.find({meta_id : 1 ,lang_id:2}).populate('lang_id').populate('meta_id').exec(function(er,res){ console.log(res); });
undefined
sails> [ {
meta_id:
{ code: 'default',
id: 1 },
lang_id:
{ code: 'En',
id: 2 },
title: 'My blog',
id: 2 } ]
At this point, id switch adapters to MySQL and then create the MySQL tables with the same column names as above. Create the FK_constraints and voila.
Another strict policy you can add is to set up the 'via' and dominance on each model. you can read more about that in the Association documentation and it depends on the nature of association (many-to-many etc.)
To get the same result without knowing the Ids before-hand :
sails> Meta.findOne({code : "default"}).exec(function(err,needed_meta){
..... Lang.findOne({code : "En"}).exec(function(err_lang,needed_lang){
....... Meta_lang.find({meta_id : needed_meta.id , lang_id : needed_lang.id}).exec(function(err_all,result){
......... console.log(result);});
....... });
..... });
undefined
sails> [ { meta_id: 1,
lang_id: 2,
title: 'My blog',
id: 2 } ]
Have you tried:
findCurrentTest: function (metaCode, langCode) {
Meta.findByCode(metaCode).populate('metaLangs', {where: {code:langCode}}).exec(function(err, metaLang) {
console.log('findCurrentTest');
if (err) return console.log(err);
console.log(metaLang);
});
},
As I've progressed thru the world of CompoundJS I've came across two ways of defining a schema:
First:
var Product = describe('Product', function () {
property('upc', String);
property('name', String);
set('restPath', pathTo.products);
});
Second:
var Schema = require('jugglingdb').Schema;
var schema = new Schema('memory');
var Product = schema.define('Product', {
upc: { type: Number, index: true },
name: { type: String, limit: 150, index: true },
createdAt: { type: Date, default: Date.now },
modifiedAt: { type: Date, default: Date.now }
}, {
restPath: pathTo.products
});
The first, works, but looks like an old design. The second, does not work, but this is the way to do it according to JugglingDB docs.
Which one should I use? Why wouldn't the second one work for me?
UPDATE:
This is the error I get when I use the second one:
Express
500 TypeError: Cannot call method 'all' of undefined in products controller during "index" action
at Object.index (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\app\controllers\products.js:47:13)
at Array.2 (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\flow-control.js:150:28)
at ActionContext.run [as innerNext] (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\flow-control.js:103:31)
at Controller.BaseController.next (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\base.js:107:22)
at Controller.protectFromForgery (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\helpers.js:76:21)
at Object.protectFromForgeryHook (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\app\controllers\application.js:3:13)
at Array.1 (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\flow-control.js:150:28)
at run (C:\Users\Eran\Documents\Relay Foods\nutrition-facts-editor\node_modules\compound\node_modules\kontroller\lib\flow-control.js:103:31)
... snip ...
I think that describe and define are the same. The issue here is the usage scope which is global in my first implementation and local in the second one. So I'll need it to be global in order to work with the rest of the application.