I'm using mongodb to store application error logs as json documents. I want to be able to format the error logs as HTML rather than returning the plain json to the browser. The logs are properly schemaless - they could change at any time, so it's no use trying to do this (in Jade):
- var items = jsonResults
- each item in items
h3 Server alias: #{item.ServerAlias}
p UUID: #{item.UUID}
p Stack trace: #{item.StackTrace}
h3 Session: #{item.Session}
p URL token: #{item.Session.UrlToken}
p Session messages: #{item.Session.SessionMessages}
as I don't know what's actually going to be in the JSON structure ahead of time. What I want is surely possible, though? Everything I'm reading says that the schema isn't enforced by the database but that your view code will outline your schema anyway - but we've got hundreds of possible fields that could be removed or added at any time so managing the views in this way is fairly unmanageable.
What am I missing? Am I making the wrong assumptions about the technology? Going at this the wrong way?
Edited with extra info following comments:
The json docs look something like this
{
"ServerAlias":"GBIZ-WEB",
"Session":{
"urltoken":"CFID=10989&CFTOKEN=f07fe950-53926E3B-F33A-093D-3FCEFB&jsessionid=84303d29a229d1",
"captcha":{
},
"sessionmessages":{
},
"sessionid":"84197a667053f63433672873j377e7d379101"
},
"UUID":"53934LBB-DB8F-79T6-C03937JD84HB864A338",
"Template":"\/home\/vagrant\/dev\/websites\/g-bis\/code\/webroot\/page\/home\/home.cfm, line 3",
"Error":{
"GeneratedContent":"",
"Mailto":"",
"RootCause":{
"Message":"Unknown tag: cfincflude.",
"tagName":"cfincflude",
"TagContext":[
{
"RAW_TRACE":"\tat cfhome2ecfm1296628853.runPage(\/home\/vagrant\/dev\/websites\/nig-bis\/code\/webroot\/page\/home\/home.cfm:3)",
"ID":"CFINCLUDE",
"TEMPLATE":"\/home\/vagrant\/dev\/websites\/nig-bis\/code\/webroot\/page\/home\/home.cfm",
"LINE":3,
"TYPE":"CFML",
"COLUMN":0
},
{
"RAW_TRACE":"\tat cfdisplay2ecfm1093821753.runPage(\/home\/vagrant\/dev\/websites\/nig-bis\/code\/webroot\/page\/display.cfm:6)",
"ID":"CFINCLUDE",
"TEMPLATE":"\/home\/vagrant\/dev\/websites\/nig-bis\/code\/webroot\/page\/display.cfm",
"LINE":6,
"TYPE":"CFML",
"COLUMN":0
}
]
}
}
... etc, but is likely to change depending on what the individual project that generates the log is configured to trigger.
What I want to end up with is a formatted HTML page with headers for each parent and the children listed below, iterating right through the data structure. The Jade sample above is effectively what we need to output, but without hard-coding that in the view.
Mike's analysis in the comments of the problem being that of creating a table-like structure from a bunch of collections that haven't really got a lot in common is bang-on. The data is relational, but only within individual documents - so hard-coding the schema into anything is virtually impossible as it requires you to know what the data structure looks like first.
The basic idea is what #Gates VP described. I use underscore.js to iterate through the arrays/objects.
function formatLog(obj){
var log = "";
_.each(obj, function(val, key){
if(typeof(val) === "object" || typeof(val) === "array"){
// if we have a new list
log += "<ul>";
log += formatLog(val);
log += "</ul>";
}
else{
// if we are at an endpoint
log += "<li>";
log += (key + ": " + val);
log += "</li>";
}
});
return log;
}
If you call formatLog()on the example data you gave it returns
ServerAlias: GBIZ-WEBurltoken: CFID=10989&CFTOKEN=f07fe950-53926E3B-F33A-093D-3FCEFB&jsessionid=84303d29a229d1sessionid: 84197a667053f63433672873j377e7d379101UUID: 53934LBB-DB8F-79T6-C03937JD84HB864A338Template: /home/vagrant/dev/websites/g-bis/code/webroot/page/home/home.cfm, line 3GeneratedContent: Mailto: Message: Unknown tag: cfincflude.tagName: cfincfludeRAW_TRACE: at cfhome2ecfm1296628853.runPage(/home/vagrant/dev/websites/nig-bis/code/webroot/page/home/home.cfm:3)ID: CFINCLUDETEMPLATE: /home/vagrant/dev/websites/nig-bis/code/webroot/page/home/home.cfmLINE: 3TYPE: CFMLCOLUMN: 0RAW_TRACE: at cfdisplay2ecfm1093821753.runPage(/home/vagrant/dev/websites/nig-bis/code/webroot/page/display.cfm:6)ID: CFINCLUDETEMPLATE: /home/vagrant/dev/websites/nig-bis/code/webroot/page/display.cfmLINE: 6TYPE: CFMLCOLUMN: 0
How to format it then is up to you.
This is basically a recursive for loop.
To do this with Jade you will need to use mixins so that you can print nested objects by calling the mixin with a deeper level of indentation.
Note that this whole thing is a little ugly as you won't get guaranteed ordering of fields and you may have to implement some logic to differentiate looping on arrays vs. looping on JSON objects.
You can try util.inspect. In your template:
pre
= util.inspect(jsonResults)
Related
I am currently working on a Discord bot, which uses node.js, to scrape info from Dododex.com, a database of creatures from the game ARK. I found a very useful library called table-scraper which scrapes tables from web pages and returns the data as an object (or array of objects for multiple tables). In the background, table-scraper uses another library called x-ray, along with the common web request-maker request. Keep in mind that I don't know that much about node or how request works.
What I have working is asking the user for the name of a creature to get the data for. I then check that name against a list of creatures (that I have in a file/JSON object) to see if that name is valid.
Now, what I want to get working is to scrape the relevant data from the dododex page belonging to that creature using table-scraper. Table-scraper is definitely working with dododex, but when I call it to gather the data and put it into an object, it seems to start scraping, but then lets the function return with an empty object. If the program didn't crash right after the return (caused by the attempt to access an empty object), I know from testing that it would finish scraping, even though the function the scraping was called in has already finished.
Here's where the relevant code starts:
const creature = getDinoFromName(args[0].toLowerCase(), arkdata); //Function explained below
if(creature.health.base == 404) //If the creature's name is invalid, getDinoFromName() will return a creature with health.base = 404
{
console.log("unknown");
unknownCreature(args, msg); //Tells the user that the creature is unknown
return;
}
else
{
//Outputs the data
}
And here's where the data gets gathered:
function getDinoFromName(name, arkdata)
{
for(let i = 0; i < arkdata.creatures.length; i++) //Loops through arkdata, a json object with the possible names of all the creatures
{
if(arkdata.creatures[i].possible_names.includes(name)) //If the name the user provided matches one of the possible names of a creature (creatures can have multiple possible names, i.e. rex, t-rex, tyrannosaurus rex)
{
var creature;
console.log("found");
tableScraper.get("http://www.dododex.com/taming/" + arkdata.creatures[i].possible_names[0]).then(function(tableData)
{
console.log("scraped");
var stats = tableData[1]; //tableData is an array of 3 tables matching the 3 html tables found on the site, tableData[1] is the one with the creature's stats. It looks like this: http://ronsoros.github.io/?31db928bc662538a80bd25dd1207ac080e5ebca7
creature = //Now the actual creature object, parsed from tableData[1]
{
"health":
{
"base": stats[0]["Base (Lvl 1)"], //
"perwild": stats[0]["Increase Per Lvl (Wild)"],
"pertamed": stats[0]["Increase Per Lvl (Tamed)"]
}
//There is more data to parse but I only included hp as an example
}
});
return creature;
}
}
//If the creature name is not found to be one of the possible names, returns this to signify the creature was not found
return {"health":{"base":404}};
}
Running the code result in node crashing and the error
/app/server.js:320
if(creature.health.base == 404)
TypeError: Cannot read property 'health' of undefined
at stats (/app/server.js:320:15)
at Client.client.on.msg (/app/server.js:105:7)
Something wonky is going on here, and I really can't figure it out. Any help is appreciated.
I have been searching for hours, but I cannot find anything about this.
Situation:
Backend, existing of NodeJS + Express + Mongoose (+ MongoDB ofcourse).
Frontend retrieves object from the Backend.
Frontend makes some changes (adds/updates/removes some attributes).
Now I use mongoose: PersonModel.findByIdAndUpdate(id, updatedPersonObject);
Result: added properties are added. Updated properties are updated. Removed properties... are still there!
Now I've been searching for an elegant way to solve this, but the best I could come up with is something like:
var properties = Object.keys(PersonModel.schema.paths);
for (var i = 0, len = properties.length; i < len; i++) {
// explicitly remove values that are not in the update
var property = properties[i];
if (typeof(updatedPersonObject[property]) === 'undefined') {
// Mongoose does not like it if I remove the _id property
if (property !== '_id') {
oldPersonDocument[property] = undefined;
}
}
}
oldPersonDocument.save(function() {
PersonModel.findByIdAndUpdate(id, updatedPersonObject);
});
(I did not even include trivial code to fetch the old document).
I have to write this for every Object I want to update. I find it hard to believe that this is the best way to handle this. Any suggestions anyone?
Edit:
Another workaround I found: to unset a value in MongoDB you have to set it to undefined.
If I set this value in the frontend, it is lost in the REST-call. So I set it to null in the frontend, and then in the backend I convert all null-values to undefined.
Still ugly though. There must be a better way.
You could use replaceOne() if you want to know how many documents matched your filter condition and how many were changed (I believe it only changes one document, so this may not be useful to know). Docs: https://mongoosejs.com/docs/api/model.html#model_Model.replaceOne
Or you could use findOneAndReplace if you want to see the document. I don't know if it is the old doc or the new doc that is passed to the callback; the docs say Finds a matching document, replaces it with the provided doc, and passes the returned doc to the callback., but you could test that on your own. Docs: https://mongoosejs.com/docs/api.html#model_Model.findOneAndReplace
So, instead of:
PersonModel.findByIdAndUpdate(id, updatedPersonObject);, you could do:
PersonModel.replaceOne({ _id: id }, updatedPersonObject);
As long as you have all the properties you want on the object you will use to replace the old doc, you should be good to go.
Also really struggling with this but I don't think your solution is too bad. Our setup is frontend -> update function backend -> sanitize users input -> save in db. For the sanitization part, we use a helper function where we integrate your approach.
private static patchModel(dbDocToUpdate: IModel, dataFromUser: Record<string, any>): IModel {
const sanitized = {};
const properties = Object.keys(PersonModel.schema.paths);
for (const key of properties) {
if (key in dbDocToUpdate) {
sanitized[key] = data[key];
}
}
Object.assign(dbDocToUpdate, sanitized);
return dbDocToUpdate;
}
That works smoothly and sets the values to undefined. Hence, they get removed from the document in the db.
The only problem that remains for us is that we wanted to allow partial updates. With that solution that's not possible and you always have to send everything to the backend.
EDIT
Another workaround we found is setting the property to an empty string in the frontend. Mongo then also removes the property in the database
Consider following Employee document structure
{
"_id":...,
"rev":...,
"type":"Employee",
"fName":...,
"lName":...,
"designation":...,
"department":...,
"reportingTo":...,
"isActive":..,
more attributes
more attributes
}
And following map function in a View named "Employee"
function(doc) {
if (doc.type=="Employee") {
emit({
"EID":doc._id,
"FirstName":doc.fName,
"LastName":doc.lName,
"Designation":doc.designation,
"Department":doc.department,
"ReportingTo":doc.reportingTo,
"Active":doc.isActive
},
null
);
}
};
I want to query this view based on any combination & order of emitted attributes ( a query may include few random attributes may be like duck typing ). Is it possible? If so kindly let me know some samples or links.
Thanks
I've ran into the same problem a few times; you can, but you'll have to index each by itself (not all in one hash like you've done). But you could through the whole thing in the value for emit. It can be fairly inefficient, but gets the job done. (See this link: View Snippets)
I.e.:
function(doc) {
if (doc.type=="Employee") {
emit(["EID",doc.values.EID], doc.values);
emit(["FirstName", doc.fName], doc.values);
emit(["LastName", doc.lName], doc.values);
emit(["Designation", doc.designation], doc.values);
emit(["Department", doc.department], doc.values);
emit(["ReportingTo", doc.reportingTo], doc.values);
emit(["Active", doc.isActive], doc.values);
}
}
This puts all "EID" things in the same part of the tree, etc., I'm not sure if that is good or bad for you.
If you start needing a lot of functionality with [field name:]value searches, its probably worth it to move towards a Lucene-CouchDB setup. Several exist, but are a little immature.
I'm displaying a list of articles in a page that are fetched using the Ember Data RESTAdapter. I need to implement a bootstrap'esque paginator (see: http://twitter.github.com/bootstrap/components.html#pagination) and cant seem to find a sane pattern for returning pagination data such as, page count, article count, current page, within a single request.
For example, I'd like the API to return something like:
{
articles: [{...}, {...}],
page: 3,
article_count: 4525,
per_page: 20
}
One idea was to add an App.Paginator DS.Model so the response could look like:
{
articles: [{...}, {...}],
paginator: {
page: 3,
article_count: 4525,
per_page: 20
}
}
But this seems like overkill to hack together for something so trivial. Has anyone solved this problem or found a particular pattern they like? Is there a simple way to manage the RESTAdapter mappings to account for scenarios such as this?
Try to use Ember Pagination Support Mixin and provide your own implementation of the following method. Instead of loading all the content, you can fetch the required content when the user is navigating the pages. All what you need initially is the total account of your records.
didRequestRange: function(rangeStart, rangeStop) {
var content = this.get('fullContent').slice(rangeStart, rangeStop);
this.replace(0, this.get('length'), content);
}
With ember-data-beta3 you can pass a meta-property in your result. The default RESTSerializer looks for that property and stores it.
You can access the meta-data like this:
var meta = this.get("store").metadataFor("post");
If you are not able to change the JSON returned from the server you could override the extractMeta-hook on the ApplicationSerializer (or any other Model-specific serializer).
App.ApplicationSerializer = DS.RESTSerializer.extend({
extractMeta: function(store, type, payload) {
if (payload && payload.total) {
store.metaForType(type, { total: payload.total }); // sets the metadata for "post"
delete payload.total; // keeps ember data from trying to parse "total" as a record
}
}
});
Read more about meta-data here
I am using Struts 2 and want to include an editable server side paging and sorting grid.
I need to sublclass the QueryReadStore to implement the write and notification APIs. I do not want to inlcude server side REST services so i do not want to use JsonRest store. Any idea how this can be done.? What methods do i have to override and exactly how. I have gone through many examples but i am not getting how this can be done exactly.
Also is it possible to just extend the ItemFileWriteStore and just override its methods to include server side pagination? If so then which methods do i need to override. Can i get an example about how this can be done?
Answer is ofc yes :)
But do you really need to subclass ItemFileWriteStore, does it not fit your needs? A short explaination of the .save() follows.
Clientside does modify / new / delete in the store and in turn those items are marked as dirty. While having dirty items, the store will keep references to those in a has, like so:
store._pending = { _deletedItems: [], _modifiedItems: [], _newItems: [] };
On call save() each of these should be looped, sending requests to server BUT, this does not happen if neither _saveEverything or _saveCustom is defined. WriteStore simply resets its client-side revert feature and saves in client-memory.
See source search "save: function"
Here is my implementation of a simple writeAPI, must be modified to use without its inbuilt validation:
OoCmS._storeAPI
In short, follow this boiler, given that you would have a CRUD pattern on server:
new ItemFileWriteStore( {
url: 'path/to/c**R**ud',
_saveCustom: function() {
for(var i in this._pending._newItems) if(this._pending._deletedItems.hasOwnProperty(i)) {
item = this._getItemByIdentity(i);
dxhr.post({ url: 'path/to/**C**rud', contents: { id:i }});
}
for(i in this._pending._modifiedItems) if(this._pending._deletedItems.hasOwnProperty(i)) {
item = this._getItemByIdentity(i);
dxhr.post({ url: 'path/to/cr**U**d', contents: { id:i }});
}
for(i in this._pending._deletedItems) if(this._pending._deletedItems.hasOwnProperty(i)) {
item = this._getItemByIdentity(i);
dxhr.post({ url: 'path/to/cru**D**', contents: { id:i }});
}
});
Now; as for paging, ItemFileWriteStore has the pagination in it from its superclass mixins.. You just need to call it with two setups, one being directly on store meaning server should only return a subset - or on a model with query capeabilities where server returns a full set.
var pageSize = 5, // lets say 5 items pr request
currentPage = 2; // note, starting on second page (with *one* being offset)
store.fetch({
onComplete: function(itemsReceived) { },
query: { foo: 'bar*' }, // optional filtering, server gets json urlencoded
count: pageSize, // server gets &count=pageSize
start: currentPage*pageSize-pageSize // server gets &start=offsetCalculation
});
quod erat demonstrandum