I have a app perpetuating data in Postgresql/Express/Knex/Objection. I am looking for. way t track changes in my models, so that I can manage and revert versions similar to paper_trail in rails or this port for sequelize: https://github.com/nielsgl/sequelize-paper-trail
Is there something I could use for this in Knex/Objection or at the db level to track changes
Answer: There is not any generic way to do it in Objection nor knex.
Random rambling:
You need to design what kind of changes you like to track and write some code for example to Model hooks in objection how to track the changes.
One way to implement it would be for example by adding a separate table where all the tracked changes are written for example in JSONB object where updated fields or old values are stored and indexed or something like that. I'm pretty sure you don't want to add tracking of all the data in the database, since it will blow up the DB size very fast.
Anyways implementation depends what it is actually why you like or need to track the data and what are actual use cases that you need to support.
Also this might work for you: https://wiki.postgresql.org/wiki/Audit_trigger
Related
Recently I learned about change streams in mongodb and how powerful they are, but I’m in need of getting the previous values before an update. However, I seem to have learned through some research that it’s impossible to get the previous values? So this got me thinking about what alternatives exist in order to retrieve those previous values?
What I want to achieve is a logging system such as, “Record A field has changed from {old_value} to {new_value}.” I’m using socket.io to push these updates to a react front-end client. The updates to records would be happening from a completely different system and not on the same blackened server where the change streams would be listening so I won’t be able to query the document before updating.
So I started to think of a different solution…maybe I could have two databases? One contains the old records and the other the updated records but this sounds like a duplication of data. And I can’t imagine having thousands of records.
I need some guidance as I really don’t know what the best option is? Is there really no way you can use change streams and get the previous values? Is it possible to somehow query the document before a change stream event? Thank you.
Not sure how I missed this but the solution is versioning the data.
The MongoDB docs discuss modelling computed data saying:
The application can either recompute the value with every write that changes the computed value’s source data, or as part of a periodic job.
Updating the data periodically is straight forward but not real-time enough for our use-case, so I wanted to know the best practice for hooking into every DB write in order to recompute some derived data?
Our application is written in NodeJS, using the native MongoDB driver.
I understand Mongoose offers a 'save' hook which sounds ideal for this, however it falls down pretty quickly as it doesn't apply to .update, .updateMany etc. (and Mongoose.prototype.save is pretty terrifying imo).
Does MongoDB support change events at the DB level which would be suitable, or perhaps the native driver (or another client?) exposes hooks designed for re-computing data?
This is a general best practice question:
I am building a MEAN (mongo, express, angular, node) website. I have a user object that can have a gender [Mr or Miss] and a city [Paris, New York, Anything]
So this is quite a common problem: where should I store those lists that rarely change and never exceed, let's say, 50 rows.
1/ Is it better to have them stored in the database (mongo) with a foreign key in the user table. And so I have a gender table and a city table. But everytime I access these lists I need to read the base?
2/ Is it better to have them store in a file or in a controller? But this is a bit dangerous I think.
3/ Maybe there is another way that I don't know about.
I am not sure what is the best solution.
Are you concerned about an extra database call to get a list out?
If it was me I'd pick option 1 and I'd be storing it in a database. If you store value descriptions only front-end you'll run the risk of discrepancies if you end up updating your database's foreign keys but forget to update your controller or file and it seems rather untrustworthy. It also makes it more difficult to provide internationalization, because you'll have to start storing names and genders in files or controllers in multiple languages. Storing things is what a database is for and an additional call to get a list out is really not that big an impact on your performance.
Angular's $http object, which you are probably using to call your API has a caching option, which means you'll only need to retrieve the list once per app instantiation.
You could alternatively have a look at this post by Josh who found a way to pre populate a directive with JSON from the server before loading it.
Simple question: I would like to react to some changes in a database, but only to those changes which are causing modifications in a certain view1. That is, I am not interested in all changes in the database, just those changes which are affecting view1. I am not talking about filter here, just about view+changes. Something like this (although this is probably not correct):
http://localhost:5984/db/_design/doc1/_view/view1/_changes
Is this at all supported by CouchDB? Does this makes sense at all?
It's possible, but in a little another way. Since 1.1.0 release CouchDB is able to use map function as filters for changes feed. This works as like regular filters: if key-value pair was emitted at least once for changed document it means that he passes filter and _changes yields the record about him. If you need get only new updates for specific view, you need to specify staring since seq number - it could be easily retrieved from _design/ddoc-name/_info resource from field view_index/update_seq. Since 1.3 release you may also specify since=now to listen updates from current point of time.
Note, that this view filters doesn't uses view index and doesn't updates him while new changes occurs. Also, there is set of patches that improves view filters in the way that you may be also interested.
I need to replicate in CouchDB data from one database to another but in the process I want to alter the documents being replicated over,
mostly stripping out particular fields (but other applications mentioned in comments).
The replication would always be 100% one way (but other applications mentioned in comments could use bi-directional and sync)
I would prefer if this process did not increment their revision ID but that might be asking for too much.
But I don't see any of the design document functions that do what I am trying to do.
As it seems doesn't do this, what plans are there for adding this? And meanwhile, what workarounds are there?
No, there is no out-of-the-box solution, as this would defy the whole purpose and logic of multi-master, MVCC logic.
The only option I can see here is to create your own solution, but I would not call this a replication, but rather ETL (Extract, Transform, Load). And for ETL there are tools available that will let you do the trick, like (mixing open source and commercial here):
Scriptella
CloverETL
Pentaho Data Integration, or to be more specific Kettle
Jespersoft ETL
Talend have some tools as well
There is plenty more of ETL tools on the market.
I believe the best approach here would be to break out the fields you want to filter out into a separate document and then filter out the document during replication.
Of course the best way would be to have built-support for this, but a workaround which occurs to me would be, instead of here using the built-in replication, to code and use a custom replication which will do the additional needed alterations/transformations, still using rather than going beneith, the other built-ins, and with good coding, in many situations (especially if each master can push to its slaves), it feels this could be nearly as efficient.
This requires efficient triggers be put on each source/master to detect any changes, which I believe CouchDB does offer (or at least PouchDB appears to), which would then copy the changes to another location also doing the full alterations.
If the source of the change is unable to push the change to the final destination, this fixed store may to be local to it where the destination can pull from -- which could get pretty expensive especially in multi-master, as each location has to not only store & maintain its own data but also the data (being sent) of everyone it sends to.
This replicate would also place each source document's revision ID in the the document's copy...
...that is ideally, including essential if the copy was to be {updated, aka a master}, too.
...in form of either:
ideally the normal "_rev" property. Indeed this looks quite possible per it ("preserve their revisions ID") already done by the normal replication algorithm using the builtin "Bulk Docs API" which seemingly our varient would use, too
otherwise have a new copy object (with its own _rev) plus another field as "_rev_original" ntelling the original rev. But well that would work?
Clearly such copy could be created no problem.
Probably no big if the destination is just reading the data.
Seems hairy if the destination is also writing the data. As we'd now have to merge with these non-standard revisions. But doable.
Relevant to this (coding an a custom/improved replication (to do this apparently-missing functionality) ideally without altering Pouch and especially Couch source code), as starter/basis material (the standard method), here's the normal Couch replication algorithm which unfortunately doens't clearly say it only uses builtin ops but it looks like it, and also the official overview of what it does; I'm suspecting Pouch implements this, likely in Pouch's replicate.js (latest release as of 2014.07).
Futher implementation particulars? - those who would know, please put it here.
This is a "community wiki" answer so please extend it.
Also please comment links & details of anyone/system already doing or trying to do this or similar.