I want to build a simple ERP app for my company,i came from PHP MYSQL background, i left PHP and now moving to NODE.JS because it overmatches PHP in many ways, i thought i have to learn NODE.JS,
my company is a meat and poultry importer and has hundreds of product and since it is CATCH WEIGHT INVENTORY SYSTEM, each carton has to be recorded as an individual data, they can be millions of cartons, besides INVENTORY, my app must also has other ERP functionalities such as SALES,PURCHASING,HRD,LOGISTIC and many others.
what i already have in mind, i would use :
-Node.js then
-Express / Meteor as backend
-Angular/React as template engine + D3js (i'm not sure) to visualize various reports
-various DBMS (i think i cannot rely onto just one DBMS, i have to harness the power both of RDBMS and NoSQL DBMS)
MySQL for storing user data
Key/Value pair DBMS (ElasticSearch / Redis) for storing hardly ever change data but used often and needs fast access such as data product
Graph DBMS (Neo4j) for storing user relation, warehouse structure and product movement
Document DBMS (MongoDB) for storing tally weight,product detail and many other document based data.
well chose the right tools and make a good plan is halfway to achieve goal, so i need enlightment from all of you Sir and Mam, give me suggestion what should/shouldn't i choose.
Your stack is okay but you need to re-consider the amount of database in use, you don't need the graph database Mysql can handle that for you and as for your key/value pair DBMS go with MongoDb it is a good key/value pair and document store.No need for the extra database.
Related
I am developing ERP system. I am using AWS lambda(Node) with Dynamo DB. I am new in dynamo. We are using multiple tables.
My one small use case is like this,
I have one company table in which company's address will be store. In that address fields there is city, state, country fields which will have primary key of master tables(City, state, country).
So I need to fetch city,state and country also when i fetch company details in single query. Also filter, sorting should be done from master table.
So can anyone help me with this that there is any way to implement this? And if possible also provide some node js query code for reference.
If possible provide some official doc so I can review that.
Thanks for any help :)
DynamoDB is a fantastic technology. However, it is completely different than SQL databases. As such, you'll need to approach the entire process differently than you would with a SQL database.
Entire books have been written on the subject, so I won't try to cover all the differences here. However, I'll pass along a few pointers to get you in the right direction.
Single table design is considered a best practice in DynamoDB. Having multiple tables isn't necessarily an anti-pattern, but spend some time investigating why this is the case. A good place to start: DynamoDB does not have a SQL join operation!
The DynamoDB Book is the best book on this topic, hands down. Spending just a few hours reading this book will drastically reduce the learning curve.
Watch AWS Re:Invent talks on DynamoDB. This talk from 2019 is a great place to start. (This talk is from Alex DeBrie, the author of the DynamoDB Book).
The rules of SQL databases do not apply when working with NoSQL databases. In other words, forget everything you know about SQL databases when working with NoSQL. It's a paradigm shift, and I find it best to set aside any preconceived notions about databases and approach the topic with an open mind. Easier said than done, but it's my advice anyway :)
DynamoDB requires that you build your data model to match the use cases of your application, aka your application "access patterns". Throwing your data into DynamoDB before thinking about how you need to access the data is the wrong approach. Instead, ask yourself "what are all the ways my application needs to get data?" and design your data model accordingly.
NoSQL data modeling has a steep learning curve, but it's awesome when you get it right! Good luck!
i'm walking my first steps with nosql databases, but so far my knowledge is very basic. I try to set up a database for a small invoice system.
In SQL i'd create 4 Tables: Products, Customers , Invoices, and a match table for Invoice and the produts.
But how to do this with nosql? Do i even build relations or just build 1 document for each invoice.
You should keep in mind that NoSQL design is not only based on data structure but also strongly on data function. So you should first ask yourself what kind of queries you need to do over your data and take it from there.
First figure out how far you want to go with denormalization and aggregation. For instance: what sets of data will often require to query or update at once? And try to keep that to a single document even if it means duplicating data from other entities (i.e. Storing customer data along with the invoice data).
So ask yourself why you want to use non relational databases, and how will you use that data. Then decide which modeling techniques to apply and how far. The highly scalable blog has a great article about NoSQL data modeling if you care to give it a read.
... or just build 1 document for each invoice.
Yes, do that for the beginning. Imagine your data in the CouchDB as read-only copy of your data in the relational database. The docs are like the result of your SQL queries.
Do i even build relations?
Of course you can, its the same as in your SQL tables. You including ids of foreign docs and name the property regarding to the relation you want to express e.g. doc.customer_id in an invoice doc can point to the doc._id of a customer doc.
Its helpful you imagine the CouchDB views as "relations" e.g. you can create a view called InvoicesByCustomer with the example above.
But summarized i would recommend to begin with the 1 document for each invoice.-approach and follow #JavoSN hint ...
So you should first ask yourself what kind of queries you need to do over your data and take it from there
... when you know that clearly its time to dig deeper into your possibilities of document designs.
I'm considering a setup where I have entities stored both in a document db (e.g. CouchDB) and a graph db (e.g. Neo4j).
The rationale is storing each entity information (data, blobs, values, complex internal structure) in the document db while storing the entity relations (parents, children, associated entities) in the graph db.
Has anyone done / seen / been bitten by a setup like this? What kind of issues should I expect? First thing that come to mindaka the 2-phase commit. But backups are problematic too here.
You may check out the book "Seven DBs in Seven Weeks". 8th chapter talks about building up a polyglot structure via CouchDB, Neo4j and Redis.
Ran,
Since CouchDB and most (all?) of the document/ kv stores do not support transactions, you would need to stop worrying about 2-phase-commits. You can do XA transactions between Neo4j and MySQL for example, but not CouchDB or it's relatives.
Indeed, for simplicity's sake, why not a pure graph database architecture? You get the better expressiveness and transactions - what is the rationale of adding another moving part in the form of a second store type?
Now I have a project with ads exchange service (something like google double click) and I have to pick a high-scalable database. I'm thinking about mongodb or cassandra.
Cassandra:
fit with our write-intensive system. (+)
looks hard to do aggregate(very important for analytics) (is there a good way? Just read slide about Twitter rainbird, seem good) (?)
I dont prefer java much. (-)
MongoDB:
Seem easier to do analytics. (have build-in aggregate functions) (+)
more RAM-consuming? (because of document-oriented vs key-value Cassandra) (?)
write perfomance compare to Cassandra? (?)
javascript shell and natural fit with node.js(one important part in our project) (+)
http://pastebin.com/raw.php?i=FD3xe6Jt - This article make me cautious. (-)
Can you guys help me to pick the one or answer some of my questions above
Thanks.
I don't know about Cassandra, but MongoDB has some advantages for using it for analytics: high concurrency, sharding, storing everything about an event in a single document, features like upsert and $inc.
For more detailed explanations check the following resources:
MongoDB Analytics - videos
http://blog.mongodb.org/post/171353301/using-mongodb-for-real-time-analytics
http://www.mongodb.org/display/DOCS/Use+Cases
http://www.slideshare.net/jrosoff/scalable-event-analytics-with-mongodb-ruby-on-rails
http://nosql.mypopescu.com/post/3508305955/fast-asynchronous-analytics-with-mongodb
http://blog.opengovernment.org/2011/02/24/fast-asynchronous-analytics-with-mongodb/
http://blog.10gen.com/post/4416876632/london-startup-ubervu-on-storing-5tb-of-data-in-mongodb
It depends a lot on your domain, most cases one would probably choose Mongo.
For example http://square.github.com/cube/ is built on Mongo.
Cube is an open-source system for visualizing time series data, built on MongoDB, Node and D3. If you send Cube timestamped events (with optional structured data), you can easily build realtime visualizations of aggregate metrics for internal dashboards. For example, you might use Cube to monitor traffic to your website, counting the number of requests in 5-minute intervals:
Most use cases of Cassandra draw from the need oh high availability that's the main feature of it afaik. Your needs seem to be centered around having a cheap way to shove queryable data in a scale-out DB, and mongo almost matches RDBMS in regards to querying. Mongo is also probably easier to deal with.
I think cassandra is a good fit for this problem.
You don't need to know much java to get it running (other than install java), as long as there is a client library in your chosen language.
Cassandra 0.8+ now has atomic counter support - perfect for impressions/click tracking.
You could also run hadoop on top of cassandra, giving you a proven platform for writing map reduce jobs to do analytics/aggregations and store the results back to Cassandra too.
Check out this slideshow about cassandra and hadoop: http://www.slideshare.net/jeromatron/cassandrahadoop-4399672
I hope that helps.
I am very newbie to this world of document db.
So... why this db are better than RDBMS ( like mysql or postgresql ) for very large amount of data ?
She have implement good indexing to carry this types of file, and this is designed for. This solution is better for Document Database, because is for it. Normal database is not designed to saving "documents", in this option you must hard work to search over your documents data, because each can be in other format this is a lot of work. If you choice document db solution you have all-in-one implemented because this database is for only "docuemnts", because this have implementation of these needed for it functions.
You want to distribute your data over multiple machines when you have a lot of data. That means that joins become really slow because joining between data on different machines means a lot of data communication between those machines.
You can store data in a mongodb/couchdb document in a hierarchical way so there is less need for joins.
But is is dependent on you use case(s). I think that relational databases do a better job when it comes to reporting.
MongoDB and CouchDB don't support transactions. Do you or your customers need transactions?
What do you want to do? Analyzing a lot of data (business intelligence/reporting) or a lot of small modifications per second "HVSP (High Volume Simple Processing)"?