I have been looking for a database that can be embedded and also be file-based, like Sqlite.
I wanted a NoSQL type of database with this kind of feature.
The language is Python, and ArangoDB has binding for Python, and many other languages.
I am finding conflicting facts about ArangoDB.
In some cases I have seen articles say it is not an embedded DB, or can't be embedded, then see others that imply it is embedded.
Also on the website it says that it stores its data in a special binary format, and then I see an article saying its mainly an In-Memory database.
So its been very confusing.
1)So the question is, can this database run embedded in a python app?
If not, if it runs as a separate process, runs as a server, can this be generated/managed in Python with "zero configuration" on the part of the user, for the sake of deploying a desktop app based on this.
2) Does the database data etc get stored on disk.
SO that is it!
No, you can't embedd ArangoDB in the way you embedd SQLite.
ArangoDB offers the Foxx framework, which you can use to implement RESTfull microservices in JavaScript close to the database core like you would use python with SQLite. However, with AQL ArangoDB also offers a query language as SQLite does with sql.
There are currently several python drivers available that grant you access to ArangoDB from python in a compfortable manner.
The ArangoDB download page offers several packages, which you could use to deploy ArangoDB alongside your app. We offer a windows zip package that you could install by yourselves without user interaction; For linux distributions you'd probably want to use the respective package for that distribution. Easy deployability is one of our core goals.
Regarding the database and your data itself, this gets persisted to disk. This works via memory mapped files. However, the index and other structures are built up during the startup, which is why we refer to ourselves as mostly in memory.
Regular access to ArangoDB (and foxx) is done via the http interface and you get json documents as response. The drivers abstract that interface for you. If you implement foxx apps, you may need to formulate requests on your own.
ArangoDB Datafiles aren't intended to be moved across machines; though it may work as long as you have the same OS & Architectures on both sides. The proper way of doing this is to use ArangoDump on the first machine and ArangoRestore on the second. These are mostly json inside (one json document per line) so they're portable and even simple to load in python - you could even directly access the dump facility from python, and prepare an email for the user with the content.
The most sustainable way of running ArangoDB would be as a service; please note that you may need elevated privileges to register & re/start new services in Windows. The service then binds a tcp port, which you may access from other nodes in the network.
Related
As a new developer, I am always confused about using NodeJS and MongoDB. So I hope I will get an actual answer. Thanks...
MongoDB and NodeJS are two different technologies. MonogDB is a database system which gives you a chance to efficiently store documents in a database and to perform operations like data updates, or to search documents by some criterias.
NodeJS's responsibilty is especially to execute your application.
Nodejs is a Javascript engine that you can write any application you want with (by programming in the Javascript language). It runs your Javascript code. Most commonly, it is used to build servers that can respond to web requests, though it can be used for lots of other types of code too.
MongoDB is a database engine. Code within some application or server uses MongoDB to save, query or update data in a database. There are many web servers built with nodejs that will then use MongoDB for storing data.
MongoDB offers an API library that runs within a Nodejs application to give you programmatic access to MongoDB so you can create databases and then add, query, update or delete data from the MongoDB database. MongoDB also has API libraries for other programming environments such as Python, Java, etc...
These two technologies are for different parts of a typical web server system. You don't substitute one for the other. Instead, you can use them together.
When should we use Nodejs?
Any project needs a programming environment and a runtime library that offers you basic programming tools/support and can compile and/or interpret your code. Nodejs is such as tool for the Javascript programming language. There are other similar tools for other languages such as Python, Java, PHP, C#, C++, Go, etc...
So, if you want to write some kind of stand-alone program or server in Javascript, then you can use nodejs for it.
When should we use MongoDB?
If your application needs the ability to persistently store data in a way that you can efficiently query or update it later, then you would typically use some form of database. There are dozens of popular databases. MongoDB is one such database. MariaDB, MySql, CouchDB, DynamoDB (on AWS), Postgres are examples of other databases. Different databases have different strengths (things they are best at) and different ways of using them so it's a whole different question to choose the right/best database for what you're doing.
Here is a Blog link that will tell you when you should use NodeJS.
Here is another link to understand when you should use mongoDB.
Well, NodeJS and MongoDB are two different process. Here I tried to explain as simpler as possible:
NodeJS
Shortly we can say, NodeJS is a JavaScript runtime environment. It's actually helps JavaScript to run outside of server. It's used in server side development.
MongoDB
But, MongoDB is NoSQL database which is document oriented. It represents data as of JSON documents. It's used for store data.
Summary
The summary is MongoDB is a database where we can store data and NodeJS helps us to to connect our client site to database by it's server site.
Example:
Suppose you are building a website and you need a database to store the data or information so here you can use MongoDB but to be connected with MongoDB you need a connector, so here you can use NodeJS which will help your website to run outside of server.
Nodejs: Node.js is an interpreter or runtime/ running environment for JavaScript. built on Chrome's V8 JavaScript engine. responsibility is especially to execute your application.
MongoDB: is a No-SQL database for storing data. when you need high availability of data with automatic, fast, and instant data recovery.
I've begun to dive into developing a desktop application with electron. I have been interested in pairing this application with a NoSQL database to create users, display data, and do CRUD operations. I've considered databases such as MongoDB and CouchDB, and I'm curious if creating a desktop application that communicates with a database hosted elsewhere is a feasible goal.
I'm hoping that someone here can help direct me to great resources on creating a desktop application that works with a remote NoSQL database. Any advice here would be greatly appreciated!
I recommend the use of CouchDB, which uses a JSON based document format. CouchDB bundles the server and data storage functionality in a single product, providing a REST-like HTTP interface for document insertion, updates, retrieval and deletion.
Therefore, you'll be able to interact with CouchDB directly from within the Electron desktop application. Apache CouchDB Nano is the official Node.js library for accessing CouchDB.
The following additional factors speach for CouchDB:
It is open source.
It has comprehensive documentation.
It is available for Linux, macOS and Windows.
It's easily installed and quickly set up.
It can be installed on your local computer (for development), on your own servers, or in the cloud.
It supports Mango querying language (inspired from MongoDB).
It is highly scalable.
It is shipped with Fauxton web interface that lets you create, update, delete, view and query documents on the fly.
etc.
I've produced a Proof of concept to automate the export of data out of an external database and save it as a file to then import into ArangoDB using arangoimp.exe. While this method is certainly functional it is unfortunately not going to work due to my company not being able to utilize the file system in the external database or of the local file system that ArangoDB has access to.
The Question: Is there an alternative method or mechanism to import data from within ArangoDB?
Is there any inherent tool I can make use of such as:
User Defined Functions (UDFs)
Foxx
I've read about both of these features in ArangoDB however I'm curious if either of these two features can do what I need.
I need to be able to automate from within ArangoDB a procedure/action that connects to an External Database and can then run SQL Queries OR run Stored Procedures in that external database and store that data directly into ArangoDB Collections.
Alternative: Do I need to code/develop my own program?
Many years ago I created a Win Forms app that can connect to several databases. Basically it was my first attempt at learning connection strings and sql injection. That project didn't end up going beyond just that but I've had thoughts in the back of my mind that are telling me I might have to develop an intermediary application to facilitate the data transfer I'm attempting to make happen.
My fear in the latter is that this just opens up a brand new project that needs to be maintained and developed internally which means resources will need to be devoted to it.
arangoimp has --server.endpoint parameter, which would allow one to import the data on a remote machine to an arangodb server. The two machines would just have to be on the same network.
I'm learning more about ArangoDB and it's Foxx framework. But it's not clear to me what I gain by using that framework over building my own stand alone nodejs app for API/access control, logic, etc.
What does Foxx offer that a regular nodejs app wouldn't?
Full disclosure: I'm an ArangoDB core maintainer and part of the Foxx team.
I would recommend taking a look at the webinar I gave last year for a detailed overview of the differences between Foxx and Node and the advantages of using Foxx when you are using ArangoDB. I'll try to give a quick summary here.
If you apply ideas like the Single Responsibility Principle to your architecture, your server-side code has two responsibilities:
Backend: persist and query data using the backend data storage (i.e. ArangoDB or other databases).
Frontend: transform the query results into a format acceptable for the client (e.g. HTML, JSON, XML, CSV, etc).
In most conventional applications, these two responsibilities are fulfilled by the same monolithic application code base running in the same process.
However the task of interacting with the data storage usually requires writing a lot of code that is specific to the database technology. You need to write queries (e.g. using SQL, AQL, ReQL or any other technology-specific language) or use database-specific drivers.
Additionally in many non-trivial applications you need to interact with things like stored procedures which are also part of the "backend code" but live in the database. So in addition to having the application server do two different tasks (storage and rendering), half the code for one of the tasks ends up living somewhere else, often using an entirely different language.
Foxx lets you solve this problem by allowing you to move the logic we identified as the "backend" of your server-side code into ArangoDB. Not only can you hide all the nitty gritty of query languages, edges and collections behind a more application-specific API, you also eliminate the network overhead often necessary to handle requests that would cause more than a single roundtrip to the database.
For trivial applications this may mean that you can eliminate the Node server completely and access your Foxx API directly from the client. For more complicated scenarios you may want to use Node to build external micro services your Foxx service can tap into (e.g. to interface with external non-HTTP APIs). Or you just put your conventional Node app in front of ArangoDB and use Foxx to create an HTTP API that better represents your application's problem domain than the database's raw HTTP API.
It's also worth keeping in mind that structurally Foxx services aren't entirely dissimilar from Node applications. You can use NPM dependencies and split your code up into modules and it can all live in version control and be deployed from zip bundles. If you're not convinced I'd suggest giving it a try by implementing a few of your most frequent queries as Foxx endpoints and then deciding whether you want to move more of your logic over or not.
I'm hoping someone can validate or correct my conclusions here.
I'm looking into writing a small side project. I want to create a desktop application for taking notes that will synchronise to a web-server so that multiple installations can be kept in step and data shared and also so that it can be accessed via a browser if necessary.
I've kind of been half-listening to the noises about CouchDB and I've heard mention of "offline functionality", of desktop-couchdb and of moves to utilise its ability to handle intermittent communications to enable distributed applications in the mobile market. This all led me to believe that it might be an interesting option to look at for providing my data storage and also handling my synchronisation needs, but after spending some time looking around for info on how to get started my conclusion is that I've got completely the wrong end of the stick and the reality is that:
There's no way of packaging up a CouchDB instance, distributing it as part of a desktop application and running it in the context of that application to provide local storage and synchronisation to a central database.
Am I correct here? If so is there any technology out there that does this sort of thing or am I left just rolling my own local storage and maybe still using CouchDB on the server?
Update (2012/05): check out the new TouchDB projects from Couchbase if you are targeting Mac OS X and/or iOS or Android. These actually use SQLite under the hood (at least for now) but can replicate to/from a "real" CouchDB server. Another clientside alternative that is finally starting to mature is PouchDB, which runs in IndexedDB capable browser engines. Using these or using them to inspire similar port to another desktop platform is now becoming a better-trod path.
Original answer:
There's no way of packaging up a
CouchDB instance, distributing it as
part of a desktop application and
running it in the context of that
application to provide local storage
and synchronisation to a central
database.
At this point in time, your statement is practically correct although it is possible to include CouchDB in an app — for an example see CouchDBX.app which is a thin wrapper around a prefixed bundle of CouchDB and all its dependencies.
The easiest way to build a CouchDB app is to assume that the user will already have a CouchDB server running. This is easier than it sounds, especially with Couchone's hosting or a prebuilt app like CouchDBX on OS X or DesktopCouch on Ubuntu. This latter is especially interesting, because if I understand correctly it is included by default with Ubuntu these days, and automatically spins up a CouchDB server per-user when you query its port via D-Bus. Something similar could (and should) be done on OS X using launchd and Bonjour.
So as you write, you either would design your app to store data in a local format and optionally sync with a CouchDB service you provide or you'd have to build and bundle all of Erlang, SpiderMonkey and CouchDB together with your app along with some scripts to make sure it was running when needed. This is possible but obviously neither of these are ideal, and believe me you're not the only one wanting a simpler solution for desktop-oriented apps!