I am new to Arango DB. I am trying to achieve the following:
Use a single Arango DB instance with multiple services.
Keep the data separate for each service. I would like each service to handle its own data(query, collections, graphs, ...).
My questions are:
Is it possible to do this with ArangoDB java driver out of the box?
Would I need to create some kind of "middleware" service to do that?
Would there be a performance issue with this?
Related
I am new to Elasticsearch and also confused how do I actually start implementing it. I have developed an office management software where on a daily basis tasks and other information based to that task belonging to a specific clients are stored. I have written API's in nodejs and the front-end in vuejs and MySQL db is used. So I want to implement a search functionality using Elasticsearch wherein user can search the tasks with any parameters they would like to.
Listed below are some of my questions
Now do Elasticsearch will work as an another db. If so, then how do I keep the record updated in Elasticsearch db as well.
Would it effect the efficiency in any way.
Also what is kibana and logstash in simple terms.
Is implementing Elasticsearch on client side is a good idea? Is Yes, then how can I implement Elasticsearch and kibana using vuejs?
I am confused with all the above things, can anyone kindly share their knowledge on the above listed questions and also tell which articles/docs/videos should I refer for implementing Elasticsearch in the best possible way?
Elasticsearch
It is a data store, all the JSON data will(Single Record/Row) be stored in indexes(Tables)
Update the records in elasticsearch using your backend only, even though we have packages available to connect the frontend to Elasticsearch.
Efficiency, nothing gets affected except the new stack in your application.
Implementing elasticsearch on the client-side is not a recommended option, Same code same API can be used till your MySQL DB connection, add a function to save update any data along with MySQL save call.
Example : MySQLConfig.SaveStudent(student)
ElasticsearchConfig.SaveStudent(student)
Till here there is no code change needed to save/update/delete/getByPrimaryID/GetByParamSearch,
For `getByPrimaryID/GetByParamSearch` search, you have to create a different API either to elasticsearch to MySQL but not both.
Kibana
GUI for your Elasticsearch - Look at it like dbForge Studio, MySQL Workbench, phpMyAdmin
Other than GUI it has a lot of other functionalities like cluster monitoring, all the elastic stack monitoring, analytics, and so on.
Logstash
It ships many files and save it into elasticsearch index, this is not needed until u need it for use cases like
application-prod.log to searchable index
Kafka topic to searchable index
MySQL Table to searchable index
Huge list of use-cases available to ship anything and make it a searchable index
To understand clearly about index, mappings, document in elasticsearch vs database, table, scheme, record in MySQL read from here
I've produced a Proof of concept to automate the export of data out of an external database and save it as a file to then import into ArangoDB using arangoimp.exe. While this method is certainly functional it is unfortunately not going to work due to my company not being able to utilize the file system in the external database or of the local file system that ArangoDB has access to.
The Question: Is there an alternative method or mechanism to import data from within ArangoDB?
Is there any inherent tool I can make use of such as:
User Defined Functions (UDFs)
Foxx
I've read about both of these features in ArangoDB however I'm curious if either of these two features can do what I need.
I need to be able to automate from within ArangoDB a procedure/action that connects to an External Database and can then run SQL Queries OR run Stored Procedures in that external database and store that data directly into ArangoDB Collections.
Alternative: Do I need to code/develop my own program?
Many years ago I created a Win Forms app that can connect to several databases. Basically it was my first attempt at learning connection strings and sql injection. That project didn't end up going beyond just that but I've had thoughts in the back of my mind that are telling me I might have to develop an intermediary application to facilitate the data transfer I'm attempting to make happen.
My fear in the latter is that this just opens up a brand new project that needs to be maintained and developed internally which means resources will need to be devoted to it.
arangoimp has --server.endpoint parameter, which would allow one to import the data on a remote machine to an arangodb server. The two machines would just have to be on the same network.
I am trying to host a simple Node.js app connected to MongoDB, I look on the web, I've found 2 different methods and some guides to do so:
Different pods:
Example from the official Kubernetes documentation.
Example from the official Google Cloud Platform documentation.
Same pod:
Example from the official IBM documentation.
This confused me a bit, which is the best practise to achieve that?
You should put them in different pods.
Being in different pods you can then scale up the number of replicas of the web application part to as many instances as you need and you will still have the one instance of MongoDB.
If you had them in the same pod, you wouldn't be able to scale up to multiple instances as each replica would have its own MongoDB instance and thus separate data. If doing data updates to MongoDB from the web application, that would cause all sorts of problems as subsequent requests could hit a difference instance and so see different data.
I need some advice as to how a search needs to be implemented to search keywords within relational databases owned by microservices.
I have some microservices with their own relational DB. These microservices are likely to be deployed in a docker container.
What would be the best way to use a search engine like Apache SOLR so that each of the microservices' database can be indexed and we can achieve keyword search
Thanks in advance
This seems to be an architectural question and while it makes sense, the question is also a little open ended depending on what your system requires. A couple of things that come off the top of my head is:
Use the DataImportHandler from Apache SOLR.
Use a message queue like Kafka or Kinesis and have the independent services consume from it to propagate to their data stores in this case, a search service backed by Apache SOLR and another service backed by MySQL.
Personally, I've never used the DataImportHandler myself but my initial thoughts are that it couples Apache SOLR to MySQL. Setting up the DataImportHandler requires Apache SOLR to know the MySQL schema, the access credentials, etc. Because of this, I would advise the second option which moves towards a shared-nothing architecture.
I'm going to call the service that is backed by MySQL the "entity" service as it sounds like its going to be the canonical service for saving some particular type of object. The entity service and the search service will have its own particular consumer that ingests events from Kinesis or Kafka into their data stores, the search service to Apache SOLR and the entity service to MySQL.
This helps decouple the services from knowing that each other exists and also allow each of them to scale independently from each other. It'll be a redundancy in data but it should be alright because the data access patterns are different.
The other caveat that I'd like to mention is that it assumes that the entity for which you're saving is allowed to be asynchronous. Notice that messages in this system doesn't require it to be persisted in MySQL at the moment which in this case is the entity service. However, you may change it to your liking such that a message persists in the entity service and then is propagated through a queue to a search service to index. After its been index, you can just add additional endpoints to search Apache SOLR. Hope this gives you some insight on how some other architectures may come into play. If you give a little more insight as to your system and the entities that involved, you might be able to get a better answer.
Good day! I have created an application using nodejs + mongoose and now I want to make something like a superuser application. I need my admin panel application to connect to the same database. So, i have a question.
Should i store the same Schema file in both applications to have an ability to use my Schema methods? In other words, what is the best way to create one more API using the same db?
Thank you!
If I'm not mistaken, why not create another service that only interacts with the database? That way, the systems will refer to the same schema/DB regardless of which application you want to connect to it. So the superuser application and the normal application will just query the DB microservice that interacts the database.
Pro: source of truth for the schema for all applications and the DB queries will just be API calls
Con: additional overhead in creating your ecosystem
If you are using the same DB from two different applications, you will want to make sure those schemas are the same between the two. If one changes its inputs, the other might need to change its display (or risk not expecting all that information). Keep all this in mind during your release process.
I would suggest making the schemas an external library to both, or have the admin panel require the current app. You'll avoid getting two sets out of sync and know to look at one place for the schema definitions.