My NodeJS project is based on SailsJS, itself using ExpressJS.
Its API will be used by mobile apps to fetch their data from it.
The tricky part is I don't want the client apps to fetch the whole data tree every time there is a change in the database.
The client only needs to download a differential between the data it's already got and the data on the server.
To achieve that I thought of using git on the server. That is create a repository and save all endpoints as a json file in the repo. Each save will trigger an automatic commit.
Then I could create a specific API endpoint that will accept a commit sha as a parameter and return a diff between that and git HEAD.
This post by William Benton comforted me with this idea.
I'm now looking for any tips that could help me get this working based on the language and frameworks cited above :
I'd like to see a proof of concept of this in action but couldn't find one
I couldn't find an easy way to use git with NodeJS yet.
I'm not sure how to parse the returned diff on client apps developed with the IONIC framework, so AngularJS.
Note : The api will only be readable. All DB movement will be triggered by a custom web back-end used by few users.
I used the ideas in that post for an experimental configuration-management service. That code is in Erlang and I can't offer Node-specific suggestions, but I have some general advice.
Calling out to git itself wasn't a great option at the time from any of the languages I was interested in using. Using git as a generic versioned-object store actually works surprisingly well, but using git plumbing commands is a pain (and slow, due to all of the forking) and there were (again, at the time) limitations to all of the available native git libraries.
I wound up implementing my own persistent trie data structure and put a git-like library interface atop it. The nice thing about doing this is that your diffs can be sensitive to the data format you're storing; if you call out to git, you're stuck with finding a serialization format for your data that is amenable to standard diffs. (Without a diffable format, though, you can still send back a sequence of operations to the client to replay on whatever stale objects they have.)
Related
I'm looking for a solution that allows my team to collaborate in the development of a single Dialogflow agent, because the typical scenario is that 2 or more developers work on the same agent.
Usually for other kind of technologies we adopt git for source code versioning (also for webhooks in example) applying the proper branching strategy.
We tried to use git also for agents exports, but in this case it seems to be impossible because we are facing issues in merging intents ids (that seems to be generated with a not well known way).
Then in this scenario, we need to synchronize each other, for consistency checks, before exporting and committing the zip to git, losing the chance to leverage the power of git.
Searching with Google I got that we can solve the problem start using such framework like Narratory.
Said that, my questions are:
Do you know if is there any way ther to accomplish our need without using Narratoy?
Do you know if exist alternative to Narratory?
In BLoC Architecture we have Data Provider and Repository , In many examples I see that
Repository just called Data Provider and it is really cumbersome to create Repository , Why Repository exists? what is purpose
This is actually something that comes from Adopting Clean Architecture, where a repository is an interface which provides data which is from a source to the UI.
The sources are usually Remote & Data where Remote refers to fetching data from a remote source (this could be other apps, REST API, Websocket connect) and data which is from a local source (something akin to a database.) The idea behind having two separate classes for this is to provide adequate separation of concerns.
Imagine an App like Instagram, where the App manages both offline data and online data. It would make sense then to have the logic handled for each separately, and then use the repository which is what your viewmodel/bloc takes in to access the data. The bloc doesn't need to know where the source of data came from, it only needs the data. Repository Implementation doesn't need to know what is used to make an API call, it just needs to consume the fetched data. Similarly, the Repository Implementation doesn't need to know where the local data is fetched from, it just needs to consume it. This way every bit is abstracted adequately and changes in one class, doesn't affect other classes because everything is an Interface.
All of this helps in testing the code better since mocking and stubbing becomes easier.
I am kind of confused about when an API is needed. I have recently created a mobile app with flutter and cloud firestore as the database where i simply queried and wrote to the database when needed. Now i am learning full stack web development and I recently watched a tutorial where he built like an Express API with GET, POST, and DELETE functionality for a simple item in the database.
Coming from a background where i just directly accessed the database i am not sure why an API in this case is necessary, is it so I wouldnt have to rewrite the queries every time? This is a very simple project so he's definitely not making a 3rd party api for other developers to use. Am i misunderstanding what an API does exactly?
It was really simple, there was one collection in a MongoDB database and he was using postman to read and write to and from the database to check if it works.
API is a standard way with which your front-end (web/mobile) stores/gets information for your application. Your front-end can/should not directly access database ever. Understand the purpose of front-end which is to just display the interface and should do minimal processing. All the application logic should be at your backend (API server) which is exposed to your frontend via API (GET, POST etc) calls. So to store an item in your database, you will write data storing logic in your backend, and expose an API end-point which when triggered will perform the storing operation. That API call should be used by your front-end to trigger the storing process. In this way your logic of storing/database or any other thing is not exposed, only the API URL is. The purpose of front-end is to be exposed whereas backend/database should never be exposed and used from front-end
May be for you, an API is not necessary. But, the use-cases of an API is a lot.
For example:
You don't have to write business logic for every platform. (iOS, Android, Web, Whatever)
Your app will be lightweight since some computation would be offloaded to server.
Your app can be reverse engineered to get secret informations. (or, Your secret algorithm may be?)
What if you need to store something in filesystem that you want share with others?
Also a good read: Why we should use REST?
In your case, you are using a pre-written SDK which knows how to connect to Firestore, does caching and updates application data when needed, and provides a standard method of reading, writing and deleting data in Firestore (with associated documentation and example data from google).
Therefore, using an API (as described for the mongoDB) is not required and is undesirable.
There are some cases where you might want to have no read or write access to a firestore collection or document, and in this case, you could write a cloud function which your app calls with parameters, that receives the data that you want to write and does some sort of checking or manipulation beyond the capabilities of cloud firestore rules (although these can get pretty sophisticated). See https://firebase.google.com/docs/firestore/security/get-started
Todd (in the video contained in this link) does a few good videos on this subject.
However, this is not really working in the same was as the API you mentioned in your question.
So in the case of using Firestore, you should use the SDK and not re-invent the wheel by creating your own API.
If you want to share photos for example, you can also store them in firebase storage and then provide a URL for other devices to access them without your app being installed.
If you want to write something to firestore which is then sent to all other users then you can use listeners on each app, and the data will be sent to the apps after it arrives at Firestore.
https://firebase.google.com/docs/firestore/query-data/listen gives an overview of this.
One thing to always look at with firebase is the cost of doing anything. Cloud functions cost more than doing a read of a firestore document.
This gives an overview of pricing for different capabilities within the firebase set of capabilities.
https://firebase.google.com/pricing
Another most important factor is coupling. To add to #Dijkstra API provides a way to decouple the logic from each other, thus allowing for more application reliability, maintainability, fault-tolerance and if required scalability.
Thus there is no right or wrong here, or the comparison of API vs DB call is in itself not justified for the fact that fetching the data from Database is the ultimate aim. Even if you use a REST API or Query a database.
The means to achieve the same can differ based on specific requirements. For example, fetching water from the well.
You can always climb down the well and fetch a bucket of water if you need 1 bucket per day and you are the only user.
But if there are many users you would want to install a pull and wheel where people use it to pour fetched water into their bucket, yet again this will depend if there are 100 users per day using or more than that. As this will not work in the case of more than 100 users.
IF the case is that an entire community of say 1000 user are going to need the water you would go with a more complex solution of installing a motorized water pump to pump out the water and supply it to the user's home via a pipeline. This solution has many benefits like fast supply, easy to use, filtered water, scheduled, etc. But the cost and effort to achieve the solution is higher as well.
All in all, It comes down to the cost-vs-benefit ratio which you and only you can chart out, for different solutions vs the particular problem, as you are the best judge of scale and future user flow.
While doing that you can ask the following question about the solution to help decide :
Is the solution satisfying the primary requirement of the problem?
How much time is it going to take to build it?
For the time we spend to build a solution, is it going to working at more than 75% or more of its capacity?
If not is there a simpler solution that I can use to satisfy the problem and scale it as the requirement increases?
HTH.
I am developing an social app on iOS that have many-to-many relation, local persistency, and user interaction. I have tried using native Parse API in iOS and find it too cumbersome to do all the client-server logic. So my focus shifted to finding a syncing solution.
After some research I found AFIncrementalStore quite easy to use and it's highly integrated in CoreData. I just started to work on this and I have two questions to ask:
1) How to do the authentication process? Is it in AFRESTClient?
2) How to set up AFRESTClient to match Parse's REST API? (an example would be great!)
P.S. I also found FTASync, which seems to be another solution. Any thought on this framework?
Any general suggestion on client-server syncing solutions will be highly appreciated!
Thanks,
Lei Zhang
Back with iOS 5 Apple silently rolled out NSIncrementalStore to manage connection between APIs and persistent stores. Because I couldn't word it better myself:
NSIncrementalStore is an abstract subclass of NSPersistentStore designed to "create persistent stores which load and save data incrementally, allowing for the management of large and/or shared datasets". And while that may not sound like much, consider that nearly all of the database adapters we rely on load incrementally from large, shared data stores. What we have here is a goddamned miracle.
Source: http://nshipster.com/nsincrementalstore/
That being said, I've been working on my own NSIncrementalStore (built specifically for Parse and utilizing the Parse iOS/OS X SDK) and you're welcome to check out/use/contribute to the project at https://github.com/sbonami/PFIncrementalStore.
Take a look at this StackOverflow question and at Chris Wagner's article on raywenderlich.com.
The linked SO question has examples for how to include the authentication token with each request to Parse. So you'll just need to have the user log in first, and store their token to include it with each subsequent request.
Chris Wagner's tutorial has a sample AFHTTPClient named SDAFParseApiClient to communicate with the Parse REST API. You'd have to adapt it to be an AFRESTClient subclass, but it should give you a start.
Some other thoughts between the two solutions you're considering:
AFIncrementalStore does not allow the user to make any changes without a network connection, while FTASync keeps a full Core Data SQLite store locally and syncs changes to the server when you tell it to.
FTASync requires you to make all your synched managed objects subclasses of FTASyncParent, with extra properties for sync metadata. AFIncrementalStore keeps its metadata behind the scenes, not in your model.
FTASync appears not to be widely used and hasn't been updated in over a year; if you use it you will likely be maintaining it.
I want to start testing CouchDB and Node.js but having problems to figure out what tools to use.
Can I do everything I want with cradle and node.js? Or do I need node.couchapp.js? Also is this follow application re-implementation of something cradle does too?
Some birth pains still visible as CouchDB and Node.js are both pretty new, and tools and utilities are still evolving. Relying on some random github projects is not amusing, so trying to keep these dependencies low.
cradle is for talking to a couchdb, inserting and getting documents and so on. couchapp is for something else, you can use it to put some logic (views) or even entire webinferfaces with images and so on into the database. You'll probably need both.
CouchDB has a simple HTTP API, then you can use any HTTP client library to talk with CouchDB. Cradle is sometimes handy, sometimes not, depending on the use-case. The important thing to keep in mind is that, whatever library you choose as CouchDB client library, you can always fall back to using "pure HTTP", and it's often very easy to do: it's only JSON and HTTP.
For writing couchapps, the reasoning is the same: you can start as simple as writing a JSON with the design document and PUTting it in a db; you can use a minimalistic tool as node.couchapp.js; or you can use something more sophisticated as erica. However, under the hood, you are always doing simple HTTP calls to send and receive some JSON.