To start: I've tried Loopback. Loopback is nice but does not allow for relations across multiple REST data services, but rather makes a call to the initial data service and passes query parameters that ask it to perform the joined query.
Before I go reinventing the wheel and writing a massive wrapper around Loopback's loopback-rest-connector, I need to find out if there are any existing libraries or frameworks that already tackle this. My extensive Googling has turned up nothing so far.
In a true microservice environment, there is a service per database.
http://microservices.io/patterns/data/database-per-service.html
From this article:
Implementing queries that join data that is now in multiple databases
is challenging. There are various solutions:
Application-side joins - the application performs the join rather than
the database. For example, a service (or the API gateway) could
retrieve a customer and their orders by first retrieving the customer
from the customer service and then querying the order service to
return the customer’s most recent orders.
Command Query Responsibility Segregation (CQRS) - maintain one or more
materialized views that contain data from multiple services. The views
are kept by services that subscribe to events that each services
publishes when it updates its data. For example, the online store
could implement a query that finds customers in a particular region
and their recent orders by maintaining a view that joins customers and
orders. The view is updated by a service that subscribes to customer
and order events.
EXAMPLE:
I have 2 data microservices:
GET /pets - Returns an object like
{
"name":"ugly",
"type":"dog",
"owner":"chris"
}
and on a completely different microservice....
GET /owners/{OWNER_NAME} - Returns the owner info
{
"owner":"chris",
"address":"under a bridge",
"phone":"123-456-7890"
}
And I have an API-level microservice that is going to call these two data services. This is the microservice that I will be applying this at.
I'd like to be able to establish a model for Pet such that, when I query pet, upon a successful response from GET /pets, it will "join" with owners (send a GET /owners/{OWNERS_NAME} for all responses), and to the user, simply return a list of pets that includes their owner's data.
So GET /pets (maybe something like Pets.find()) would return
{
"name":"ugly",
"type":"dog",
"owner": "chris",
"address": "under a bridge",
"phone": "123-456-7890"
}
Applying any model/domain logic on your API-gateway is bad decision, and considered as bad practice. API Gateway should only do your systems's CAS (with relying onto Auth service which holds the logic), And convert incoming external requests into inner system requests (different headers/ requester payload data) and proxy formatted requests to services for any other work, recieves them, cares about encapsulating errors, and presents every response in proper external form.
Another point, if there is alot of joins between two models required for application core flow (validation/scoping etc) then perhaps you should reconsider to which Business Domain your models/services are bound. If it's same BD perhaps they should be together. Priciples of Domain-Driven-Design helped me to understand where real boundaries between micro-services are.
If you work with loopback (like we are and face same problem we faced - that loopback have no proper join implementation) you can have separate Report/Combined data service, which is only one who can access to all the service databases and does it only for READ purposes - i.e. queries. Provide it with separately set-up read-only wide access to the db - instead of having only one datasource being set up (single database) it should be able to read from all the databases which are in scope of this query-join db user.
Such service should able to generate proper joins with expected output schema from configuration json - like loopback models (thats what I did in same case). Once abstraction is done it's pretty simple to build/add any equery with any complex joins. It's clean, and it's easy to reason about. Also, it's DBA friendly. For me such approach worked well so far.
Related
There are many excellent resources about the Unit of Work pattern. My understanding is that it's main purpose is to provide a way to ensure that the effects of a piece of code will not persist if an error occurs. There are plenty of examples of this usage for databases in most languages.
There are very few resources I can find about using such patterns to query and use external APIs while maintaining some level of data integrity during an error. Generally repositories are about data persistence but a lot of API's do concern such things especially in a microservice architecture. Clean Architecture: Where to make API calls suggests that such a microservice architecture should abstract calls to other microservices using a repository, and there are many public APIs that can be thought of in a similar manner.
In my specific case, I am looking to plug in the Todoist API for Task items into my application which works with its own version of a Todo entity. I have successfully adapted my TodoRepository for the Todoist API and can see my tasks from Todoist displayed in my UI - I now face the issue that if a call fails then I could be adding, deleting or updating a Task in the Todoist API when an error occurs after the call, which is not ideal for data integrity reasons.
There seems to be some distinction between an API that can act as a repository and one that cannot. Seemingly, if the API is able to perform general CRUD on a similar entity in the modelled entity then it may be a good candidate for a repository adapter, but if it were something like retrieving the weather forecast, determining if a name is the same as some celebrity, working with the google maps API (if your application wasn't a map itself), etc, then these are handled differently.
Under the assumption that I have not yet confirmed that all API adapters/facades will be implemented in the Infrastructure layer of a project, what context does the interface that defines the API usage exist? If I want to query to see if a name is also a celebrity name, would I have an Application or Domain service interface that looks something like
public interface CelebrityService {
Celebrity LinkNameToCelebrity(string first_name, string last_name);
}
Where Celebrity is a Domain entity. This feels out of place if the Celebrity entity has been made only for this call.
Similarly for a weather API,
public interface WeatherService {
Weather GetWeatherForDay(datetime day);
}
I am quite new to Jhipster and have problem understanding some of its functionalities. Hence here is my question.
I have the following two microservices.
Microservice 1 (MS1) has the following data structures in Java:
Lead {
Customer customer;
Deal deal;
}
Customer{
Integer phoneNumber;
etc...
}
Deal{
Integer value;
etc...
}
Microservice 2 (MS2) is a JHipster generated database.
The DB only has the following SQL tables :
CUSTOMER
LEAD
When changes happen in Microservice 1, I send 2 separate PUT requests from MS1 to MS2.
first a request to update CUSTOMER through the /customer API in MS2
if update is OK, then send a request to update DEAL /deal API in MS2
For a successful update for Lead, PUT requests to Customer, Deal should all be OK. If updating one table fails, all should fail.
Hence, I would like to avoid sending 2 separate requests to avoid a case where CUSTOMER request is OK and DEAL request fails for whatever reason.
If possible, I would like to send one single transaction throught an API such as /lead that udpates the two table..
What is the best way I can achieve this without creating an extra table for LEAD?
e.g., a layer/service that I should generate using Jhipster.
If possible (but not necessary), I would like to avoid touching code that are frequently regenerated. (e.g., Customer, Deal)
Please kindly direct me to a documentation too if one already exist. They are quite hard to understand so I am not sure if any current one specifically addresses this problem. Thank you.
This is a common issue when directly exposing JPA entities from a CRUD REST API.
Your persistence model does not need to be your API model.
If 2 entities are related and should be updated within same transaction, it means that they should be updated with one atomic API request.
So, you could define a new resource with a DTO combining your 2 entities, exposed by a new API that you would code manually (so no need for an additional table).
As you are using microservices architecture, you could have similar situation also between MS1 and MS2 and here you could not use a transaction, you could then have to implement remediation.
I want to make a webservice and it looks like Loopback is good starting point.
To explain my question, I will describe situation
I have 2 MySQL Tables:
Users
Companies
Every User has it's Company. It's like master User for it's company.
I wish to create Products table for each company next way:
company1_Products,
company2_Products,
company3_Products
Each company have internal Users, like:
company1_Users,
company2_Users,
company3_Users
Internal users are logging in from corresponding subdomain, like
company1.myservice.com
company2.myservice.com
For the API, I want datasource to get Products from the corresponding table. So the question is, how to change datasource dynamically?
And how to handle Users? Storing in one table is not good, because internal company users could be in different companies...
Maybe there's better way to do such models?
Disclaimer: I am co-author and one of current maintainers of LoopBack.
how to change datasource dynamically?
The following StackOverflow answer describes a solution how to attach a single model (e.g. Product) to multiple datasources: https://stackoverflow.com/a/28327323/69868 This solution would work if you were creating one MySQL database per company instead of using company's name as the prefix of Product table name.
To achieve what you described, you can use model subclassing. For each company, define a new company-specific Product model inheriting from the shared Product model and changing the table name.
// common/models/company1-product.json
{
"name": "Company1_Product",
"base": "Product",
"mysql": {
"tableName": "company1_Products"
}
// etc.
}
You can even create these models on the fly using app.registry.createModel() and app.model() APIs, and then run dataSource.autoupdate to create SQL tables for the new model(s).
And how to handle Users? Storing in one table is not good, because internal company users could be in different companies...
I suppose you can use the same approach as you do for Products and as you described in your question.
Maybe there's better way to do such models?
The problem you are facing is calling multi-tenancy. I am afraid we haven't figured out an easy to use solution yet. There are many possible ways how to implement multi-tenancy.
For example, you can create one LoopBack application for each Company (tenant) and then create a top-level LoopBack or Express application to route incoming requests to appropriate tenant-specific LB app instance. See the following repository for a proof-of-concept implementation: https://github.com/strongloop/loopback-multitenant-poc
As it says in the documentation for the Microsoft Bot Framework, they have different types of data. One of them is the dialogData, privateConversationData, conversationData and userData.
By default, it seems the userData is/should be prepared to handle the persistency across nodes, however the dialogData should be used for temporary data.
As it says here: https://learn.microsoft.com/en-us/bot-framework/nodejs/bot-builder-nodejs-dialog-waterfall
If the bot is distributed across multiple compute nodes, each step of
the waterfall could be processed by a different node, therefore it's
important to store bot data in the appropriate data bag
So, basically, if I have two nodes, how/why should I used dialogData at all, as I cannot guarantee it will be kept across nodes? It seems that if you have more than one node, you should just use userData.
I've asked the docs team to remove the last portion of the sentence: "therefore it's important to store bot data in the appropriate data bag". It is misleading. The Bot Builder is restful and stateless. Each of the dialogData, privateConversationData, conversationData and userData are stored in the State Service: so any "compute node" will be able to retrieve the data from any of these objects.
Please note: the default Connector State Service is intended only for prototyping, and should not be used with production bots. Please use the Azure Extensions or implement a custom state client.
This blog post might also be helpful: Saving State data with BotBuilder-Azure in Node.js
Trying to sell a move to ServiceStack from traditional ASP.Net /SOAP web services with the management team.
I am struggling with a some RPC'ish issues. Requirement is that I support SOAP (even backhandedly) in the hope of selling my service consumers on REST.
Take for example a service called "ReplaceItem" which basically requires:
Close out item number
Replacement item number
Store Number
Bunch of other replacement item data
Should I create a ReplacementItem DTO? It seems to be if I have a number of these type of functions I am just going to have tons of DTOs instead of tons of RPC methods. Plus what is the "id" in this case and what REST method would I be using?
I get that REST/SS gives me basic CRUD functionality for domain level structures like Items/Customers/etc, but how do I handle non-CRUD methods in SS.
I am also having issues with multiple parameters making up the primary key for a certain service. Almost all Inventory tables are structured by Item Number AND Store Number. I'd rather not dump the creation of some composite string on the service client. How do I handle this?
Thanks.
ServiceStack promotes a SOA-like message-based design that is optimal and provides many natural benefits for remote services.
My initial thoughts would look something like
POST {CloseItemNumber} /item/1/close
POST {ItemNumber} /item/1?replace=true
POST {ItemNumber} /item/1
POST {ItemNumber} /item/1 i.e. same DTO/service different values.
Where ItemNumber and CloseItemNumber are separate Request DTOs and services.
Designing Service APIs
I prefer to structure my services around 'resources/nouns' and design my service APIs as actions that apply operations to them.
If the operation requires more information than storing the Resource DTO I would create a separate service with the additional metadata.
i.e. Here's how I would convert Amazons 'RPC' service to be more REST-ful:
https://ec2.amazonaws.com/?Action=AttachVolume
&VolumeId=vol-4d826724
&InstanceId=i-6058a509
&Device=/dev/sdh
&AUTHPARAMS
Into how I prefer to write it:
POST https://ec2.amazonaws.com/volumes/vol-4d826724/attach
FormData: InstanceId=i-6058a509&Device=/dev/sdh&AUTHPARAMS
Which would still use an explicit AttachVolume Request DTO.
Another example I use to showcase the different between WCF RPC and ServiceStack's coarse-grained message-based approach is in: https://gist.github.com/1386381
Difference between an RPC-chatty and message-based API:
This is a typical API that WCF encourages:
public interface IService
{
Customer GetCustomerById(int id);
Customer[] GetCustomerByIds(int[] id);
Customer GetCustomerByUserName(string userName);
Customer[] GetCustomerByUserNames(string[] userNames);
Customer GetCustomerByEmail(string email);
Customer[] GetCustomerByEmails(string[] emails);
}
This is an equivalent message-based API we encourage in ServiceStack:
public class Customers {
int[] Ids;
string[] UserNames;
string[] Emails;
}
public class CustomersResponse {
Customer[] Results;
}
Note: If you want your same services to support a both SOAP and a REST-based API, you will need to structure your services slightly differently to overcome SOAP's limitation of tunnelling all operations through HTTP POST.
Problem i still have when deciding to switch from 1 chatty RPC api to a REST api is that instead of having several functions easy to maintain, i find myself with either 2 solutions :
multiplying DTOs and services that makes internal code for the services being chatty and complex
or
putting into a single route (OnGet) all the code managing the service but this way i have to parse the different parameters to 'discover' which parameters have been requested (to simplify instead of having multiple simple functions with pre-defined parameters i now have only one function that has to determine which parameters are meaningful ... but that is VERY hard to maintain - code is more complex to me now).
In the proposed solution to solve GetCustomerById, GetCustomersByEmails etc. the point to me is that instead of having simple queries, we now have to dynamically construct the query based on the filled parameters - that can make the code tricky and hard to maintain - having to manage possible combinations of multiple parameters - some combinations not being possible too.
Feeling little bit sad about that as i REALLY don't like WCF at all.
Mixing WCF and REST seems summum of the complexity - the worst for me (complexity of defining a REST api + complexity of WCF).
Are my feelings shared or did i miss something ?