I'm currently building an application with actix-web and sqlx. What I've build as an architecture is very similar to this source.
This is basically a trait wrapping the db access, so far so good. But this assumes every single method will get a connection from the pool and execute it. There's no way to share a connection for a transactional workflow (e.g. SELECT FOR UPDATE ... process ... UPDATE).
With which achitecture or library could I achieve this?
Related
I'm trying to build a Rust web framework using Actix that needs to query HBase backend. We have chosen to use the Thrift code generator to generate the APIs with this file. However we are having some troubles figuring out how to pass the connection to our web-tier query functions. The official Actix way is to use an extractor that extracts the application state, in this case an HBase connection. More specifically, we are able to create objects of type THBaseServiceSyncClient, which is the application data we wish to pass around that keeps an open connection to HBase and allows us to do queries.
The official way is to clone this data, for each running thread. The first issue we encountered is that, this type does not implement the Clone trait. We were able to implement our own Clone function, only to realize that it also does not implement the DerefMut trait. This is slightly harder, and cannot be circumvented due to the function definitions in the API linked above. The usual way to go about doing this, is to wrap the object with a Mutex. We have experimented with that, and it performed very poorly. The contention was way too high, and we simply cannot have one global connection for all the threads to use.
We researched how other popular databases connections are handled in Rust. We realize that a thread_pool is usually used, where a pool of active connections is kept, and a manager keeps track of the alive/dead connections and spins up more if needed. We found this r2d2 crate, which claims to provide a generic connection pool for Rust. Unfortunately there is no thrift support, and we experimented by implementing our very simple pool manager similar to the mysql variant here. The result was very underwhelming. The throughput was not nearly what we need, and a lot of the time is wasted on the pool manager, according to some simple flamegraph profiling.
Is there some more obvious ways to achieve this goal that we're missing here? I'm wondering if anyone has experienced similar issues and can provide some inside as to what's the best way to go about doing this. Much appreciated.
I know that Corda is in the process of removing its web server module and in the documentation they suggest the use of other frameworks.
In one example ("spring-observable-stream") they use Spring Boot for the server-side APIs and use an RPC call to the actual running Corda node. That's fine and comparable with what I have to do.
In that example, the author import the specific Corda's RPC code, along with the code of the actual flow (and states) needed.
What I want to ask here is if it's possible to avoid that tangle and keep the web server APIs independent from the actual Corda/CordApp's code by using a general RPC library (any advice ?).
If, instead, I must import the Corda-specific code (there is a reason ?), I'd like to ask you:
What is the minimum necessary to do so from a Gradle perspective ?
Is it possible to implement some sort of plugin on the CordApp to reduce that tangle ?
To be honest, I'm interested in a more general way for interacting with the CordApp (e.g. from Python), but I know that due to the AMQP integration not yet ready, we have to stay on the JVM for now. So, feel free to answer just about what we need to do as of today from Kotlin (which I have to use for a short-term PoC)…
Thank you in advance!
Currently, your server has to depend on the Corda RPC library to interact with nodes via RPC. Corda doesn't yet expose an independent format for sending and receiving messages via RPC.
Your server also needs to depend on any CorDapps that contain flows that the server will start via RPC, or that contain types that will be returned via RPC. Otherwise, your server will not be able to start the flows or deserialise the returned types.
If you're using Gradle, here's what a minimal dependencies block might look like:
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"
cordaCompile "net.corda:corda-rpc:$corda_release_version"
compile "com.github.corda:cordapp-example:release-V1-SNAPSHOT"
}
Here, we're depending on the corda-rpc library, and also using JitPack to depend on the CorDapp where we define the flows and states that we want to start/return via RPC.
If you want, you can modularise the CorDapp so that all the classes you need to depend on for RPC are included in a separate module, and only depend on that module.
Here is my situation. I have an extensive REST based API that connects to a MongoDB database using Mongoose. The API is written as a standard "MEAN" stack application.
Currently, when a developer queries the API they're always connecting to the live production database. What I want to do is have an exact duplicate database as a "staging" database, where new data will be added first, vetted over a period of time, and then move to the live database. Then I want developers to be able to query either one simply by modifying their query.
I started looking into this with the Mongoose documentation, and it appears as though the models are tied to the DB connection, and if I want to have multiple connections I also have to have multiple models, one for each connection. This would be a nightmare of WET code and not the path I want to take.
What I want to do is not touch any of my code at all and simply have a switch that changes to the proper database for a given query. So my question is, how can I achieve this? Is it possible? The documentation seems to imply it is not.
Rather than trying to maintain connections two environments in the same code base have you considered setting up stage version of your application? Which database it connects to could be set through an environment variable or some other configuration option.
The developers would still then only have to make a change to query one or the other and you could migrate data from the stage database to production/live database once you have finished your vetting process.
I'm currently building a rest api and I'm struggling to find the best way to unit test each route.
A route handler performs various things and one of them is to execute a query to mongodb. I can unit test the route handler by using stubs, but if I'm testing the query I cannot stub the query itself, I need to have an in-memory mongodb that I could reset and insert new data for each test.
How do you test queries? I'm thinking that the only real way to ensure that the query does what I need is to use a real mongodb database installed in the testing machine (typically in the same machine used for developing).
yes, just as for relation databases, you need to have real base. if mongo offers in-memory auto-created version then it's easy. if not, then each developer has to have running mongo before he runs integration tests. for CI you can have one single dedicated mongo but than you have to prevent concurrent access (schema creation, multiple transactions etc). you should also implement automatic creation of schema if needed and empty database before each test. in relational db rollback is usually enough. when it's not enough then trimming all tables helps. although we had to implement it manually as we couldn't find any existing tools
I have seen this both ways. When writing a Repository, should the methoods be passed in a connection string or should the repositpry be "self-contained", in other words, know internally how to get to the database? In case it helps, my Repository is not true DDD, but is the Repository pattern surrounding methods that call Oracle SPs (that's the way it is ar work here)?
Repositories should normally not work in their own independent transactional unit, so they most often use the 'existing' database connection. This way you can do multiple repository (database!) operations in a single transaction.
How to implement this depends on your developing platform. Java EE for example has ways to inject the current Entity Manager into objects or ways to get it by code. You can also implement this manually by storing a reference in a thread local storage.