My code runs oracle select query on a server. The server may be free or it may be busy at certain times.Now what I am making is a timeout mechanism in Pro*C that times out after 10 seconds and cancels the thread that was running oracle query (if it is still running).
Now, the problem which I'm facing is that the timeout function is scheduled to terminate the threads after 10 seconds, and it does so very well. However it is not able to make out whether the oracle query was still in waiting state at the time of termination or it was returning results/executing procedure/blah-blah...
What I want is a function/mechanism/anything that would be able to query the status of the oracle query initiated and act as following: after 10 seconds,
IF (query is still in waiting state)
DO terminate all threads;
ELSE IF (query is fetching data/doing some processing)
DO wait for the fetching/processing to complete and then terminate all threads;
A Pro*C or Oracle function call would be the best option instead of using complex code, if possible.
What does "fetching data/ doing some processing" mean to you? In general, since Oracle executes the query as the application fetches data, a query will alternate between actively running (which would involve being in a wait state) and returning data to the client. If you are sending 50 rows at a time to the client, for example, Oracle only executes the query far enough to identify the first 50 rows, it sends those rows to the client, and it waits for the client to request the next 50 rows before continuing to execute the query to pull the next 50 rows.
Related
Probably a newbie question:
(I know that there is gettimeremaining property in context).
I have a job that is querying an API (say Marketo), however I need to plan for the scenario where the API does not return the required data in 15 mins.
I can check the remaining time in ms from context, however how do i accomplish this along with my data pull tasks as 2 concurrent tasks?
Meaning, do I fire off something first that says keep check of time, and then i proceed to my actual work and when time is 14 mins the first thing/thread returns back and I can proceed accodingly?
Thanks
Since you are calling an external API you will be using a client to make a connection; set connection timeout to be less than maximum time you can wait. In-case API doesn't return response in time your connection with throw timeout exception which than you can handle gracefully.
https://docs.python-requests.org/en/latest/user/advanced/#timeouts
Blocking execute fethod from com.datastax.driver.core.Session
public ResultSet execute(Statement statement);
Comment on this method:
This method blocks until at least some result has been received from
the database. However, for SELECT queries, it does not guarantee that
the result has been received in full. But it does guarantee that some
response has been received from the database, and in particular
guarantee that if the request is invalid, an exception will be thrown
by this method.
Non-blocking execute fethod from com.datastax.driver.core.Session
public ResultSetFuture executeAsync(Statement statement);
This method does not block. It returns as soon as the query has been
passed to the underlying network stack. In particular, returning from
this method does not guarantee that the query is valid or has even
been submitted to a live node. Any exception pertaining to the failure
of the query will be thrown when accessing the {#link
ResultSetFuture}.
I have 02 questions about them, thus it would be great if you can help me to understand them.
Let's say I have 1 million of records and I want all of them to be arrived in the database (without any lost).
Question 1: If I have n number of threads, all threads will have the same amount of records they need to send to the database. All of them continue sending multiple insert queries to cassandra using blocking execute call. If I increase the value of n, will it also helps to speed up the time that I need to insert all records to cassandra?
Will this cause performance problem for cassandra? Does Cassandra have to make sure that for every single insert record, all the nodes in the clusters should know about the new record immediately? In order to maintain the consistency in data. (I assume cassandra node won't even think about using the local machine time for controlling the record insertion time).
Question 2: With non-blocking execute, how can I assure that all of the insertions is successful? The only way I know is waiting for the ResultSetFuture to check the execution of the insert query. Is there any better way I can do ? Is there a higher chance that non-blocking execute is easier to fail then blocking execute?
Thank you very much for your helps.
If I have n number of threads, all threads will have the same amount of records they need to send to the database. All of them continue sending multiple insert queries to cassandra using blocking execute call. If I increase the value of n, will it also helps to speed up the time that I need to insert all records to cassandra?
To some extent. Lets divorce the client implementation details a bit and look at things from the perspective of "Number of concurrent requests", as you don't need to have a thread for each ongoing request if you use executeAsync. In my testing I have found that while there is a lot of value in having a high number of concurrent requests, there is a threshold for which there are diminishing returns or performance starts to degrade. My general rule of thumb is (number of Nodes *native_transport_max_threads (default: 128)* 2), but you may find more optimal results with more or less.
The idea here is that there is not much value in enqueuing more requests than cassandra will handle at a time. While reducing the number of inflight requests, you limit unnecessary congestion on the connections between your driver client and cassandra.
Question 2: With non-blocking execute, how can I assure that all of the insertions is successful? The only way I know is waiting for the ResultSetFuture to check the execution of the insert query. Is there any better way I can do ? Is there a higher chance that non-blocking execute is easier to fail then blocking execute?
Waiting on the ResultSetFuture via get is one route, but if you are developing a fully async application, you want to avoid blocking as much as possible. Using guava, your two best weapons are Futures.addCallback and Futures.transform.
Futures.addCallback allows you to register a FutureCallback that gets executed when the driver has received the response. onSuccess gets executed in the success case, onFailure otherwise.
Futures.transform allows you to effectively map the returned ResultSetFuture into something else. For example if you only want the value of 1 column you could use it to transform ListenableFuture<ResultSet> to a ListenableFuture<String> without having to block in your code on the ResultSetFuture and then getting the String value.
In the context of writing a dataloader program, you could do something like the following:
To keep things simple use a Semaphore or some other construct with a fixed number of permits (that will be your maximum number of inflight requests). Whenever you go to submit a query using executeAsync, acquire a permit. You should really only need 1 thread (but may want to introduce a pool of # cpu cores size that does this) that acquires the permits from the Semaphore and executes queries. It will just block on acquire until there is an available permit.
Use Futures.addCallback for the future returned from executeAsync. The callback should call Sempahore.release() in both onSuccess and onFailure cases. By releasing a permit, this should allow your thread in step 1 to continue and submit the next request.
To further improve throughput, you might want to consider using BatchStatement and submitting requests in batches. This is a good option if you keep your batches small (50-250 is a good number) and if your inserts in a batch all share the same partition key.
Besides the above answer,
Looks like execute() calls executeAsync(statement).getUninterruptibly(), so whether you manage your own "n thread pool" using execute() and block yourself until execution completes up to a max of n running threads OR using executeAsync() on all records, cassandra side performance should be roughly same, depending on execution time/count + timeouts.
They executions will all run connections borrowed from a pool, each execution has a streamId on client side and gets notified you via future when the response comes back for this streamId, limited by total requests per connection on client side and total requests limited by read threads on each node that was picked to execute your request, any higher number will be buffered in a queue (not blocked) limited by the connection maxQueueSize and maxRequestsPerConnection, any higher than this should fail. The beauty of this is that executeAsync() does not run on a new thread per request/execution.
So, there has to be a limit on how many requests can run via execute() or executeAsync(), in execute() you are avoiding beyond these limits.
Performance wise, you will start seeing a penalty beyond what each node can handle so execute() with a good size pool makes sense to me. Even better, use a reactive architecture to avoid creating so many threads that are doing nothing but waiting, so large number of threads will cause wasted context switching on client side. For smaller number of requests, executeAsync() will be better by avoiding thread pools.
DefaultResultSetFuture future = new DefaultResultSetFuture(..., makeRequestMessage(statement, null));
new RequestHandler(this, future, statement).sendRequest();
I am trying to debug an issue with the `node-pg-cursor' module in node.js against a postgresql server (version 9.3)
This module allows for sequential reads of N rows in a select and works by sending
cur.read(N): 'Execute' on portal=unnamed, rows=N
this command fetches up to N rows and we can continue fetching rows incrementally until the end, where we receive
CommandComplete
ReadyForQuery
Now my problem is that I want to bail out of the extended command before fetching all the rows and reaching the end of the Execute sequence: I would like to incrementally fetch N rows, N rows, N rows,.. and at one point decide that I have enough.
When I do that (stop fetching via Execute), the query seem to never reach CommandComplete or ReadyForQuery. This seems normal since nothing tells the extended query that I am never going to ask rows from it again.
Apart from closing the connection, is there a command to reach CommandComplete, or ReadyForQuery while not fetching all the rows from the portal ?
I tried to send Close and received CloseComplete, but it did not go to ReadyForQuery.
If I force an ErrorResponse by sending garbage on the protocol, I reach ReadyForQuery but that does not seem very clean ...
I think you're referring to this, in the documentation:
If Execute terminates before completing the execution of a portal (due to reaching a nonzero result-row count), it will send a PortalSuspended message; the appearance of this message tells the frontend that another Execute should be issued against the same portal to complete the operation. The CommandComplete message indicating completion of the source SQL command is not sent until the portal's execution is completed. Therefore, an Execute phase is always terminated by the appearance of exactly one of these messages: CommandComplete, EmptyQueryResponse (if the portal was created from an empty query string), ErrorResponse, or PortalSuspended.
Presumably, you're getting PortalSuspended and you want to discard the portal without executing any more of it or consuming any more results.
If so, I think you can just send a Sync message:
At completion of each series of extended-query messages, the frontend should issue a Sync message. This parameterless message causes the backend to close the current transaction if it's not inside a BEGIN/COMMIT transaction block ("close" meaning to commit if no error, or roll back if error). Then a ReadyForQuery response is issued.
You may wish to issue a Close against the portal first:
The Close message closes an existing prepared statement or portal and releases resources.
so what I think you need to do is, in message flow terms:
Parse
Bind a named portal
Describe
Loop:
Execute with rowcount limit to fetch some rows
If no more rows needed; then
Close the portal
Break out of the loop
If CommandComplete received:
Break out of the loop
Sync
Wait for ReadyForQuery
It sounds like you might want to be using the asynchronous query processing API, if your driver is a libpq wrapper. If it's a native implementation the source code for libpq may offer you clues.
Overall, it looks like you'll need to cancel the query using a new connection, then continue to consume input until the buffer is empty. You'll receive however much result data was buffered, then an error message indicating the query was cancelled (if it didn't buffer all its output before you cancelled it) and finally a ReadyForQuery.
I quote the libpq manual:
A client that uses PQsendQuery/PQgetResult can also attempt to cancel a command that is still being processed by the server; see Section 31.6. But regardless of the return value of PQcancel, the application must continue with the normal result-reading sequence using PQgetResult. A successful cancellation will simply cause the command to terminate sooner than it would have otherwise.
Systems usually have quite big TCP send buffers, and they're typically dynamic. See Linux's tcp(7), the SO_SNDBUF option to setsockopt(2), etc. So quite a lot of data might be buffered before the PostgreSQL server blocks on writing to the socket. PostgreSQL doesn't offer per-connection control of the send buffer size, or even a global config option; you must do it on the operating system level. (That said, it'd be trivial to patch PostgreSQL to set a send buffer size with setsockopt and SO_SENDBUF if you wanted to).
PostgreSQL can't just flush the output buffer when you cancel a query. Even if it were safe to do so and the platform supported it, Pg doesn't know for sure that the buffer has emptied of results from prior queries and other relevant messages, since you might have piplined multiple queries.
So all you can really do is reduce the maximum size of the TCP output buffer. That'll reduce the amount of data you must read and throw away, but it may impact the performance of other queries that send bulk data.
Instead of trying to run the query and cancelling it when you've seen enough, I suggest reading rows in batches, requesting a new batch when you've consumed the current one. You can do this by using protocol-level cursors. That way you can control how much data the server queues up and you don't have to mess with buffer sizes. You may already be doing this - using a named portal, and sending an Execute with a maximum row-count, waiting for the PortalSuspended to say there are more rows to read.
I've come up with a fancy issue of synchronization in node.js, which I've not able to find an elegant solution:
I setup a express/node.js web app for retrieving statistics data from a one row database table.
If the table is empty, populate it by a long calculation task
If the record in table is older than 15 minutes from now, update it by a long calculation task
Otherwise, respond with a web page showing the record in DB.
The problem is,
when multiple users issue requests simultaneously, in case the record is old, the long calculation task would be executed once per request, instead of just once.
Is there any elegant way that only one request triggers the calculation task, and all others wait for the updated DB record?
Yes, it is called locks.
Put an additional column in your table say lock which will be of timestamp type. Once a process starts working with that record put a now+timeout time into it (by the rule of thumb I choose timeout to be 2x the average time of processing). When the process stops processing update that column with NULL value.
At the begining of processing check that column. If the value > now condition is satisfied then return some status code to client (don't force client to wait, it's a bad user experience, he doesn't know what's going on unless processing time is really short) like 409 Conflict. Otherwise start processing (also ideally processing takes place in a separate thread/process so that user won't have to wait: respond with an appropriate status code like 202 Accepted).
This now+timeout value is needed in case your processing process crashes (so we avoid deadlocks). Also remember that you have to "check and set" this lock column in transaction because of race conditions (might be quite difficult if you are working with MongoDB-like databases).
I have 2 processes that connect to the same DB.
The first one is used to read from the DB and the second is used to write to the DB.
The first process sends write procedures for executing to the second process via message-queue on linux.
Every SQL-statement is taken in the prepare, step, finalize routine; Where the prepare and step are made in loop of 10000 times till it succedd (did this to overcome DB locked issues).
To add a table i do the next procedure:
the first process sends request via msg-q to the second process to add a table and insert garbage in it's rows in a journal_mode=OFF mode.
then the first process checks for the existing table so it could continue in its algorithm. (It checks it in a loop with usleep command between iterations.)
The problem is that the second process is stuck in the step execute of 'PRAGMA journal_mode=OFF;' because it says the DB is locked (Here too, i use a loop of 10000 iterations with usleep to check 10000 time for the DB to be free, as i mentioned before).
When i add to the first process in the 'check for existing table' loop, the operation of closing the connection, the second process is ok. But now when i add tables and values sometime i get 'callback requested query abort' in the Step Statement.
Any help of whats happening here ?
Use WAL mode. It allows one writer and any number of readers without any problems. You don't need to check for the locked state and do retrys etc.
WAL limitation: The DB has to be on the local drive.
Performance: Large transactions (1000s of inserts or similar) are slower than classic rollback journal, but apart of that the speed is very similar, sometimes even better. Perceived performance (UI waiting for DB write to finish) improves dramatically.
WAL is a new technology, but already used in Firefox, Adroid/iOS phones etc. I did tests with 2 threads running at full speed - one writing and the other one reading - and did not encounter a single problem.
You may be able to simplify your app when adopting the WAL mode.