For multithreaded env in our application we implemented oracle skip locked where no two threads pick up the same record in the database for processing(We add 'waiting' to 'working' flag).
Now, we have a modification where if two records in the database which are queued for processing have the same ID(workid) should not picked up the same time.(i.e - the other record status should not be updated to WORKING if already one record has a flag was 'WORKING'
Can someone help as to how this can be achieved?
Below is the procedure for the same where single record is locked without comparison.
create or replace PROCEDURE DEQUEUE_MANAGER(
v_managerName IN Queue.manager%TYPE,
v_workid IN VARCHAR2,
v_key OUT NUMBER,
v_datatablekey OUT Queue.DATA_TABLE_KEY%TYPE,
v_tasktype OUT Queue.TASK_TYPE%TYPE,
v_no_of_attempts OUT Queue.ATTEMPT%TYPE,
v_result OUT NUMBER,
v_error_desc OUT VARCHAR2)
IS
v_recordrow Queue%ROWTYPE;
v_qeuuestatus VARCHAR2(255);
v_updatedstaus VARCHAR2(255);
CURSOR c IS
select *
from QUEUE
WHERE MANAGER =v_managerName
AND STATUS =v_qeuuestatus
AND workid=v_workid
AND DATE_RELEASE<=SYSDATE
FOR UPDATE SKIP LOCKED;
BEGIN
v_result := -1;
v_qeuuestatus :='WAITING';
v_updatedstaus:='WORKING';
v_tasktype :='';
v_datatablekey:=-1;
v_key:=-1;
v_error_desc:='No Data Found';
v_no_of_attempts:=0;
OPEN c;
FOR i IN 1..1 LOOP
FETCH c INTO v_recordrow;
EXIT WHEN c%NOTFOUND;
select v_recordrow.key into v_key
from QUEUE
where key = v_recordrow.key
for update;
UPDATE Queue
SET STATUS=v_updatedstaus
WHERE KEY=v_recordrow.key;
COMMIT;
v_datatablekey:=v_recordrow.data_table_key;
v_tasktype := v_recordrow.task_type;
v_no_of_attempts := v_recordrow.attempt;
v_result := 0;
IF (v_no_of_attempts IS NULL) THEN
v_no_of_attempts:=0;
END IF;
END LOOP;
CLOSE c;
EXCEPTION
WHEN NO_DATA_FOUND THEN
v_datatablekey:=-1;
v_tasktype:='';
v_key:=-1;
v_no_of_attempts:=0;
v_result :=-1;
v_error_desc:='No Rows Found';
WHEN OTHERS THEN
DBMS_OUTPUT.put_line ('Exception Occurred');
v_datatablekey:=0;
v_tasktype:='';
v_key:=0;
v_no_of_attempts:=0;
v_result := -2;
v_error_desc := SQLERRM;
ROLLBACK;
END;
The purpose of the FOR UPDATE syntax is to lock records we want to update at some point in the future. We want to get the records now so that we can be sure our subsequent update won't fail because another session has locked the records.
That's not what your code does. Instead it selects the record, updates it and then issues a commit. The commit ends the transaction, which releases the lock. Your process would work the same without the FOR UPDATE.
Now we have your additional requirement: if a queued record for a given workid is being processed no other record for the same workid can be processed by another session. You say that all instances of a workid have the same values of queue status or manager. So that means the initial SELECT FOR UPDATE grabs all the records you want to lock. The snag is the SKIP LOCK allows other sessions to update any other records for that workid (only the first record is actually locked because that's the only one you've updated). Not that it matters sd the commit releases those locks.
The simplest solution would be to remove the SKIP LOCKED and the COMMIT. That would keep all the related records locked up until your processing transaction commits. But this may create further problems elsewhere.
Concurrent programming is really hard to get right. It's an architectural problem; you can't solve it at the level of an individual program unit.
Related
[Question posted by a user on YugabyteDB Community Slack]
I am running YugabyteDB 2.12 single node and would like to know if it is possible to create a temporary table such that it is automatically dropped upon committing the transaction in which it was created.
In “vanilla” PostgreSQL it is possible to specify ON COMMIT DROP option when creating a temporary table. In the YugabyteDB documentation for CREATE TABLE no such option is mentioned, however, when I tried it from ysqlsh it did not complain about the syntax. Here is what I tried from within ysqlsh:
yugabyte=# begin;
BEGIN
yugabyte=# create temp table foo (x int) on commit drop;
CREATE TABLE
yugabyte=# insert into foo (x) values (1);
INSERT 0 1
yugabyte=# select * from foo;
x
---
1
(1 row)
yugabyte=# commit;
ERROR: Illegal state: Transaction for catalog table write operation 'pg_type' not found
The CREATE TABLE documentation for YugabyteDB mentions the following for temporary tables:
Temporary tables are only visible in the current client session or transaction in which they are created and are automatically dropped at the end of the session or transaction.
When I create a temporary table (without the ON COMMIT DROP option), indeed the table is automatically dropped at the end of the session, but it is not automatically dropped upon commit of the transaction. Is there any way that this can be accomplished (apart from manually dropping the table just before the transaction is committed)?
Your input is greatly appreciated.
Thank you
See these two GitHub issues:
#12221: The create table doc section doesn’t mention the ON COMMIT clause for a temp table
and
#7926 CREATE TEMP … ON COMMIT DROP writes data into catalog table outside the DDL transaction
You cannot (yet, through YB-2.13.0.1) use the ON COMMIT DROP feature. But why not use ON COMMIT DELETE ROWS and simply let the temp table remain in place until the session ends?
Saying this raises a question: how do you create the temp table in the first place? Your stated goal implies that you’d need to create it before every use. But why? You could, instead, have dedicated initialization code to create the ON COMMIT DELETE ROWS temp table that you call from the client for this purpose at (but only at) the start of a session.
If you don’t want to have this, then (back to a variant of your present thinking) you could just do this before every intended use the table:
drop table if exists t;
create temp table t(k int) on commit delete rows;
After all, how else (without dedicated initialization code) would you know whether or not the temp table exists yet?
If you prefer, you could use this logic instead:
do $body$
begin
if not
(
select exists
(
select 1 from information_schema.tables
where
table_type='LOCAL TEMPORARY' and
table_name='t'
)
)
then
create temp table t(k int) on commit delete rows;
end if;
end;
$body$;
I want to update two database tables using hibernate entityManager. Currently I am updating 2nd table after verifying that data has been updated in 1st table.
My question is how to Rollback 1st table if data is not updated in 2nd table.
This is how I am updating individual table.
try {
wapi = getWapiUserUserAuthFlagValues(subject, UserId);
wapi.setFlags((int) flags);
entityManager.getTransaction().begin();
entityManager.merge(wapi);
entityManager.flush();
entityManager.getTransaction().commit();
} catch (NoResultException nre) {
wapi = new Wapi();
wapi.setSubject(merchant);
wapi.setUserId(UserId);
wapi.setFlags((int) flags);
entityManager.getTransaction().rollback();
}
Note - I am calling separate methods to update each table data
Thanks
I got the solution. basically I was calling two methods to update 2 db table and these two methods I am calling from one method.
Ex - I am method p and q from method r.
Initially I was calling begin, merge, flush and commit entitymanager methods in both p and q.
Now I am calling begin and commit in r and merge and flush in p and q.
So now my tables are getting updated together and rollback is also simple.
Hope it will help someone. Because I wasted my time for this, probably it can save someone else.
Thanks
Initially our flow of cimmunicating with google Pub/Sub was so:
Application accepts message
Checks that it doesn't exist in idempotencyStore
3.1 If doesn't exist - put it into idempotency store (key is a value of unique header, value is a current timestamp)
3.2 If exist - just ignore this message
When processing is finished - send acknowledge
In the acknowledge successfull callback - remove this msg from metadatastore
The point 5 is wrong because theoretically we can get duplicated message even after message has processed. Moreover we found out that sometimes message might not be removed even although successful callback was invoked( Message is received from Google Pub/Sub subscription again and again after acknowledge[Heisenbug]) So we decided to update value after message is proccessed and replace timestamp with "FiNISHED" string
But sooner or later we will encounter that this table will be overcrowded. So we have to cleanup messages in the MetaDataStore. We can remove messages which are processed and they were processed more 1 day.
As was mentioned in the comments of https://stackoverflow.com/a/51845202/2674303 I can add additional column in the metadataStore table where I could mark if message is processed. It is not a problem at all. But how can I use this flag in the my cleaner? MetadataStore has only key and value
In the acknowledge successfull callback - remove this msg from metadatastore
I don't see a reason in this step at all.
Since you say that you store in the value a timestamp that means that you can analyze this table from time to time to remove definitely old entries.
In some my project we have a daily job in DB to archive a table for better main process performance. Right, just because we don't need old data any more. For this reason we definitely check some timestamp in the raw to determine if that should go into archive or not. I wouldn't remove data immediately after process just because there is a chance for redelivery from external system.
On the other hand for better performance I would add extra indexed column with timestamp type into that metadata table and would populate a value via trigger on each update or instert. Well, MetadataStore just insert an entry from the MetadataStoreSelector:
return this.metadataStore.putIfAbsent(key, value) == null;
So, you need an on_insert trigger to populate that date column. This way you will know in the end of day if you need to remove an entry or not.
Im create new trigger audit using store procedure cause want flexible column in the trigger audit
im using Oracle 12 C ..
CREATE OR REPLACE PROCEDURE DBADMIN.TEST3 (OUTPUT OUT SYS_REFCURSOR,
TABLE_NAME IN VARCHAR2)
IS
N NUMBER;
BEGIN
N := 0;
EXECUTE IMMEDIATE '
CREATE OR REPLACE TRIGGER DBADMIN.TA_EMPLOYEES3
AFTER INSERT OR DELETE OR UPDATE
ON DBADMIN.EMPLOYEES
FOR EACH ROW
DECLARE
SID VARCHAR2 (30);
BEGIN
SELECT SYS_CONTEXT ('' USERENV '', '' IP_ADDRESS '') INTO IP FROM DUAL;
SELECT SEQ#
INTO SID1
FROM v$session
WHERE audsid = (SELECT USERENV ('' SESSIONID '') FROM DUAL);
IF INSERTING
THEN
INSERT INTO DBADMIN.DBLOG_MONITORING_DETAIL2 (SID,
COLUMNS,
OLDVALUE,
NEWVALUE)
VALUES (SID1,
i.COLUMN_NAME,
'for row in (SELECT column_name from user_tab_columns where table_name=''EMPLOYEES'' loop
execute immediate '':old.row.column_name '';
end loop;
/
32 26 PLS-00103: Encountered the symbol "FOR"
i think im bad logic in my script .. can give me better logic or repair my script its better ?? .
In Oracle, you should really rarely use dynamic SQL to create database objects. People sometimes abuse that functionality which is - in my opinion - what you're trying to do. The fact that you can do it doesn't mean that you should do it, e.g. you can poke your eye with a pencil, but you shouldn't do that.
Rule of thumb with dynamic SQL:
forget about EXECUTE IMMEDIATE
Create a local variable (a large VARCHAR2 or CLOB, depending on what you're doing)
compose the statement (CREATE TRIGGER in your case) and
store it into that variable
display it on the screen using DBMS_OUTPUT.PUT_LINE
copy/paste it and run it as a standalone statement
if it succeeds, you've done a good job so you can now use it in EXECUTE IMMEDIATE
if it fails, you'll have to try a little bit harder, debug it, fix errors and repeat the cycle
As of the error you got: this:
for row in (SELECT column_name from user_tab_columns where table_name=''EMPLOYEES'' loop
execute immediate '':old.row.column_name '';
end loop;
is invalid as the INSERT INTO target; you can't mix SQL and PL/SQL that way. It won't work at all, regardless dynamic SQL you use. Besides, it is obvious that it is wrong (missing closing bracket, what exactly are you executing immediately? old_row.column_name? How would you execute a column name?
In my opinion (once again), you shouldn't do it that way. If you want to create a trigger, do it - but not dynamically.
I am working on Oracle 10gR2.
And here is my problem -
I have a procedure, lets call it *proc_parent* (inside a package) which is supposed to call another procedure, lets call it *user_creation*. I have to call *user_creation* inside a loop, which is reading some columns from a table - and these column values are passed as parameters to the *user_creation* procedure.
The code is like this:
FOR i IN (SELECT community_id,
password,
username
FROM customer
WHERE community_id IS NOT NULL
AND created_by = 'SRC_GLOB'
)
LOOP
user_creation (i.community_id,i.password,i.username);
END LOOP;
COMMIT;
user_Creation procedure is invoking a web service for some business logic, and then based on the response updates a table.
I need to find a way by which I can use multi-threading here, so that I can run multiple instances of this procedure to speed up things. I know I can use *DBMS_SCHEDULER* and probably *DBMS_ALERT* but I am not able to figure out, how to use them inside a loop.
Can someone guide me in the right direction?
Thanks,
Ankur
what you can do is submit lots of jobs in the same time. See Example 28-2 Creating a Set of Lightweight Jobs in a Single Transaction
This fills a pl/sql table with all jobs you want to submit in one tx, all at the same time. As soon as they are submitted (enabled) they will start running, as many as the system can handle, or as many as are allowed by a resource manager plan.
The overhead that the Lightweight jobs have is very ... minimal/light.
I would like to close this question. DBMS_SCHEDULER as well as DBMS_JOB (though DBMS_SCHEDULER is preferred) can be used inside the loop to submit and execute the job.
For instance, here's a sample code, using DBMS_JOB which can be invoked inside a loop:
...
FOR i IN (SELECT community_id,
password,
username
FROM customer
WHERE community_id IS NOT NULL
AND created_by = 'SRC_GLOB'
)
LOOP
DBMS_JOB.SUBMIT(JOB => jobnum,
WHAT => 'BEGIN user_creation (i.community_id,i.password,i.username); END;'
COMMIT;
END LOOP;
Using a commit after SUBMIT will kick off the job (and hence the procedure) in parallel.