How do I insert big JSON data into an Oracle database(insert 15000 rows with 1 query)? - cx-oracle

I have an API that gives me 15000 rows of data in json every 10 seconds. I use this tutorial and when I insert 5 rows or more inserting is OK, but when I insert 15000 row that get error PLS-00172: string literal too long.
My Code:
create table jt_test (
CUST_NUM int, SORT_ORDER int, CATEGORY varchar2(100)
);
DECLARE
myJSON CLOB := '[
{"CUST_NUM": 12345, "SORT_ORDER": 1, "CATEGORY": "ICE CREAM"},
{"CUST_NUM": 12345, "SORT_ORDER": 2, "CATEGORY": "ICE CREAM"},
{"CUST_NUM": 12345, "SORT_ORDER": 3, "CATEGORY": "ICE CREAM"}
]';
BEGIN
insert into jt_test
select * from json_table ( myjson, '$[*]'
columns (
CUST_NUM, SORT_ORDER, CATEGORY
)
);
END;
Notes: I before use MSSql this link and work perfect even for 20000 rows in one query.
I use Oracle19C and connect and insert to db with cx_Oracle module python
Edit:
I test this tutorial but not work.

Instead of creating a large literal string in PL/SQL (which isn't allowed) you can create that string in Python instead and simply pass it through. Something like this should do the trick:
long_json_string = '''[
{"CUST_NUM": 12345, "SORT_ORDER": 1, "CATEGORY": "ICE CREAM"},
{"CUST_NUM": 12345, "SORT_ORDER": 2, "CATEGORY": "ICE CREAM"},
{"CUST_NUM": 12345, "SORT_ORDER": 3, "CATEGORY": "ICE CREAM"}
]'''
sql = '''
declare
myJSON CLOB := :long_json_string;
begin
insert into jt_test
select * from json_table ( myjson, '$[*]'
columns (
CUST_NUM, SORT_ORDER, CATEGORY
)
);
END;'''
cursor.execute(sql, long_json_string=long_json_string)

Related

CosmosDB - list in aggregate query response

I have following document structure:
{
"id": "1",
"aId": "2",
"bId": "3",
....
},
{ "id":"2",
"aId": "2",
"bId": "4"
}
How do i return for that JSON that has aId that has list of all bIds of the same aId, and as additional field: count of such bIds? So for example above and condtion: "WHERE aId="2" response would be:
{
"aId": "2",
"bIds" : ["4","3"],
"bIds count" : 2
}
Assuming i only pass one aId as parameter.
I tried something like:
select
(select 'something') as aId,
(select distinct value c.bId from c where c.aId='something') as bIds
from TableName c
But for love of me i cant figure out how to get that list + its count + hardcoded aId in single JSON response (single row)
For example this query:
select
(select distinct value 'someId') as aId,
(select distinct value c.bId) as bIds
from c where c.aId='someId'
will return
{ { 'aId': 'someId', 'bIds':'2'},{'aId':'someId','bIds':'4'}}
while what i acutally want is
{ {'aId':''someId', 'bIds':['2','4']}}
Here is query that is closest to what i want:
select
c.aId as aId,
count(c2) as bIdCount,
array(select distinct value c2.bId from c2)
from c join (select c.bId from c) as c2
where c.aId = 'SOME_ID'
Only thing line with array make this query fail if i delete this line it works (correctly returns id and count in one row). But i need to select content of this list also, and i ma lost why its not working, example is almost copypasted from "How to perform array projection Cosmos Db"
https://azurelessons.com/array-in-cosmos-db/#How_to_perform_array_projection_Azure_Cosmos_DB
Here is how you'd return an array of bId:
SELECT distinct value c.bId
FROM c
where c.aId = "2"
This yields:
[
"3",
"4"
]
Removing the value keyword:
SELECT distinct c.bId
FROM c
where c.aId = "2"
yields:
[
{ "bId" : "3" },
{ "bId" : "4" }
]
From either of these, you can count the number of array elements returned. If your payload must include count and aId, you'll need to add those to your JSON output.

Dash DataTable Filter is not working for numeric columns

I created a table using Python Dash DataTable and added a filter to each column. The data table looks like this:
Text_Column Numeric_Column
abcde 12345
dfjke 34928
Each column has a filter but it seems like the filter works only when the column has text values. It's not working for the numeric column. How do I make it to work for both text and numeric columns?
dash_table.DataTable(
id="table",
columns=[
{"name": i, "id": i} for i in df.columns
],
data=df.to_dict('records'),
filter_action="native",
sort_action="native",
style_table={
"overflowX": "scroll",
},
row_selectable="multi",
style_cell={
"height": "auto",
"maxWidth": "200px",
"whiteSpace": "normal",
"font-family": "Arial",
'textAlign': 'center',
},
)
),
So I just had the same issue. I fixed it by correctly setting the column's type to "numeric".
Like this:
columns = []
for col in df.columns:
col_options = {"name": col, "id": col}
if col == "MyNumericColumn":
col_options["type"] = "numeric"
columns.append(col_options)
my_table = dash_table.DataTable(id="table", columns=columns)
Please note that some filter operations currently seem to have an issue related to the case sensitivity of the column. See this bug report: https://github.com/plotly/dash/issues/1793
So, a slight alternative if you wish to make any of your non string column to be filtered for any data you have.
columns = []
for col in df.columns:
col_options = {"name": col, "id": col}
for value in df[col]:
if not (isinstance(value, str)):
col_options["type"] = "numeric"
columns.append(col_options)
return columns

How do I determine rowsAffected array elements in a T-SQL query?

I have a stored procedure which I execute to insert data from NodeJs which returns a value as the rowsAffected. The rowsAffected is an array in this case of 3, is there anywhere some docs to determine the logic of the this array?
I am trying to determine the rows updated, my best guess would be the 2nd element in array , 1st one being the NewID(), 2nd the actual Insert and 3rd SELECT #guid, am I correct or am I missing something?
Create PROCEDURE sp_FarmFollowUp_INS
(#p_notes_guid VARCHAR(50),
#p_property_guid VARCHAR(50),
#p_contact_name VARCHAR(150),
#p_contact_date DATETIME,
#p_contact_type VARCHAR(50),
#p_contact_email VARCHAR(250),
#p_contact_phone VARCHAR(20),
#p_contact_reason VARCHAR(50),
#p_notes VARCHAR(1000),
#p_created_by VARCHAR(50))
AS
declare #guid Varchar(50)
set #guid = NEWID()
INSERT INTO dbo.tbl_FarmFollowUp (
guid,
notes_guid,
property_guid,
contact_name,
contact_date,
contact_type,
contact_email,
contact_phone,
contact_reason,
notes,
created_by,
created_on)
VALUES
(
#guid,
#p_notes_guid,
#p_property_guid,
#p_contact_name,
#p_contact_date,
#p_contact_type,
#p_contact_email,
#p_contact_phone,
#p_contact_reason,
#p_notes,
#p_created_by,
getDate()
)
SELECT #guid as FollowUpiId
So this returns me the following result
{ recordsets: [ [ [Object] ] ],
recordset: [ { FollowUpiId: '2839D43D-CF9E-466D-B512-1DCD73833380' } ],
output: {},
rowsAffected: [ 1, 1, 1 ],
returnValue: 0 }
You have three separate queries in your stored procedure - one to set the GUID (one row affected), the INSERT (one row affected), and the SELECT to return the GUID (one row affected). Thus, you have three 'Rows Affected' values.
If you wish to shut off that array, add:
SET NOCOUNT ON
as the first line of your stored procedure.

The 'default' dialect with current database version settings does not support in-place multirow inserts

I have data in a list of dict,i tried to insert the data into my db then i get error sqlalchemy.exc.CompileError: The 'default' dialect with current database version settings does not support in-place multirow inserts.
from sqlalchemy import create_engine,Table,Column,String,Integer,MetaData,ForeignKey
from sqlalchemy_utils import database_exists,create_database,drop_database
from sqlalchemy.orm import join
engine = create_engine("mysql+mysqldb://root:268454#localhost:3306/sqlalchemy")
conn = engine.connect()
if database_exists(engine.url):
drop_database(engine.url)
create_database(engine.url)
metaData = MetaData()
userTable = Table("user", metaData
, Column("Id", Integer, primary_key = True)
, Column("Name", String(25)))
orderTable = Table("order", metaData
, Column("Id", Integer, primary_key = True)
, Column("UserId", Integer, ForeignKey("user.Id"))
, Column("Desc", String(250)))
try:
metaData.create_all(engine)
except Exception as ex:
print(ex)
users = [
{"Id": 1,
"Name": "user1"},
{"Id": 2,
"Name": "user2"},
{"Id": 3,
"Name": "user3"},
{"Id": 4,
"Name": "user4"}
]
orders = [
{"Id": 1,
"UserId": 1,
"Desc": "desc1"},
{"Id": 2,
"UserId": 1,
"Desc": "desc2"},
{"Id": 3,
"UserId": 2,
"Desc": "desc3"},
{"Id": 4,
"UserId": 2,
"Desc": "desc4"},
]
sql = userTable.insert().values(users)
print(sql) #this line causes the exception
conn.execute(sql)
sql = orderTable.insert().values(orders)
print(sql)
conn.execute(sql)
sql = userTable.select()
print(sql)
returnUsers = conn.execute(sql)
print(returnUsers)
sql = orderTable.select()
print(sql)
returnOrders = conn.execute(sql)
print(returnOrders)
conn.close()
update:without print(sql) it can works perfectly.
Try if it works if you iterate over the list of dicts and pass in the dicts individually. What you are trying to do uses a special syntax, that is not available in all dialects.
for user in users:
sql = userTable.insert().values(user)
print(sql) #this line causes the exception
conn.execute(sql)
Admittedly the sqlalchemy documentation states, that the MySQL backend should support this. Personally I would not rely on something that is not portable or non-standard and stick with one insert call per row, or the explicit bulk_insert methods.
Edit Regarding print statement
As I mentioned in my comment, the method print calls on this representation of a query is not dialect aware.
You can however retrieve a dialect version of your query like this:
from sqlalchemy.dialects import mysql
print str(q.statement.compile(dialect=mysql.dialect()))
Credit where credit is due: Blog post

JSONB Postgres 9.4

I have a jsonb storing my order product:
CREATE TABLE configuration (
documentid text PRIMARY KEY
, data jsonb NOT NULL
);
Records:
(1, [{"itemid": "PROD001", "qty": 10}, {"itemid": "PROD002", "qty": 20}]),
(2, [{"itemid": "PROD001", "qty": 5}, {"itemid": "PROD003", "qty": 6}, {"itemid": "PROD004", "qty": 7}]),
(2, [{"itemid": "PROD002", "qty": 8}])
I already index data using GIN.
How do I:
Select all sales that has PROD001
Select all sales that has itemid LIKE P%1
Select all sales that has qty > 10
get each product total qty
Postgres documentation regarding JSON functionality is really clear and worth your attention. As for the use cases you provided, answers may be the following:
-- Answer 1
SELECT sales_item
FROM configuration,jsonb_array_elements(data) AS sales_item
WHERE sales_item->>'itemid' = 'PROD001'; -- "->>" returns TEXT value
-- Answer 2
SELECT sales_item FROM configuration,jsonb_array_elements(data) AS sales_item
WHERE sales_item->>'itemid' LIKE 'P%1'; -- just use LIKE
-- Answer 3
SELECT sales_item FROM configuration,jsonb_array_elements(data) AS sales_item
WHERE (sales_item->>'qty')::INTEGER > 10; -- convert TEXT to INTEGER
For last example you just need to use Postgres Window function:
-- Answer 4
SELECT DISTINCT documentid,sum((sales_item->>'qty')::INTEGER)
OVER (PARTITION BY documentid)
FROM configuration,jsonb_array_elements(data) as sales_item;
Examples 1,3 with JsQuery extension
SELECT * FROM
(SELECT jsonb_array_elements(data) as elem FROM configuration
WHERE data ## '#.itemid = "PROD001"') q
WHERE q.elem ## 'itemid = "PROD001"'
SELECT * FROM
(SELECT jsonb_array_elements(data) as elem FROM configuration
WHERE data ## '#.qty > 10') q
WHERE q.elem ## 'qty > 10'
Inner query WHERE clause filters rows doesn't have any array element that matches requirements. Than jsonb_array_elements func. is applied only for needed rows of "configuration" table.

Resources