ServiceStack.OrmLite Select<> throws npgsql syntax error when using WITH CTE - servicestack

From the error I thought this was an issue with Npgsql (see closed issue), however the error is with OrmLite Select<> as it's changing the executed sql.
Question:
Other than not using the WITH CTE is there another way around this error in OrmLite?
Is db.Select<> the wrong command to be using?
Note: WITH CTE works with OrmLite.Scalar
Postgres WITH CTE: http://www.postgresql.org/docs/current/static/queries-with.html
UPDATE: Issue seems to be with OrmLite preparing the SQL statement and it not starting with "SELECT" causes OrmLite to treat the SQL as a "WHERE" param.
[Test]
public void with_cte_ormlite_obj()
{
using (var db = DbConnection)
{
var sql = "WITH w_cnt AS (SELECT 5 AS cnt, 'me' AS name) SELECT cnt, name FROM w_cnt";
// An exception of type 'Npgsql.NpgsqlException' occurred in Npgsql.dll
// ERROR: 42601: syntax error at or near "WITH w_cnt"
// Actual Exec Sql:
// SELECT "cnt", "name" FROM "my_with_cte_obj" WHERE WITH w_cnt AS (SELECT 5 AS cnt, 'me' AS name) SELECT cnt, name FROM w_cnt
var cnt = db.Select<MyWithCteObj>(sql);
var first = cnt.First();
Assert.AreEqual(5, first.Cnt);
Assert.AreEqual("me", first.Name);
}
}
public class MyWithCteObj
{
public int Cnt { get; set; }
public string Name { get; set; }
}

The db.Select<T>() API should only by used for SQL SELECT statements.
The db.SqlList<T>() API should be used for non-SELECT queries, e.g:
using (var db = DbConnection)
{
var cnt = db.SqlList<MyWithCteObj>(
"WITH w_cnt AS (SELECT 5 AS cnt, 'me' AS name) SELECT cnt, name FROM w_cnt");
}
See the docs for more custom SQL APIs examples.

Related

Cannot insert into Cassandra table, getting SyntaxError

I have an assignment where I have to build a Cassandra database. I have connected Cassandra with IntelliJ, i'm writing in java and the output is shown in the command line.
My keyspace farm_db contains a couple of tables in wish i'm would like to insert data. I would like to insert the data with two columns and a list all in one row, in the table 'farmers'. This is a part of my database so far:
cqlsh:farm_db> use farm_db;
cqlsh:farm_db> Describe tables;
farmers foods_dairy_eggs foods_meat
foods_bread_cookies foods_fruit_vegetables
cqlsh:farm_db> select * from farmers;
farmer_id | delivery | the_farmer
-----------+----------+------------
This is what i'm trying to do:
[Picture of what i'm trying to do][1]
I need to insert the collection types 'list' and 'map' in 'farmers' but after a couple of failed attempts with that I tried using hashmap and arraylist instead. I think this could work but i seem to have an error in my syntax and I have no idea what the problem seem to be:
Exception in thread "main" com.datastax.driver.core.exceptions.SyntaxError: line 1:31 mismatched input 'int' expecting ')' (INSERT INTO farmers (farmer_id [int]...)
Am I missing something or am I doing something wrong?
This is my code:
public class FarmersClass {
public static String serverIP = "127.0.0.1";
public static String keyspace = "";
//Create db
public void crateDatabase(String databaseName) {
Cluster cluster = Cluster.builder()
.addContactPoints(serverIP)
.build();
keyspace = databaseName;
Session session = cluster.connect();
String create_db_query = "CREATE KEYSPACE farm_db WITH replication "
+ "= {'class':'SimpleStrategy', 'replication_factor':1};";
session.execute(create_db_query);
}
//Create table
public void createFarmersTable() {
Cluster cluster = Cluster.builder()
.addContactPoints(serverIP)
.build();
Session session = cluster.connect("farm_db");
String create_farm_table_query = "CREATE TABLE farmers(farmer_id int PRIMARY KEY, the_farmer Map <text, text>, delivery list<text>); ";
session.execute(create_farm_table_query);
}
//Insert data in table 'farmer'.
public void insertFarmers(int id, HashMap< String, String> the_farmer, ArrayList <String> delivery) {
Cluster cluster = Cluster.builder()
.addContactPoints(serverIP)
.build();
Session session = cluster.connect("farm_db");
String insert_query = "INSERT INTO farmers (farmer_id int PRIMARY KEY, the_farmer, delivery) values (" + id + "," + the_farmer + "," + delivery + ");";
System.out.println(insert_query);
session.execute(insert_query);
}
}
public static void main(String[] args) {
FarmersClass farmersClass = new FarmersClass();
// FarmersClass.crateDatabase("farm_db");
// FarmersClass.createFarmersTable();
//Collection type map
HashMap<String, String> the_farmer = new HashMap<>();
the_farmer.put("Name", "Ana Petersen ");
the_farmer.put("Farmhouse", "The great farmhouse");
the_farmer.put("Foods", "Fruits & Vegetables");
//Collection type list
ArrayList<String> delivery = new ArrayList<String>();
String delivery_1 = "Village 1";
String delivery_2 = "Village 2";
delivery.add(delivery_1);
delivery.add(delivery_2);
FarmersClass.insertFarmers(1, the_farmer, delivery);
}
The problem is the syntax of your CQL INSERT query:
String insert_query = \
"INSERT INTO farmers (farmer_id int PRIMARY KEY, the_farmer, delivery) \
values (" + id + "," + the_farmer + "," + delivery + ");";
You've incorrectly added int PRIMARY KEY in the list of columns.
The correct format is:
INSERT INTO table_name (pk, col2, col3) VALUES ( ... )
For details and examples, see CQL INSERT. Cheers!

SQL Parser Visitor + Metabase + Presto

I'm facing what seems to be a quite easy problem, but I'm not able to put my head around the problem to find a suitable solution.
Problem:
I need to append the schema into my SQL statement, in a "weird"(with schema in double quotes) way.
FROM "SCHEMA".tableB tableB
LEFT JOIN "SCHEMA".tableC tableC
Context
Basically, we are hosting and exposing a Metabase tool that will connect and perform query on our Hive database using Presto SQL.
Metabase allow the customer to write SQL statements and some customers, they just don't type the schema on statements. Today we are throwing and error for those queries, but I could easily retrieve the schema value from the Authorization header, since in our multi-tenant product the schema is the tenant id where this user is logged, and with that information in hands, I could append to the customer SQL statement and avoid the error.
Imagine that the customer typed the follow statement:
SELECT tableA.*
, (tableA.valorfaturado + tableA.valorcortado) valorpedido
FROM (SELECT from_unixtime(tableB.datacorte / 1000) datacorte
, COALESCE((tableB.quantidadecortada * tableC.preco), 0) valorcortado
, COALESCE((tableB.quantidade * tableC.preco), 0) valorfaturado
, tableB.quantidadecortada
FROM tableB tableB
LEFT JOIN tableC tableC
ON tableC.numeropedido = tableB.numeropedido
AND tableC.codigoproduto = tableB.codigoproduto
AND tableC.codigofilial = tableB.codigofilial
LEFT JOIN tableD tableD
ON tableD.numero = tableB.numeropedido
WHERE (CASE
WHEN COALESCE(tableB.codigofilial, '') = '' THEN
tableD.codigofilial
ELSE
tableB.codigofilial
END) = '10'
AND from_unixtime(tableB.datacorte / 1000) BETWEEN from_iso8601_timestamp('2020-07-01T03:00:00.000Z') AND from_iso8601_timestamp('2020-08-01T02:59:59.999Z')) tableA
ORDER BY datacorte
I should convert this into (adding the "SCHEMA"):
SELECT tableA.*
, (tableA.valorfaturado + tableA.valorcortado) valorpedido
FROM (SELECT from_unixtime(tableB.datacorte / 1000) datacorte
, COALESCE((tableB.quantidadecortada * tableC.preco), 0) valorcortado
, COALESCE((tableB.quantidade * tableC.preco), 0) valorfaturado
, tableB.quantidadecortada
FROM "SCHEMA".tableB tableB
LEFT JOIN "SCHEMA".tableC tableC
ON tableC.numeropedido = tableB.numeropedido
AND tableC.codigoproduto = tableB.codigoproduto
AND tableC.codigofilial = tableB.codigofilial
LEFT JOIN "SCHEMA".tableD tableD
ON tableD.numero = tableB.numeropedido
WHERE (CASE
WHEN COALESCE(tableB.codigofilial, '') = '' THEN
tableD.codigofilial
ELSE
tableB.codigofilial
END) = '10'
AND from_unixtime(tableB.datacorte / 1000) BETWEEN from_iso8601_timestamp('2020-07-01T03:00:00.000Z') AND from_iso8601_timestamp('2020-08-01T02:59:59.999Z')) tableA
ORDER BY datacorte
Still trying to find a solution that uses only presto-parser and Visitor + Instrumentation solution.
Also, I know about JSQLParser and I tried, but I alway come back to try to find a "plain" solution scared that JSQLParser will not be able to support all the Presto/Hive queries, that are a little bit different than standard SQL;
I create a little project on GitHub with test case to validate..
https://github.com/genyherrera/prestosqlerror
But for those that don't want to clone a repository, here are the classes and dependencies:
import java.util.Optional;
import com.facebook.presto.sql.SqlFormatter;
import com.facebook.presto.sql.parser.ParsingOptions;
import com.facebook.presto.sql.parser.SqlParser;
public class SchemaAwareQueryAdapter {
// Inspired from
// https://github.com/prestodb/presto/tree/master/presto-parser/src/test/java/com/facebook/presto/sql/parser
private static final SqlParser SQL_PARSER = new SqlParser();
public String rewriteSql(String sqlStatement, String schemaId) {
com.facebook.presto.sql.tree.Statement statement = SQL_PARSER.createStatement(sqlStatement, ParsingOptions.builder().build());
SchemaAwareQueryVisitor visitor = new SchemaAwareQueryVisitor(schemaId);
statement.accept(visitor, null);
return SqlFormatter.formatSql(statement, Optional.empty());
}
}
public class SchemaAwareQueryVisitor extends DefaultTraversalVisitor<Void, Void> {
private String schemaId;
public SchemaAwareQueryVisitor(String schemaId) {
super();
this.schemaId = schemaId;
}
/**
* The customer can type:
* [table name]
* [schema].[table name]
* [catalog].[schema].[table name]
*/
#Override
protected Void visitTable(Table node, Void context) {
List<String> parts = node.getName().getParts();
// [table name] -> is the only one we need to modify, so let's check by parts.size() ==1
if (parts.size() == 1) {
try {
Field privateStringField = Table.class.getDeclaredField("name");
privateStringField.setAccessible(true);
QualifiedName qualifiedName = QualifiedName.of("\""+schemaId+"\"",node.getName().getParts().get(0));
privateStringField.set(node, qualifiedName);
} catch (NoSuchFieldException | SecurityException | IllegalArgumentException | IllegalAccessException e) {
throw new SecurityException("Unable to execute query");
}
}
return null;
}
}
import static org.testng.Assert.assertEquals;
import org.gherrera.prestosqlparser.SchemaAwareQueryAdapter;
import org.testng.annotations.Test;
public class SchemaAwareTest {
private static final String schemaId = "SCHEMA";
private SchemaAwareQueryAdapter adapter = new SchemaAwareQueryAdapter();
#Test
public void testAppendSchemaA() {
String sql = "select * from tableA";
String bound = adapter.rewriteSql(sql, schemaId);
assertEqualsFormattingStripped(bound,
"select * from \"SCHEMA\".tableA");
}
private void assertEqualsFormattingStripped(String sql1, String sql2) {
assertEquals(sql1.replace("\n", " ").toLowerCase().replace("\r", " ").replaceAll(" +", " ").trim(),
sql2.replace("\n", " ").toLowerCase().replace("\r", " ").replaceAll(" +", " ").trim());
}
}
<dependencies>
<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-parser</artifactId>
<version>0.229</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.10</version>
<scope>test</scope>
</dependency>
</dependencies>
PS: I was able to add the schema without the doubles quotes, but them I got into identifiers must not start with a digit; surround the identifier with double quotes error. Basically this error comes from SqlParser$PostProcessor.exitDigitIdentifier(...) method..
Thanks
I was able to find a solution for my case, either way will share on Presto Slack my finding to see if that is expected behavior.
So, if you want to append with double quote your schema, you will need to create your own Vistor class and you'll need to override the method visitTable and when you Qualify the name of your table with schema, (here's the tick), pass the schema as UPPERCASE, so it will not match the regex pattern on class SqlFormatter on method formatName and it will add the double-quote..
public class SchemaAwareQueryVisitor extends DefaultTraversalVisitor<Void, Void> {
private String schemaId;
public SchemaAwareQueryVisitor(String schemaId) {
super();
this.schemaId = schemaId;
}
#Override
protected Void visitTable(Table node, Void context) {
try {
Field privateStringField = Table.class.getDeclaredField("name");
privateStringField.setAccessible(true);
QualifiedName qualifiedName = QualifiedName.of(schemaId, node.getName().getParts().get(0));
privateStringField.set(node, qualifiedName);
} catch (NoSuchFieldException
| SecurityException
| IllegalArgumentException
| IllegalAccessException e) {
throw new SecurityException("Unable to execute query");
}
return null;
}
}

IllegalArgumentException: Table xyz does not exist in keyspace my_ks

I am developing an application, where I am trying to create a table if not exists and making a Query on it. It is working fine in normal cases. But for the first time , when the table is created , then when trying to Query the same table, the application is throwing :
IllegalArgumentException: Table xyz does not exist in keyspace my_ks
Same happens if I drop the table, and when my code recreates the table again.
For other cases, when the table exists, it is working fine. Is it some kind of replication issue, or should use a timeout from some time when the table is created for first time.
Following is the code snippet:
// Oredr 1: First this will be called
public boolean isSchemaExists() {
boolean isSchemaExists = false;
Statement statement = QueryBuilder
.select()
.countAll()
.from(keyspace_name, table_name);
statement.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);
try {
Session session = cassandraClient.getSession(someSessionKey);
ResultSet resultSet = null;
resultSet = session.execute(statement);
if (resultSet.one() != null) {
isSchemaExists = true;
}
} catch (all exception handling)
}
return isSchemaExists;
}
// Oredr 2: if previous method returns false then this will be get called
public void createSchema(String createTableScript) {
Session session = cassandraClient.getSession(someSessionKey);
if (isKeySpaceExists(keyspaceName, session)) {
session.execute("USE " + keyspaceName);
}
session.execute(createTableScript);
}
//Oredr 3: Now read the table, this is throwing the exception when the table
// is created for first time
public int readTable(){
Session session = cassandraClient.getSession(someSessionKey);
MappingManager manager = new MappingManager(session);
Mapper<MyPojo> mapper = manager.mapper(MyPojo.class);
Statement statement = QueryBuilder
.select()
.from(keyspaceName, tableName)
.where(eq("col_1", someValue)).and(eq("col_2", someValue));
statement.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);
ResultSet resultSet = session.execute(statement);
result = mapper.map(resultSet);
for (MyPojo myPojo : result) {
return myPojo.getCol1();
}
}
In isSchemaExists function use system.tables.
SELECT * FROM system.tables WHERE keyspace_name='YOUR KEYSPACE' AND table_name='YOUR TABLE'
Corresponding Java Code:
Statement statement = QueryBuilder
.select()
.from("system", "tables")
.where(eq("keyspace_name", keyspace)).and(eq("table_name", table));
It seems like in isSchemaExists you are using actual table and keyspace which will not exist when dropped or not created. That's the reason it is throwing you error table does not exist.

Entity Framework 5 - Implementing SQL Server "Execute As User"

I am writing a database application using Visual Studio 2012 with Entity Framework 5 and SQL Server 2008. I would like Entity Framework to impersonate a SQL Server user (i.e. user without a login). I have created a new constructor for the DB context MyDatabaseEntities which includes an argument for the name of the user to impersonate. Here is the code that I've written:
public partial class MyDatabaseEntities
{
private String _impersonateUser = null;
public MyDatabaseEntities(String impersonateUser)
: base("MyConnectionString")
{
_impersonateUser = impersonateUser;
this.Database.Connection.StateChange += Connection_StateChange;
}
void Connection_StateChange(object sender, StateChangeEventArgs e)
{
if (e.CurrentState == ConnectionState.Open && e.OriginalState != ConnectionState.Open)
{
using (var cmd = this.Database.Connection.CreateCommand())
{
cmd.CommandType = CommandType.Text;
cmd.Parameters.Add(new SqlParameter("user", _impersonateUser));
cmd.CommandText = "EXECUTE AS USER = #user";
cmd.ExecuteNonQuery();
}
}
}
I had to add the check...
if (e.CurrentState == ConnectionState.Open && e.OriginalState != ConnectionState.Open)
...because the method Connection_StateChange method seems to execute even when the state hasn't changed. Then problem is that when I run the code twice,
public void RunSimpleQuery()
{
using (MyDatabaseEntities context = new MyDatabaseEntities("UserName"))
{
var result = context.TableName.ToList();
}
}
...Entity Framework throws a SqlException:
A severe error occurred on the current command. The results, if
any, should be discarded.\r\nA severe error occurred on the current
command. The results, if any, should be discarded.
Any ideas?
Update 1
I in my code above, I changed...
cmd.CommandText = "EXECUTE AS USER = #user;";
...to...
cmd.CommandText = "REVERT; EXECUTE AS USER = #user;";
...and I still get the same SqlException error.
The problem is that EF closes connection when it doesn't need it and returns it back to the pool. So when it executes some SQL again it request new connection from the pool where your event may not be initialized. But again I believe that you should try to solve this with manually controlling connection lifetime to have both benefit of connection pooling and be able to meet your requirements.
I know is an old question, but maybe will be useful for someone.
I did in a different way, using your code...
Instead of
Connection_StateChanged event
I create two methods in the same class:
public void ChangeUser(String sUser)
{
if(Database.Connection.State != ConnectionState.Open)
Database.Connection.Open();
using (var cmd = Database.Connection.CreateCommand())
{
cmd.CommandType = CommandType.Text;
cmd.Parameters.Add(new SqlParameter("user", sUser));
cmd.CommandText = "EXECUTE AS USER = #user;";
cmd.ExecuteNonQuery();
}
}
public void Revert()
{
if (Database.Connection.State != ConnectionState.Open)
Database.Connection.Open();
using (var cmd = Database.Connection.CreateCommand())
{
cmd.CommandType = CommandType.Text;
cmd.CommandText = "REVERT;";
cmd.ExecuteNonQuery();
}
}
I use it before and after execute stored procedure,
using (var db = new MyDatabaseEntities())
{
db.ChangeUser(model.Username);
var result = db.Something();
db.Revert();
return result;
}
It works fine with SPs and it doesn't throw an exception even after many executions. If I could catch an event after command execute, maybe all be encapsulated on MyDatabaseEntities.

Reengineer SQL into SubSonic LINQ

I have 11 records in the Menu table, only 1 with a PageID set however if I use
var test = Menu.All().Where(
x => x.WebPages.Any(
pages => pages.Roles.Contains(Roles.GetRolesForUser()[0])
I get 11 records as the SQL run is this
SELECT [t0].[CategoryID], [t0].[CreatedBy], [t0].[CreatedOn],
[t0].[ID], [t0].[ImageID], [t0].[ImageIDHover], [t0].[Locale],
[t0].[ModifiedBy], [t0].[ModifiedOn], [t0].[OrderID], [t0].[PageID],
[t0].[ParentID], [t0].[Title], [t0].[URL], [t0].[Visible]
FROM [dbo].[Menu] AS t0
WHERE EXISTS(
SELECT NULL
FROM [dbo].[WebPage] AS t1
WHERE ([t1].[Roles] LIKE '%' + 'User' + '%')
)
If I run this I get the 1 record
var test = Menu.All().Where(
x => x.WebPages.Any(
pages => pages.Roles.Contains(
Roles.GetRolesForUser()[0]) && pages.ID == x.PageID));
The SQL for this is
SELECT [t0].[CategoryID], [t0].[CreatedBy], [t0].[CreatedOn],
[t0].[ID], [t0].[ImageID], [t0].[ImageIDHover], [t0].[Locale],
[t0].[ModifiedBy], [t0].[ModifiedOn], [t0].[OrderID], [t0].[PageID],
[t0].[ParentID], [t0].[Title], [t0].[URL], [t0].[Visible]
FROM [dbo].[Menu] AS t0
WHERE EXISTS (
SELECT NULL
FROM [dbo].[WebPage] AS t1
WHERE (([t1].[Roles] LIKE '%' + 'User' + '%') AND
([t1].[ID] = [t0].[PageID]))
)
The problem with Any() is that in the SQL as long as one record exits, doesn't matter which record it will return data.
I think effectively I am wanting an UNION SQL like below but I don't know how I re-engineer that into C#/Subsonic
select m.* from menu m where pageid is null
union
select m.* from menu m
join webpage p
on p.id = m.pageid
where p.roles like '%User%'
I want to return all menu records and for those with a PageID set that the corresponding WebPage has the user's role in it. If the user's role is not in the WebPage then I don't want to see it in my results.
Here are my cutodwn classes that Subsonic generates
public partial class Menu: IActiveRecord
{
public int ID {get; set;}
public int CategoryID {get;set;}
public bool Visible {get;set;}
public int PageID {get;set;}
public IQueryable<WebPage> WebPages
{
get
{
var repo=NorthCadburyWebsite.Models.WebPage.GetRepo();
return from items in repo.GetAll()
where items.ID == _PageID
select items;
}
}
}
public partial class WebPage: IActiveRecord
{
public int ID {get;set;}
public string Roles {get;set;}
}
It sure seems like you could just add an alternative test to the where clause to do this.
var test = Menu.All()
.Where(x => x.PageID == null
|| x.WebPages
.Any(pages => pages.Roles.Contains(Roles.GetRolesForUser()[0])
&& pages.ID == x.PageID));
EDIT:
Try using the the Intersect method to see if there is an overlap between the roles in the DB and the roles for the user.
var test = Menu.All()
.Where(x => x.PageID == null
|| x.WebPages
.Any(pages => pages.Roles.Intersect(Roles.GetRolesForUser().Split(','))
&& pages.ID == x.PageID));

Resources