The generated Webapp "crashes" while bootstrapping the backend, because Liquibase won't be able to reach a consistent state. Crash means here, the webapp itself runs, but it won't be usable because the backend cannot authorize you. I read other posts on SO regarding Liquibase and no suggestion works.
Performed steps:
visit Jhipster Online to generate a vanilla Webapp as starting point.
download and extract the ZIP file to a directory of my choice
chmod +x ./mvnw
npm i
run ./mvnw -P-webapp
System Info
Ubuntu 20.04.03 LTS
node v14.16.0
javac openJDK 11.0.11
java:
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
My steps to solve the issue
create new directories
switch node versions using nvm use --lts or i.e. nvm use [version no.]
run jhipster and select some stuff on each project
run ./mvnw -P-webapp --> "watch LiqourBase burn :-("
alternatively change java-version to i.e. 16.x
reboots
cursing, repeating upper steps with different selections
I don't know why it stops working. I observed the issue with some other recently made projects using Jhipster, too. What is it? Do I have to wipe complete .m2 simply that way?
The Log:
2021-11-09 14:22:14.210 DEBUG 22360 --- [kground-preinit] org.jboss.logging : Logging Provider: org.jboss.logging.Log4j2LoggerProvider
2021-11-09 14:22:14.253 INFO 22360 --- [ restartedMain] dev.semo.some.Webapp : Starting Webapp84 using Java 11.0.11 on computer with PID 22360 (/home/semo/dev/work/playground/target/classes started by semo in /home/semo/dev/work/playground)
2021-11-09 14:22:14.254 DEBUG 22360 --- [ restartedMain] dev.semo.some.Webapp : Running with Spring Boot v2.5.5, Spring v5.3.10
2021-11-09 14:22:14.255 INFO 22360 --- [ restartedMain] dev.semo.some.Webapp : The following profiles are active: dev,api-docs
[...omitted...]
2021-11-09 14:22:18.725 DEBUG 22360 --- [ restartedMain] d.b.semo.some.config.AsyncConfiguration : Creating Async Task Executor
2021-11-09 14:22:18.853 DEBUG 22360 --- [ restartedMain] d.b.s.some.config.LiquibaseConfiguration : Configuring Liquibase
2021-11-09 14:22:19.076 WARN 22360 --- [ebapp-84-task-1] t.j.c.liquibase.AsyncSpringLiquibase : Starting Liquibase asynchronously, your database might not be ready at startup!
2021-11-09 14:22:20.770 ERROR 22360 --- [ebapp-84-task-1] t.j.c.liquibase.AsyncSpringLiquibase : Liquibase could not start correctly, your database is NOT ready: liquibase.exception.MigrationFailedException: Migration failed for change set config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster:
Reason: liquibase.exception.DatabaseException: Syntax Fehler in SQL Befehl "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"
Syntax error in SQL statement "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"; SQL statement:
IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON [42000-200] [Failed SQL: (42000) IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON]
liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for change set config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster:
Reason: liquibase.exception.DatabaseException: Syntax Fehler in SQL Befehl "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"
Syntax error in SQL statement "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"; SQL statement:
IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON [42000-200] [Failed SQL: (42000) IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON]
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:124)
at liquibase.Liquibase.lambda$null$0(Liquibase.java:265)
at liquibase.Scope.lambda$child$0(Scope.java:177)
at liquibase.Scope.child(Scope.java:186)
at liquibase.Scope.child(Scope.java:176)
at liquibase.Scope.child(Scope.java:155)
at liquibase.Scope.child(Scope.java:239)
at liquibase.Liquibase.lambda$update$1(Liquibase.java:264)
at liquibase.Scope.lambda$child$0(Scope.java:177)
at liquibase.Scope.child(Scope.java:186)
at liquibase.Scope.child(Scope.java:176)
at liquibase.Scope.child(Scope.java:155)
at liquibase.Liquibase.runInScope(Liquibase.java:2404)
at liquibase.Liquibase.update(Liquibase.java:211)
at liquibase.Liquibase.update(Liquibase.java:197)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:314)
at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:269)
at org.springframework.boot.autoconfigure.liquibase.DataSourceClosingSpringLiquibase.afterPropertiesSet(DataSourceClosingSpringLiquibase.java:46)
at tech.jhipster.config.liquibase.AsyncSpringLiquibase.initDb(AsyncSpringLiquibase.java:118)
at tech.jhipster.config.liquibase.AsyncSpringLiquibase.lambda$afterPropertiesSet$0(AsyncSpringLiquibase.java:93)
at tech.jhipster.async.ExceptionHandlingAsyncTaskExecutor.lambda$createWrappedRunnable$1(ExceptionHandlingAsyncTaskExecutor.java:78)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster:
Reason: liquibase.exception.DatabaseException: Syntax Fehler in SQL Befehl "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"
Syntax error in SQL statement "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"; SQL statement:
IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON [42000-200] [Failed SQL: (42000) IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON]
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:695)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:49)
at liquibase.changelog.ChangeLogIterator$2.lambda$null$0(ChangeLogIterator.java:111)
at liquibase.Scope.lambda$child$0(Scope.java:177)
at liquibase.Scope.child(Scope.java:186)
at liquibase.Scope.child(Scope.java:176)
at liquibase.Scope.child(Scope.java:155)
at liquibase.changelog.ChangeLogIterator$2.lambda$run$1(ChangeLogIterator.java:110)
at liquibase.Scope.lambda$child$0(Scope.java:177)
at liquibase.Scope.child(Scope.java:186)
at liquibase.Scope.child(Scope.java:176)
at liquibase.Scope.child(Scope.java:155)
at liquibase.Scope.child(Scope.java:239)
at liquibase.changelog.ChangeLogIterator$2.run(ChangeLogIterator.java:94)
at liquibase.Scope.lambda$child$0(Scope.java:177)
at liquibase.Scope.child(Scope.java:186)
at liquibase.Scope.child(Scope.java:176)
at liquibase.Scope.child(Scope.java:155)
at liquibase.Scope.child(Scope.java:239)
at liquibase.Scope.child(Scope.java:243)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:66)
... 23 common frames omitted
Caused by: liquibase.exception.DatabaseException: Syntax Fehler in SQL Befehl "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"
Syntax error in SQL statement "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"; SQL statement:
IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON [42000-200] [Failed SQL: (42000) IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON]
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:393)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:82)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:150)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1279)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1261)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:660)
... 43 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Syntax Fehler in SQL Befehl "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"
Syntax error in SQL statement "IF[*] EXISTS(SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'jhi_user'
AND TABLE_SCHEMA = 'PUBLIC'
AND COLUMNPROPERTY(OBJECT_ID(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.JHI_USER ON"; SQL statement:
IF EXISTS(select TABLE_NAME
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'jhi_user'
and TABLE_SCHEMA = 'PUBLIC'
and COLUMNPROPERTY(object_id(TABLE_SCHEMA + '.' + TABLE_NAME), COLUMN_NAME, 'IsIdentity') = 1)
SET IDENTITY_INSERT PUBLIC.jhi_user ON [42000-200]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:453)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
at org.h2.message.DbException.get(DbException.java:205)
at org.h2.message.DbException.get(DbException.java:181)
at org.h2.message.DbException.getSyntaxError(DbException.java:229)
at org.h2.command.Parser.getSyntaxError(Parser.java:1051)
at org.h2.command.Parser.parsePrepared(Parser.java:1013)
at org.h2.command.Parser.parse(Parser.java:843)
at org.h2.command.Parser.parse(Parser.java:819)
at org.h2.command.Parser.prepareCommand(Parser.java:738)
at org.h2.engine.Session.prepareLocal(Session.java:657)
at org.h2.engine.Session.prepareCommand(Session.java:595)
at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1235)
at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:212)
at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:201)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:389)
... 48 common frames omitted
2021-11-09 14:22:21.028 DEBUG 22360 --- [ restartedMain] d.b.semo.some.security.jwt.TokenProvider : Using a Base64-encoded JWT secret key
2021-11-09 14:22:21.940 INFO 22360 --- [ restartedMain] o.a.k.clients.producer.ProducerConfig : ProducerConfig values:
[...omitted...]
2021-11-09 14:22:22.057 INFO 22360 --- [ restartedMain] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.7.1
2021-11-09 14:22:22.057 INFO 22360 --- [ restartedMain] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 61dbce85d0d41457
2021-11-09 14:22:22.058 INFO 22360 --- [ restartedMain] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1636464142055
2021-11-09 14:22:22.072 WARN 22360 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[... omitted ...]
2021-11-09 14:22:22.910 DEBUG 22360 --- [ restartedMain] t.j.c.apidoc.SpringfoxAutoConfiguration : Starting OpenAPI docs
2021-11-09 14:22:22.928 DEBUG 22360 --- [ restartedMain] t.j.c.apidoc.SpringfoxAutoConfiguration : Started OpenAPI docs in 18 ms
2021-11-09 14:22:22.953 DEBUG 22360 --- [ restartedMain] d.b.s.kea.config.DatabaseConfiguration : H2 database is available on port 18501
2021-11-09 14:22:23.795 INFO 22360 --- [ restartedMain] org.jboss.threads : JBoss Threads version 3.1.0.Final
2021-11-09 14:22:24.112 INFO 22360 --- [ restartedMain] dev.semo.some.Webapp : Started Webapp84 in 10.846 seconds (JVM running for 11.77)
2021-11-09 14:22:24.116 INFO 22360 --- [ restartedMain] dev.semo.some.Webapp :
----------------------------------------------------------
Application 'webapp84' is running! Access URLs:
Local: http://localhost:8501/
External: http://127.0.1.1:8501/
Profile(s): [dev, api-docs]
----------------------------------------------------------
Related
In pgAdmin/the cli, the following query:
UPDATE wq SET l_id = NULL, v_id = NULL WHERE w_id = 'cf93bc71-88c1-4bba-9e5c-fdc58d0ed14e';
works fine. However, when calling the same with the pg package in node:
const w_id_val = 'cf93bc71-88c1-4bba-9e5c-fdc58d0ed14e';
/*
(here client is the result of calling const pool = pg.Pool({...}) ,
then let client = await pool.connect());
*/
const result = await client.query(
`UPDATE wq
SET l_id = null,
v_id = null
WHERE w_id = $1`,
[w_id_val]
);
I get the following error:
{
"message":"column \"w_id\" does not exist",
"stack":"error: column \"w_id\" does not exist\n at Connection.parseE (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:569:11)\n at Connection.parseMessage (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:396:17)\n at Socket.<anonymous> (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:132:22)\n at Socket.emit (events.js:314:20)\n at Socket.EventEmitter.emit (domain.js:483:12)\n at addChunk (_stream_readable.js:297:12)\n at readableAddChunk (_stream_readable.js:272:9)\n at Socket.Readable.push (_stream_readable.js:213:10)\n at TCP.onStreamRead (internal/stream_base_commons.js:188:23)\n at TCP.callbackTrampoline (internal/async_hooks.js:126:14)",
"name":"error",
"length":112,
"severity":"ERROR",
"code":"42703",
"position":"68",
"file":"parse_relation.c",
"line":"3514",
"routine":"errMissingColumn"
}
I can confirm the column exists with this query:
SELECT table_schema, table_name, column_name, data_type
FROM information_schema.columns
WHERE table_name = 'wq';
public wq id uuid
public wq w_id uuid
public wq l_id uuid
public wq v_id uuid
I can also confirm that the column (w_id) should be recognized by pg as when using pg to query the table with a SELECT statement, I get this back in the fields property in the result object returned:
fields: [
Field {
name: 'id',
tableID: 26611,
columnID: 1,
dataTypeID: 2950,
dataTypeSize: 16,
dataTypeModifier: -1,
format: 'text'
},
Field {
name: 'w_id',
tableID: 26611,
columnID: 3,
dataTypeID: 2950,
dataTypeSize: 16,
dataTypeModifier: -1,
format: 'text'
},
...
I've also confirmed this isn't a case issue; i.e. the column name is all lowercase and using double quotes around the column name has no effect.
In Databricks I tried this :
storageName = "..."
accessKey = "..."
containerName = '...'
#try:
dbutils.fs.mount(
source = "wasbs://"+containerName+"\#"+storageName+".blob.core.windows.net/",
mount_point = "/mnt/foldername",
extra_configs = {"fs.azure.account.key."+storageName+".blob.core.windows.net": accessKey})
#except:
# print("hello")
#pass
And I had this error:
java.io.IOException: Failed to perform 'getMountFileState(forceRefresh=true)' for mounts after 3 attempts. Please, retry the operation.
Original exception: 'shaded.databricks.org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation.
What can I do to avoid this ?
I have the below 'generic' code that sets the diagnostic setting for all my Azure resources. However, I am not able to use it for Azure Databricks, as the Azure databricks does not have 'MetricsCategory' but just the log category.
How can I bypass the 'MetricCategory' in the '$diagParams' parameter. I do not want to change the entire function just for Azure Databricks. It defeats the very purpose I created the Generic Function.
Code:
$diagParams = #{
Name = "coemonitoreu"
ResourceId = "/subscriptions/-f30a-4bfd-a6be-1c59594b8592/resourcegroups/manjug-dev/providers/Microsoft.Databricks/workspaces/manjug-adbv-dev"
StorageAccountId = "/subscriptions/-2ad9-4167-b9f2-45be0e48c465/resourcegroups/-monitor-dev/providers/microsoft.storage/storageaccounts/monitorblobdev"
WorkspaceId = "/subscriptions/-2ad9-4167-b9f2-45be0e48c465/resourcegroups/-monitor-dev/providers/microsoft.operationalinsights/workspaces/diag-dev"
MetricCategory = #("AllMetrics")
Category = #("dbfs", "clusters", "accounts", "jobs", "notebook", "ssh", "workspace", "secrets", "sqlPermissions", "instancePools")
Enabled = $true
RetentionEnabled = $true
RetentionInDays = 0
}
$DebugPreference = 'continue'
Set-AzDiagnosticSetting #diagParams
I get the error, because I am mentioning 'metrics' in my parameter set. And this is not consumed by Azure databricks diagnostic setting.
Set-AzDiagnosticSetting : Exception type: ErrorResponseException, Message: Null/Empty, Code: BadRequest, Status code:BadRequest, Reason phrase: Bad Request
At line:14 char:1
+ Set-AzDiagnosticSetting #diagParams
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Set-AzDiagnosticSetting], PSInvalidOperationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Insights.Diagnostics.SetAzureRmDiagnosticSettingCommand
DEBUG: AzureQoSEvent: CommandName - Set-AzDiagnosticSetting; IsSuccess - False; Duration - 00:00:01.8132017; Exception - System.Management.Automation.PSInvalidOperationEx
ception: Exception type: ErrorResponseException, Message: Null/Empty, Code: BadRequest, Status code:BadRequest, Reason phrase: Bad Request ---> Microsoft.Azure.Management
.Monitor.Models.ErrorResponseException: Operation returned an invalid status code 'BadRequest'
at Microsoft.Azure.Management.Monitor.DiagnosticSettingsOperations.<CreateOrUpdateWithHttpMessagesAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Azure.Management.Monitor.DiagnosticSettingsOperationsExtensions.<CreateOrUpdateAsync>d__3.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.Azure.Commands.Insights.MonitorCmdletBase.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 5:38:51 AM - SetAzureRmDiagnosticSettingCommand end processing.
DEBUG: 5:38:51 AM - SetAzureRmDiagnosticSettingCommand end processing.
Well, to create an exclusion you should use if statement:
if ($diagParam.resourceId -match 'DataBricks') {
$diagParams.Remove('MetricCategory')
}
I am using embedded cassandra to run unit tests. I notice that if any cql statements fail, then I don't see any descriptive reason for failure. For eg. I am running the following two statements which fails because I am trying to add a table without switching to a keyspace
val statement1 =
"""
|CREATE KEYSPACE test
| WITH REPLICATION = {
| 'class' : 'SimpleStrategy',
| 'replication_factor' : 1
| };
""".stripMargin
val statement3 =
"""
|CREATE TABLE users (
| bucket int,
| email text,
| firstname text,
| lastname text,
| authprovider text,
| password text,
| confirmed boolean,
| id UUID,
| hasher text,
| salt text,
| PRIMARY KEY ((bucket, email), authprovider,firstname, lastname) )
""".stripMargin
val cqlStatements:CqlStatements = new CqlStatements(statement1,statement3)
")
val testCassandra = repoTestEnv.testCassandra
try {
testCassandra.start()
testCassandra.executeScripts(cqlStatements)
} finally testCassandra.stop()
But I don't see correct error. I see the following which doesn't tell exactly what is the problem
[info] c.g.n.e.c.l.WindowsCassandraNode - Apache Cassandra Node '7276' is started
[info] c.g.n.e.c.l.LocalCassandraDatabase - Apache Cassandra '3.11.1' is started (20811 ms)
[warn] c.d.d.c.Connection - /127.0.0.1:9042 did not send an authentication challenge; This is suspicious because the driver expects authentication (configured auth provider = com.datastax.driver.core.PlainTextAuthProvider)
[warn] c.d.d.c.Connection - /127.0.0.1:9042 did not send an authentication challenge; This is suspicious because the driver expects authentication (configured auth provider = com.datastax.driver.core.PlainTextAuthProvider)
[debug] c.g.n.e.c.t.u.CqlUtils - Executing Script: CqlStatements [
CREATE KEYSPACE test
WITH REPLICATION = {
'class' : 'SimpleStrategy',
'replication_factor' : 1
};
,
CREATE TABLE users (
bucket int,
email text,
firstname text,
lastname text,
authprovider text,
password text,
confirmed boolean,
id UUID,
hasher text,
salt text,
PRIMARY KEY ((bucket, email), authprovider,firstname, lastname) )
]
[debug] c.g.n.e.c.t.u.CqlUtils - Executing Statement:
CREATE KEYSPACE test
WITH REPLICATION = {
'class' : 'SimpleStrategy',
'replication_factor' : 1
};
[info] c.g.n.e.c.Cassandra - INFO [Native-Transport-Requests-1] 2019-05-29 07:50:00,788 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=test, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
[debug] c.g.n.e.c.t.u.CqlUtils - Executing Statement:
CREATE TABLE users (
bucket int,
email text,
firstname text,
lastname text,
authprovider text,
password text,
confirmed boolean,
id UUID,
hasher text,
salt text,
PRIMARY KEY ((bucket, email), authprovider,firstname, lastname) )
[debug] c.g.n.e.c.t.TestCassandra - Stop TestCassandra 3.11.1
[info] c.g.n.e.c.l.LocalCassandraDatabase - Stop Apache Cassandra '3.11.1'
[debug] c.g.n.e.c.l.RunProcess - Execute 'powershell -ExecutionPolicy Unrestricted C:\Users\manu\AppData\Local\Temp\embedded-cassandra\3.11.1\0d155e04-97d5-4927-87ac-d46824a77c32\bin\stop-server.ps1 -p C:\Users\manu\AppData\Local\Temp\embedded-cassandra\3.11.1\0d155e04-97d5-4927-87ac-d46824a77c32\1da63488-2624-4141-a49e-174203b7edc4' within a directory 'C:\Users\manu\AppData\Local\Temp\embedded-cassandra\3.11.1\0d155e04-97d5-4927-87ac-d46824a77c32'
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:03,926 HintsService.java:220 - Paused hints dispatch
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:03,933 Server.java:176 - Stop listening for CQL clients
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:03,934 Gossiper.java:1532 - Announcing shutdown
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:03,938 StorageService.java:2268 - Node localhost/127.0.0.1 state jump to shutdown
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:05,941 MessagingService.java:984 - Waiting for messaging service to quiesce
[info] c.g.n.e.c.Cassandra - INFO [ACCEPT-localhost/127.0.0.1] 2019-05-29 07:50:05,948 MessagingService.java:1338 - MessagingService has terminated the accept() thread
[info] c.g.n.e.c.Cassandra - INFO [StorageServiceShutdownHook] 2019-05-29 07:50:06,076 HintsService.java:220 - Paused hints dispatch
[info] c.g.n.e.c.l.WindowsCassandraNode - Successfully sent ctrl+c to process with id: 7276.
[info] c.g.n.e.c.l.WindowsCassandraNode - Apache Cassandra Node '7276' is stopped
[info] c.g.n.e.c.l.LocalCassandraDatabase - Apache Cassandra '3.11.1' is stopped (3490 ms)
[info] c.g.n.e.c.l.LocalCassandraDatabase - The working directory 'C:\Users\manu\AppData\Local\Temp\embedded-cassandra\3.11.1\0d155e04-97d5-4927-87ac-d46824a77c32' was deleted.
[debug] c.g.n.e.c.t.TestCassandra - TestCassandra 3.11.1 is stopped
Unable to start TestCassandra 3.11.1
com.github.nosan.embedded.cassandra.CassandraException: Unable to start TestCassandra 3.11.1
at com.github.nosan.embedded.cassandra.test.TestCassandra.start(TestCassandra.java:128)
at UnitSpecs.RepositorySpecs.UsersRepositorySpecs.$anonfun$new$3(UsersRepositorySpecs.scala:146)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
Ideally I should get the error similar to what I would get if I was using cqlsh
Is there a way to get more descriptive errors?
I have tried to reproduce your issue, but no luck.
import com.github.nosan.embedded.cassandra.cql.CqlScript;
import com.github.nosan.embedded.cassandra.test.TestCassandra;
class Scratch {
public static void main(String[] args) {
TestCassandra testCassandra = new TestCassandra(CqlScript.statements(createKeyspace(),
createUserTable()));
testCassandra.start();
try {
System.out.println(testCassandra.getSettings());
}
finally {
testCassandra.stop();
}
}
private static String createUserTable() {
return "CREATE TABLE users ( bucket int, "
+ "email text, "
+ "firstname text, "
+ "lastname text, "
+ "authprovider text, "
+ "password text, "
+ "confirmed boolean, "
+ "id UUID, hasher text, "
+ "salt text, "
+ "PRIMARY KEY ((bucket, email), authprovider,firstname, lastname) )";
}
private static String createKeyspace() {
return "CREATE KEYSPACE test WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 1}";
}
}
Output:
Exception in thread "main" com.github.nosan.embedded.cassandra.CassandraException: Unable to start TestCassandra 3.11.4
at com.github.nosan.embedded.cassandra.test.TestCassandra.start(TestCassandra.java:156)
at com.github.nosan.embedded.cassandra.Scratch.main(Scratch.java:27)
Caused by: com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename
at com.datastax.oss.driver.api.core.servererrors.InvalidQueryException.copy(InvalidQueryException.java:48)
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:113)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:207)
at com.datastax.oss.driver.api.core.CqlSession.execute(CqlSession.java:47)
at com.datastax.oss.driver.api.core.CqlSession.execute(CqlSession.java:56)
at com.github.nosan.embedded.cassandra.test.util.CqlUtils.execute(CqlUtils.java:68)
at com.github.nosan.embedded.cassandra.test.util.CqlUtils.execute(CqlUtils.java:47)
at com.github.nosan.embedded.cassandra.test.util.CqlSessionUtils.execute(CqlSessionUtils.java:43)
at com.github.nosan.embedded.cassandra.test.CqlSessionConnection.execute(CqlSessionConnection.java:60)
at com.github.nosan.embedded.cassandra.test.DefaultConnection.execute(DefaultConnection.java:53)
at com.github.nosan.embedded.cassandra.test.TestCassandra.executeScripts(TestCassandra.java:256)
at com.github.nosan.embedded.cassandra.test.TestCassandra.doStart(TestCassandra.java:285)
at com.github.nosan.embedded.cassandra.test.TestCassandra.start(TestCassandra.java:147)
I haven't been able to find my Caused isn't printed but I have found this workaround
try {
testCassandra.start()
println(s"cassandra state is ${testCassandra.getState}")
testCassandra.executeScripts(cqlStatements)
//println(s"result of execution is ${result}")
//val settings = testCassandra.getSettings
//println(s"settings are ${settings}")
} catch {
case e:Exception => {
println(s"exception ${e} caused by ${e.getCause}")
//println(s"caused by ${e.getCause()}")
fail( new Throwable(e.getCause))
}
}finally {
testCassandra.stop()
}
the above prints
org.scalatest.exceptions.TestFailedException was thrown.
ScalaTestFailureLocation: UnitSpecs.RepositorySpecs.UsersRepositorySpecs at (UsersRepositorySpecs.scala:157)
...
Caused by: java.lang.Throwable: com.datastax.driver.core.exceptions.InvalidQueryException: No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename
I found the reason. I wasn't using TestCassandra correctly it seems. I didn't realize that if I create TestCassandra and also specify the cql statements at time of instantiation, the start method runs the queries as well. In my code, I was creating TestCassandra as follows
new TestCassandra(factory,cqlStatements)})
and was calling both start and executeScripts
testCassandra.start()
testCassandra.executeScripts(cqlStatements)
I commented that executeScripts line and I now see both Exception and Caused
I think it would be better if the APIs clearly mention that start has side effect of executing statements as well.
I'm trying to convert HQL to Spark.
I have the following query (Works in Hue with Hive editor):
select reflect('java.util.UUID', 'randomUUID') as id,
tt.employee,
cast( from_unixtime(unix_timestamp (date_format(current_date(),'dd/MM/yyyy HH:mm:ss'), 'dd/MM/yyyy HH:mm:ss')) as timestamp) as insert_date,
collect_set(tt.employee_detail) as employee_details,
collect_set( tt.emp_indication ) as employees_indications,
named_struct ('employee_info', collect_set(tt.emp_info),
'employee_mod_info', collect_set(tt.emp_mod_info),
'employee_comments', collect_set(tt.emp_comment) )
as emp_mod_details,
from (
select views_ctr.employee,
if ( views_ctr.employee_details.so is not null, views_ctr.employee_details, null ) employee_detail,
if ( views_ctr.employee_info.so is not null, views_ctr.employee_info, null ) emp_info,
if ( views_ctr.employee_comments.so is not null, views_ctr.employee_comments, null ) emp_comment,
if ( views_ctr.employee_mod_info.so is not null, views_ctr.employee_mod_info, null ) emp_mod_info,
if ( views_ctr.emp_indications.so is not null, views_ctr.emp_indications, null ) employees_indication,
from
( select * from views_sta where emp_partition=0 and employee is not null ) views_ctr
) tt
group by employee
distribute by employee
First, What I'm trying is to write it in spark.sql as follow:
sparkSession.sql("select reflect('java.util.UUID', 'randomUUID') as id, tt.employee, cast( from_unixtime(unix_timestamp (date_format(current_date(),'dd/MM/yyyy HH:mm:ss'), 'dd/MM/yyyy HH:mm:ss')) as timestamp) as insert_date, collect_set(tt.employee_detail) as employee_details, collect_set( tt.emp_indication ) as employees_indications, named_struct ('employee_info', collect_set(tt.emp_info), 'employee_mod_info', collect_set(tt.emp_mod_info), 'employee_comments', collect_set(tt.emp_comment) ) as emp_mod_details, from ( select views_ctr.employee, if ( views_ctr.employee_details.so is not null, views_ctr.employee_details, null ) employee_detail, if ( views_ctr.employee_info.so is not null, views_ctr.employee_info, null ) emp_info, if ( views_ctr.employee_comments.so is not null, views_ctr.employee_comments, null ) emp_comment, if ( views_ctr.employee_mod_info.so is not null, views_ctr.employee_mod_info, null ) emp_mod_info, if ( views_ctr.emp_indications.so is not null, views_ctr.emp_indications, null ) employees_indication, from ( select * from views_sta where emp_partition=0 and employee is not null ) views_ctr ) tt group by employee distribute by employee")
But I got the following exception:
Exception in thread "main" org.apache.spark.SparkException: Job
aborted due to stage failute: Task not serializable:
java.io.NotSerializableException:
org.apache.spark.unsafe.types.UTF8String$IntWrapper
-object not serializable (class : org.apache.spark.unsafe.types.UTF8String$IntWrapper, value:
org.apache.spark.unsafe.types.UTF8String$IntWrapper#30cfd641)
If I'm trying to run my query without collect_set function its work, It's can fail because struct column types in my table?
How can I write my HQL query in Spark / fix my exception?