I get these message when I start sonarqube.
INFO web[org.sonar.INFO] Security realm: Crowd
INFO web[c.a.c.s.c.ClientPropertiesImpl] Loading property: 'application.name' : 'sonar'
INFO web[c.a.c.s.c.ClientPropertiesImpl] Failed to find value for property: application.login.url
INFO web[c.a.c.s.c.ClientPropertiesImpl] Loading property: 'session.tokenkey' : 'session.tokenkey'
INFO web[c.a.c.s.c.ClientPropertiesImpl] Loading property:
'session.lastvalidation' : 'session.lastvalidation'
INFO web[c.a.c.s.c.ClientPropertiesImpl] Loading property: 'session.validationinterval' : '1'
INFO web[c.a.c.s.c.ClientPropertiesImpl] Failed to find value for property: cookie.domain
INFO web[o.s.p.c.CrowdRealm] Crowd configuration is valid, connection test successful.
INFO web[org.sonar.INFO] Security realm started
and Sonarqube start, but the users are only sonar and not crowd.
I just suppress and reinstall crowd plugin.
Before that, it was run correctly.
We faced similar trouble while upgrading. If Crowd 1.0 is still compatible please use it.
If you are going to use new crowd plugin remember that the type of user (admin/generic) has to be configured on the crowd server and not sonar.
Related
In my Spark-history some applications are "incomplete" for a week now. I've tried to kill them, close sparkContext(), kill main .py process, but nothing helped.
For example,
yarn application -status <id>
shows:
...
State: FINISHED
Final-State: SUCCEDED
...
Log Aggregation Status: TIME_OUT
...
But in Spark-History I still see it in incomplete section of my applications. If I open this application there, I can see 1 Active job with 1 Alive executor, but they are doing nothing for all week. This seems like a logging bug, but as I know this problem is only with me, other coworkers don't have this problem.
This thread doesn't helped me, because I dont have access to start-history-server.sh.
I suppose this problem because of
Log Aggregation Status: TIME_OUT
because my "completed" applications have
Log Aggregation Status: SUCCEDED
What can I do to fix this? Right now I have 90+ incomplete applications.
I've found clear description of my problem with same situation (yarn, spark, etc.), but there is no solution: What is 'Active Jobs' in Spark History Server Spark UI Jobs section
From Spark Monitoring and Instrumentation:
...
3. Applications which exited without registering themselves as completed will be listed as incomplete --even though they are no
longer running. This can happen if an application crashes.
...
Meaning:
History Server's UI shows only those Spark applications whose event logs it can find in its spark.eventLog.dir directory (a config typically set to /user/spark/applicationHistory in Hadoop). If a log doesn't end with the special ApplicationEnd event
:
{"Event":"SparkListenerApplicationEnd","Timestamp":1667223930402}
...the application is considered incomplete (even if it is no longer running) and will be displayed on the Incomplete Applications page.
To your question it means that "moving" application to the Completed Apps page won't be trivial, and will require manually editing eventlog and re-uploading it to SHS directory in Hadoop. Moreover, it won't solve anything, since most likely, your application keeps crashing before it can write that final message, and its next run will end up on the same Incomplete page again.
To diagnose the reason why it fails, perhaps you can look at the application driver logs for any clues -- errors or exception messages. Graceful shutdown looks different depending on what kind of resource manager and what deploy mode your app is using. For deploy-mode=cluster and YARN RM, it would look something like this:
:
22/10/31 11:11:11 INFO spark.SparkContext: Successfully stopped SparkContext
22/10/31 11:11:11 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
22/10/31 11:11:11 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
22/10/31 11:11:11 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
22/10/31 11:11:11 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://.../.../.sparkStaging/application_<appId>
22/10/31 11:11:11 INFO util.ShutdownHookManager: Shutdown hook called
22/10/31 11:11:11 INFO util.ShutdownHookManager: Deleting directory /.../.../appcache/application_<appId>/spark-<guid>
I am trying to use databricks connect to run the spark job on databricks cluster from intellj .I followed the below link documentation.
https://docs.databricks.com/dev-tools/databricks-connect.html
However I could not make it work with intellj and it throws below exception
21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
Exception in thread "main" java.lang.NoSuchFieldError: JAVA_9
at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:95)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:443)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:384)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:432)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:262)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:291)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:495)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2834)
I could not find a workaround this as the documentation does not say anything clearly I cross checked from intellj its pointed to correct jar directory returned by (databricks-connect get-jar-dir).Any clue on this will be helpful?
Note:databricks-connect test is returning success
I'm performing millions of operations using Google Dataproc with one problem, the logging data size.
I do not perform any show or any other kind of print, but the 7 lines of INFO, multiplied by millions gets a really big logging size.
Is there any way to avoid Google Dataproc from logging?
Already tried without success in Dataproc:
https://cloud.google.com/dataproc/docs/guides/driver-output#configuring_logging
These are the 7 lines I want to get rid off:
18/07/30 13:11:54 INFO org.spark_project.jetty.util.log: Logging initialized #...
18/07/30 13:11:55 INFO org.spark_project.jetty.server.Server: ....z-SNAPSHOT
18/07/30 13:11:55 INFO org.spark_project.jetty.server.Server: Started #...
18/07/30 13:11:55 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector#...
18/07/30 13:11:56 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: ...
18/07/30 13:11:57 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ...
18/07/30 13:12:01 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_...
What you are looking for is an exclusion filter: you need to browse from your Console to Stackdriver Logging > Logs ingestion > Exclusions and click on "Create exclusion". As explained there:
To create a logs exclusion, edit the filter on the left to only match
logs that you do not want to be included in Stackdriver Logging. After
an exclusion has been created, matched logs will no longer be
accessible in Stackdriver Logging.
In your case, the filter should be something like this:
resource.type="cloud_dataproc_cluster"
textPayload:"INFO org.spark_project.jetty.util.log: Logging initialized"
...
There are two approaches to control logging. One is via log4j.properties and another via controlling it programmatically. I have tried both:
Via log4j.properties file:
# disable logging for spark libraries
log4j.additivity.org=false
log4j.additivity.org.apache=false
#log4j.logger.org.apache=ERROR, NOAPPENDER
log4j.logger.org=ERROR, NOAPPENDER
and via programmatically:
org.apache.log4j.Logger logger = LogManager.getLogger(pkgName);
logger.setLevel(Level.ERROR);
I was able to suppress other logs but there are few INFO logs which are still getting printed:
INFO metastore: Connected to metastore.
INFO Hive: Registering function addfunc ca.nextpathway.hive.UDFToDate
and
INFO ContextHandler: Started o.s.j.s.ServletContextHandler#17f9344b{/static,null,AVAILABLE}
I want to suppress all the INFO logs except for few specific packages. But I think I am nowhere near it. If anyone knows what could be the problem here please let me know.
Try using the below. This should work.
Logger.getLogger("org.apache.hadoop.hive").setLevel(Level.ERROR);
The code
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java has a bug. It creates the LOg as below:
Logger LOG = LoggerFactory.getLogger("hive.ql.metadata.Hive");
So the regular filter with org.apache.hadoop.hive does not work. Instead, you have to use "hive.ql.metadata.Hive". For example:
org.apache.log4j.Logger.getLogger("hive.ql.metadata.Hive").setLevel(Level.WARN);
So I have configured and initialized accumulo as given in the Accumulo User Manual.
In the conf/accumulo-site.xml, I set my trace.user to accumulo_tracer, and I also create a system user with the same name. Additionally, I store the password in the same file under trace.token.property.password property.
However, after ./bin/start-all.sh, everything starts up fine, including the accumulo UI. But the following error is displayed in the logs/tracer_localhost.log file.
2015-05-28 10:58:46,229 [watcher.MonitorLog4jWatcher] INFO : Enabled log-forwarding
2015-05-28 10:58:46,254 [server.Accumulo] INFO : tracer starting
2015-05-28 10:58:46,254 [server.Accumulo] INFO : Instance 48f5f9cf-f08d-4736-b504-335b044a2d88
2015-05-28 10:58:46,255 [server.Accumulo] INFO : Data Version 6
2015-05-28 10:58:46,255 [server.Accumulo] INFO : Attempting to talk to zookeeper
2015-05-28 10:58:46,430 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS
2015-05-28 10:58:46,430 [server.Accumulo] INFO : Connected to HDFS
2015-05-28 10:58:46,432 [watcher.MonitorLog4jWatcher] INFO : Changing monitor log4j address to localhost:4560
2015-05-28 10:58:46,433 [watcher.MonitorLog4jWatcher] INFO : Enabled log-forwarding
2015-05-28 10:58:46,510 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2015-05-28 10:58:46,638 [tracer.TraceServer] INFO : Waiting to checking/create the trace table.
org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS for user accumulo_tracer - Username or Password is Invalid
at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:65)
at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:66)
at org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:156)
at org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:201)
at org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:303)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: ThriftSecurityException(user:accumulo_tracer, code:BAD_CREDENTIALS)
at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result$authenticate_resultStandardScheme.read(ClientService.java:15613)
at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result$authenticate_resultStandardScheme.read(ClientService.java:15591)
at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result.read(ClientService.java:15535)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticate(ClientService.java:500)
at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticate(ClientService.java:486)
at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:69)
at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:66)
at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:100)
at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:63)
... 10 more
2015-05-28 10:58:47,469 [server.Accumulo] WARN : System swappiness setting is greater than ten (60) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement.
Any help or guidance where I might have missed something, would be very helpful!
Thanks in advance!
You don't need to create a system user with that name. What you need is to create an Accumulo user with that name. You can do this in the accumulo shell as the (Accumulo) root user $ACCUMULO_HOME/bin/accumulo shell -u root. You'll also need to grant the user the table permission to create tables. See help grant in the shell to learn how to set the CREATE_TABLE system permission for that user.
So here's what solved my issue: a fresh init!!
So during the init phase, I give a password for the root user and the same password I am supposed to mention in the conf/accumulo-site.xml
<property>
<name>instance.secret</name>
<value>your_own_secret_password</value>
<description>A secret unique to a given instance that all servers must know in order to communicate with one another.
Change it before initialization. To
change it later use ./bin/accumulo org.apache.accumulo.server.util.ChangeSecret --old [oldpasswd] --new [newpasswd],
and then update this file.
</description>
</property>
After that everything just worked like a charm. Hope that helped. Post any queries, would be happy to help.