I am using simba cassandra jdbc driver to connect to cassandra. My jdbc url is: jdbc:cassandra://127.0.0.1:9042?ssl=true. How to disable the ssl validation, like in postgress we can do sslfactory=org.postgresql.ssl.NonValidatingFactory . I am looking for similar thing for cassandra. Any pointers would help
<dependency>
<groupId>com.simba.cassandra.jdbc42.Driver</groupId>
<artifactId>jdbc42</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/jar/CassandraJDBC42.jar</systemPath>
</dependency>
Related
I'm using Apache-Spark 3.2.3.
To connect to Hive JDBC, HiveServer2 is configured as http transport mode.
hive-site.xml:
<property>
<name>hive.server2.transport.mode</name>
<value>http</value>
<description>
Expects one of [binary, http].
Transport mode of HiveServer2.
</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'.</description>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
<description>Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'http'.</description>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
<description>Path component of URL endpoint when in HTTP mode.</description>
</property>
For Amazon QuickSight, "http" transport mode is failing. I tried ports 10000 and 10001.
If I change transportation mode to binary, QuickSight works with the port 10000. But now the hive jdbc connection fails.
I'M NOT USING CLOUDERA, but it this topic gives a good idea.
https://community.cloudera.com/t5/Support-Questions/hive-Enable-HTTP-Binary-transport-modes-in-HiveServer2/td-p/94401
Is it possible to configure Hive "config group" to allow multiple instances in hive-site.xml manually? or any other idea about how to configure Thrift in Apache-Spark to work with binary and http transportation mode at the same time?
Does anyone have a working example of Stork DNS Service Discovery working in Quarkus.... I see lots of examples using consul but not using DNS and I cannot get the properties or Inject beans when using DNS.
#ApplicationScoped
#RegisterRestClient(baseUri = "stork://auth-service/")
#RegisterProvider(OidcClientRequestFilter.class)
public interface KeycloakService {
I placed the following in my properties file but I get a compiler warning saying it is being ignored.
quarkus.stork.auth-service.service-discovery.type=dns
and my POM
<dependency>
<groupId>io.smallrye.stork</groupId>
<artifactId>stork-service-discovery-dns</artifactId>
<version>1.2.0</version>
</dependency>
The error I am getting is
Caused by: java.lang.IllegalArgumentException: The value of URL was invalid stork://auth-service
I use jooq-codegen-maven version <3.10.5>
It works with Postgresql 10.4 but doesn't work with PostgreSQL 9.4.6 gives warning:
[WARNING] No schemata were loaded : Please check your connection settings, and whether your database (and your database version!) is really supported by jOOQ. Also, check the case-sensitivity in your configured <inputSchema/> elements : {=[schema_name]}
Is there a compatibility table for jooq(code generator) and db versions?
My plugin configuration is:
<plugin>
<groupId>org.jooq</groupId>
<artifactId>jooq-codegen-maven</artifactId>
<version>3.10.5</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.1-901.jdbc3</version>
</dependency>
</dependencies>
<configuration>
<!-- JDBC connection parameters -->
<jdbc>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://X.X.X.106:5432/postgres</url>
<user>xxxx</user>
<password>xxxx</password>
</jdbc>
<!-- Generator parameters -->
<generator>
<name>org.jooq.util.DefaultGenerator</name>
<database>
<name>org.jooq.util.postgres.PostgresDatabase</name>
<includes>.*</includes>
<inputSchema>somedata</inputSchema>
<excludes></excludes>
</database>
<target>
<packageName>com.xxxx.xxxx.jooq.generated</packageName>
<directory>target/generated-sources/jooq</directory>
</target>
</generator>
</configuration>
</plugin>
I got the same error when I copy pasted the pom.xml plugin configuration from jooq website. Later I changed inputSchema in the pom.xml to the name of the database I created (It was 'public' ) before and it generated the code
I changed
<inputSchema>public</inputSchema>
to
<inputSchema>library</inputSchema>
where 'library' is the name of the database I created
In my case jdbc url's db part(jdbc:postgresql://X.X.X.106:5432/postgres) was wrong for my new database. So changing it with the right one solved my issue.
I have a need to be able to run spark on my local machine to access azure wasb and adl urls, but I can't get it to work. I have a stripped down example here:
maven pom.xml (Brand-new pom, only the dependencies have been set):
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.8.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure-datalake</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage</artifactId>
<version>6.0.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-data-lake-store-sdk</artifactId>
<version>2.2.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage</artifactId>
<version>7.0.0</version>
</dependency>
Java code (Doesn't need to be java - could be scala):
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.sql.SparkSession;
public class App {
public static void main(String[] args) {
SparkConf config = new SparkConf();
config.setMaster("local");
config.setAppName("app");
SparkSession spark = new SparkSession(new SparkContext(config));
spark.read().parquet("wasb://container#host/path");
spark.read().parquet("adl://host/path");
}
}
No matter what I try I get:
Exception in thread "main" java.io.IOException: No FileSystem for scheme: wasb
Same for adl. Every doc I can find on this either just says to add the azure-storage dependency, which I have done, or says to use HDInsight.
Any thoughts?
I figured this out and decided to post a working project since that is always what I look for. It is hosted here:
azure-spark-local-sample
The crux of it though is as #Shankar Koirala suggested:
For WASB, set the property to allow the url scheme to be recognized:
config.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem");
Then set the property which authorizes access to the account. You will need one of these for each account you need to access. These are generated through the Azure Portal under the Access Keys section of the Storage Account blade.
config.set("fs.azure.account.key.[storage-account-name].blob.core.windows.net", "[access-key]");
Now for adl, assign the fs scheme as with WASB:
config.set("spark.hadoop.fs.adl.impl", "org.apache.hadoop.fs.adl.AdlFileSystem");
// I don't know why this would be needed, but I saw it
// on an otherwise very helpful page . . .
config.set("spark.fs.AbstractFileSystem.adl.impl", "org.apache.hadoop.fs.adl.Adl");
. . . and finally, set the client access keys in these properties, again for each different account you need to access:
config.set("fs.adl.oauth2.access.token.provider.type", "ClientCredential");
/* Client ID is generally the application ID from the azure portal app registrations*/
config.set("fs.adl.oauth2.client.id", "[client-id]");
/*The client secret is the key generated through the portal*/
config.set("fs.adl.oauth2.credential", "[client-secret]");
/*This is the OAUTH 2.0 TOKEN ENDPOINT under the ENDPOINTS section of the app registrations under Azure Active Directory*/
config.set("fs.adl.oauth2.refresh.url", "[oauth-2.0-token-endpoint]");
I hope this is helpful, and I wish I could give credit to Shankar for the answer, but I also wanted to get the exact details out there.
I am not sure about the adl haven't tested but for the wasb you need to define the file system to be used in the underlying Hadoop configurations.
Since you are using spark 2.3 you can use spark session to create a entry point as
val spark = SparkSession.builder().appName("read from azure storage").master("local[*]").getOrCreate()
Now define the file system
spark.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
spark.sparkContext.hadoopConfiguration.set("fs.azure.account.key.yourAccount.blob.core.windows.net", "yourKey ")
Now read the parquet file as
val baseDir = "wasb[s]://BlobStorageContainer#yourUser.blob.core.windows.net/"
val dfParquet = spark.read.parquet(baseDir + "pathToParquetFile")
Hope this helps!
Configured secure HBase-1.1.2 with Hadoop-2.7.1 on Windows. When i enable authorization referring Configuring HBase Authorization, getting ERROR: DISABLED: Security features are not available exception.
I have set the authorization configurations as below,
Configuration
<property>
<name>hbase.security.authorization</name>
<value>true</value>
</property>
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
But HBase Authorization works fine when i tried with HBase-0.98.13 version. Some one help me to enable HBase Authorization in a correct way.
I was encountered with the same problem as I was not able to grant privileges to any other users. Mine was Kerberized Hadoop cluster I did following changes to make it work.
hbase.security.authentication=kerberos
hbase.security.authorization=true
Then re-deployed the configurations then it worked fine.
I was encountered with the same problem as I was not able to grant privileges to any other users. Mine was Kerberized Hadoop cluster.In addition to,My zookeeper was kerberized.So I do the following things:
firstly,you need stop your hbase.
Add the following to {$ZOOKEEPER_CONF_DIR}/jaas.conf:
Client{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/var/local/hadoop/zookeeper-3.4.8/conf/keytabs/hbase.keytab"
storeKey=true
useTicketCache=true
principal="hbase/zte1.zdh.com#ZDH.COM";
};
(My hbase principal is:hbase/zte1.zdh.com#ZDH.COM,username must be same)
then,use zkCli.sh command Line,next you can use: rmr /hbase to rmove the hbase directory,then start your hbase service,you will solve this problem.