logstash input jdbc plugin Apache Drill - logstash

I'm trying to configure input for logstash 5 with Apache Drill JDBC (https://drill.apache.org/docs/using-the-jdbc-driver/)
Below is my input jdbc configuration for logstash.
input {
jdbc {
jdbc_driver_library => "jdbc_jars/drill-jdbc-all-1.10.0.jar"
jdbc_driver_class => "org.apache.drill.jdbc.Driver"
jdbc_connection_string => "jdbc:drill:zk=local"
jdbc_user=> "dfs"
schedule => "* * * * *"
statement => "select * from `sample.json`;"
}
}
I essentially get logstash WARN of "Failed test_connection". Hence, although logstash is launching, the DB connection is failing.
Any suggestions?

I see a few problems with your configuration.
You need to provide a valid IP address and port for a zookeeper node that Drill is using. The line you provided to logstash jdbc_connection_string => "jdbc:drill:zk=local" is telling logstash that zookeeper is running on the same node as logstash. What you need to provide instead is jdbc_connection_string => "jdbc:drill:zk=zk_hostname_or_ip:zk_port". Talk to the guy who setup your drill cluster to figure out the hostname or ip and port of your zookeeper node.
dfs is not a drill user, it is the name of one of Drill's storage plugins. If you want to run your query on a file stored on hdfs change
statement => "select * from `sample.json`;"
to
statement => "select * from dfs.`/path/to/sample.json`;"
If you do not have authentication configured for Drill your config should look like this.
input {
jdbc {
jdbc_driver_library => "jdbc_jars/drill-jdbc-all-1.10.0.jar"
jdbc_driver_class => "org.apache.drill.jdbc.Driver"
jdbc_connection_string => "jdbc:drill:zk=zk_hostname_or_ip:zk_port"
schedule => "* * * * *"
statement => "select * from `dfs./path/to/sample.json`;"
}
}
If you have authentication configured for Drill and you know your Drill username and password your config should look like this.
input {
jdbc {
jdbc_driver_library => "jdbc_jars/drill-jdbc-all-1.10.0.jar"
jdbc_driver_class => "org.apache.drill.jdbc.Driver"
jdbc_connection_string => "jdbc:drill:zk=zk_hostname_or_ip:zk_port"
schedule => "* * * * *"
statement => "select * from `dfs./path/to/sample.json`;"
jdbc_user => "myusername"
jdbc_password => "mypassword"
}
}

Related

Web3.js >> w3.eth.sendSignedTransaction the txn is submitted but not succesful

Description::
I've a signed txn and when I submit it using w3.eth.sendSignedTransaction , I see the following logs in my Geth log file
Geth logs are as follows:
INFO [05-24|12:01:44] Submitted transaction fullhash=0xd6ad180c709ce93f5884070f28488925e9b944a24fc6ab737c79d8e66dfd9dca recipient=0xF06c0a4A9fafddA7b8B25F986e1C0dfEC62e1E84
I obtain the txn hash as shown above but now when I try to search my txn using the hash , following is what I get >>
My question is :: Why the block hash is >> 0x0000 . . . . . ?
What could be wrong here ?
The code which is use to send this txn is as follows >>
w3.eth.sendSignedTransaction( data ).once( 'transactionHash', (hash) => {
console.log(hash)
}).on('receipt', (receipt) => {
console.log('receipt');
}).on('confirmation', (confirmationNumber, receipt) => {
console.log('confirmation');
}).on('error', (err) => {
console.log(err);
}).then( (receipt) => {
console.log('finally got the receipt!');
})
.catch(e => {
console.log('err');
})
I had the same issue.
This problem comes when our nonce is not one higher than the count of transactions from the account.
Check whether the following equation is true in your case >>
NONCE = count_of_transactions_from_account + 1
Hope that helps!!!

Bad GET URL when using the Logstash http_poller input plugin

Trying to pull data from a public API using the Logstash http_poller input plugin:
input {
http_poller {
urls => {
method => "GET"
url => "https://api.example.com/v1/service/"
}
request_timeout => 60
schedule => { cron => "0 * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter {
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
Keep on getting a bad get URL error:
[ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid URL GET>...]
Any idea what's causing this? The URL for the API is correct...
Turns out it was the method => "GET" line. Removing it worked like a charm.

How to defer query execution in NHibernate QueryOver

I am trying to execute 3 queries but want to execuate all at one database call. I gone through https://ayende.com/blog/3979/nhibernate-futures but it still executes each query individually. I am using QueryOver instead of CreateCriteria.
Can anyone help how to achieve this?
my query is like
var changedScriptsInHeader = _session.QueryOver<ProgramHeader>()
.Where(x => x.ModifiedTime.IsBetween(changedFrom).And(changedTo))
.Select(x => x.ScriptNumber)
.Future<string>();
var changedScriptsInDetail = _session.QueryOver<ProgramDetail>()
.Where(x => x.UpdatedDate.IsBetween(changedFrom).And(changedTo))
.SelectList(list => list.SelectGroup(pr => pr.ScriptNumber))
.Future<string>();
var changedScriptsInReplay = _session.QueryOver<ProgramReplay>()
.Where(x => x.UpdatedDate.IsBetween(changedFrom).And(changedTo))
.SelectList(list => list.SelectGroup(pr => pr.ScriptNumber))
.Future<string>();
Thanks

How to chain classes with commands

I have the following class which installs mysql and sets up a user called user but when the create-database commands runs the user has not been created yet.
How do i chain the commands so that the user is created before create-database tries to use it?
class { '::mysql::server':
package_name => 'mariadb-server.x86_64',
root_password => 'root',
remove_default_accounts => true,
override_options => $override_options,
restart => true,
users => {
'user#%' => {
ensure => 'present',
max_connections_per_hour => '0',
max_queries_per_hour => '0',
max_updates_per_hour => '0',
max_user_connections => '0',
password_hash => '...',
}
},
grants => {
'user#%/*.*' => {
ensure => 'present',
options => ['GRANT'],
privileges => ['ALL'],
table => '*.*',
user => 'user#%',
},
}
}->
exec { 'create-database':
creates => '/opt/dbinstalled',
command => '/usr/bin/mysql -u user -puser < /create-db.sql'
}
I am using the puppetlabs-mysql package to install mysql.
You should take a look at the documentation for the require, before, subscribe, notify metaparameters. They are used to describe resource ordering (before, notify), or resource ordering and failure if the dependency fails (require, subscribe). Note the subscribe, notify metaparameters are only available for some resource types (exec, service, etc.).
In this instance, you would do the following to chain a class:
exec { 'create-database':
creates => '/opt/dbinstalled',
command => '/usr/bin/mysql -u user -puser < /create-db.sql',
require => Class[::mysql::server],
}
But you really only need the dependency on the user resource:
exec { 'create-database':
creates => '/opt/dbinstalled',
command => '/usr/bin/mysql -u user -puser < /create-db.sql',
require => User[username or array of users],
}
Also you probably only want to create the database once, so we can give it a subscribe/refreshonly for idempotence:
exec { 'create-database':
creates => '/opt/dbinstalled',
command => '/usr/bin/mysql -u user -puser < /create-db.sql',
subscribe => User[username or array of users],
refreshonly => true,
}
Note that if you change the user resource that the create-database is subscribed to this will rerun the exec resource, so look into the unless, onlyif parameters for exec as other methods to establish idempotence.

Puppet: is there a way to catch failure to apply a resource?

is there a way in Puppet to catch a failure when resource is applied, for example, when declaration like
file { '/var/tmp/test':
ensure => file,
mode => '0755',
}
fails, invoke something like
exec { 'Register some failure':
command => '/var/tmp/register failure for /var/tmp/test',
}
?
You can try this :
exec { 'Notify a failure' :
command => "/var/tmp/register failure for /var/tmp/test",
path => "/bin:",
subscribe => File["/var/tmp/test"],
}

Resources