how to start druid from puppet script - puppet
I am trying to run druid on a local vagrant machine. I use puppet to get archives, extract them etc. However I get a problem when trying to run historical and overlord node.
I use following code to start servers:
file_line { "configure_historical_server":
path => '/usr/share/druid-services-0.6.160/config/historical/runtime.properties',
line => 'druid.extensions.coordinates=["io.druid.extensions:druid-s3- extensions:0.6.147","io.druid.extensions:druid-hdfs-storage:0.6.147"]',
match => '^druid.extensions.coordinates*',
require => [ Exec["run_coordinator"] ],
}
exec { "run_historical":
cwd => "/usr/share/druid-services-0.6.160/",
command => "nohup java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/*:/usr/lib/hadoop/client/*:config/historical io.druid.cli.Main server historical&",
path => ["/bin", "/usr/bin"],
require => [ File_Line["configure_historical_server"] ],
}
file_line { "configure_overlord_server":
path => '/usr/share/druid-services-0.6.160/config/overlord/runtime.properties',
line => 'druid.extensions.coordinates=["io.druid.extensions:druid-kafka-seven:0.6.147","io.druid.extensions:druid-hdfs-storage:0.6.147"]',
match => '^druid.extensions.coordinates*',
require => [ Exec["run_historical"] ],
}
exec { "run_overlord":
cwd => "/usr/share/druid-services-0.6.160/",
command => "nohup java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/*:/usr/lib/hadoop/client/*:config/overlord io.druid.cli.Main server overlord&",
path => ["/bin", "/usr/bin"],
require => [ File_Line["configure_overlord_server"] ],
}
but both overlord and historical server fails due to the following error:
Caused by: java.io.FileNotFoundException: /home/vagrant/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.3.0/aether-e687f19b-733b-4348-a06f-e67797a26748-hadoop-hdfs-2.3.0.jar-in-progress (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:139)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:214)
at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.rename(AetherRepositoryConnector.java:624)
at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:404)
at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:232)
... 8 more
any idea how to fix this? when I start those servers from command line one after another (I wait until historical is started then I start overlord) everything works fine.
Related
MSSQL JDBC Driver library path is not recognized when using ~/ when running Logstash manually
Currently trying to populate the employee index with the below settings:
CONF
input {
jdbc {
jdbc_driver_library => "~/sqljdbc_6.2/enu/mssql-jdbc-6.2.1.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://SERVER;user=USER;password=PASSWORD"
jdbc_user => "DB_USER"
jdbc_password => "DB_PASSWORD"
jdbc_validate_connection => true
jdbc_validation_timeout => -1
statement => "SELECT * FROM [dbo].Employee ORDER BY ID"
type => "employee"
}
}
filter {
}
output {
}
NOTE: filter and output sections of the conf file is purposely blank
LINUX COMMAND
sudo /usr/share/logstash/bin/logstash -f /home/ubuntu/Employee-pipeline.conf --path.settings /etc/logstash/ --path.data /var/lib/logstash_new
RESULT
Looks like logstash does not know or don't have access to ~/sqljdbc...*.jar
I also confirmed that the mssql-jdbc-6.2.1.jre8.jar exists
However, when I changed the path to /home/ubuntu/sqljdbc_6.2/enu/mssql-jdbc-6.2.1.jre8.jar, it runs successfully.
So ~/ is the same as /home/ubuntu
This started to occur after upgrading our Elastic Stack from v5.5 to v5.6. Also, note that this does not occur if we run the same conf file with the logstash service.
Puppet : exec[] : wget returned 8 instead of 0
I am completely new to Puppet and this is my first time writing code in puppet. I want to get a tar.gz file and then untar it to create the folder.
Here is my code:
file{ "${::filename}.tar.gz":
ensure => 'file',
mode => '0644',
notify => Exec['untar-file'],
}
exec{ 'download-file' :
command => "wget URL_FOR_TAR_GZ",
cwd => "PATH_WHERE_TO_STORE",
user => "my_name",
group => "our company name",
}
exec { 'untar-file':
command => "/bin/tar -xzvf tar_file_name",
cwd => "file_path",
creates => "foldername_to_be_createdc",
user => "my_name",
group => "our company name",
require => Exec['download-file']
}
As soon as I run this I get an error:
wget returned 8 instead of one of [0]" and "/Exec[download-file]/returns: change from notrun to 0 failed"
Where am I going wrong?
puppet not able to run shell script from master to client
I am new to puppet. I want to run a shell script call crfs.sh located under /myscripts on a RHEL linux puppet master server. How do I execute this script on a client or target server?
What you want can be solved using the file and the exec modules of puppet.
class mymodule::myclass {
file { 'my_bash_script':
ensure => 'file',
source => 'puppet:///modules/mymodule/my_bash_script.sh',
path => '/usr/local/bin/my_bash_script.sh',
owner => 'root'
group => 'root'
mode => '0744', # Use 0700 if it is sensitive
notify => Exec['run_my_script'],
}
exec { 'run_my_script':
command => '/usr/local/bin/my_bash_script.sh',
refreshonly => true,
}
}
Sequence of Execs in Puppet
I have a sequence of exec in my Puppet manifest:
The first one downloads ZIP file with binary (unless the binary has already been installed) and saves it to /tmp.
The second one unzips it.
When I apply the manifest for the first time, it works correctly. However, when I clean my /tmp and apply the manifest again, it fails because the first exec doesn't executed (that is correct), but the second still tries to execute and fails because ZIP file is not present.
How do I modify the manifest to skip the second exec if the first one doesn't download file?
exec { 'ngrok-download':
command => 'wget https://dl.ngrok.com/linux_386/ngrok.zip -O /tmp/ngrok.zip',
unless => 'which ngrok',
path => ['/bin', '/usr/bin'],
}
exec { 'ngrok-unzip':
command => 'unzip ngrok.zip',
cwd => '/tmp',
path => ['/usr/bin'],
require => Exec['ngrok-download'],
}
Try this:
exec { 'ngrok-download':
command => 'wget https://dl.ngrok.com/linux_386/ngrok.zip -O /tmp/ngrok.zip',
unless => 'which ngrok',
path => ['/bin', '/usr/bin'],
notify => Exec['ngrok-unzip'],
}
exec { 'ngrok-unzip':
command => 'unzip ngrok.zip',
cwd => '/tmp',
path => ['/usr/bin'],
refreshonly => true,
require => Exec['ngrok-download'],
}
This will result in the unzip exec only running when the wget exec actually does something -- which it won't if ngrok is found.
Normally I would wget it to a more permanent location and leave it there. Then instead of the unless => 'which ngrok' check, replace with creates => '/path/to/zip.file'. The result being as long as the file is still there, none of the execs fire.
Comes in handy when you version the zip files and want to change versions.
You could also try easier approach:
exec { 'ngrok-download':
command => 'wget https://dl.ngrok.com/linux_386/ngrok.zip -O /tmp/ngrok.zip',
unless => 'which ngrok',
path => ['/bin', '/usr/bin'],
} ~>
exec { 'ngrok-unzip':
command => 'unzip ngrok.zip',
cwd => '/tmp',
path => ['/usr/bin'],
refreshonly => true,
}
Where Exec['ngrok-download'] notifies Exec['ngrok-unzip'] if applied and Exec['ngrok-unzip'] refresh its state only if needed
Same thing can be achieved by doing following:
exec { 'ngrok-download':
command => 'wget https://dl.ngrok.com/linux_386/ngrok.zip -O /tmp/ngrok.zip',
unless => 'which ngrok',
path => ['/bin', '/usr/bin'],
}
exec { 'ngrok-unzip':
command => 'unzip ngrok.zip',
cwd => '/tmp',
path => ['/usr/bin'],
refreshonly => true,
}
Exec['ngrok-download'] ~> Exec['ngrok-unzip']
Hope this helps.
Puppet and composer project
A small puppet question
I am creating a composer project like so.
composer::project { 'project-test':
ensure => 'latest', #or installed?
target => '/home/test/www',
dev => false,
require => Package ['php', 'apache']
}
And then from an exec I want to require it as a resource. How can I?
Example of exec:
exec { 'generate-tests' :
command => 'php tests.php',
path => '/usr/bin/',
cwd => "/home/test/www/bin",
logoutput => 'true',
#require => composer::project['project-test']
}
Since what you're requiring is a resource, it should be capitalized as follows: require => Composer::Project['project-test']