I'm concerned about security of database passwords in sqoop batches (no interactive input).
In the old days, for a sqoop batch, the only thing you could do was to pass it on the command line using --password, but then the password was easy to read with a simple ps command.
Now we have that --password-file option, but it requires to store the password unencrypted on the disk and that's not really a "secure" practice, nor is it very convenient to have individual files for individual parameters.
I was thinking of storing the encrypted password in a configuration file, and dynamically decrypt it, store it in a temporary file, setting the rights (using a chmod command), calling sqoop, and then deleting the file... But I may miss a less cumbersome way ? How do you deal with it ?
#abeaamase has the best answer at this time in his stackoverflow 23916985 response from 3/2015 found here. Essentially, we should upgrade to sqoop > 1.4.5 and use a java keystore (JKES), org.apache.sqoop.util.password.CryptoFileLoader, or a loader defined in our own class.
The provided CryptoFileLoader has the disadvantage of presuming the crypto passphrase and salt will be provided as -D parameters to drive system properties (which are open to snooping via ps) or in plain text in the configuration XML.
I initially discovered these parameters in this blog from 3/2015 having missed it earlier (it lacks a heading but you'll find it if you look at step 3 part 2).
Surprisingly, it is not the recommended practice, and does not appear in the sqoop 1.4.5 docs.
Before the availibilty of the --password-file option, I made a patch for sqoop to read the password from in input stream in a non interactive way when using the -P command.
That way, I could unencrypt the password from a configuration file, and call sqoop with that password using a stdin pipe, without using a file or a command line where the plain password could be seen.
Edit file src/java/org/apache/sqoop/SqoopOptions.java
Replace the securePasswordEntry function code by
private String securePasswordEntry() {
try {
return new String(System.console().readPassword("Enter password: "));
}
// PATCH Bouygues Telecom - read password from pipe if launched in non-interactive mode
catch (NullPointerException e) {
try {
final java.io.BufferedReader reader = new java.io.BufferedReader(
new java.io.InputStreamReader(System.in));
return reader.readLine();
}
catch (java.io.IOException excep) {
LOG.error("It seems that you have launched a Sqoop metastore job via");
LOG.error("Oozie with sqoop.metastore.client.record.password disabled.");
LOG.error("But this configuration is not supported because Sqoop can't");
LOG.error("prompt the user to enter the password while being executed");
LOG.error("as Oozie tasks. Please enable sqoop.metastore.client.record");
LOG.error(".password in sqoop-site.xml, or provide the password");
LOG.error("explicitly using --password in the command tag of the Oozie");
LOG.error("workflow file.");
}
return null;
}
}
What is cumbersome is to have to re-patch every new release of Sqoop... I should maybe submit a jira (with a low confidence that my patch will be taken into account), or move to the --password-file option the way you wanted to.
Related
I try to manage a user password files with bcrypt hashed passwords. The password hashes will be created via puppet on each run.
So now i have the problem that the file content of the user file is changed on each run.
My first idea was to manage a check file which contains a md5sum of the complete user hash to monitor any change on it. And i try to change the user file only if the check file was changed.
But this does not work. This is because puppet file resources doesn't react on refresh events i guess?
This was my last try with an exec as extra instance.
$md5sum = md5(String($input_users_hash))
file { "${filename}.serial" :
ensure => present,
content => $md5sum,
notify => Exec['md5sum_check']
}
exec { 'md5sum_check':
command => '/bin/true',
unless => "test `cat /opt/tomcat/shared/classes/users.properties.serial` == ${md5sum}",
notify => File[$filename],
}
file { $filename :
content => template($users_properties_template),
require => Exec['md5sum_check']
}
Any useful hints to solve this and to prevent a flapping resource?
In this case, my recommendation is that you store the bcrypt hashes in a hiera yaml on puppet server, and they are generated by something other than puppet (i.e. during the point where you create the new users).
If that isn't possible, and your avoiding directory service account management, then you need to find a bcrypt implementation that lets you choose set the salt. Then for each username, you can hash it, and use it to stablize the salt for a given user, which should stabilize the passwourd output.
Is there any fast way to remove all data from the local database? Like SQL 'drop database'?
I was looking through the documentation but haven't found anythig interesting yet.
The "CLI" way
Using the provided fdbcli interface, you can clear all the keys in the database using a single clearrange command, like this:
fdb> writemode on
fdb> clearrange "" \xFF
Committed (68666816293119)
Be warned that it executes instantly and that there is no undo possible!
Also, any application still connected to the database may continue reading/writing data using cached directory subspace prefixes, which may introduce data corruption! You should make sure to only use this method when nothing is actively using the cluster.
This method requires that your cluster be in a working state, and it will not immediately reclaim the space used on disk, and also will not reset the cluster's read version.
The "hard" way
If you have a single-node cluster, you can stop the fdb service, remove all files in its data_dir folder, restart the service, and then using fdbcli, execute the configure new single ssd command.
This will reclaim the disk space used previously, and reset everything back to the post-install state.
You can do this by clearing the entire range of keys.
In Python, it looks like this:
Database.clear_range('', '\xFF')
Where '' is the default slice begin, and '\xFF' is the default slice end, according to the clear_range documentation.
You can find the more information on clear_range for the API you're using in the documentation.
To do this programmatically in Java:
db.run(tx -> {
final byte[] st = new Subspace(new byte[]{(byte) 0x00}).getKey();
final byte[] en = new Subspace(new byte[]{(byte) 0xFF}).getKey();
tx.clear(st, en);
return null;
});
I want to create a user for a service (postgres, rabbitmq...) using a random generated password. This password should then be written to a file on the host. This file, containing env vars is then used by an application to connect to those services.
I don't want to store these passwords elsewhere.
postgresql::server::db { $name:
user => $name,
password => postgresql_password($name, random_password(10)),
}
Then i want to insert this password in the form PG_PASS='the same password' into a config file but the whole thing should happen only if the user is not already present.
In pure Puppet
A trick is to define a custom type somehow like :
define authfile($length=24,$template,$path) {
$passwordfile = "/etc/puppet/private/${::hostname}/${::title}"
$password = file($passwordfile,'/dev/null')
##exec { "generate-${title}":
command => "openssl rand -hex -out '$passwordfile' 24",
creates => $passwordfile,
tag => 'generated_password'
}
file { $path:
content => template($template);
}
}
And on your puppetmaster, have something like :
Exec<|| tag = 'generated_password' ||>
You can then pass in the $template variable the name of a template that will have the variable $password available. You will need to be careful about how these authfile types are defined (as it creates files on the puppetmaster, you will want to take care of malicious facts), and you will need to run puppet once on the host (so that the exported resources is created), once on the puppetmaster (so that the secret file is generated), then once again on the host (so that the secret file is read) for it to work.
With a custom function
Another solution is to write a custom function, random_password, that will use the fqdn and sign it with a secret that is stored on the puppetmaster (using a HMAC) to seed the password. That way you will have a password that can't be guessed without getting the secret, and no extra puppet roundtrips.
I haven't tried it myself yet. But trocla looks exactly like what you're looking for. Here's a little intro.
EDIT: After having now tried out trocla I can tell you that it works like a charm :-)
You need to have the trocla module installed to use it from puppet.
I am attempting to connect to a firebird database running in windows from a linux client, and when trying to attache the database I get the following error:
bad parameters on attach or create database, CHARACTER SET UTF8 is not defined
I have googled and searched the answers here but can't seem to find a solution.
Any suggestions on how this can be overcome from the client side, or does it require the databse to be rebuilt with UTF8 support ?
Client-side I am using node-js with the node-firebird module, server side engine version is 2.5, ODS Version is 11.2
function dbConnect(cb){
fb.attach({
host: '192.168.42.233',
database: 'gi',
user: 'SYSDBA',
password: 'xxxxx'
}, function(err, db){
if (err) console.log(err.message);
else cb(db);
})
}
Harriv's comment prompted me to test my idea that this might be caused by a missing entry in RDB$CHARACTER_SETS. If I manually delete UTF8 from this system table I get the same error when I try to connect with UTF8:
SQL Message : -924
Connection error
Engine Code : 335544325
Engine Message :
bad parameters on attach or create database
CHARACTER SET UTF8 is not defined
The solution is to backup the database and restore it again. That will recreate the RDB$CHARACTER_SETS system table to include UTF8 again.
Note that this will only solve your problem if UTF8 is missing from RDB$CHARACTER_SETS. If it does, you should ask yourself why it was missing in the first place. Maybe a DBA or other developer deleted entries from RDB$CHARACTER_SETS, if so it is probably a good idea to find out why that was done.
For example: Maybe the database uses NONE as the default characterset (and for every column), and this was a way to ensure people only connect with the 'right' characterset by deleting all other options (a normal Firebird 2.5 database contains 52 entries in RDB$CHARACTER_SETS). If that is the case, make sure you fix this issue before connecting with UTF8, otherwise you will probably experience some form of data corruption (on read or on write) by incorrect transliteration.
The fix for this would be to create a new database with the right default characterset (and for the columns), and pump the data from the old to the new (making sure the read and write is done with the appropriate character set). After that Firebird can take care of transliterating between the database (or column) character set and the connection character set (with the exception of applications connecting using NONE as the connection character set).
I added to options encoding key 'NONE':
options.encoding = 'NONE'
I have an account on a coldfusion/SQL Server shared server hosting service and have been targeted by what I believe to be a SQL injection attack. I have a absolute path to a javascript file in nearly every field of every table. It seems the script hides majority of the text and features on every page it is on. I'm lucky enough to be able to restore the database to a reasonable point, but I need to prevent this from happening again. Is there anything else I can do besides <cfqueryparam></cfqueryparam> ?
The recommended solution is to review ALL your code and make sure that you don't build your sql statements via string concatenation. Instead, use parametrized queries and sanitize user input. That's recommended for any kind of database you use.
Take a look at this post.
EDIT - Providing C# Example as requested:
string sql = "insert into table_a (cola,colb,colc) values (#value_a,#value_v,#value_c)";
using(SqlConnection conn = new SqlConnection("ConnectionString"))
{
conn.Open();
SqlCommand command = new SqlCommand(sql,conn);
command.CommandType = Commandtype.Text;
command.Parameters.AddWithValue("#value_a",txtFieldAFromForm.Text);
command.Parameters.AddWithValue("#value_b",txtFieldBFromForm.Text);
command.Parameters.AddWithValue("#value_c",txtFieldCFromForm.Text);
command.ExecuteNonQuery();
}
Checkout <cfqueryparams> tag: http://www.adobe.com/livedocs/coldfusion/7/htmldocs/wwhelp/wwhimpl/common/html/wwhelp.htm?context=ColdFusion_Documentation&file=part_cfm.htm
This is how to clean up user-submitted input for SQL queries.