I do a number of projects that involve automatically submitting forms and/or retrieving data from websites. Some of these sites require username/password authentication. (These sites do not have APIs, so I am relying on screen scraping.)
Most of the tutorials I've seen store the username and password in the source code like any other POST data, e.g.:
string username = "someUserName";
string password = "somePassword";
// submit POST data...
But I know storing passwords in plain text is generally frowned upon. Is there an alternative method I should use?
The common way of storing a password is by hashing it. As most algorithms for hashing passwords are destructive, that is they can't be reversed, this wouldn't work for you.
An option would be to use a reversible hash, such as to base64 encode the password, but it isn't really a lot safer than storing it in plain text.
The best solution as far as I can see, would be to store the passwords in a database. If you are really worried about someone getting the usernames and passwords, you could encrypt them in the DB with encryption functions, or you could use a SQLite database which you would encrypt directly on the disk.
This way your code and login credentials are separated, and you can safely share your code with others without worrying about security.
A pattern we use is:
In your database table you have an encrypted column. This column contains data encrypted with a system-wide, long (128 bit), random secret key (usually stored in a configuration file).
The data in this encrypted column contains a separate (random) secret key used for each thirdparty service. With this password we encrypt the authentication details related to this thirdparty service.
Why this double encrypting?
You reduce the amount of passwords in plain text to a single one (the system-wide password). Because of this, key management is easier.
We create a long random secret key for each thirdparty service so that we can selectively decrypt the credentials for each thirdparty service and transfer them between systems if necessary. Having one of our secret keys stored outside the database also reduces the risk associated with both SQL-injection attacks (they 'only' get the database data) and with backups (configuration files are not included in the regular backup data).
The weakness is obviously the system-wide password. It needs to be in memory somewhere.
I'm no cryptographer, and I'm pretty sure the above is sub-optimal. However, it works, is manageable and lot safer than just storing the thirdparty service credentials in plain text.
I have a scraping project that needed this problem solved. My setup includes two separate servers. The first is the user front end web app. the second is a nodejs server that handles the scraping.
I handle encryption with openssl key pair encryption. I generate a key pair for the nodejs machine and give the public key to the front end web app. When a user registers their 3rd party credentials those credentials are encrypted with the public key and stored in a database.
The web app regularly selects a user's encrypted credentials and sends them to the node server where they are decrypted with the private key and used with the 3rd party for scraping.
After a quick search I found this article about using openssl and encrypting strings.
I realize this is a very old post but hopefully it helps the next person that stumbles onto this problem.
A very simple way to encrypt and decrypt is extended tiny encription algorithm (XTEA). I'm pasting the C++ code from wikipedia here, but keep in mind anyone could have changed it there.
#include <stdint.h>
/* take 64 bits of data in v[0] and v[1] and 128 bits of key[0] - key[3] */
void encipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
unsigned int i;
uint32_t v0=v[0], v1=v[1], sum=0, delta=0x9E3779B9;
for (i=0; i < num_rounds; i++) {
v0 += (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
sum += delta;
v1 += (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
}
v[0]=v0; v[1]=v1;
}
void decipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
unsigned int i;
uint32_t v0=v[0], v1=v[1], delta=0x9E3779B9, sum=delta*num_rounds;
for (i=0; i < num_rounds; i++) {
v1 -= (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
sum -= delta;
v0 -= (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
}
v[0]=v0; v[1]=v1;
}
You have to do two things:
1. Use HTTPS for login pages (if necessary)
2. Use password Encryption right after receiving it. An encoder is something like this:
private static String passwordEncryption(String oldPass){
String newPass = "";
try {
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
messageDigest.update(oldPass.getBytes(), 0, oldPass.length());
newPass = new BigInteger(1,messageDigest.digest()).toString(16);
if (newPass.length() < 32) {
newPass = "0" + newPass;
}
return newPass;
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
return newPass;
}
And use MD5() function of MySql to compare the received password with the stored one.
There's no way to do it. it will need to be available to the script somewhere as plain text (or "reversible encryption").
Many Apis (including Amazon Web Services for example) will recommend setting credentials in a environment variable and this is probably as much safety as you can hope for.
Put it in your .bash_profile, double check perrmissions, and at least you can be sure it won't end up on github in a public repo.
Related
I've have the following ElGamal encryption scheme
const forge = require('node-forge');
const bigInt = require("big-integer");
// Generates private and public keys
function keyPairGeneration(p, q, g) {
var secretKey = bigInt.randBetween(2, q.minus(2));
var publicKey = g.modPow(secretKey, p);
const keys = {
secret: secretKey,
public: publicKey
}
return keys;
}
// Generates a proxy and a user key
function generateProxyKeys(secretKey) {
const firstKey = bigInt.randBetween(1, secretKey);
const secondKey = secretKey.minus(firstKey);
const keys = {
firstKey: firstKey,
secondKey: secondKey
}
return keys;
}
// Re-encrypts
function preEncrypt(p, q, g, m, publicKey) {
const k = bigInt.randBetween(1, q.minus(1));
const c1 = g.modPow(k, p);
// g^x = publicKey
// m.publicKey^k
const c2 = bigInt(m).multiply(publicKey.modPow(k, p)).mod(p);
const c = {
c1: c1,
c2: c2
}
return c;
}
function preDecrypt(p, c1, c2, key) {
// (mg^xr) / (g^rx1)
var decrypt = c2.multiply(c1.modPow(key, p).modInv(p)).mod(p);
return decrypt;
}
Which works fine with numbers. However, I want to be able to use it to encrypt strings (btw, it's not a regular ElGamal, I don't think the difference is that relevant in this context but for more details see this question I asked)
I thought about converting the string to an integer, running the encryption, and converting back to a string whenever I needed it. I couldn't find a way of doing this in JS (there was this question posted here but the code didn't work). There is another similar question but it's in Java and the method mentioned there is not provided by the BigInt implementation in JS.
Is there any easy way of converting a string to a BigInt?
Arbitrarily long messages
Asymmetric encryption should not be used to encrypt messages of arbitrary length, because it is much slower than symmetric encryption. So, we can use symmetric encryption for the actual message and asymmetric encryption for the key that encrypted the message.
There are basically two ways for arbitrary sized messages:
If prime p is big enough that it fits a common key size of a symmetric cipher such as AES, then you can simply generate a random AES key (128, 192 or 256 bit) and use an AES-derived scheme such as AES-GCM to encrypt your message. Afterwards, you decode a number from the AES key (use fromArray) to be used as m in your ElGamal-like encryption scheme. This is called hybrid encryption.
Regardless how big prime p is, you can always generate a random m number in the range of 1 to p-1 and use that to produce your asymmetric ciphertext. Afterwards, you can take the previously generated m, encode it into a byte array (use toString(16) to produce a Hex-encoded string and then simply parse it as Hex for the hashing) and hash it with a cryptographic hash function such as SHA-256 to get your AES key. Then you can use the AES key to encrypt the message with a symmetric scheme like AES-GCM. This is called key encapsulation.
The main remaining thing that you have to look out for is data format: How do you serialize the data for the asymmetric part and the symmetric part of the ciphertext? How do you read them back that you can always tell them apart? There are many possible solutions there.
Short messages
If the messages that you want to encrypt have a maximum size that is smaller than the prime that you use, then you don't need the two approaches above. You just need to take the byte representation of the message and convert it to a big integer. Something like this:
var arr = Array.prototype.slice.call(Buffer.from("some message"), 0);
var message = bigInt.fromArray(arr, 256);
This is a big endian encoding.
This makes only sense if your prime is big enough which it should be for security.
I need to Generate Random Unique hashes to be used in password reset links in .Net Core. In ASP.NET Identity, I found this to be helpful, I am wondering if RNGCryptoServiceProvider still assures the randomness of the hashes generated.
using (RNGCryptoServiceProvider rngCsp = new RNGCryptoServiceProvider())
{
var data = new byte[4];
for (int i = 0; i < 10; i++)
{
//filled with an array of random numbers
rngCsp.GetBytes(data);
//this is converted into a character from A to Z
var randomchar = Convert.ToChar(
//produce a random number
//between 0 and 25
BitConverter.ToUInt32(data, 0) % 26
//Convert.ToInt32('A')==65
+ 65
);
token.Append(randomchar);
}
}
What is the best way to achieve that using .net core and using which classes?
RNGCryptoServiceProvider is not in .NET Core. Use RandomNumberGenerator.Create() to get an instance of a CSPRNG that will work on the correct platform. Its API is the same as RNGCryptoServiceProvider, except that GetNonZeroBytes is missing (which I'd argue shouldn't have even been there).
On Windows, this will boil down to BCryptGenRandom in CNG, and on Linux it will use OpenSSL.
This question already has answers here:
Is it possible to reverse a SHA-1 hash?
(11 answers)
Closed 8 years ago.
Is it possible to decrypt(retain the actual string) the password which is saved in db using SHA1 algorithm.
Example:If password is "password" and it is stored in db as "sha1$4fb4c$2bc693f8a86e2d87f757c382a32e3d50fc945b24",is any chance to retain the same "password"(string) from "sha1$4fb4c$2bc693f8a86e2d87f757c382a32e3d50fc945b24"
SHA1 is a cryptographic hash function, so the intention of the design was to avoid what you are trying to do.
However, breaking a SHA1 hash is technically possible. You can do so by just trying to guess what was hashed. This brute-force approach is of course not efficient, but that's pretty much the only way.
So to answer your question: yes, it is possible, but you need significant computing power. Some researchers estimate that it costs $70k - $120k.
As far as we can tell today, there is also no other way but to guess the hashed input. This is because operations such as mod eliminate information from your input. Suppose you calculate mod 5 and you get 0. What was the input? Was it 0, 5 or 500? You see, you can't really 'go back' in this case.
SHA1 is a one way hash. So you can not really revert it.
That's why applications use it to store the hash of the password and not the password itself.
Like every hash function SHA-1 maps a large input set (the keys) to a smaller target set (the hash values). Thus collisions can occur. This means that two values of the input set map to the same hash value.
Obviously the collision probability increases when the target set is getting smaller. But vice versa this also means that the collision probability decreases when the target set is getting larger and SHA-1's target set is 160 bit.
Jeff Preshing, wrote a very good blog about Hash Collision Probabilities that can help you to decide which hash algorithm to use. Thanks Jeff.
In his blog he shows a table that tells us the probability of collisions for a given input set.
As you can see the probability of a 32-bit hash is 1 in 2 if you have 77163 input values.
A simple java program will show us what his table shows:
public class Main {
public static void main(String[] args) {
char[] inputValue = new char[10];
Map<Integer, String> hashValues = new HashMap<Integer, String>();
int collisionCount = 0;
for (int i = 0; i < 77163; i++) {
String asString = nextValue(inputValue);
int hashCode = asString.hashCode();
String collisionString = hashValues.put(hashCode, asString);
if (collisionString != null) {
collisionCount++;
System.out.println("Collision: " + asString + " <-> " + collisionString);
}
}
System.out.println("Collision count: " + collisionCount);
}
private static String nextValue(char[] inputValue) {
nextValue(inputValue, 0);
int endIndex = 0;
for (int i = 0; i < inputValue.length; i++) {
if (inputValue[i] == 0) {
endIndex = i;
break;
}
}
return new String(inputValue, 0, endIndex);
}
private static void nextValue(char[] inputValue, int index) {
boolean increaseNextIndex = inputValue[index] == 'z';
if (inputValue[index] == 0 || increaseNextIndex) {
inputValue[index] = 'A';
} else {
inputValue[index] += 1;
}
if (increaseNextIndex) {
nextValue(inputValue, index + 1);
}
}
}
My output end with:
Collision: RvV <-> SWV
Collision: SvV <-> TWV
Collision: TvV <-> UWV
Collision: UvV <-> VWV
Collision: VvV <-> WWV
Collision: WvV <-> XWV
Collision count: 35135
It produced 35135 collsions and that's the nearly the half of 77163. And if I ran the program with 30084 input values the collision count is 13606. This is not exactly 1 in 10, but it is only a probability and the example program is not perfect, because it only uses the ascii chars between A and z.
Let's take the last reported collision and check
System.out.println("VvV".hashCode());
System.out.println("WWV".hashCode());
My output is
86390
86390
Conclusion:
If you have a SHA-1 value and you want to get the input value back you can try a brute force attack. This means that you have to generate all possible input values, hash them and compare them with the SHA-1 you have. But that will consume a lot of time and computing power. Some people created so called rainbow tables for some input sets. But these do only exist for some small input sets.
And remember that many input values map to a single target hash value. So even if you would know all mappings (which is impossible, because the input set is unbounded) you still can't say which input value it was.
Since SHA-1 maps several byte sequences to one, you can't "decrypt" a hash, but in theory you can find collisions: strings that have the same hash.
It seems that breaking a single hash would cost about 2.7 million dollars worth of computer time currently, so your efforts are probably better spent somewhere else.
I am trying to sign a message which contains in inBuffer byte array using my own derived key S (also byte array). The snippet of the function from javacard (jc) applet module is given below. I am using javacard2.2.2 library for developing jc applet. I am using android application for sending process request. I am reciving return code '6A81' which means 'function not supported'. Now, I have no clue that how to proceed as I failed to understand that it is mentioning about HMAC_SHA256 not supported or I am making some mistake in the function. Please help.
Signature m_sessionMAC = null;
HMACKey keyType = null;
Sign = new byte[64];
bytesRead = apdu.setIncomingAndReceive();
// Create HMAC Key Used in Mac
m_sessionMAC = Signature.getInstance(Signature.ALG_HMAC_SHA_256, false);
// Create HMAC Key Used in Mac
keyType = (HMACKey) KeyBuilder.buildKey(KeyBuilder.TYPE_HMAC, KeyBuilder.LENGTH_HMAC_SHA_256_BLOCK_64, false);
keyType.setKey(S,(short) 0, (short) S.length);
m_sessionMAC.init(keyType, Signature.MODE_SIGN);
//Generate Signature on inBuffer (received data to sign)
echoOffset = m_sessionMAC.sign(inBuffer, ISO7816.OFFSET_CDATA, ISO7816.OFFSET_LC, Sign , (short)0);
Util.arrayCopyNonAtomic(Sign, ( short ) 0, inBuffer, ( short ) 0, echoOffset);
apdu.setOutgoingAndSend( ( short ) 0, (short) echoOffset );
Please help me in this regards or also provide any pointers for implementing HMAC_SHA256 or HMAC_SHA1 symmetric crypto. in javacard applet.
Thank you in advance.
Most cryptographic algorithms are optional for a JavaCard. Therefore it may be that your card does not support Signature.ALG_HMAC_SHA_256. But HMAC algorithm isn't very complex therefore you should check if your card supports MessageDigest.ALG_SHA_256.
If it is supported you can follow RFC2104 and implement HMAC yourself:
K = HMAC key of length 32
ipad = the byte 0x36 repeated 32 times
opad = the byte 0x5C repeated 32 times.
To compute HMAC over the data `text' we perform
H(K XOR opad, H(K XOR ipad, text))
You can test your implementation by comparing your result with the test vectors noted in RFC 4231
In addition to Robert answer I'd like to highlight that you have to check CryptoException when you are calling getInstance() method. As Robert already mentioned algorithms could be optional hence it is good practice to check before like:
try {
Signature.getInstance(Signature.ALG_HMAC_SHA_256, false);
} catch (CryptoException e) {
if (e.getReason() == CryptoException.NO_SUCH_ALGORITHM) {
// Do something to treat algorithm absebce
}
}
I want to add a simple kind of MAC to some of my URL parameters. This is only intended as an additional line of defense against application bugs and caching related problems/bugs, and not intended as any form of replacement of the actual login security in the application. A given business-object-id is already protected by backends to be limited to a single user.
So basically I'd like to add a short authentication code to my url parameters, on the size of 2-4 characters. I think I'd like to have a reversible function along the lines of f(business-data-id + logged-on-user-id + ??) = hash, but I am open to suggestions.
The primary intention is to stop id guessing, and to make sure that url's are fairly distinct per logged on user. I also don't want something big and clunky like an MD5.
Since you aren't looking for cryptographic quality, maybe a 24-bit CRC would fit your needs. While MD5 is "fast" in absolute terms, CRC is, relatively, "blindingly fast". Then the 3-byte CRC could be text-encoded into four characters with Base-64 encoding.
Here's a Java implementation of the check used for OpenPGP ASCII-armor checksums:
private static byte[] crc(byte[] data)
{
int crc = 0xB704CE;
for (int octets = 0; octets < data.length; ++octets) {
crc ^= (data[octets] & 0xFF) << 16;
for (int i = 0; i < 8; ++i) {
crc <<= 1;
if ((crc & 0x1000000) != 0)
crc ^= 0x1864CFB;
}
}
byte[] b = new byte[3];
for (int shift = 16, idx = 0; shift >= 0; shift -= 8) {
b[idx++] = (byte) (crc >>> shift);
}
return b;
}
I would hash a secret key (which is known only by the server), together with whatever you want to protect—probably the combination of object identifier and user identifier.
If what you want is basically MD5 but smaller, why not just use MD5 but just the last 4 characters? This doesn't add a huge blob to your urls, it's always 4 nice hex digits.
A quick question for which I'm sure there's a good answer for, but why not store this information in a cookie?
Then you could use something big and clunky like MD5 and your URLs would still be pretty.