I'm working on a project which involves a lot of encrypted data. Basically, these are JSON objects serialized into a String, then encrypted with AES256 into a Cyphertext, and then have to be stored in Mongo.
I could of course do this the way described above, which will store the cyphertext as String into a BSON document. However, this way, if for some reason along the way the Cyphertext isn't treated properly (for instance, different charset or whatever reason), the cyphertext is altered and I cannot rebuild the original string anymore. With millions of records, that's unacceptable (it's also slow).
Is there a proper way to save the cyphertext in some kind of native binary format, retrieve it binary and then return it to the original string? I'm used to working with strings, my skills with binary format are pretty rusty. I'm very interested in hearing your thoughts on the subject.
Thanks everyone for your input,
Fabian
yes :)
var Binary = require('mongodb').Binary;
var doc = {
data: new Binary(new Buffer(256))
}
or with 1.1.5 of the driver you can do
var doc = {
data: new Buffer(256)
}
The data is always returned as a Binary object however and not a buffer. The link to the docs is below.
http://mongodb.github.com/node-mongodb-native/api-bson-generated/binary.html
Related
So with lots of different services around now, Google APIs, Twitter API, Facebook API, etc etc.
Each service has an API key, like:
AIzaSyClzfrOzB818x55FASHvX4JuGQciR9lv7q
All the keys vary in length and the characters they contain, I'm wondering what the best approach is for generating an API key?
I'm not asking for a specific language, just the general approach to creating keys, should they be an encryption of details of the users app, or a hash, or a hash of a random string, etc. Should we worry about hash algorithm (MSD, SHA1, bcrypt) etc?
Edit:
I've spoke to a few friends (email/twitter) and they recommended just using a GUID with the dashes stripped.
This seems a little hacky to me though, hoping to get some more ideas.
Use a random number generator designed for cryptography. Then base-64 encode the number.
This is a C# example:
var key = new byte[32];
using (var generator = RandomNumberGenerator.Create())
generator.GetBytes(key);
string apiKey = Convert.ToBase64String(key);
API keys need to have the properties that they:
uniquely identify an authorized API user -- the "key" part of "API key"
authenticate that user -- cannot be guessed/forged
can be revoked if a user misbehaves -- typically they key into a database that can have a record deleted.
Typically you will have thousands or millions of API keys not billions, so they do not need to:
Reliably store information about the API user because that can be stored in your database.
As such, one way to generate an API key is to take two pieces of information:
a serial number to guarantee uniqueness
enough random bits to pad out the key
and sign them using a private secret.
The counter guarantees that they uniquely identify the user, and the signing prevents forgery. Revocability requires checking that the key is still valid in the database before doing anything that requires API-key authorization.
A good GUID generator is a pretty good approximation of an incremented counter if you need to generate keys from multiple data centers or don't have otherwise a good distributed way to assign serial numbers.
or a hash of a random string
Hashing doesn't prevent forgery. Signing is what guarantees that the key came from you.
Update, in Chrome's console and Node.js, you can issue:
crypto.randomUUID()
Example output:
'4f9d5fe0-a964-4f11-af99-6c40de98af77'
Original answer (stronger):
You could try your web browser console by opening a new tab, hitting CTRL + SHIFT + i on Chrome, and then entering the following immediately invoked function expression (IIFE):
(async function (){
let k = await window.crypto.subtle.generateKey(
{name: "AES-GCM", length: 256}, true, ["encrypt", "decrypt"]);
const jwk = await crypto.subtle.exportKey("jwk", k)
console.log(jwk.k)
})()
Example output:
gv4Gp1OeZhF5eBNU7vDjDL-yqZ6vrCfdCzF7HGVMiCs
References:
https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/exportKey
I'll confess that I mainly wrote this for myself for future reference...
I use UUIDs, formatted in lower case without dashes.
Generation is easy since most languages have it built in.
API keys can be compromised, in which case a user may want to cancel their API key and generate a new one, so your key generation method must be able to satisfy this requirement.
If you want an API key with only alphanumeric characters, you can use a variant of the base64-random approach, only using a base-62 encoding instead. The base-62 encoder is based on this.
public static string CreateApiKey()
{
var bytes = new byte[256 / 8];
using (var random = RandomNumberGenerator.Create())
random.GetBytes(bytes);
return ToBase62String(bytes);
}
static string ToBase62String(byte[] toConvert)
{
const string alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
BigInteger dividend = new BigInteger(toConvert);
var builder = new StringBuilder();
while (dividend != 0) {
dividend = BigInteger.DivRem(dividend, alphabet.Length, out BigInteger remainder);
builder.Insert(0, alphabet[Math.Abs(((int)remainder))]);
}
return builder.ToString();
}
An API key should be some random value. Random enough that it can't be predicted. It should not contain any details of the user or account that it's for. Using UUIDs is a good idea, if you're certain that the IDs created are random.
Earlier versions of Windows produced predictable GUIDs, for example, but this is an old story.
I have an object that I'd like to hash with sha256 in Node. The contents of the object are simple Javascript types. For example's sake, let's say:
var payload = {
"id": "3bab3f00-7d55-11e7-9b0a-4c32759242a5",
"foo": "a message",
"version": 7,
};
I create a hash like this:
const crypto = require('crypto');
var hash = crypto.createHash('sha256');
hash.update( ... ).digest('hex');
The question is, what to pass to update? The documentation for crypto says you can pass a <string> | <Buffer> | <TypedArray> | <DataView>, which seems to suggest an object is not a good thing to pass.
I can't use toString() because that prints "[object Object]". I could use JSON.stringify, however I have read elsewhere that the output from stringify is not guaranteed to be deterministic for the same input.
Are there any other options? I do not want to download a package from NPM.
The right terms are "canonical" and the action is called "canonicalization" (I'm assuming EN-US here), you can find a stringify that produces canonical output here.
Beware that you must make sure that the output also has the right character set (UTF-8 should be preferred) and line endings. Spurious data should not be present, e.g. a byte order mark or NUL termination string is enough to void the hash value.
After that you can pass it as string I suppose.
You can of course use any canonical encoding. Note that XML has defined XML-digsig, which contains canonicalization during signature generation and signing, which means that the verification will even succeed if the XML code is altered (without altering the structure or contents of course, but whitespace / indentation will not matter).
I'd still recommend regression testing between implementations and even version updates of the libraries.
Hello I have this Java code which uses the following encryption method to encrypt password.
MessageDigest digester = MessageDigest.getInstance("SHA-1");
value = digester.digest(password.getBytes());
digester.update(email.getBytes());
value = digester.digest(value);
This returns base64 encoded string like qXO4aUUUyiue6arrcLAio+TBNwQ= This is sample not exact.
I am converting this to NodeJs not sure how to handle this. I have tried like
var crypto = require('crypto');
var shasum = crypto.createHash('sha1');
var value = shasum.update('hello');
shasum.update('abc#xyz.com');
value = shasum.digest(value).toString('base64');
console.log(value);
The string base64 I get in node js is not similar to get from java. Not sure why?. I need to have same encoding as java as its old system migrated to new one cant lose old details.
Can someone help me how I can achieve same base64 string.
In Java you're calculating the first value as the hash of the password alone, then overwrite it with hash of the email alone. (digest gives the result and resets the hash, in Java).
In Javascript, on the other hand, you're having an undefined value, then overwrite it with the hash of (password concatenated with email).
PS that hash is conceptually wrong: you should always put a separator between two fields, to avoid ambiguity and, thus, possible attacks.
I use nodejs as my backend and store my data in MongoDB. I'm interested how should I check incomming data before saving into database.
I need to check as pure strings like:
"some xss test"
and object of strings:
{
"name": "xss name",
"age": 25
}
What library should I use for my task?
It is a general practice to verify the data when you are outputting it, not storing. Doing so, you do not need to worry, what if the XSS data got into database using other routes?
But your question still stands, how would a programmer check if something contains XSS or not. There is a validator module exactly for doing this job:
var validator = require('validator');
var escaped_string = validator.escape(someString);
To verify the object of strings, you might have to iterate manually through the list.
If you are actually intersted in outputting html code, but worry for XSS, then you need to use a more sophisticated XSS validator which is kept up-to-date. Example would be Google Caja
So with lots of different services around now, Google APIs, Twitter API, Facebook API, etc etc.
Each service has an API key, like:
AIzaSyClzfrOzB818x55FASHvX4JuGQciR9lv7q
All the keys vary in length and the characters they contain, I'm wondering what the best approach is for generating an API key?
I'm not asking for a specific language, just the general approach to creating keys, should they be an encryption of details of the users app, or a hash, or a hash of a random string, etc. Should we worry about hash algorithm (MSD, SHA1, bcrypt) etc?
Edit:
I've spoke to a few friends (email/twitter) and they recommended just using a GUID with the dashes stripped.
This seems a little hacky to me though, hoping to get some more ideas.
Use a random number generator designed for cryptography. Then base-64 encode the number.
This is a C# example:
var key = new byte[32];
using (var generator = RandomNumberGenerator.Create())
generator.GetBytes(key);
string apiKey = Convert.ToBase64String(key);
API keys need to have the properties that they:
uniquely identify an authorized API user -- the "key" part of "API key"
authenticate that user -- cannot be guessed/forged
can be revoked if a user misbehaves -- typically they key into a database that can have a record deleted.
Typically you will have thousands or millions of API keys not billions, so they do not need to:
Reliably store information about the API user because that can be stored in your database.
As such, one way to generate an API key is to take two pieces of information:
a serial number to guarantee uniqueness
enough random bits to pad out the key
and sign them using a private secret.
The counter guarantees that they uniquely identify the user, and the signing prevents forgery. Revocability requires checking that the key is still valid in the database before doing anything that requires API-key authorization.
A good GUID generator is a pretty good approximation of an incremented counter if you need to generate keys from multiple data centers or don't have otherwise a good distributed way to assign serial numbers.
or a hash of a random string
Hashing doesn't prevent forgery. Signing is what guarantees that the key came from you.
Update, in Chrome's console and Node.js, you can issue:
crypto.randomUUID()
Example output:
'4f9d5fe0-a964-4f11-af99-6c40de98af77'
Original answer (stronger):
You could try your web browser console by opening a new tab, hitting CTRL + SHIFT + i on Chrome, and then entering the following immediately invoked function expression (IIFE):
(async function (){
let k = await window.crypto.subtle.generateKey(
{name: "AES-GCM", length: 256}, true, ["encrypt", "decrypt"]);
const jwk = await crypto.subtle.exportKey("jwk", k)
console.log(jwk.k)
})()
Example output:
gv4Gp1OeZhF5eBNU7vDjDL-yqZ6vrCfdCzF7HGVMiCs
References:
https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/exportKey
I'll confess that I mainly wrote this for myself for future reference...
I use UUIDs, formatted in lower case without dashes.
Generation is easy since most languages have it built in.
API keys can be compromised, in which case a user may want to cancel their API key and generate a new one, so your key generation method must be able to satisfy this requirement.
If you want an API key with only alphanumeric characters, you can use a variant of the base64-random approach, only using a base-62 encoding instead. The base-62 encoder is based on this.
public static string CreateApiKey()
{
var bytes = new byte[256 / 8];
using (var random = RandomNumberGenerator.Create())
random.GetBytes(bytes);
return ToBase62String(bytes);
}
static string ToBase62String(byte[] toConvert)
{
const string alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
BigInteger dividend = new BigInteger(toConvert);
var builder = new StringBuilder();
while (dividend != 0) {
dividend = BigInteger.DivRem(dividend, alphabet.Length, out BigInteger remainder);
builder.Insert(0, alphabet[Math.Abs(((int)remainder))]);
}
return builder.ToString();
}
An API key should be some random value. Random enough that it can't be predicted. It should not contain any details of the user or account that it's for. Using UUIDs is a good idea, if you're certain that the IDs created are random.
Earlier versions of Windows produced predictable GUIDs, for example, but this is an old story.