Is there public API for using the Google Authenticator (two factor authentication) on self-running (e.g. LAMP stack) web apps?
The project is open source. I have not used it. But it's using a documented algorithm (noted in the RFC listed on the open source project page), and the authenticator implementations support multiple accounts.
The actual process is straightforward. The one time code is, essentially, a pseudo random number generator. A random number generator is a formula that once given a seed, or starting number, continues to create a stream of random numbers. Given a seed, while the numbers may be random to each other, the sequence itself is deterministic. So, once you have your device and the server "in sync" then the random numbers that the device creates, each time you hit the "next number button", will be the same, random, numbers the server expects.
A secure one time password system is more sophisticated than a random number generator, but the concept is similar. There are also other details to help keep the device and server in sync.
So, there's no need for someone else to host the authentication, like, say OAuth. Instead you need to implement that algorithm that is compatible with the apps that Google provides for the mobile devices. That software is (should be) available on the open source project.
Depending on your sophistication, you should have all you need to implement the server side of this process give the OSS project and the RFC. I do not know if there is a specific implementation for your server software (PHP, Java, .NET, etc.)
But, specifically, you don't need an offsite service to handle this.
The algorithm is documented in RFC6238. Goes a bit like this:
your server gives the user a secret to install into Google Authenticator. Google do this as a QR code documented here.
Google Authenticator generates a 6 digit code by from a SHA1-HMAC of the Unix time and the secret (lots more detail on this in the RFC)
The server also knows the secret / unix time to verify the 6-digit code.
I've had a play implementing the algorithm in javascript here: http://blog.tinisles.com/2011/10/google-authenticator-one-time-password-algorithm-in-javascript/
There are a variety of libraries for PHP (The LAMP Stack)
PHP
https://code.google.com/p/ga4php/
http://www.idontplaydarts.com/2011/07/google-totp-two-factor-authentication-for-php/
You should be careful when implementing two-factor auth, you need to ensure your clocks on the server and client are synchronized, that there is protection in place against brute-force attacks on the token and that the initial seed used is suitably large.
You can use my solution, posted as the answer to my question (there is full Python code and explanation):
Google Authenticator implementation in Python
It is rather easy to implement it in PHP or Perl, I think. If you have any problems with this, please let me know.
I have also posted my code on GitHub as Python module.
I found this: https://github.com/PHPGangsta/GoogleAuthenticator. I tested it and works fine for me.
Not LAMP but if you use C# this is the code I use:
Code originally from:
https://github.com/kspearrin/Otp.NET
The Base32Encoding class is from this answer:
https://stackoverflow.com/a/7135008/3850405
Example program:
class Program
{
static void Main(string[] args)
{
var bytes = Base32Encoding.ToBytes("JBSWY3DPEHPK3PXP");
var totp = new Totp(bytes);
var result = totp.ComputeTotp();
var remainingTime = totp.RemainingSeconds();
}
}
Totp:
public class Totp
{
const long unixEpochTicks = 621355968000000000L;
const long ticksToSeconds = 10000000L;
private const int step = 30;
private const int totpSize = 6;
private byte[] key;
public Totp(byte[] secretKey)
{
key = secretKey;
}
public string ComputeTotp()
{
var window = CalculateTimeStepFromTimestamp(DateTime.UtcNow);
var data = GetBigEndianBytes(window);
var hmac = new HMACSHA1();
hmac.Key = key;
var hmacComputedHash = hmac.ComputeHash(data);
int offset = hmacComputedHash[hmacComputedHash.Length - 1] & 0x0F;
var otp = (hmacComputedHash[offset] & 0x7f) << 24
| (hmacComputedHash[offset + 1] & 0xff) << 16
| (hmacComputedHash[offset + 2] & 0xff) << 8
| (hmacComputedHash[offset + 3] & 0xff) % 1000000;
var result = Digits(otp, totpSize);
return result;
}
public int RemainingSeconds()
{
return step - (int)(((DateTime.UtcNow.Ticks - unixEpochTicks) / ticksToSeconds) % step);
}
private byte[] GetBigEndianBytes(long input)
{
// Since .net uses little endian numbers, we need to reverse the byte order to get big endian.
var data = BitConverter.GetBytes(input);
Array.Reverse(data);
return data;
}
private long CalculateTimeStepFromTimestamp(DateTime timestamp)
{
var unixTimestamp = (timestamp.Ticks - unixEpochTicks) / ticksToSeconds;
var window = unixTimestamp / (long)step;
return window;
}
private string Digits(long input, int digitCount)
{
var truncatedValue = ((int)input % (int)Math.Pow(10, digitCount));
return truncatedValue.ToString().PadLeft(digitCount, '0');
}
}
Base32Encoding:
public static class Base32Encoding
{
public static byte[] ToBytes(string input)
{
if (string.IsNullOrEmpty(input))
{
throw new ArgumentNullException("input");
}
input = input.TrimEnd('='); //remove padding characters
int byteCount = input.Length * 5 / 8; //this must be TRUNCATED
byte[] returnArray = new byte[byteCount];
byte curByte = 0, bitsRemaining = 8;
int mask = 0, arrayIndex = 0;
foreach (char c in input)
{
int cValue = CharToValue(c);
if (bitsRemaining > 5)
{
mask = cValue << (bitsRemaining - 5);
curByte = (byte)(curByte | mask);
bitsRemaining -= 5;
}
else
{
mask = cValue >> (5 - bitsRemaining);
curByte = (byte)(curByte | mask);
returnArray[arrayIndex++] = curByte;
curByte = (byte)(cValue << (3 + bitsRemaining));
bitsRemaining += 3;
}
}
//if we didn't end with a full byte
if (arrayIndex != byteCount)
{
returnArray[arrayIndex] = curByte;
}
return returnArray;
}
public static string ToString(byte[] input)
{
if (input == null || input.Length == 0)
{
throw new ArgumentNullException("input");
}
int charCount = (int)Math.Ceiling(input.Length / 5d) * 8;
char[] returnArray = new char[charCount];
byte nextChar = 0, bitsRemaining = 5;
int arrayIndex = 0;
foreach (byte b in input)
{
nextChar = (byte)(nextChar | (b >> (8 - bitsRemaining)));
returnArray[arrayIndex++] = ValueToChar(nextChar);
if (bitsRemaining < 4)
{
nextChar = (byte)((b >> (3 - bitsRemaining)) & 31);
returnArray[arrayIndex++] = ValueToChar(nextChar);
bitsRemaining += 5;
}
bitsRemaining -= 3;
nextChar = (byte)((b << bitsRemaining) & 31);
}
//if we didn't end with a full char
if (arrayIndex != charCount)
{
returnArray[arrayIndex++] = ValueToChar(nextChar);
while (arrayIndex != charCount) returnArray[arrayIndex++] = '='; //padding
}
return new string(returnArray);
}
private static int CharToValue(char c)
{
int value = (int)c;
//65-90 == uppercase letters
if (value < 91 && value > 64)
{
return value - 65;
}
//50-55 == numbers 2-7
if (value < 56 && value > 49)
{
return value - 24;
}
//97-122 == lowercase letters
if (value < 123 && value > 96)
{
return value - 97;
}
throw new ArgumentException("Character is not a Base32 character.", "c");
}
private static char ValueToChar(byte b)
{
if (b < 26)
{
return (char)(b + 65);
}
if (b < 32)
{
return (char)(b + 24);
}
throw new ArgumentException("Byte is not a value Base32 value.", "b");
}
}
Theres: https://www.gauthify.com that offers it as a service
Yes, need no network service, because Google Authenticator app won't communicate with the google server, it just keeps synced with the initital secret that your server generate(input into your phone from QR code) while the time pass.
For those using Laravel, this https://github.com/sitepoint-editors/google-laravel-2FA is a nice way to solve this problem.
For C# user, run this simple Console App to understand how to verify the one time token code. Note that we need to install library Otp.Net from Nuget package first.
static string secretKey = "JBSWY3DPEHPK3PXP"; //add this key to your Google Authenticator app
private static void Main(string[] args)
{
var bytes = Base32Encoding.ToBytes(secretKey);
var totp = new Totp(bytes);
while (true)
{
Console.Write("Enter your code from Google Authenticator app: ");
string userCode = Console.ReadLine();
//Generate one time token code
string tokenInApp = totp.ComputeTotp();
int remainingSeconds = totp.RemainingSeconds();
if (userCode.Equals(tokenInApp)
&& remainingSeconds > 0)
{
Console.WriteLine("Success!");
}
else
{
Console.WriteLine("Failed. Try again!");
}
}
}
Related
I am using Microsoft Azure Text To Speech with Unity. But there will be broken sounds at the beginning and end of the playing sound. Is this normal, or result.AudioData is broken. Below is the code.
public AudioSource audioSource;
void Start()
{
SynthesisToSpeaker("你好世界");
}
public void SynthesisToSpeaker(string text)
{
var config = SpeechConfig.FromSubscription("[redacted]", "southeastasia");
config.SpeechSynthesisLanguage = "zh-CN";
config.SpeechSynthesisVoiceName = "zh-CN-XiaoxiaoNeural";
// Creates a speech synthesizer.
// Make sure to dispose the synthesizer after use!
SpeechSynthesizer synthesizer = new SpeechSynthesizer(config, null);
Task<SpeechSynthesisResult> task = synthesizer.SpeakTextAsync(text);
StartCoroutine(CheckSynthesizer(task, config, synthesizer));
}
private IEnumerator CheckSynthesizer(Task<SpeechSynthesisResult> task,
SpeechConfig config,
SpeechSynthesizer synthesizer)
{
yield return new WaitUntil(() => task.IsCompleted);
var result = task.Result;
// Checks result.
string newMessage = string.Empty;
if (result.Reason == ResultReason.SynthesizingAudioCompleted)
{
var sampleCount = result.AudioData.Length / 2;
var audioData = new float[sampleCount];
for (var i = 0; i < sampleCount; ++i)
{
audioData[i] = (short)(result.AudioData[i * 2 + 1] << 8
| result.AudioData[i * 2]) / 32768.0F;
}
// The default output audio format is 16K 16bit mono
var audioClip = AudioClip.Create("SynthesizedAudio", sampleCount,
1, 16000, false);
audioClip.SetData(audioData, 0);
audioSource.clip = audioClip;
audioSource.Play();
}
else if (result.Reason == ResultReason.Canceled)
{
var cancellation = SpeechSynthesisCancellationDetails.FromResult(result);
}
synthesizer.Dispose();
}
The default audio format is Riff16Khz16BitMonoPcm, which has a riff header in the beginning of result.AudioData. If you pass the audioData to audioClip, it will play the header, then you hear some noise.
You can set the format to a raw format without header by speechConfig.SetSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Raw16Khz16BitMonoPcm);, see this sample for details.
I have coded the following solution, and it works, except for when there is punctuation. I was wondering if there are O(1) space complexity and O(length of string) time complexity solutions, without using reverse() method, or anything similar that makes this task too easy. Any answer that can also handle punctuation correctly would be great.
Example:
Given string: "I love chocolate"
Return string should be: "I evol etalocohc"
For clarity, when I say handle punctuation correctly, I mean punctuation should not move around.
// reverse the letters of every word in a sentence (string), and return the result
public static String reverse(String x)
{
String[] str = x.split(" ");
StringBuilder rev = new StringBuilder("");
for (int i = 0; i < str.length; i++)
{
for (int s = str[i].length()-1; s >= 0; s--)
{
rev.append(str[i].charAt(s));
}
rev.append(" ");
}
return rev.toString();
}
Here is my output for some tests of mine:
public static void main(String[] args)
{
System.out.println(reverse("I love chocolate"));//this passes
System.out.println(reverse("Geeks for Geeks"));//this passes
System.out.println(reverse("You, are awesome"));//not handling puncutation mark correctly, gives me ",uoY era emosewa", instead of "uoY, era emosewa"
System.out.println(reverse("Geeks! for Geeks."));//not handling puncutation marks correctly, gives me "!skeeG rof .skeeG", instead of "skeeG! rof skeeG."
}
This would probably work with the punctuation:
public static String reverse(String x)
{
String[] str = x.split("\\W"); //Split on non-word characters.
StringBuilder rev = new StringBuilder("");
int currentPosition = 0;
for (int i = 0; i < str.length; i++)
{
for (int s = str[i].length()-1; s >= 0; s--)
{
rev.append(str[i].charAt(s));
currentPosition++;
}
while (currentPosition < x.length() && Character.toString(x.charAt(currentPosition)).matches("\\W"))
rev.append(x.charAt(currentPosition++)); //Add the actual character there.
}
return rev.toString();
}
Haven't coded in Java in a while now so I know it's probably not best practices here.
Complexity is O(n) (space and time).
If you start off with a string builder you might be able to lower space complexity by using in-place character swaps instead of appending, but you'd need to preemptively find where all the non-word characters are.
Here is an implementation which uses in-place reversing of words requiring only O(1) storage. In addition, the reversing algorithm itself is O(N) with the length of the string.
The basic algorithm is to walk down the string until hitting a space. Then, the swap() method is called to reverse that particular word in place. It only needs at any moment one extra character.
This approach may not be as performant as the accepted answer due to its heavy use of StringBuilder manipulations. But this might something to consider in an environment like an Android application where space is very precious.
public static void swap(StringBuilder input, int start, int end) {
for (int i=0; i <= (end - start) / 2; ++i) {
char ch = input.charAt(start + i);
input.setCharAt(start + i, input.charAt(end - i));
input.setCharAt(end - i, ch);
}
}
public static void main(String[] args) {
StringBuilder sb = new StringBuilder("I love chocolate");
int start = 0;
int end = 0;
while (true) {
while (end <= sb.length() - 1 && sb.charAt(end) != ' ') {
++end;
}
swap(sb, start, end - 1);
start = end + 1;
end = start;
if (end > sb.length() - 1) {
break;
}
}
System.out.println(sb);
}
Demo here:
Rextester
A compact solution is
public static String reverse(String x) {
Matcher m = Pattern.compile("\\w+").matcher(x);
if(!m.find()) return x;
StringBuffer target = new StringBuffer(x.length());
do m.appendReplacement(target, new StringBuilder(m.group()).reverse().toString());
while(m.find());
return m.appendTail(target).toString();
}
The appendReplacement loop + appendTail on a Matcher is the manual equivalent of String.replaceAll(regex), intended to support exactly such cases where the replacement is more complex than a simple string with placeholders, like reversing the found words.
Unfortunately, we have to use the outdated StringBuffer here, as the API is older than StringBuilder. Java 9 is going to change that.
A potentially more efficient alternative is
public static String reverse(String x) {
Matcher m = Pattern.compile("\\w+").matcher(x);
if(!m.find()) return x;
StringBuilder target = new StringBuilder(x.length());
int last=0;
do {
int s = m.start(), e = m.end();
target.append(x, last, s).append(new StringBuilder(e-s).append(x, s, e).reverse());
last = e;
}
while(m.find());
return target.append(x, last, x.length()).toString();
}
This uses StringBuilder throughout the operation and usese the feature to append partial character sequences, not creating intermediate strings for the match and replacement. It also elides the search for placeholders in the replacement, which appendReplacement does internally.
I just have made changes on your code, not additional methods or loops, just some variables and if conditions.
/* Soner - The methods reverse a string with preserving punctiations */
public static String reverse(String x) {
String[] str = x.split(" ");
boolean flag = false;
int lastCharPosition;
StringBuilder rev = new StringBuilder("");
for (int i = 0; i < str.length; i++) {
flag = false;
lastCharPosition = str[i].length()-1;
if (str[i].charAt(lastCharPosition) == '.' || str[i].charAt(lastCharPosition) == '!'
|| str[i].charAt(lastCharPosition) == ',') { // you can add new punctiations
flag = true;
lastCharPosition = str[i].length()-2;
}
for (int s = lastCharPosition; s >= 0; s--) {
rev.append(str[i].charAt(s));
}
if (flag) rev.append(str[i].charAt(lastCharPosition + 1));
rev.append(" ");
}
return rev.toString();
}
I am working on migration of code form java to Nodejs. I have one requirement to encrypt the text with private key using "DESede/ECB/NoPadding" algorithm. Currently code is written in Java and now I need to migrate to Nodejs. Since encrypted key is sent to other application therefore I can't change the algorithm or key here. Following is approach used in java
1. Stored the private key in hex string. I.e. 48 chars hex string as below which is equivalent to 24 bytes reuquired for 3des
73AD9CEC99816AA6A4D82FB273AD9CEC99816AA6A4D82FB2
2. Following is code written in java
https://github.com/dilipkumar2k6/3des/blob/master/TripleDes.java
import java.security.GeneralSecurityException;
import javax.crypto.Cipher;
import javax.crypto.spec.SecretKeySpec;
public class TripleDes {
// Crypto library related keys
private static final String ALGO_NAME = "DESede/ECB/NoPadding";
private static final int PADDING_BLOCK = 8;
// Test Data
private static final String PLAIN_TEXT = "Hello World";
private static final String SHARED_KEY = "73AD9CEC99816AA6A4D82FB273AD9CEC99816AA6A4D82FB2";
public static void main(String[] arg) {
try {
// Get Algorithm name
String desAlgoName = getDESAlgorithmName(ALGO_NAME);
// Create Cipher object
Cipher cipher = Cipher.getInstance(ALGO_NAME);
//Actual DES algo needs 56 bits key, which is equivalent to 1byte (0 at 0th position) Get 8*3 byets key
byte [] key = hexFromString(SHARED_KEY);
System.out.println("DES Algorithm shared key size in bytes >> "+key.length);
// Create SecretKeySpec
SecretKeySpec secretKeySpec = new SecretKeySpec(key, desAlgoName);
//Encrypt bytes
byte [] encryptedBytes = encryptIntoBytes(cipher, secretKeySpec, PLAIN_TEXT.getBytes(), 0, PLAIN_TEXT.getBytes().length);
String encryptedString= hexToString(encryptedBytes);
System.out.println(encryptedString);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static byte[] encryptIntoBytes(Cipher cipher, SecretKeySpec secretKeySpec, byte[] dct, int offset, int len) throws GeneralSecurityException {
cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec);
byte[] ect = cipher.doFinal(addPadding(dct, offset, len));
return ect;
}
public static String getDESAlgorithmName(String algoName) {
System.out.println("getDESAlgorithmName algoName >> "+algoName);
String desAlgoName = null;
int i = algoName.indexOf("/");
if (i != -1)
desAlgoName = algoName.substring(0, i);
else
desAlgoName = algoName;
return desAlgoName;
}
/**
* Adds padding characters to the data to be encrypted. Also adds random
* Initial Value to the beginning of the encrypted data when using Triple
* DES in CBC mode (DES-EDE3/CBC).
*
* #param inData
* Array of bytes to be padded
* #param offset
* Offset to starting point within array
* #param len
* Number of bytes to be encrypted
* #return Padded array of bytes
*/
public static byte[] addPadding(byte[] inData, int offset, int len) {
System.out.println("addPadding offset >> "+offset+", len >> "+len);
byte[] bp = null;
int padChars = PADDING_BLOCK; // start with max padding value
int partial = (len + 1) % padChars; // calculate how many extra bytes
// exist
if (partial == 0) {
padChars = 1; // if none, set to only pad with length byte
} else {
padChars = padChars - partial + 1; // calculate padding size to
// include length
}
System.out.println("addPadding >> Add padding of "+padChars);
/*
* Create a byte array large enough to hold data plus padding bytes The
* count of padding bytes is placed in the first byte of the data to be
* encrypted. That byte is included in the count.
*/
bp = new byte[len + padChars];
bp[0] = Byte.parseByte(Integer.toString(padChars));
System.arraycopy(inData, offset, bp, 1, len);
return bp;
}
public static byte[] hexFromString(String hex) {
int len = hex.length();
byte[] buf = new byte[((len + 1) / 2)];
int i = 0, j = 0;
if ((len % 2) == 1)
buf[j++] = (byte) fromDigit(hex.charAt(i++));
while (i < len) {
buf[j++] = (byte) ((fromDigit(hex.charAt(i++)) << 4) | fromDigit(hex
.charAt(i++)));
}
return buf;
}
public static int fromDigit(char ch) {
if (ch >= '0' && ch <= '9')
return ch - '0';
if (ch >= 'A' && ch <= 'F')
return ch - 'A' + 10;
if (ch >= 'a' && ch <= 'f')
return ch - 'a' + 10;
throw new IllegalArgumentException("invalid hex digit '" + ch + "'");
}
public static String hexToString(byte[] ba) {
return hexToString(ba, 0, ba.length);
}
public static final char[] hexDigits = { '0', '1', '2', '3', '4', '5',
'6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' };
public static String hexToString(byte[] ba, int offset, int length) {
char[] buf = new char[length * 2];
int j = 0;
int k;
for (int i = offset; i < offset + length; i++) {
k = ba[i];
buf[j++] = hexDigits[(k >>> 4) & 0x0F];
buf[j++] = hexDigits[k & 0x0F];
}
return new String(buf);
}
}
I need to migrate this code to Nodejs and facing multiple issues. I refeered http://mygo.iteye.com/blog/2018882 to get the basic idea on nodejs way to do the encryption in des3. However I see following difference in JAVA way and Nodejs way.
1. JAVA is using Hex string of 48 lenght as key, since one char in hex is 4 bits therfore final size is equivalent to 24 bytes length which meets DES3 requirement.
2. In Java code, final key is being used as bytes (as needed by DES) which made indpendent of the way we store the key
3. In node js, key is stored as character i.e. to use des3 I have to use 24 bytes which is equivalent to 24 chars key as 73AD9CEC99816AA6A4D82FB2. Here this is string of 24 chars and since one char is one byte thereofore total length is 24 bytes which meets DES3 requirement.
4. Following is nodejs code for reference
https://github.com/dilipkumar2k6/3des/blob/master/Crypto.js
'use strict';
/*
* Offers related services.
*/
var crypto = require("crypto");
module.exports = {
encrypt: function (plainText) {
return encrypt({
alg: 'des-ede3', //3des-ecb
autoPad: true,
key: '73AD9CEC99816AA6A4D82FB2',
plaintext: 'Hello World',
iv: null
});
}
};
function encrypt(param) {
var key = new Buffer(param.key);
var iv = new Buffer(param.iv ? param.iv : 0);
var plaintext = param.plaintext;
var alg = param.alg;
var autoPad = param.autoPad;
//encrypt
var cipher = crypto.createCipheriv(alg, key, iv);
cipher.setAutoPadding(autoPad); //default true
var ciph = cipher.update(plaintext, 'utf8', 'hex');
ciph += cipher.final('hex');
console.log(alg, ciph);
return ciph;
}
function decrypt(param) {
var key = new Buffer(param.key);
var iv = new Buffer(param.iv ? param.iv : 0)
var alg = param.alg;
var autoPad = param.autoPad;
//decrypt
var decipher = crypto.createDecipheriv(alg, key, iv);
cipher.setAutoPadding(autoPad);
var txt = decipher.update(ciph, 'hex', 'utf8');
txt += decipher.final('utf8');
console.log(alg, txt);
return txt;
}
Following is my problem.
1. How can i convert my existing hex code into string? I used "hexToString" method (please check the java code)to convert hex into string. However getting weired character (this is also expected but problem is how i can use this transformed key in nodejs.
2. Can I pass byte array as key to Nodejs? It will make problem easy as I can easily convert my hex key into bytes array and I store my bytes array key in nodejs code.
3. In my javacode, I have custom padding logic, how can i write same logic in nodejs?
4. Most importantly, can I achieve same encryption logic in nodejs (similar to java)?
Artjom B. helped me to get the insight of nodejs and des3 algorithm. I have edited my post to clarify my exact requirement.
I think my main problem is, how can i feed byte[] as key to nodejs crypto for DES3?
I am kind of stuck. Please help.
Running crypto.getCiphers() shows you the available ciphers. Triple DES (EDE) in ECB mode with two keys (16 byte key) can be used as des-ede. If you have three part key (24 byte key) you should use des-ede3. ecb probably does not appear in the cipher description, because it's the most basic form.
Triple DES-EDE has different ways to use a key. EDE means encrypt-decrypt-encrypt with three different keys. If you only have for example one 8 byte key, this suggests that you use the same key for every phase of EDE. It's clear from your Java code that you have a 24 byte key (48 hex encoded chars). You have to use the same key.
The crypto module uses PKCS7 padding by default, so you will need to set the auto padding to false and do the padding yourself. I leave that task up to you.
module.exports = {
encrypt: function (plainText) {
return encrypt({
alg: 'des-ede3', //3des-ecb
autoPad: false,
key: '73AD9CEC99816AA6A4D82FB273AD9CEC99816AA6A4D82FB2',
plaintext: 'Hello World',
iv: null
});
}
};
function mypad(buf){
// TODO: do the padding
// replicate padding as in Java
return buf;
}
function myunpad(buf){
// TODO: do the unpadding
// read the first *byte* and remove as many trailing *bytes*
return buf;
}
function encrypt(param) {
var key = new Buffer(param.key);
var iv = new Buffer(param.iv ? param.iv : 0);
var plaintext = mypad(new Buffer(param.plaintext));
var alg = param.alg;
var autoPad = param.autoPad;
//encrypt
var cipher = crypto.createCipheriv(alg, key, iv);
cipher.setAutoPadding(autoPad); //default true
var ciph = cipher.update(plaintext, 'utf8', 'hex');
ciph += cipher.final('hex');
console.log(alg, ciph);
return ciph;
}
function decrypt(param) {
var key = new Buffer(param.key);
var iv = new Buffer(param.iv ? param.iv : 0)
var alg = param.alg;
var autoPad = param.autoPad;
//decrypt
var decipher = crypto.createDecipheriv(alg, key, iv);
cipher.setAutoPadding(autoPad);
var txt = decipher.update(ciph, 'hex', 'utf8');
txt += decipher.final('utf8');
console.log(alg, txt);
return myunpad(new Buffer(txt, 'hex'));
}
Word of caution:
Don't use (3)DES especially with only one 8 byte key! Don't use ECB mode! Don't use NoPadding for block modes! Use AES-256 with GCM mode (no padding since it's a streaming mode).-
This is my decrypt function based on Artjom's answer. use 'des-ede3' if you have a 24byte key.
internals.decrypt = function (message, key) {
var message = Buffer.from(message, 'base64');
var decipher = crypto.createDecipher('des-ede', key);
var decryptedMessage = decipher.update(message, 'hex', 'utf8');
decryptedMessage += decipher.final('utf8');
return decryptedMessage.toString();
}
In previous versions of Wordpress images could be automatically scaled to 60% with one click. Now in Wordpress 3.9 the only automatic scaling is Thumbnail, Medium, and Full Size. I could chose Custom Size or drag it to the approximate size I want, but Custom Size requires me to compute 60% myself and dragging it is inexact.
All of my images are different heights and widths. They are images of a written font so the font size needs to be the same for every image, even though the height and width are different. In the past I just made all of my images display at 60%. Is there a way to do that in Wordpress 3.9?
Since I couldn't find a better long term method, I just wrote some Java code to do the job. This way I can just cut and paste from the HTML view.
import java.util.Scanner;
public class ImageResizer {
public static void main(String[] args) {
Scanner userInput = new Scanner(System.in);
String inputString;
String outputString;
System.out.print("Default resize is 60%\n");
System.out.print("Type \"exit\" to quit: \n\n");
do {
// input something like:
// width="208" height="425"
System.out.print("Enter html: ");
inputString = userInput.nextLine();
outputString = resize(inputString);
System.out.print(outputString + "\n");
} while (!inputString.equals("exit"));
}
public static String resize(String myString) {
final int RESIZE_PERCENT = 60;
// parse input string
String delims = "[ ]+"; // spaces
String[] tokens = myString.split(delims);
for (int i = 0; i < tokens.length; i++) {
if (tokens[i].startsWith("width") || tokens[i].startsWith("height")) {
// extract the height/width number
String subDelims = "[\"]"; // quote mark
String[] subTokens = tokens[i].split(subDelims);
int number = Integer.parseInt(subTokens[1]);
number = number * RESIZE_PERCENT / 100;
// rebuild string
tokens[i] = subTokens[0] + "\"" + number + "\"";
}
}
StringBuilder sb = new StringBuilder();
for (int i = 0; i < tokens.length; i++) {
sb.append(tokens[i]);
if ((i + 1) < tokens.length) {
sb.append(" ");
}
}
return sb.toString();
}
}
I have a list of files, which need to be read, in chunks, into a byte[], which is then passed to a hashing function. The tricky part is this: if I reach the end of a file, I need to continue reading the next file untill I fill the buffer, like so:
read 16 bits as an example:
File 1: 00101010
File 2: 01101010111111111
would need to be read as 0010101001101010
The point is: these files can be as large as several gigabytes, and I don't want to completely load them into memory. Loading pieces into a buffer of, like, 30 MB would be perfectly fine.
I want to use threading, but would it be efficient to thread reading a file? I don't know if Disc I/O is such a large bottleneck that this would be worth it. Would the hashing be sped up sufficently if I only thread that part, and lock on the read of each chunk? It is important the hashes get saved in the correct order.
The second thing I need to do, is to generate the MD5sum from each file as well. Is there anyway to do this more efficiently than doing this as a separate step?
(This question has some overlap with Is there a built-in way to handle multiple files as one stream?, but I thought this differed enough)
I am really stumped what approach to take, as I am fairly new to C#, as well as to threading. I already tried the approaches listed below, but they do not suffice for me.
As I am new to C# I value every kind of input on any aspect of my code.
This piece of code was threaded, but does not 'append' the streams, and as such generates invalid hashes:
public void DoHashing()
{
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = numThreads;
options.CancellationToken = cancelToken.Token;
Parallel.ForEach(files, options, (string f, ParallelLoopState loopState) =>
{
options.CancellationToken.ThrowIfCancellationRequested();
using (BufferedStream fileStream = new BufferedStream(File.OpenRead(f), bufferSize))
{
// Get the MD5sum first:
using (MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider())
{
md5.Initialize();
md5Sums[f] = BitConverter.ToString(md5.ComputeHash(fileStream)).Replace("-", "");
}
//setup for reading:
byte[] buffer = new byte[(int)pieceLength];
//I don't know if the buffer will f*ck up the filelenghth
long remaining = (new FileInfo(f)).Length;
int done = 0;
while (remaining > 0)
{
while (done < pieceLength)
{
options.CancellationToken.ThrowIfCancellationRequested();
//either try to read the piecelength, or the remaining length of the file.
int toRead = (int)Math.Min(pieceLength - done, remaining);
int read = fileStream.Read(buffer, done, toRead);
//if read == 0, EOF reached
if (read == 0)
{
remaining = 0;
break;
}
//offsets
done += read;
remaining -= read;
}
// Hash the piece
using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
{
sha1.Initialize();
byte[] hash = sha1.ComputeHash(buffer);
hashes[f].AddRange(hash);
}
done = 0;
buffer = new byte[(int)pieceLength];
}
}
}
);
}
This other piece of code isn't threaded (and doesn't calculate MD5):
void Hash()
{
//examples, these got handled by other methods
List<string> files = new List<string>();
files.Add("a.txt");
files.Add("b.blob");
//....
long totalFileLength;
int pieceLength = Math.Pow(2,20);
foreach (string file in files)
{
totalFileLength += (new FileInfo(file)).Length;
}
//Reading the file:
long remaining = totalFileLength;
byte[] buffer = new byte[Math.min(remaining, pieceSize)];
int index = 0;
FileStream fin = File.OpenRead(files[index]);
int done = 0;
int offset = 0;
while (remaining > 0)
{
while (done < pieceLength)
{
int toRead = (int)Math.Min(pieceLength - offset, remaining);
int read = fin.Read(buffer, done, toRead);
//if read == 0, EOF reached
if (read == 0)
{
index++;
//if last file:
if (index > files.Count)
{
remaining = 0;
break;
}
//get ready for next round:
offset = 0;
fin.OpenRead(files[index]);
}
done += read;
offset += read;
remaining -= read;
}
//Doing the piece hash:
HashPiece(buffer);
//reset for next piece:
done = 0;
byte[] buffer = new byte[Math.min(remaining, pieceSize)];
}
}
void HashPiece(byte[] piece)
{
using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
{
sha1.Initialize();
//hashes is a List
hashes.Add(sha1.ComputeHash(piece));
}
}
Thank you very much for your time and effort.
I'm not looking for completely coded solutions, any pointer and idea where to go with this would be excellent.
Questions & remarks to yodaj007's answer:
Why if (currentChunk.Length >= Constants.CHUNK_SIZE_IN_BYTES)? Why not ==? If the chunk is larger than the chunk size, my SHA1 hash gets a different value.
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = 0,
Length = (int)Math.Min(fi.Length, (long)needed)
});
Is a really interesting idea. Postpone reading untill you need it. Nice!
chunks.Add(currentChunk = new Chunk());
Why do this in the if (currentChunk != null) block and in the for (int i = 0; i < (fi.Length - offset) / Constants.CHUNK_SIZE_IN_BYTES; i++) block? Isn't the first a bit redundant?
Here is my complete answer. I tested it on one of my anime folders. It processes 14 files totaling 3.64GiB in roughly 16 seconds. In my opinion, using any sort of parallelism is more trouble than it is worth here. You're being limited by disc I/O, so multithreading will only get you so far. My solution can be easily parallelized though.
It starts by reading "chunk" source information: source file, offset, and length. All of this is gathered very quickly. From here, you can process the "chunks" using threading however you wish. Code follows:
public static class Constants
{
public const int CHUNK_SIZE_IN_BYTES = 32 * 1024 * 1024; // 32MiB
}
public class ChunkSource
{
public string Filename { get; set; }
public int StartPosition { get; set; }
public int Length { get; set; }
}
public class Chunk
{
private List<ChunkSource> _sources = new List<ChunkSource>();
public IList<ChunkSource> Sources { get { return _sources; } }
public byte[] Hash { get; set; }
public int Length
{
get { return Sources.Select(s => s.Length).Sum(); }
}
}
static class Program
{
static void Main()
{
DirectoryInfo di = new DirectoryInfo(#"C:\Stuff\Anime\Shikabane Hime Aka");
string[] filenames = di.GetFiles().Select(fi=> fi.FullName).OrderBy(n => n).ToArray();
var chunks = ChunkFiles(filenames);
ComputeHashes(chunks);
}
private static List<Chunk> ChunkFiles(string[] filenames)
{
List<Chunk> chunks = new List<Chunk>();
Chunk currentChunk = null;
int offset = 0;
foreach (string filename in filenames)
{
FileInfo fi = new FileInfo(filename);
if (!fi.Exists)
throw new FileNotFoundException(filename);
Debug.WriteLine(String.Format("File: {0}", filename));
//
// First, start off by either starting a new chunk or
// by finishing a leftover chunk from a previous file.
//
if (currentChunk != null)
{
//
// We get here if the previous file had leftover bytes that left us with an incomplete chunk
//
int needed = Constants.CHUNK_SIZE_IN_BYTES - currentChunk.Length;
if (needed == 0)
throw new InvalidOperationException("Something went wonky, shouldn't be here");
offset = needed;
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = 0,
Length = (int)Math.Min(fi.Length, (long)needed)
});
if (currentChunk.Length >= Constants.CHUNK_SIZE_IN_BYTES)
{
chunks.Add(currentChunk = new Chunk());
}
}
else
{
offset = 0;
}
//
// Note: Using integer division here
//
for (int i = 0; i < (fi.Length - offset) / Constants.CHUNK_SIZE_IN_BYTES; i++)
{
chunks.Add(currentChunk = new Chunk());
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = i * Constants.CHUNK_SIZE_IN_BYTES + offset,
Length = Constants.CHUNK_SIZE_IN_BYTES
});
Debug.WriteLine(String.Format("Chunk source created: Offset = {0,10}, Length = {1,10}", currentChunk.Sources[0].StartPosition, currentChunk.Sources[0].Length));
}
int leftover = (int)(fi.Length - offset) % Constants.CHUNK_SIZE_IN_BYTES;
if (leftover > 0)
{
chunks.Add(currentChunk = new Chunk());
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = (int)(fi.Length - leftover),
Length = leftover
});
}
else
{
currentChunk = null;
}
}
return chunks;
}
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
foreach (var chunk in chunks)
{
MemoryMappedFile mms = null;
byte[] buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES];
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
lock (files)
{
if (!files.TryGetValue(source.Filename, out mms))
{
Debug.WriteLine(String.Format("Opening {0}", source.Filename));
files.Add(source.Filename, mms = MemoryMappedFile.CreateFromFile(source.Filename, FileMode.Open));
}
}
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(buffer, 0, source.Length);
}
Debug.WriteLine("Done reading sources in {0}ms", sw.Elapsed.TotalMilliseconds);
sw.Restart();
MD5 md5 = MD5.Create();
chunk.Hash = md5.ComputeHash(buffer);
sw.Stop();
Debug.WriteLine(String.Format("Computed hash: {0} in {1}ms", String.Join("-", chunk.Hash.Select(h=> h.ToString("X2")).ToArray()), sw.Elapsed.TotalMilliseconds));
}
foreach (var x in files.Values)
{
x.Dispose();
}
}
}
I don't guarantee everything is spotlessly free of bugs. But I did have fun working on it. Look at the output window in Visual Studio for the debug information. It looks like this:
File: C:\Stuff\Anime\Shikabane Hime Aka\Episode 02.mkv
Chunk source created: Offset = 26966010, Length = 33554432
Chunk source created: Offset = 60520442, Length = 33554432
Chunk source created: Offset = 94074874, Length = 33554432
Chunk source created: Offset = 127629306, Length = 33554432
Chunk source created: Offset = 161183738, Length = 33554432
Chunk source created: Offset = 194738170, Length = 33554432
Chunk source created: Offset = 228292602, Length = 33554432
...
Opening C:\Stuff\Anime\Shikabane Hime Aka\Episode 02.mkv
Done reading sources in 42.9362ms
The thread '' (0xc10) has exited with code 0 (0x0).
Computed hash: 3C-81-A5-2C-90-02-24-23-42-5B-19-A2-15-56-AB-3F in 94.2481ms
Done reading sources in 0.0053ms
Computed hash: 58-F0-6D-D5-88-D8-FF-B3-BE-B4-6A-DA-63-09-43-6B in 98.9263ms
Done reading sources in 29.4805ms
Computed hash: F7-19-8D-A8-FE-9C-07-6E-DB-D5-74-A6-E1-E7-A6-26 in 85.0061ms
Done reading sources in 28.4971ms
Computed hash: 49-F2-CB-75-89-9A-BC-FA-94-A7-DF-E0-DB-02-8A-99 in 84.2799ms
Done reading sources in 31.106ms
Computed hash: 29-7B-18-BD-ED-E9-0C-68-4B-47-C6-5F-D0-16-8A-44 in 84.1444ms
Done reading sources in 31.2204ms
Computed hash: F8-91-F1-90-CF-9C-37-4E-82-68-C2-44-0D-A7-6E-F8 in 84.2592ms
Done reading sources in 31.0031ms
Computed hash: 65-97-ED-95-07-31-BF-C8-3A-BA-2B-DA-03-37-FD-00 in 95.331ms
Done reading sources in 33.0072ms
Computed hash: 9B-F2-83-E6-A8-DF-FD-8D-6C-5C-9E-F4-20-0A-38-4B in 85.9561ms
Done reading sources in 31.6232ms
Computed hash: B6-7C-6B-95-69-BC-9C-B2-1A-07-B3-13-28-A8-10-BC in 84.1866ms
Here is the parallel version. It's basically the same really. Using parallelism = 3 cut the processing time down to 9 seconds.
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
Parallel.ForEach(chunks, new ParallelOptions() { MaxDegreeOfParallelism = 2 }, (chunk, state, index) =>
{
MemoryMappedFile mms = null;
byte[] buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES];
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
lock (files)
{
if (!files.TryGetValue(source.Filename, out mms))
{
Debug.WriteLine(String.Format("Opening {0}", source.Filename));
files.Add(source.Filename, mms = MemoryMappedFile.CreateFromFile(source.Filename, FileMode.Open));
}
}
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(buffer, 0, source.Length);
}
Debug.WriteLine("Done reading sources in {0}ms", sw.Elapsed.TotalMilliseconds);
sw.Restart();
MD5 md5 = MD5.Create();
chunk.Hash = md5.ComputeHash(buffer);
sw.Stop();
Debug.WriteLine(String.Format("Computed hash: {0} in {1}ms", String.Join("-", chunk.Hash.Select(h => h.ToString("X2")).ToArray()), sw.Elapsed.TotalMilliseconds));
});
foreach (var x in files.Values)
{
x.Dispose();
}
}
EDIT
I found a bug, or what I think is a bug. Need to set the read offset to 0 if we're starting a new file.
EDIT 2 based on feedback
This processes the hashes in a separate thread. It's necessary to throttle the I/O. I was running into OutOfMemoryException without doing so. It doesn't really perform that much better, though. Beyond this... I'm not sure how it can be improved any further. Perhaps by reusing the buffers, maybe.
public class QueueItem
{
public Chunk Chunk { get; set; }
public byte[] buffer { get; set; }
}
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
foreach (var filename in chunks.SelectMany(c => c.Sources).Select(c => c.Filename).Distinct())
{
files.Add(filename, MemoryMappedFile.CreateFromFile(filename, FileMode.Open));
}
AutoResetEvent monitor = new AutoResetEvent(false);
ConcurrentQueue<QueueItem> hashQueue = new ConcurrentQueue<QueueItem>();
CancellationToken token = new CancellationToken();
Task.Factory.StartNew(() =>
{
int processCount = 0;
QueueItem item = null;
while (!token.IsCancellationRequested)
{
if (hashQueue.TryDequeue(out item))
{
MD5 md5 = MD5.Create();
item.Chunk.Hash = md5.ComputeHash(item.buffer);
if (processCount++ > 1000)
{
processCount = 0;
monitor.Set();
}
}
}
}, token);
foreach (var chunk in chunks)
{
if (hashQueue.Count > 10000)
{
monitor.WaitOne();
}
QueueItem item = new QueueItem()
{
buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES],
Chunk = chunk
};
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
MemoryMappedFile mms = files[source.Filename];
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(item.buffer, 0, source.Length);
}
sw.Restart();
sw.Stop();
hashQueue.Enqueue(item);
}
foreach (var x in files.Values)
{
x.Dispose();
}
}
I'm new to C# too, but I think what your are looking for is System.IO.MemoryMappedFiles namespace since C# 4.0
Using this API functions the operating system itself takes care how to manage the current file region in memory.
In stead of copy&paste code here, continue reading this article: http://www.developer.com/net/article.php/3828586/Using-Memory-Mapped-Files-in-NET-40.htm
Regarding the MD5 use the System.Security.Cryptography.MD5CryptoServiceProvider class. Maybe it's faster.
In your case where you have to go over the "boundaries" of one file, do it. Let the operating system handle how the memory mapped files are represented in memory. Work as you would do with "small" sized buffers.
In .Net 4 you now have System.IO.MemoryMappedFiles
You can create a ViewAccessor of a particular chuck size to match your hash function, and then just keep filling your hash function buffer from the current ViewAccessor, when you run out of file, start chunking the next file using the current hash chuck offset as your ViewAccessor offset