If I'm creating an object and I want it to only last for the current code block, after which it will be destroyed (or at least marked for destruction) automatically, I can use the scope keyword:
using System;
class Program
{
public void Program()
{
let basicString = String("Basic String");
for (let i = 0 ; i < 4 ; i++)
{
let modifiedString = scope String(s);
if (i%2 == 0)
{
modifiedString.ToUpper();
}
else
{
modifiedString.ToLower();
}
Console.WriteLine(modifiedString);
// modifiedString is marked for destruction
}
}
}
But, if I need the object to remain until after the block, such as for the whole method or for a block outside of the one it was created, how can I specify that scope? For instance, how can I ensure that the strings exist in the second loop below?
using System;
using System.Collections.Generic;
class Program
{
public void Program()
{
let basicString = String("Basic String");
let modifiedStringList = scope List<String>();
for (let i = 0 ; i < 4 ; i++)
{
let modifiedString = scope String(s);
if (i%2 == 0)
{
modifiedString.ToUpper();
}
else
{
modifiedString.ToLower();
}
modifiedStringList.Append(modifiedString);
Console.WriteLine(modifiedString);
// somehow prevent modifiedString from being marked for destruction
}
modifiedStringList.RemoveAt(1);
for (let s in modifiedStringList)
{
Console.WriteLine(s);
}
// allow all of the modifiedString to be marked for destruction here
}
}
You can use scope:: to not let the variable be marked for destruction for the entire method:
using System;
using System.Collections.Generic;
class Program
{
public void Program()
{
let basicString = String("Basic String");
let modifiedStringList = scope List<String>();
for (let i = 0 ; i < 4 ; i++)
{
// modifiedString won't be destroyed until after Program() exits.
let modifiedString = scope:: String(s);
if (i%2 == 0)
{
modifiedString.ToUpper();
}
else
{
modifiedString.ToLower();
}
modifiedStringList.Append(modifiedString);
Console.WriteLine(modifiedString);
}
modifiedStringList.RemoveAt(1);
for (let s in modifiedStringList)
{
Console.WriteLine(s);
}
}
}
If you need to specify an arbitrary scope between the method and the current block, you can use a named block with BlockName: { ... } and use scope:BlockName:
using System;
using System.Collections.Generic;
class Program
{
public void Program()
{
for (let repeat=0; repeat<10; repeat++)
RepeatBlock:
{
let basicString = String("Basic String");
let modifiedStringList = scope List<String>();
for (let i = 0 ; i < 4 ; i++)
{
// modifiedString won't be destroyed until after
// the block named RepeatBlock block exits.
let modifiedString = scope:RepeatBlock String(s);
if (i%2 == 0)
{
modifiedString.ToUpper();
}
else
{
modifiedString.ToLower();
}
modifiedStringList.Append(modifiedString);
Console.WriteLine(modifiedString);
}
modifiedStringList.RemoveAt(1);
for (let s in modifiedStringList)
{
Console.WriteLine(s);
}
}
}
}
I recently started learning Kotlin and the one thing I noticed is the for-loop syntax of Kotlin is different from traditional for-loop syntax and for me it is a bit confusing...I tried to search it on google but didn't get my answer.
How would I duplicate the following Java for loop?
for (int i = 0; i <= 100; i++) {
System.out.println(i);
}
Here is a Java for loop to iterate 100 times:
for (int i = 0; i <= 100; i++) {
System.out.println(i);
}
Here is the Kotlin equivalent:
for (i in 0..100) {
println(i)
}
Here is a Java for loop that will iterate through a list:
for (int i = 0; i < list.size(); i++) {
Object item = list.get(i);
// Do something with item
}
Kotlin equivalent:
for (i in list.indices) {
val item = list[i]
// Do something with item
}
Here is another Kotlin equivalent for iterating a list:
for (i in 0 until list.size) {
val item = list[i]
// Do something with item
}
Java for-each loop:
for (Object item : list) {
// Do something with item
}
Kotlin for-each loop:
for (item in list) {
// Do something with item
}
val scanner = Scanner(System.`in`)
var nos = Array<Int>(5) { 0 }
for (i in 1..3) {
nos[i] = scanner.nextInt()
}
println("Given values $nos")
Here, you can see i in 1..3 and you do not need to declare var i : Int = 1 as it'll be declared for you in the loop. Nor do you need the i = i+1 inside the loop for that matter.
I want to concatenate a bunch of different files of a single type into one large file. For example, many javascript files into one large file, many css files down to one etc. I want to create a sourcemap of the files pre concatenation, but I do not know where to start. I am working in Node, but I am also open to solutions in other environments.
I know there are tools that can do this, but they seem to be on a language by language basis (uglifyjs, cssmin or whatever its called these days), but I want a tool that is not language specific.
Also, I would like to define how the files are bound. For example, in javascript I want to give each file its own closure with an IIFE. Such as:
(function () {
// File
}());
I can also think of other wrappers I would like to implement for different files.
Here are my options as I see it right now. However, I don't know which is best or how to start any of them.
Find a module that does this (I'm working in a Node.js environment)
Create an algorithm with Mozilla's source-map module. For that I also see a couple options.
Only map each line to the new line location
Map every single character to the new location
Map every word to its new location (this options seems way out of scope)
Don't even worry about source maps
What do you guys think about these options. I've already tried options 2.1 and 2.2, but the solution seemed way too complicated for a concatenation algorithm and it did not perform perfectly in the Google Chrome browser tools.
I implemented code without any dependencies like this:
export interface SourceMap {
version: number; // always 3
file?: string;
sourceRoot?: string;
sources: string[];
sourcesContent?: string[];
names?: string[];
mappings: string | Buffer;
}
const emptySourceMap: SourceMap = { version: 3, sources: [], mappings: new Buffer(0) }
var charToInteger = new Buffer(256);
var integerToChar = new Buffer(64);
charToInteger.fill(255);
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/='.split('').forEach((char, i) => {
charToInteger[char.charCodeAt(0)] = i;
integerToChar[i] = char.charCodeAt(0);
});
class DynamicBuffer {
buffer: Buffer;
size: number;
constructor() {
this.buffer = new Buffer(512);
this.size = 0;
}
ensureCapacity(capacity: number) {
if (this.buffer.length >= capacity)
return;
let oldBuffer = this.buffer;
this.buffer = new Buffer(Math.max(oldBuffer.length * 2, capacity));
oldBuffer.copy(this.buffer);
}
addByte(b: number) {
this.ensureCapacity(this.size + 1);
this.buffer[this.size++] = b;
}
addVLQ(num: number) {
var clamped: number;
if (num < 0) {
num = (-num << 1) | 1;
} else {
num <<= 1;
}
do {
clamped = num & 31;
num >>= 5;
if (num > 0) {
clamped |= 32;
}
this.addByte(integerToChar[clamped]);
} while (num > 0);
}
addString(s: string) {
let l = Buffer.byteLength(s);
this.ensureCapacity(this.size + l);
this.buffer.write(s, this.size);
this.size += l;
}
addBuffer(b: Buffer) {
this.ensureCapacity(this.size + b.length);
b.copy(this.buffer, this.size);
this.size += b.length;
}
toBuffer(): Buffer {
return this.buffer.slice(0, this.size);
}
}
function countNL(b: Buffer): number {
let res = 0;
for (let i = 0; i < b.length; i++) {
if (b[i] === 10) res++;
}
return res;
}
export class SourceMapBuilder {
outputBuffer: DynamicBuffer;
sources: string[];
mappings: DynamicBuffer;
lastSourceIndex = 0;
lastSourceLine = 0;
lastSourceCol = 0;
constructor() {
this.outputBuffer = new DynamicBuffer();
this.mappings = new DynamicBuffer();
this.sources = [];
}
addLine(text: string) {
this.outputBuffer.addString(text);
this.outputBuffer.addByte(10);
this.mappings.addByte(59); // ;
}
addSource(content: Buffer, sourceMap?: SourceMap) {
if (sourceMap == null) sourceMap = emptySourceMap;
this.outputBuffer.addBuffer(content);
let sourceLines = countNL(content);
if (content.length > 0 && content[content.length - 1] !== 10) {
sourceLines++;
this.outputBuffer.addByte(10);
}
let sourceRemap = [];
sourceMap.sources.forEach((v) => {
let pos = this.sources.indexOf(v);
if (pos < 0) {
pos = this.sources.length;
this.sources.push(v);
}
sourceRemap.push(pos);
});
let lastOutputCol = 0;
let inputMappings = (typeof sourceMap.mappings === "string") ? new Buffer(<string>sourceMap.mappings) : <Buffer>sourceMap.mappings;
let outputLine = 0;
let ip = 0;
let inOutputCol = 0;
let inSourceIndex = 0;
let inSourceLine = 0;
let inSourceCol = 0;
let shift = 0;
let value = 0;
let valpos = 0;
const commit = () => {
if (valpos === 0) return;
this.mappings.addVLQ(inOutputCol - lastOutputCol);
lastOutputCol = inOutputCol;
if (valpos === 1) {
valpos = 0;
return;
}
let outSourceIndex = sourceRemap[inSourceIndex];
this.mappings.addVLQ(outSourceIndex - this.lastSourceIndex);
this.lastSourceIndex = outSourceIndex;
this.mappings.addVLQ(inSourceLine - this.lastSourceLine);
this.lastSourceLine = inSourceLine;
this.mappings.addVLQ(inSourceCol - this.lastSourceCol);
this.lastSourceCol = inSourceCol;
valpos = 0;
}
while (ip < inputMappings.length) {
let b = inputMappings[ip++];
if (b === 59) { // ;
commit();
this.mappings.addByte(59);
inOutputCol = 0;
lastOutputCol = 0;
outputLine++;
} else if (b === 44) { // ,
commit();
this.mappings.addByte(44);
} else {
b = charToInteger[b];
if (b === 255) throw new Error("Invalid sourceMap");
value += (b & 31) << shift;
if (b & 32) {
shift += 5;
} else {
let shouldNegate = value & 1;
value >>= 1;
if (shouldNegate) value = -value;
switch (valpos) {
case 0: inOutputCol += value; break;
case 1: inSourceIndex += value; break;
case 2: inSourceLine += value; break;
case 3: inSourceCol += value; break;
}
valpos++;
value = shift = 0;
}
}
}
commit();
while (outputLine < sourceLines) {
this.mappings.addByte(59);
outputLine++;
}
}
toContent(): Buffer {
return this.outputBuffer.toBuffer();
}
toSourceMap(sourceRoot?: string): Buffer {
return new Buffer(JSON.stringify({ version: 3, sourceRoot, sources: this.sources, mappings: this.mappings.toBuffer().toString() }));
}
}
I, at first, implemented "index map" from that spec, only to find out that it is not supported by any browser.
Another project that could be useful to look at is magic string.
I was working on massive collision detection for my game(more than 1000 sprites is massive for my game), and i was searching to find a way to implement this, then i reached to quad tree:
http://en.wikipedia.org/wiki/Quadtree
Well it's approach to reduce the number of objects that should be check for collision by dividing them to the groups of objects which have more chance to collide.
I found a java version of quad tree here:
http://gamedev.tutsplus.com/tutorials/implementation/quick-tip-use-quadtrees-to-detect-likely-collisions-in-2d-space/
Then i change it and use it for my javafx game. but the performance wasn't really good for huge number of objects, so i made some optimisation on it.
Well i used AnimationTimer for each tree to check for collisions which has improved performance so much. i think Animation Timer use GPU to process because when i run my code CPU usage doesn't go hight(3% to 5% - 1640 sprites). but if i use Thread instead of AnimationTimer it use much more CPU(about 40% to 50% - 1640 sprites).
import java.util.ArrayList;
import java.util.List;
import javafx.animation.AnimationTimer;
import javafx.scene.layout.Region;
import javafx.scene.layout.RegionBuilder;
import javafx.scene.paint.Color;
import viwofx.sprit.Sprite;
import viwofx.ui.GameScene;
public class QuadTree
{
private int MAX_OBJECTS = 10;
private int MAX_LEVELS = 5;
private int level;
private ArrayList<Sprite> sprites;
private ArrayList<Sprite> unAllocatedSprites;
private Region bounds;
private QuadTree[] nodes;
private QuadTree parent;
private AnimationTimer detection;
private boolean detecting = false;
private QuadTree getqt()
{
return this;
}
public QuadTree(QuadTree p, int pLevel, Region pBounds)
{
this.parent = p;
level = pLevel;
sprites = new ArrayList<>(0);
unAllocatedSprites = new ArrayList<>(0);
bounds = pBounds;
nodes = new QuadTree[4];
detection = new AnimationTimer()
{
#Override
public void handle(long l)
{
// This for happens when this node has child nodes and there is some object which can not fit whitin the bounds of child nodes
// these object being checked till they can fit inside the bounds of child nodes then they will be added to correspinding child node,
// or object is out of bounds then it will be pushed to the parent node
for (int i = 0; i < unAllocatedSprites.size(); i++)
{
if (!isInside(unAllocatedSprites.get(i)))
{
pushToParent(unAllocatedSprites.get(i));
continue;
}
int index = getIndex(unAllocatedSprites.get(i));
if (index != -1)
{
nodes[index].add(unAllocatedSprites.remove(i));
}
}
for (int i = 0; i < sprites.size(); i++)
{
Sprite ts = sprites.get(i);
if (isInside(ts))
{
int ii = 0;
for (ii = 0; ii < sprites.size(); ii++)
{
Sprite ts2 = sprites.get(ii);
if (ts != ts2)
{
Your collision detection logic
}
}
if (parent != null)
{
for (ii = 0; ii < parent.getUnAllocatedSprites().size(); ii++)
{
Sprite ts2 = parent.getUnAllocatedSprites().get(ii);
if (ts != ts2 && isInside(ts2))
{
Your collision detection logic
}
}
}
}
else
{
pushToParent(ts);
}
}
}
};
}
public int getLevel()
{
return level;
}
public ArrayList<Sprite> getUnAllocatedSprites()
{
return unAllocatedSprites;
}
// Split the node into 4 subnodes
private void split()
{
double subWidth = (bounds.getPrefWidth() / 2);
double subHeight = (bounds.getPrefHeight() / 2);
double x = bounds.getLayoutX();
double y = bounds.getLayoutY();
nodes[0] = new QuadTree(this, level + 1, RegionBuilder.create().layoutX(x).layoutY(y).prefWidth(subWidth).prefHeight(subHeight).build());
nodes[1] = new QuadTree(this, level + 1, RegionBuilder.create().layoutX(x + subWidth).layoutY(y).prefWidth(subWidth).prefHeight(subHeight).build());
nodes[2] = new QuadTree(this, level + 1, RegionBuilder.create().layoutX(x).layoutY(y + subHeight).prefWidth(subWidth).prefHeight(subHeight).build());
nodes[3] = new QuadTree(this, level + 1, RegionBuilder.create().layoutX(x + subWidth).layoutY(y + subHeight).prefWidth(subWidth).prefHeight(subHeight).build());
}
private int getIndex(Sprite s)
{
int index = -1;
double verticalMidpoint = bounds.getLayoutX() + (bounds.getPrefWidth() / 2);
double horizontalMidpoint = bounds.getLayoutY() + (bounds.getPrefHeight() / 2);
double spriteMaxX = (s.getNode().getTranslateX() + s.getWidth());
double spriteMaxY = (s.getNode().getTranslateY() + s.getHeight());
// Object can completely fit within the top quadrants
boolean topQuadrant = (spriteMaxY < horizontalMidpoint);
// Object can completely fit within the bottom quadrants
boolean bottomQuadrant = (s.getNode().getTranslateY() >= horizontalMidpoint);
// Object can completely fit within the left quadrants
if (s.getNode().getTranslateX() >= bounds.getLayoutX() && spriteMaxX < verticalMidpoint)
{
if (topQuadrant)
{
index = 0;
}
else if (bottomQuadrant)
{
index = 2;
}
}
// Object can completely fit within the right quadrants
else if (s.getNode().getTranslateX() >= verticalMidpoint && (s.getNode().getTranslateX() + s.getWidth()) < (bounds.getLayoutX() + bounds.getPrefWidth()))
{
if (topQuadrant)
{
index = 1;
}
else if (bottomQuadrant)
{
index = 3;
}
}
return index;
}
public boolean isInside(Sprite s)
{
double maxX = bounds.getLayoutX() + bounds.getPrefWidth();
double maxY = bounds.getLayoutY() + bounds.getPrefHeight();
// Object can completely fit within the left quadrants
if (s.getNode().getTranslateX() >= bounds.getLayoutX() && (s.getNode().getTranslateX() + s.getWidth()) < maxX && s.getNode().getTranslateY() >= bounds.getLayoutY() && (s.getNode().getTranslateY() + s.getHeight()) < maxY)
{
return true;
}
if (parent != null && parent.getUnAllocatedSprites().contains(s))
{
return true;
}
return false;
}
public void pushToParent(Sprite s)
{
sprites.remove(s);
unAllocatedSprites.remove(s);
if (parent == null)
{
//System.out.println("parent");
if (!unAllocatedSprites.contains(s))
{
unAllocatedSprites.add(s);
}
return;
}
parent.add(s);
if (sprites.size() < 1 && unAllocatedSprites.size() < 1)
{
stopDetection();
}
}
public void add(viwofx.sprit.Sprite sprite)
{
// if sprite is not fit in the bounds of node, it will be pushed to the parent node.
// this is a optimization for when child node push a object to this node and object still is not fit in the bounds this node,
// so it will be pushed to the parent node till object can be fited whitin the node bounds
// this if prevent of out of bounds object to being added to unAllocatedSprites and then being pushed to parent
if (!isInside(sprite))
{
pushToParent(sprite);
return;
}
// if tree has been splited already add sprite to corrosponding child
if (nodes[0] != null)
{
int index = getIndex(sprite);
if (index != -1)
{
nodes[index].add(sprite);
return;
}
else
{
unAllocatedSprites.add(sprite);
return;
}
}
sprites.add(sprite);
if (!detecting)
{
startDetection();
}
if (sprites.size() > MAX_OBJECTS && level < MAX_LEVELS)
{
if (nodes[0] == null)
{
split();
}
int i = 0;
while (i < sprites.size())
{
int index = getIndex(sprites.get(i));
if (index != -1)
{
nodes[index].add(sprites.remove(i));
}
else
{
unAllocatedSprites.add(sprites.remove(i));
}
}
}
}
public List<Sprite> retrieve(List<Sprite> returnObjects, Sprite pRect)
{
int index = getIndex(pRect);
if (index != -1 && nodes[0] != null)
{
nodes[index].retrieve(returnObjects, pRect);
}
returnObjects.addAll(sprites);
return returnObjects;
}
public void startDetection()
{
detecting = true;
detection.start();
}
public void stopDetection()
{
//detecting = false;
//detection.stop();
}
}
I hope this will be helpful for you.
I have a list of files, which need to be read, in chunks, into a byte[], which is then passed to a hashing function. The tricky part is this: if I reach the end of a file, I need to continue reading the next file untill I fill the buffer, like so:
read 16 bits as an example:
File 1: 00101010
File 2: 01101010111111111
would need to be read as 0010101001101010
The point is: these files can be as large as several gigabytes, and I don't want to completely load them into memory. Loading pieces into a buffer of, like, 30 MB would be perfectly fine.
I want to use threading, but would it be efficient to thread reading a file? I don't know if Disc I/O is such a large bottleneck that this would be worth it. Would the hashing be sped up sufficently if I only thread that part, and lock on the read of each chunk? It is important the hashes get saved in the correct order.
The second thing I need to do, is to generate the MD5sum from each file as well. Is there anyway to do this more efficiently than doing this as a separate step?
(This question has some overlap with Is there a built-in way to handle multiple files as one stream?, but I thought this differed enough)
I am really stumped what approach to take, as I am fairly new to C#, as well as to threading. I already tried the approaches listed below, but they do not suffice for me.
As I am new to C# I value every kind of input on any aspect of my code.
This piece of code was threaded, but does not 'append' the streams, and as such generates invalid hashes:
public void DoHashing()
{
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = numThreads;
options.CancellationToken = cancelToken.Token;
Parallel.ForEach(files, options, (string f, ParallelLoopState loopState) =>
{
options.CancellationToken.ThrowIfCancellationRequested();
using (BufferedStream fileStream = new BufferedStream(File.OpenRead(f), bufferSize))
{
// Get the MD5sum first:
using (MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider())
{
md5.Initialize();
md5Sums[f] = BitConverter.ToString(md5.ComputeHash(fileStream)).Replace("-", "");
}
//setup for reading:
byte[] buffer = new byte[(int)pieceLength];
//I don't know if the buffer will f*ck up the filelenghth
long remaining = (new FileInfo(f)).Length;
int done = 0;
while (remaining > 0)
{
while (done < pieceLength)
{
options.CancellationToken.ThrowIfCancellationRequested();
//either try to read the piecelength, or the remaining length of the file.
int toRead = (int)Math.Min(pieceLength - done, remaining);
int read = fileStream.Read(buffer, done, toRead);
//if read == 0, EOF reached
if (read == 0)
{
remaining = 0;
break;
}
//offsets
done += read;
remaining -= read;
}
// Hash the piece
using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
{
sha1.Initialize();
byte[] hash = sha1.ComputeHash(buffer);
hashes[f].AddRange(hash);
}
done = 0;
buffer = new byte[(int)pieceLength];
}
}
}
);
}
This other piece of code isn't threaded (and doesn't calculate MD5):
void Hash()
{
//examples, these got handled by other methods
List<string> files = new List<string>();
files.Add("a.txt");
files.Add("b.blob");
//....
long totalFileLength;
int pieceLength = Math.Pow(2,20);
foreach (string file in files)
{
totalFileLength += (new FileInfo(file)).Length;
}
//Reading the file:
long remaining = totalFileLength;
byte[] buffer = new byte[Math.min(remaining, pieceSize)];
int index = 0;
FileStream fin = File.OpenRead(files[index]);
int done = 0;
int offset = 0;
while (remaining > 0)
{
while (done < pieceLength)
{
int toRead = (int)Math.Min(pieceLength - offset, remaining);
int read = fin.Read(buffer, done, toRead);
//if read == 0, EOF reached
if (read == 0)
{
index++;
//if last file:
if (index > files.Count)
{
remaining = 0;
break;
}
//get ready for next round:
offset = 0;
fin.OpenRead(files[index]);
}
done += read;
offset += read;
remaining -= read;
}
//Doing the piece hash:
HashPiece(buffer);
//reset for next piece:
done = 0;
byte[] buffer = new byte[Math.min(remaining, pieceSize)];
}
}
void HashPiece(byte[] piece)
{
using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
{
sha1.Initialize();
//hashes is a List
hashes.Add(sha1.ComputeHash(piece));
}
}
Thank you very much for your time and effort.
I'm not looking for completely coded solutions, any pointer and idea where to go with this would be excellent.
Questions & remarks to yodaj007's answer:
Why if (currentChunk.Length >= Constants.CHUNK_SIZE_IN_BYTES)? Why not ==? If the chunk is larger than the chunk size, my SHA1 hash gets a different value.
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = 0,
Length = (int)Math.Min(fi.Length, (long)needed)
});
Is a really interesting idea. Postpone reading untill you need it. Nice!
chunks.Add(currentChunk = new Chunk());
Why do this in the if (currentChunk != null) block and in the for (int i = 0; i < (fi.Length - offset) / Constants.CHUNK_SIZE_IN_BYTES; i++) block? Isn't the first a bit redundant?
Here is my complete answer. I tested it on one of my anime folders. It processes 14 files totaling 3.64GiB in roughly 16 seconds. In my opinion, using any sort of parallelism is more trouble than it is worth here. You're being limited by disc I/O, so multithreading will only get you so far. My solution can be easily parallelized though.
It starts by reading "chunk" source information: source file, offset, and length. All of this is gathered very quickly. From here, you can process the "chunks" using threading however you wish. Code follows:
public static class Constants
{
public const int CHUNK_SIZE_IN_BYTES = 32 * 1024 * 1024; // 32MiB
}
public class ChunkSource
{
public string Filename { get; set; }
public int StartPosition { get; set; }
public int Length { get; set; }
}
public class Chunk
{
private List<ChunkSource> _sources = new List<ChunkSource>();
public IList<ChunkSource> Sources { get { return _sources; } }
public byte[] Hash { get; set; }
public int Length
{
get { return Sources.Select(s => s.Length).Sum(); }
}
}
static class Program
{
static void Main()
{
DirectoryInfo di = new DirectoryInfo(#"C:\Stuff\Anime\Shikabane Hime Aka");
string[] filenames = di.GetFiles().Select(fi=> fi.FullName).OrderBy(n => n).ToArray();
var chunks = ChunkFiles(filenames);
ComputeHashes(chunks);
}
private static List<Chunk> ChunkFiles(string[] filenames)
{
List<Chunk> chunks = new List<Chunk>();
Chunk currentChunk = null;
int offset = 0;
foreach (string filename in filenames)
{
FileInfo fi = new FileInfo(filename);
if (!fi.Exists)
throw new FileNotFoundException(filename);
Debug.WriteLine(String.Format("File: {0}", filename));
//
// First, start off by either starting a new chunk or
// by finishing a leftover chunk from a previous file.
//
if (currentChunk != null)
{
//
// We get here if the previous file had leftover bytes that left us with an incomplete chunk
//
int needed = Constants.CHUNK_SIZE_IN_BYTES - currentChunk.Length;
if (needed == 0)
throw new InvalidOperationException("Something went wonky, shouldn't be here");
offset = needed;
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = 0,
Length = (int)Math.Min(fi.Length, (long)needed)
});
if (currentChunk.Length >= Constants.CHUNK_SIZE_IN_BYTES)
{
chunks.Add(currentChunk = new Chunk());
}
}
else
{
offset = 0;
}
//
// Note: Using integer division here
//
for (int i = 0; i < (fi.Length - offset) / Constants.CHUNK_SIZE_IN_BYTES; i++)
{
chunks.Add(currentChunk = new Chunk());
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = i * Constants.CHUNK_SIZE_IN_BYTES + offset,
Length = Constants.CHUNK_SIZE_IN_BYTES
});
Debug.WriteLine(String.Format("Chunk source created: Offset = {0,10}, Length = {1,10}", currentChunk.Sources[0].StartPosition, currentChunk.Sources[0].Length));
}
int leftover = (int)(fi.Length - offset) % Constants.CHUNK_SIZE_IN_BYTES;
if (leftover > 0)
{
chunks.Add(currentChunk = new Chunk());
currentChunk.Sources.Add(new ChunkSource()
{
Filename = fi.FullName,
StartPosition = (int)(fi.Length - leftover),
Length = leftover
});
}
else
{
currentChunk = null;
}
}
return chunks;
}
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
foreach (var chunk in chunks)
{
MemoryMappedFile mms = null;
byte[] buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES];
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
lock (files)
{
if (!files.TryGetValue(source.Filename, out mms))
{
Debug.WriteLine(String.Format("Opening {0}", source.Filename));
files.Add(source.Filename, mms = MemoryMappedFile.CreateFromFile(source.Filename, FileMode.Open));
}
}
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(buffer, 0, source.Length);
}
Debug.WriteLine("Done reading sources in {0}ms", sw.Elapsed.TotalMilliseconds);
sw.Restart();
MD5 md5 = MD5.Create();
chunk.Hash = md5.ComputeHash(buffer);
sw.Stop();
Debug.WriteLine(String.Format("Computed hash: {0} in {1}ms", String.Join("-", chunk.Hash.Select(h=> h.ToString("X2")).ToArray()), sw.Elapsed.TotalMilliseconds));
}
foreach (var x in files.Values)
{
x.Dispose();
}
}
}
I don't guarantee everything is spotlessly free of bugs. But I did have fun working on it. Look at the output window in Visual Studio for the debug information. It looks like this:
File: C:\Stuff\Anime\Shikabane Hime Aka\Episode 02.mkv
Chunk source created: Offset = 26966010, Length = 33554432
Chunk source created: Offset = 60520442, Length = 33554432
Chunk source created: Offset = 94074874, Length = 33554432
Chunk source created: Offset = 127629306, Length = 33554432
Chunk source created: Offset = 161183738, Length = 33554432
Chunk source created: Offset = 194738170, Length = 33554432
Chunk source created: Offset = 228292602, Length = 33554432
...
Opening C:\Stuff\Anime\Shikabane Hime Aka\Episode 02.mkv
Done reading sources in 42.9362ms
The thread '' (0xc10) has exited with code 0 (0x0).
Computed hash: 3C-81-A5-2C-90-02-24-23-42-5B-19-A2-15-56-AB-3F in 94.2481ms
Done reading sources in 0.0053ms
Computed hash: 58-F0-6D-D5-88-D8-FF-B3-BE-B4-6A-DA-63-09-43-6B in 98.9263ms
Done reading sources in 29.4805ms
Computed hash: F7-19-8D-A8-FE-9C-07-6E-DB-D5-74-A6-E1-E7-A6-26 in 85.0061ms
Done reading sources in 28.4971ms
Computed hash: 49-F2-CB-75-89-9A-BC-FA-94-A7-DF-E0-DB-02-8A-99 in 84.2799ms
Done reading sources in 31.106ms
Computed hash: 29-7B-18-BD-ED-E9-0C-68-4B-47-C6-5F-D0-16-8A-44 in 84.1444ms
Done reading sources in 31.2204ms
Computed hash: F8-91-F1-90-CF-9C-37-4E-82-68-C2-44-0D-A7-6E-F8 in 84.2592ms
Done reading sources in 31.0031ms
Computed hash: 65-97-ED-95-07-31-BF-C8-3A-BA-2B-DA-03-37-FD-00 in 95.331ms
Done reading sources in 33.0072ms
Computed hash: 9B-F2-83-E6-A8-DF-FD-8D-6C-5C-9E-F4-20-0A-38-4B in 85.9561ms
Done reading sources in 31.6232ms
Computed hash: B6-7C-6B-95-69-BC-9C-B2-1A-07-B3-13-28-A8-10-BC in 84.1866ms
Here is the parallel version. It's basically the same really. Using parallelism = 3 cut the processing time down to 9 seconds.
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
Parallel.ForEach(chunks, new ParallelOptions() { MaxDegreeOfParallelism = 2 }, (chunk, state, index) =>
{
MemoryMappedFile mms = null;
byte[] buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES];
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
lock (files)
{
if (!files.TryGetValue(source.Filename, out mms))
{
Debug.WriteLine(String.Format("Opening {0}", source.Filename));
files.Add(source.Filename, mms = MemoryMappedFile.CreateFromFile(source.Filename, FileMode.Open));
}
}
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(buffer, 0, source.Length);
}
Debug.WriteLine("Done reading sources in {0}ms", sw.Elapsed.TotalMilliseconds);
sw.Restart();
MD5 md5 = MD5.Create();
chunk.Hash = md5.ComputeHash(buffer);
sw.Stop();
Debug.WriteLine(String.Format("Computed hash: {0} in {1}ms", String.Join("-", chunk.Hash.Select(h => h.ToString("X2")).ToArray()), sw.Elapsed.TotalMilliseconds));
});
foreach (var x in files.Values)
{
x.Dispose();
}
}
EDIT
I found a bug, or what I think is a bug. Need to set the read offset to 0 if we're starting a new file.
EDIT 2 based on feedback
This processes the hashes in a separate thread. It's necessary to throttle the I/O. I was running into OutOfMemoryException without doing so. It doesn't really perform that much better, though. Beyond this... I'm not sure how it can be improved any further. Perhaps by reusing the buffers, maybe.
public class QueueItem
{
public Chunk Chunk { get; set; }
public byte[] buffer { get; set; }
}
private static void ComputeHashes(IList<Chunk> chunks)
{
if (chunks == null || chunks.Count == 0)
return;
Dictionary<string, MemoryMappedFile> files = new Dictionary<string, MemoryMappedFile>();
foreach (var filename in chunks.SelectMany(c => c.Sources).Select(c => c.Filename).Distinct())
{
files.Add(filename, MemoryMappedFile.CreateFromFile(filename, FileMode.Open));
}
AutoResetEvent monitor = new AutoResetEvent(false);
ConcurrentQueue<QueueItem> hashQueue = new ConcurrentQueue<QueueItem>();
CancellationToken token = new CancellationToken();
Task.Factory.StartNew(() =>
{
int processCount = 0;
QueueItem item = null;
while (!token.IsCancellationRequested)
{
if (hashQueue.TryDequeue(out item))
{
MD5 md5 = MD5.Create();
item.Chunk.Hash = md5.ComputeHash(item.buffer);
if (processCount++ > 1000)
{
processCount = 0;
monitor.Set();
}
}
}
}, token);
foreach (var chunk in chunks)
{
if (hashQueue.Count > 10000)
{
monitor.WaitOne();
}
QueueItem item = new QueueItem()
{
buffer = new byte[Constants.CHUNK_SIZE_IN_BYTES],
Chunk = chunk
};
Stopwatch sw = Stopwatch.StartNew();
foreach (var source in chunk.Sources)
{
MemoryMappedFile mms = files[source.Filename];
var view = mms.CreateViewStream(source.StartPosition, source.Length);
view.Read(item.buffer, 0, source.Length);
}
sw.Restart();
sw.Stop();
hashQueue.Enqueue(item);
}
foreach (var x in files.Values)
{
x.Dispose();
}
}
I'm new to C# too, but I think what your are looking for is System.IO.MemoryMappedFiles namespace since C# 4.0
Using this API functions the operating system itself takes care how to manage the current file region in memory.
In stead of copy&paste code here, continue reading this article: http://www.developer.com/net/article.php/3828586/Using-Memory-Mapped-Files-in-NET-40.htm
Regarding the MD5 use the System.Security.Cryptography.MD5CryptoServiceProvider class. Maybe it's faster.
In your case where you have to go over the "boundaries" of one file, do it. Let the operating system handle how the memory mapped files are represented in memory. Work as you would do with "small" sized buffers.
In .Net 4 you now have System.IO.MemoryMappedFiles
You can create a ViewAccessor of a particular chuck size to match your hash function, and then just keep filling your hash function buffer from the current ViewAccessor, when you run out of file, start chunking the next file using the current hash chuck offset as your ViewAccessor offset