Please can you help me with me threading. I'm trying to download a file and at the same time update a BTProgressHUD progress display. I know that the reason that it's not working is to do with the download using the main thread and not allowing me to update the UI but I can't work out how to correctly use the thread pool to allow me to update the BTProgressHUD whilst the file is downloading. Please help!!
`
BTProgressHUD.Show("Downloading...", progress);
string this_file = "example.pdf";
string file_url = "http://our_server.com/files/" + this_file;
Uri url = new Uri(file_url);
var documents = Environment.GetFolderPath (Environment.SpecialFolder.MyDocuments);
var folder = Path.Combine (documents, "", "PDF");
System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
response.Close();
Int64 iSize = response.ContentLength;
// keeps track of the total bytes downloaded so we can update the progress bar
Int64 iRunningByteTotal = 0;
// use the webclient object to download the file
using (System.Net.WebClient client = new System.Net.WebClient())
{
// open the file at the remote URL for reading
using (System.IO.Stream streamRemote = client.OpenRead(new Uri(file_url)))
{
// using the FileStream object, we can write the downloaded bytes to the file system
using (Stream streamLocal = new FileStream(folder + "/" + this_file, FileMode.Create, FileAccess.Write, FileShare.None))
{
// loop the stream and get the file into the byte buffer
int iByteSize = 0;
byte[] byteBuffer = new byte[iSize];
while ((iByteSize = streamRemote.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
{
// write the bytes to the file system at the file path specified
streamLocal.Write(byteBuffer, 0, iByteSize);
iRunningByteTotal += iByteSize;
// calculate the progress out of a base "100"
double dIndex = (double)(iRunningByteTotal);
double dTotal = (double)byteBuffer.Length;
double dProgressPercentage = (dIndex / dTotal);
int iProgressPercentage = (int)(dProgressPercentage * 100);
if (iProgressPercentage == 100)
{
var z = new UIAlertView ("Download Complete", this_file + " downloaded.", null, "OK", null);
z.Show();
BTProgressHUD.Dismiss();
}
if (iProgressPercentage % 10 == 0)
{
// THIS BUT NEVER HAPPENS!!! Cannot update the progress display
progress += 0.1f;
BTProgressHUD.Show("XXX", progress);
}
} // while..
streamLocal.Close(); // clean up the file stream
} // using stream
streamRemote.Close(); // close the connection to the remote server
} // using I.O
} // using system.net
`
Any help would be very very much appreciated.
I have used the TPL to kick of a background thread then called back to the UI by using InvokeOnMainThread. I have substituted the BTProgressHUD for a UILabel but it should work the same. Here is it working:
private void DownloadCoffeePDF()
{
Task.Factory.StartNew (() => {
InvokeOnMainThread(() => {
this.TheLabel.Text = string.Format("Downloading...{0}", progress);
});
string file_url = "http://www.pnf.org/coffeeedited041001.pdf";
Uri url = new Uri(file_url);
var documents = Environment.GetFolderPath (Environment.SpecialFolder.MyDocuments);
var folder = Path.Combine (documents, "", "PDF");
System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
response.Close();
Int64 iSize = response.ContentLength;
// keeps track of the total bytes downloaded so we can update the progress bar
Int64 iRunningByteTotal = 0;
// use the webclient object to download the file
using (System.Net.WebClient client = new System.Net.WebClient())
{
// open the file at the remote URL for reading
using (System.IO.Stream streamRemote = client.OpenRead(new Uri(file_url)))
{
// using the FileStream object, we can write the downloaded bytes to the file system
using (Stream streamLocal = new FileStream(folder + "/" + "Coffee.pdf", FileMode.Create, FileAccess.Write, FileShare.None))
{
// loop the stream and get the file into the byte buffer
int iByteSize = 0;
byte[] byteBuffer = new byte[iSize];
while ((iByteSize = streamRemote.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
{
// write the bytes to the file system at the file path specified
streamLocal.Write(byteBuffer, 0, iByteSize);
iRunningByteTotal += iByteSize;
// calculate the progress out of a base "100"
double dIndex = (double)(iRunningByteTotal);
double dTotal = (double)byteBuffer.Length;
double dProgressPercentage = (dIndex / dTotal);
int iProgressPercentage = (int)(dProgressPercentage * 100);
if (iProgressPercentage == 100)
{
InvokeOnMainThread(() => {
var z = new UIAlertView ("Download Complete", "Coffee.pdf" + " downloaded.", null, "OK", null);
z.Show();
this.TheLabel.Text = "Download Complete";
});
}
if (iProgressPercentage % 10 == 0)
{
InvokeOnMainThread(() => {
// THIS BUT NEVER HAPPENS!!! Cannot update the progress display
progress += 0.1f;
this.TheLabel.Text = string.Format("{0}", progress);
});
}
} // while..
streamLocal.Close(); // clean up the file stream
} // using stream
streamRemote.Close(); // close the connection to the remote server
} // using I.O
} // using system.net
});
}
Related
I'm trying to write an app that will listen to my computer audio and transcribe it using Google Speach Recognition.
I've been able to record the system sound using WasapiLoopbackCapture and I've been able to use google streaming recognition api with test files, but I was not able to merge the two togther.
When I stream the audio from the WasapiLoopbackCapture to google it doesn't return any result.
I've based my code on the google code sample at:
https://github.com/GoogleCloudPlatform/dotnet-docs-samples/blob/9588cee6d96bfe484c8e189e9ac2f6eaa3c3b002/speech/api/Recognize/InfiniteStreaming.cs#L225
private WaveInEvent StartListening()
{
var waveIn = new WaveInEvent
{
DeviceNumber = 0,
WaveFormat = new WaveFormat(SampleRate, ChannelCount)
};
waveIn.DataAvailable += (sender, args) =>
_microphoneBuffer.Add(ByteString.CopyFrom(args.Buffer, 0, args.BytesRecorded));
waveIn.StartRecording();
return waveIn;
}
And adjusted it to use the WasapiLoopbackCapture:
private IDisposable StartListening()
{
var waveIn = new WasapiLoopbackCapture();
//var waveIn = new WaveInEvent
//{
// DeviceNumber = 0,
// WaveFormat = new WaveFormat(SampleRate, ChannelCount)
//};
SampleRate = waveIn.WaveFormat.SampleRate;
ChannelCount = waveIn.WaveFormat.Channels;
BytesPerSecond = SampleRate * ChannelCount * BytesPerSample;
Console.WriteLine(SampleRate);
Console.WriteLine(BytesPerSecond);
waveIn.DataAvailable += (sender, args) =>
_microphoneBuffer.Add(ByteString.CopyFrom(args.Buffer, 0, args.BytesRecorded));
waveIn.StartRecording();
return waveIn;
}
But it doesn't return any transcribed text.
I've saved the input stream to a file, and it played ok - so the sound is getting there, my guess is that the waveFormat that is received from the WasapiLoopback is not compatible with what google likes - I tried some conversion and couldn't get it to work.
I've reviewed the following topics on stack overflow, but still couldn't get it to work:
Resampling WasapiLoopbackCapture
Naudio - Convert 32 bit wav to 16 bit wav
And tried combining them both:
private IDisposable StartListening()
{
var waveIn = new WasapiLoopbackCapture();
//var waveIn = new WaveInEvent
//{
//DeviceNumber = 0,
//WaveFormat = new WaveFormat(SampleRate, ChannelCount)
//};
// SampleRate = waveIn.WaveFormat.SampleRate;
// ChannelCount = waveIn.WaveFormat.Channels;
// BytesPerSecond = waveIn.WaveFormat.AverageBytesPerSecond;// SampleRate * ChannelCount * BytesPerSample;
var target = new WaveFormat(SampleRate, 16, 1);
var writer = new WaveFileWriter(#"c:\temp\xx.wav", waveIn.WaveFormat);
Console.WriteLine(SampleRate);
Console.WriteLine(BytesPerSecond);
var stop = false;
waveIn.DataAvailable += (sender, args) =>
{
var a = args;
byte[] newArray16Bit = new byte[args.BytesRecorded / 2];
short two;
float value;
for (int i = 0, j = 0; i < args.BytesRecorded; i += 4, j += 2)
{
value = (BitConverter.ToSingle(args.Buffer, i));
two = (short)(value * short.MaxValue);
newArray16Bit[j] = (byte)(two & 0xFF);
newArray16Bit[j + 1] = (byte)((two >> 8) & 0xFF);
}
var resampleStream = new NAudio.Wave.Compression.AcmStream(new WaveFormat(waveIn.WaveFormat.SampleRate
,16,waveIn.WaveFormat.Channels), target);
Buffer.BlockCopy(newArray16Bit, 0, resampleStream.SourceBuffer, 0, a.BytesRecorded/2);
int sourceBytesConverted = 0;
var bytes = resampleStream.Convert(a.BytesRecorded/2, out sourceBytesConverted);
var converted = new byte[bytes];
Buffer.BlockCopy(resampleStream.DestBuffer, 9, converted,0, bytes);
a = new WaveInEventArgs(converted,bytes);
_microphoneBuffer.Add(ByteString.CopyFrom(a.Buffer, 0, a.BytesRecorded));
if (writer != null)
{
writer.Write(a.Buffer, 0, a.BytesRecorded);
if (writer.Position > waveIn.WaveFormat.AverageBytesPerSecond * 5)
{
stop = true;
writer.Dispose();
writer = null;
Console.WriteLine("Saved file");
}
}
};
waveIn.StartRecording();
return waveIn;
}
But it doesn't work.
I'm not sure if this is the right path.
A code sample of a fix would be highly appreciated
I tried converting the bit rate etc.. but couldn't get this to work.
I am writing a sample pdf signer using itextsharp.dll n the digital signature is on a usb device.
it's working nicely only problem is that some times the signing gets completely stops for hours together.
I have found that the signing process goes to digital issuer site to get CRL to embed in the signed document increasing it's size from 100kb to 1300kb+ and also signs document very slow due to getting CRL from issuer site.
but if I disable internet the file gets signed very fast and size increase is just 5-10kb.
So my question precisely is how can I instruct the signing process not to go online to get CRL n simply sign the document fast with less size increase in original document.
public void Sign(ICollection<X509Certificate> chain, X509Certificate2 pk,
String digestAlgorithm, CryptoStandard subfilter,
String reason, String location,
int estimatedSize)
{
IList<ICrlClient> crlList = new List<ICrlClient>();
crlList.Add(new CrlClientOnline(chain));
// Creating the reader and the stamper
PdfReader reader = null;
PdfStamper stamper = null;
FileStream os = null;
//int noofpdfs = 0;
var watch = Stopwatch.StartNew();
try
{
string[] srcfiles;
if (lblSplitedFilePath.Text.Equals("Split By Page Directory"))
srcfiles = Directory.GetFiles(tbsrcpath.Text, "*.pdf");
else
if (tbsrcpath.Text.ToUpper().Contains(#"\SPLIT"))
srcfiles = Directory.GetFiles(tbsrcpath.Text, "*.pdf");
else
srcfiles = Directory.GetFiles(tbsrcpath.Text + #"\split\", "*.pdf");
//int noofpdfs = srcfilePaths.Count();
for (int i = 0; i < srcfiles.Count(); i++)
{
// Get FileName
lblOutputFile.Text = tbtgtpath.Text + #"\" + Path.GetFileName(srcfiles[i]);
lblOutputFile.Refresh();
os = new FileStream(lblOutputFile.Text, FileMode.Create);
reader = new PdfReader(srcfiles[i]);
iTextSharp.text.Rectangle pdfbox = reader.GetPageSize(1);
stamper = PdfStamper.CreateSignature(reader, os, '\0');
// Creating the appearance
PdfSignatureAppearance appearance = stamper.SignatureAppearance;
if (!string.IsNullOrWhiteSpace(tbContact.Text))
{
appearance.ReasonCaption = "Contact:";
appearance.Reason = tbContact.Text;// reason;
}
appearance.Location = location;
//Adding Image to Sign
if (cbAddImageSign.Checked)
{
var image = iTextSharp.text.Image.GetInstance(tbSignImage.Text);
appearance.Acro6Layers = true;
appearance.SignatureGraphic = image;
appearance.SignatureRenderingMode = PdfSignatureAppearance.RenderingMode.GRAPHIC_AND_DESCRIPTION;
}
int llx, lly, urx, ury;
int.TryParse(Tbllx.Text, out llx);
int.TryParse(tblly.Text, out lly);
int.TryParse(tburx.Text, out urx);
int.TryParse(tbury.Text, out ury);
pdfSigning.Properties.Settings.Default.llx = llx;
pdfSigning.Properties.Settings.Default.lly = lly;
pdfSigning.Properties.Settings.Default.urx = urx;
pdfSigning.Properties.Settings.Default.ury = ury;
appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(llx, lly, urx, ury), 1, "sig");
//Add Water mark
if (!lblWaterMarkImagePath.Text.Equals("Pdf Water Mark Image Path"))
{
var wmimage = iTextSharp.text.Image.GetInstance(lblWaterMarkImagePath.Text);
wmimage.SetAbsolutePosition(0, 0);
wmimage.ScaleToFit(100, 100);
for (var j = 0; j < reader.NumberOfPages; j++)
{
var content = stamper.GetUnderContent(j + 1);
content.AddImage(wmimage);
}
}
//appearance.setRenderingMode(PdfSignatureAppearance.RenderingMode.GRAPHIC);
// Creating the signature
try
{
IExternalSignature pks = new X509Certificate2Signature(pk, digestAlgorithm);
MakeSignature.SignDetached(appearance, pks, chain, crlList, null, null, estimatedSize,
subfilter);
}
catch (CryptographicException ex)
{
MessageBox.Show(ex.ToString());
}
//noofpdfs++;
if (cbPrintOnSign.Checked)
{
switch (tbPrintMethod.Text)
{
case "2":
SendFileToPrinter(lblOutputFile.Text, printpdf2printer);
break;
default:
SendToPrinter(lblOutputFile.Text);
break;
}
}
if (cbDeletePdfPostSign.Checked)
{
File.Delete(srcfiles[i]);
}
dgvPrintFiles.Rows.Add(srcfiles[i].ToString());
}
lblOutputFile.Text += #" Siging Over:Signed " + srcfiles.Count().ToString() + " Files";
}
finally
{
if (reader != null)
reader.Close();
if (stamper != null)
stamper.Close();
if (os != null)
os.Close();
}
watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;
MessageBox.Show("Signing Time:" + elapsedMs / 1000 + " Second");
}
I followed this NAudio Demo modified to play ShoutCast.
In my full code I have to resample the incoming audio and stream it again over the network to a network player. Since I get many "clicks and pops", I came back to the demo code and I found that these artifacts are originated after the decoding block.
If I save the incoming stream in mp3 format, it is pretty clear.
When I save the raw decoded data (without other processing than the decoder) I get many audio artifacts.
I wonder whether I am doing some error, even if my code is almost equal to the NAudio demo.
Here the function from the example as modified by me to save the raw data. It is called as a new Thread.
private void StreamMP3(object state)
{
//Configuration config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
//SettingsSection section = (SettingsSection)config.GetSection("system.net/settings");
this.fullyDownloaded = false;
string url = "http://icestreaming.rai.it/5.mp3";//(string)state;
webRequest = (HttpWebRequest)WebRequest.Create(url);
int metaInt = 0; // blocksize of mp3 data
int framesize = 0;
webRequest.Headers.Clear();
webRequest.Headers.Add("GET", "/ HTTP/1.0");
// needed to receive metadata informations
webRequest.Headers.Add("Icy-MetaData", "1");
webRequest.UserAgent = "WinampMPEG/5.09";
HttpWebResponse resp = null;
try
{
resp = (HttpWebResponse)webRequest.GetResponse();
}
catch (WebException e)
{
if (e.Status != WebExceptionStatus.RequestCanceled)
{
ShowError(e.Message);
}
return;
}
byte[] buffer = new byte[16384 * 4]; // needs to be big enough to hold a decompressed frame
try
{
// read blocksize to find metadata block
metaInt = Convert.ToInt32(resp.GetResponseHeader("icy-metaint"));
}
catch
{
}
IMp3FrameDecompressor decompressor = null;
byteOut = createNewFile(destPath, "salva", "raw");
try
{
using (var responseStream = resp.GetResponseStream())
{
var readFullyStream = new ReadFullyStream(responseStream);
readFullyStream.metaInt = metaInt;
do
{
if (mybufferedWaveProvider != null && mybufferedWaveProvider.BufferLength - mybufferedWaveProvider.BufferedBytes < mybufferedWaveProvider.WaveFormat.AverageBytesPerSecond / 4)
{
Debug.WriteLine("Buffer getting full, taking a break");
Thread.Sleep(500);
}
else
{
Mp3Frame frame = null;
try
{
frame = Mp3Frame.LoadFromStream(readFullyStream, true);
if (metaInt > 0)
UpdateSongName(readFullyStream.SongName);
else
UpdateSongName("No Song Info in Stream...");
}
catch (EndOfStreamException)
{
this.fullyDownloaded = true;
// reached the end of the MP3 file / stream
break;
}
catch (WebException)
{
// probably we have aborted download from the GUI thread
break;
}
if (decompressor == null)
{
// don't think these details matter too much - just help ACM select the right codec
// however, the buffered provider doesn't know what sample rate it is working at
// until we have a frame
WaveFormat waveFormat = new Mp3WaveFormat(frame.SampleRate, frame.ChannelMode == ChannelMode.Mono ? 1 : 2, frame.FrameLength, frame.BitRate);
decompressor = new AcmMp3FrameDecompressor(waveFormat);
this.mybufferedWaveProvider = new BufferedWaveProvider(decompressor.OutputFormat);
this.mybufferedWaveProvider.BufferDuration = TimeSpan.FromSeconds(200); // allow us to get well ahead of ourselves
framesize = (decompressor.OutputFormat.Channels * decompressor.OutputFormat.SampleRate * (decompressor.OutputFormat.BitsPerSample / 8) * 20) / 1000;
//this.bufferedWaveProvider.BufferedDuration = 250;
}
int decompressed = decompressor.DecompressFrame(frame, buffer, 0);
//Debug.WriteLine(String.Format("Decompressed a frame {0}", decompressed));
mybufferedWaveProvider.AddSamples(buffer, 0, decompressed);
while (mybufferedWaveProvider.BufferedDuration.Milliseconds >= 20)
{
byte[] read = new byte[framesize];
mybufferedWaveProvider.Read(read, 0, framesize);
byteOut.Write(read, 0, framesize);
}
}
} while (playbackState != StreamingPlaybackState.Stopped);
Debug.WriteLine("Exiting");
// was doing this in a finally block, but for some reason
// we are hanging on response stream .Dispose so never get there
decompressor.Dispose();
}
}
finally
{
if (decompressor != null)
{
decompressor.Dispose();
}
}
}
OK i found the problem. I included the shoutcast metadata to the MP3Frame.
See the comment "HERE I COLLECT THE BYTES OF THE MP3 FRAME" to locate the correct point to get the MP3 frame with no streaming metadata.
The following code runs without audio artifacts:
private void SHOUTcastReceiverThread()
{
//-*- String server = "http://216.235.80.18:8285/stream";
//String serverPath = "/";
//String destPath = "C:\\temp\\"; // destination path for saved songs
HttpWebRequest request = null; // web request
HttpWebResponse response = null; // web response
int metaInt = 0; // blocksize of mp3 data
int count = 0; // byte counter
int metadataLength = 0; // length of metadata header
string metadataHeader = ""; // metadata header that contains the actual songtitle
string oldMetadataHeader = null; // previous metadata header, to compare with new header and find next song
//CircularQueueStream framestream = new CircularQueueStream(2048);
QueueStream framestream = new QueueStream();
framestream.Position = 0;
bool bNewSong = false;
byte[] buffer = new byte[512]; // receive buffer
byte[] dec_buffer = new byte[decSIZE];
Mp3Frame frame;
IMp3FrameDecompressor decompressor = null;
Stream socketStream = null; // input stream on the web request
// create web request
request = (HttpWebRequest)WebRequest.Create(server);
// clear old request header and build own header to receive ICY-metadata
request.Headers.Clear();
request.Headers.Add("GET", serverPath + " HTTP/1.0");
request.Headers.Add("Icy-MetaData", "1"); // needed to receive metadata informations
request.UserAgent = "WinampMPEG/5.09";
// execute request
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
return;
}
// read blocksize to find metadata header
metaInt = Convert.ToInt32(response.GetResponseHeader("icy-metaint"));
try
{
// open stream on response
socketStream = response.GetResponseStream();
var readFullyStream = new ReadFullyStream(socketStream);
frame = null;
// rip stream in an endless loop
do
{
if (IsBufferNearlyFull)
{
Debug.WriteLine("Buffer getting full, taking a break");
Thread.Sleep(500);
frame = null;
}
else
{
int bufLen = readFullyStream.Read(buffer, 0, buffer.Length);
try
{
if (framestream.CanRead && framestream.Length > 512)
frame = Mp3Frame.LoadFromStream(framestream);
else
frame = null;
}
catch (Exception ex)
{
frame = null;
}
if (bufLen < 0)
{
Debug.WriteLine("Buffer error 1: exit.");
return;
}
// processing RAW data
for (int i = 0; i < bufLen; i++)
{
// if there is a header, the 'headerLength' would be set to a value != 0. Then we save the header to a string
if (metadataLength != 0)
{
metadataHeader += Convert.ToChar(buffer[i]);
metadataLength--;
if (metadataLength == 0) // all metadata informations were written to the 'metadataHeader' string
{
string fileName = "";
string fileNameRaw = "";
// if songtitle changes, create a new file
if (!metadataHeader.Equals(oldMetadataHeader))
{
// flush and close old byteOut stream
if (byteOut != null)
{
byteOut.Flush();
byteOut.Close();
byteOut = null;
}
if (byteOutRaw != null)
{
byteOutRaw.Flush();
byteOutRaw.Close();
byteOutRaw = null;
}
timeStart = timeEnd;
// extract songtitle from metadata header. Trim was needed, because some stations don't trim the songtitle
//fileName = Regex.Match(metadataHeader, "(StreamTitle=')(.*)(';StreamUrl)").Groups[2].Value.Trim();
fileName = Regex.Match(metadataHeader, "(StreamTitle=')(.*)(';)").Groups[2].Value.Trim();
// write new songtitle to console for information
if (fileName.Length == 0)
fileName = "shoutcast_test";
fileNameRaw = fileName + "_raw";
framestream.reSetPosition();
SongChanged(this, metadataHeader);
bNewSong = true;
// create new file with the songtitle from header and set a stream on this file
timeEnd = DateTime.Now;
if (bWrite_to_file)
{
byteOut = createNewFile(destPath, fileName, "mp3");
byteOutRaw = createNewFile(destPath, fileNameRaw, "raw");
}
timediff = timeEnd - timeStart;
// save new header to 'oldMetadataHeader' string, to compare if there's a new song starting
oldMetadataHeader = metadataHeader;
}
metadataHeader = "";
}
}
else // write mp3 data to file or extract metadata headerlength
{
if (count++ < metaInt) // write bytes to filestream
{
//HERE I COLLECT THE BYTES OF THE MP3 FRAME
framestream.Write(buffer, i, 1);
}
else // get headerlength from lengthbyte and multiply by 16 to get correct headerlength
{
metadataLength = Convert.ToInt32(buffer[i]) * 16;
count = 0;
}
}
}//for
if (bNewSong)
{
decompressor = createDecompressor(frame);
bNewSong = false;
}
if (frame != null && decompressor != null)
{
framedec(decompressor, frame);
}
// fine Processing dati RAW
}//Buffer is not full
SHOUTcastStatusProcess();
} while (playbackState != StreamingPlaybackState.Stopped);
} //try
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (byteOut != null)
byteOut.Close();
if (socketStream != null)
socketStream.Close();
if (decompressor != null)
{
decompressor.Dispose();
decompressor = null;
}
if (null != request)
request.Abort();
if (null != framestream)
framestream.Dispose();
if (null != bufferedWaveProvider)
bufferedWaveProvider.ClearBuffer();
//if (null != bufferedWaveProviderOut)
// bufferedWaveProviderOut.ClearBuffer();
if (null != mono16bitFsinStream)
{
mono16bitFsinStream.Close();
mono16bitFsinStream.Dispose();
}
if (null != middleStream2)
{
middleStream2.Close();
middleStream2.Dispose();
}
if (null != resampler)
resampler.Dispose();
}
}
public class QueueStream : MemoryStream
{
long ReadPosition = 0;
long WritePosition = 0;
public QueueStream() : base() { }
public override int Read(byte[] buffer, int offset, int count)
{
Position = ReadPosition;
var temp = base.Read(buffer, offset, count);
ReadPosition = Position;
return temp;
}
public override void Write(byte[] buffer, int offset, int count)
{
Position = WritePosition;
base.Write(buffer, offset, count);
WritePosition = Position;
}
public void reSetPosition()
{
WritePosition = 0;
ReadPosition = 0;
Position = 0;
}
}
private void framedec(IMp3FrameDecompressor decompressor, Mp3Frame frame)
{
int Ndecoded_samples = 0;
byte[] dec_buffer = new byte[decSIZE];
Ndecoded_samples = decompressor.DecompressFrame(frame, dec_buffer, 0);
bufferedWaveProvider.AddSamples(dec_buffer, 0, Ndecoded_samples);
NBufferedSamples += Ndecoded_samples;
brcnt_in.incSamples(Ndecoded_samples);
if (Ndecoded_samples > decSIZE)
{
Debug.WriteLine(String.Format("Too many samples {0}", Ndecoded_samples));
}
if (byteOut != null)
byteOut.Write(frame.RawData, 0, frame.RawData.Length);
if (byteOutRaw != null) // as long as we don't have a songtitle, we don't open a new file and don't write any bytes
byteOutRaw.Write(dec_buffer, 0, Ndecoded_samples);
frame = null;
}
private IMp3FrameDecompressor createDecompressor(Mp3Frame frame)
{
IMp3FrameDecompressor dec = null;
if (frame != null)
{
// don't think these details matter too much - just help ACM select the right codec
// however, the buffered provider doesn't know what sample rate it is working at
// until we have a frame
WaveFormat srcwaveFormat = new Mp3WaveFormat(frame.SampleRate, frame.ChannelMode == ChannelMode.Mono ? 1 : 2, frame.FrameLength, frame.BitRate);
dec = new AcmMp3FrameDecompressor(srcwaveFormat);
bufferedWaveProvider = new BufferedWaveProvider(dec.OutputFormat);// decompressor.OutputFormat
bufferedWaveProvider.BufferDuration = TimeSpan.FromSeconds(400); // allow us to get well ahead of ourselves
// ------------------------------------------------
//Create an intermediate format with same sampling rate, 16 bit, mono
middlewavformat = new WaveFormat(dec.OutputFormat.SampleRate, 16, 1);
outwavFormat = new WaveFormat(Fs_out, 16, 1);
// wave16ToFloat = new Wave16ToFloatProvider(provider); // I have tried with and without this converter.
wpws = new WaveProviderToWaveStream(bufferedWaveProvider);
//Check middlewavformat.Encoding == WaveFormatEncoding.Pcm;
mono16bitFsinStream = new WaveFormatConversionStream(middlewavformat, wpws);
middleStream2 = new BlockAlignReductionStream(mono16bitFsinStream);
resampler = new MediaFoundationResampler(middleStream2, outwavFormat);
}
return dec;
}
I am trying to read the stream that is on Azure Service bus. But I am getting null bytes. File size that is getting created is same as the size of the bytes that are being sent, but it contains all nulls. I have added the code that is used for converting Stream to Byte array with the function name ReadAllBytes(Stream source)
below is the code for reference:
static void Main(string[] args)
{
MemoryStream largeMessageStream = new MemoryStream();
#region ReceiveMessage
var msg = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(ConfigurationSettings.AppSettings["Microsoft.ServiceBus.ConnectionString"].ToString());
var numofmessages = msg.GetQueue(AccountDetails.QueueName).MessageCountDetails.ActiveMessageCount.ToString();
if (msg.GetQueue(AccountDetails.QueueName).RequiresSession)
{
var queueClient1 = QueueClient.CreateFromConnectionString(ConfigurationSettings.AppSettings["Microsoft.ServiceBus.ConnectionString"].ToString(), AccountDetails.QueueName);
var session = queueClient1.AcceptMessageSession();
Console.WriteLine("Message session Id: " + session.SessionId);
Console.Write("Receiving sub messages");
while (true)
{
// Receive a sub message
BrokeredMessage subMessage = session.Receive(TimeSpan.FromSeconds(5));
if (subMessage != null)
{
// Copy the sub message body to the large message stream.
Stream subMessageStream = subMessage.GetBody<Stream>();
subMessageStream.CopyTo(largeMessageStream);
// Mark the message as complete.
subMessage.Complete();
Console.Write(".");
}
else
{
// The last message in the sequence is our completeness criteria.
Console.WriteLine("Done!");
break;
}
}
// Create an aggregated message from the large message stream.
BrokeredMessage largeMessage = new BrokeredMessage(largeMessageStream, true);
Console.WriteLine("Received message");
Console.WriteLine("Message body size: " + largeMessageStream.Length);
string testFile = #"D:\Dev\csvData1.csv";
Console.WriteLine("Saving file: " + testFile);
// Save the message body as a file.
Stream resultStream = largeMessage.GetBody<Stream>();
byte[] x = ReadAllBytes(resultStream);
File.WriteAllBytes(testFile, x);
}
public static byte[] ReadAllBytes(Stream source)
{
long originalPosition = source.Position;
source.Position = 0;
try
{
byte[] readBuffer = new byte[source.Length];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = source.Read(readBuffer, totalBytesRead, readBuffer.Length - totalBytesRead)) > 0)
{
totalBytesRead += bytesRead;
if (totalBytesRead == readBuffer.Length)
{
int nextByte = source.ReadByte();
if (nextByte != -1)
{
byte[] temp = new byte[readBuffer.Length * 2];
Buffer.BlockCopy(readBuffer, 0, temp, 0, readBuffer.Length);
Buffer.SetByte(temp, totalBytesRead, (byte)nextByte);
readBuffer = temp;
totalBytesRead++;
}
}
}
byte[] buffer = readBuffer;
if (readBuffer.Length != totalBytesRead)
{
buffer = new byte[totalBytesRead];
Buffer.BlockCopy(readBuffer, 0, buffer, 0, totalBytesRead);
}
return buffer;
}
finally
{
source.Position = originalPosition;
}
}
I send a message with stream body using the following code, and then I receive the message and test your code on my side, the code works for me.
using (MemoryStream stream = new MemoryStream(File.ReadAllBytes(#"C:\Users\xxx\Desktop\source.txt")))
{
client.Send(new BrokeredMessage(stream));
}
My source.txt
"ID", "Age", "Rich", "timestamp"
1, "50", "Y", "2017-06-06 14:19:21.77"
2, "22", "N", "2017-06-06 14:19:21.77"
Byte array
Console app output
csvData1.csv
If possible, you can try to send a new message with stream body and execute your code to check if it can works fine.
I have a 2GB file in blob storage and am building a console application that will download this file into a desktop. Requirement is to split into 100MB chunks and append a number into the filename. I do not need to re-combine those files again. What I need is only the chunks of files.
I currently have this code from Azure download blob part
But I cannot figure out how to stop downloading when the file size is already 100MB and create a new one.
Any help will be appreciated.
Update: Here is my code
CloudStorageAccount account = CloudStorageAccount.Parse(connectionString);
var blobClient = account.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(containerName);
var file = uri;
var blob = container.GetBlockBlobReference(file);
//First fetch the size of the blob. We use this to create an empty file with size = blob's size
blob.FetchAttributes();
var blobSize = blob.Properties.Length;
long blockSize = (1 * 1024 * 1024);//1 MB chunk;
blockSize = Math.Min(blobSize, blockSize);
//Create an empty file of blob size
using (FileStream fs = new FileStream(file, FileMode.Create))//Create empty file.
{
fs.SetLength(blobSize);//Set its size
}
var blobRequestOptions = new BlobRequestOptions
{
RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(5), 3),
MaximumExecutionTime = TimeSpan.FromMinutes(60),
ServerTimeout = TimeSpan.FromMinutes(60)
};
long startPosition = 0;
long currentPointer = 0;
long bytesRemaining = blobSize;
do
{
var bytesToFetch = Math.Min(blockSize, bytesRemaining);
using (MemoryStream ms = new MemoryStream())
{
//Download range (by default 1 MB)
blob.DownloadRangeToStream(ms, currentPointer, bytesToFetch, null, blobRequestOptions);
ms.Position = 0;
var contents = ms.ToArray();
using (var fs = new FileStream(file, FileMode.Open))//Open that file
{
fs.Position = currentPointer;//Move the cursor to the end of file.
fs.Write(contents, 0, contents.Length);//Write the contents to the end of file.
}
startPosition += blockSize;
currentPointer += contents.Length;//Update pointer
bytesRemaining -= contents.Length;//Update bytes to fetch
Console.WriteLine(fileName + dateTimeStamp + ".csv " + (startPosition / 1024 / 1024) + "/" + (blob.Properties.Length / 1024 / 1024) + " MB downloaded...");
}
}
while (bytesRemaining > 0);
Per my understanding, you could break your blob file into your expected pieces (100MB), then leverage CloudBlockBlob.DownloadRangeToStream to download each of your chunks of files. Here is my code snippet, you could refer to it:
ParallelDownloadBlob
private static void ParallelDownloadBlob(Stream outPutStream, CloudBlockBlob blob,long startRange,long endRange)
{
blob.FetchAttributes();
int bufferLength = 1 * 1024 * 1024;//1 MB chunk for download
long blobRemainingLength = endRange-startRange;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = startRange;
while (blobRemainingLength > 0)
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
queues.Enqueue(new KeyValuePair<long, long>(offset, chunkLength));
offset += chunkLength;
blobRemainingLength -= chunkLength;
}
Parallel.ForEach(queues,
new ParallelOptions()
{
MaxDegreeOfParallelism = 5
}, (queue) =>
{
using (var ms = new MemoryStream())
{
blob.DownloadRangeToStream(ms, queue.Key, queue.Value);
lock (outPutStream)
{
outPutStream.Position = queue.Key- startRange;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
});
}
Program Main
var container = storageAccount.CreateCloudBlobClient().GetContainerReference(defaultContainerName);
var blob = container.GetBlockBlobReference("code.txt");
blob.FetchAttributes();
long blobTotalLength = blob.Properties.Length;
long chunkLength = 10 * 1024; //divide blob file into each file with 10KB in size
for (long i = 0; i <= blobTotalLength; i += chunkLength)
{
long startRange = i;
long endRange = (i + chunkLength) > blobTotalLength ? blobTotalLength : (i + chunkLength);
using (var fs = new FileStream(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, $"resources\\code_[{startRange}]_[{endRange}].txt"), FileMode.Create))
{
Console.WriteLine($"\nParallelDownloadBlob from range [{startRange}] to [{endRange}] start...");
Stopwatch sp = new Stopwatch();
sp.Start();
ParallelDownloadBlob(fs, blob, startRange, endRange);
sp.Stop();
Console.WriteLine($"download done, time cost:{sp.ElapsedMilliseconds / 1000.0}s");
}
}
RESULT
UPDATE:
Based on your requirement, I recommend that you could download your blob into a single file, then leverage LumenWorks.Framework.IO to read your large file records line by line, then check the byte size you have read and save into a new csv file with the size up to 100MB. Here is a code snippet, you could refer to it:
using (CsvReader csv = new CsvReader(new StreamReader("data.csv"), true))
{
int fieldCount = csv.FieldCount;
string[] headers = csv.GetFieldHeaders();
while (csv.ReadNextRecord())
{
for (int i = 0; i < fieldCount; i++)
Console.Write(string.Format("{0} = {1};",
headers[i],
csv[i] == null ? "MISSING" : csv[i]));
//TODO:
//1.Read the current record, check the total bytes you have read;
//2.Create a new csv file if the current total bytes up to 100MB, then save the current record to the current CSV file.
}
}
Additionally, you could refer to A Fast CSV Reader and CsvHelper for more details.
UPDATE2
Code sample for breaking large CSV file into small CSV file with the fixed bytes, I used CsvHelper 2.16.3 for the following code snippet, you could refer to it:
string[] headers = new string[0];
using (var sr = new StreamReader(#"C:\Users\v-brucch\Desktop\BlobHourMetrics.csv")) //83.9KB
{
using (CsvHelper.CsvReader csvReader = new CsvHelper.CsvReader(sr,
new CsvHelper.Configuration.CsvConfiguration()
{
Delimiter = ",",
Encoding = Encoding.UTF8
}))
{
//check header
if (csvReader.ReadHeader())
{
headers = csvReader.FieldHeaders;
}
TextWriter writer = null;
CsvWriter csvWriter = null;
long readBytesCount = 0;
long chunkSize = 30 * 1024; //divide CSV file into each CSV file with byte size up to 30KB
while (csvReader.Read())
{
var curRecord = csvReader.CurrentRecord;
var curRecordByteCount = curRecord.Sum(r => Encoding.UTF8.GetByteCount(r)) + headers.Count() + 1;
readBytesCount += curRecordByteCount;
//check bytes you have read
if (writer == null || readBytesCount > chunkSize)
{
readBytesCount = curRecordByteCount + headers.Sum(h => Encoding.UTF8.GetByteCount(h)) + headers.Count() + 1;
if (writer != null)
{
writer.Flush();
writer.Close();
}
string fileName = $"BlobHourMetrics_{Guid.NewGuid()}.csv";
writer = new StreamWriter(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, fileName), true);
csvWriter = new CsvWriter(writer);
csvWriter.Configuration.Encoding = Encoding.UTF8;
//output header field
foreach (var header in headers)
{
csvWriter.WriteField(header);
}
csvWriter.NextRecord();
}
//output record field
foreach (var field in curRecord)
{
csvWriter.WriteField(field);
}
csvWriter.NextRecord();
}
if (writer != null)
{
writer.Flush();
writer.Close();
}
}
}
RESULT