Javascript GZIP and btoa and decompress with C# - base64

i am developing an application where i compress large JSON data using pako.gzip and then use the btoa function to make it base64string in order to post the data to the server. In the javascript i wrote:
var data = JSON.stringify(JSONData);
var ZippedData = pako.gzip(data, { to: 'string' });
var base64String = btoa(ZippedData);
/* post to server*/
$http.post("URL?base64StringParam=" + base64String").then(function (response) {
//do stuff
});
the problem is that i need to decompress the data again in C# code after posting in order to do other workings on it. In the C# code i wrote:
byte[] data = Convert.FromBase64String(base64StringParam);
string decodedString = System.Text.ASCIIEncoding.ASCII.GetString(data);
Encoding enc = Encoding.Unicode;
MemoryStream stream = new MemoryStream(enc.GetBytes(decodedString));
GZipStream decompress = new GZipStream(stream, CompressionMode.Decompress);
string plainDef = "";
and i get the error here
using (var sr = new StreamReader(decompress))
{
plainDef = sr.ReadToEnd();
}
Found invalid data while decoding.
any help to decompress the data back in C# will be appreciated
EDIT:to sum up what needed to be done
javascript does the following:
Plain text >> to >> gzip bytes >> to >> base64 string
i need C# to do the reverse:
Base64 >> to >> unzip bytes >> to >> plain text

Assuming the following js:
dataToCommitString = btoa(pako.gzip(dataToCommitString, { to: "string" }));
This is the correct c# code to compress/decompress with GZip: Taken from https://stackoverflow.com/a/7343623/679334
using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace YourNamespace
{
public class GZipCompressor : ICompressor
{
private static void CopyTo(Stream src, Stream dest)
{
byte[] bytes = new byte[4096];
int cnt;
while ((cnt = src.Read(bytes, 0, bytes.Length)) != 0)
{
dest.Write(bytes, 0, cnt);
}
}
public byte[] Zip(string str)
{
var bytes = Encoding.UTF8.GetBytes(str);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(mso, CompressionMode.Compress))
{
//msi.CopyTo(gs);
CopyTo(msi, gs);
}
return mso.ToArray();
}
}
public string Unzip(byte[] bytes)
{
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(msi, CompressionMode.Decompress))
{
//gs.CopyTo(mso);
CopyTo(gs, mso);
}
return Encoding.UTF8.GetString(mso.ToArray());
}
}
}
}
Calling it as follows:
value = _compressor.Unzip(Convert.FromBase64CharArray(value.ToCharArray(), 0, value.Length));

In client use:
let output = pako.gzip(JSON.stringify(obj));
send as: 'Content-Type': 'application/octet-stream'
=====================
then in C#:
[HttpPost]
[Route("ReceiveCtImage")]
public int ReceiveCtImage([FromBody] byte[] data)
{
var json = Decompress(data);
return 1;
}
public static string Decompress(byte[] data)
{
// Read the last 4 bytes to get the length
byte[] lengthBuffer = new byte[4];
Array.Copy(data, data.Length - 4, lengthBuffer, 0, 4);
int uncompressedSize = BitConverter.ToInt32(lengthBuffer, 0);
var buffer = new byte[uncompressedSize];
using (var ms = new MemoryStream(data))
{
using (var gzip = new GZipStream(ms, CompressionMode.Decompress))
{
gzip.Read(buffer, 0, uncompressedSize);
}
}
string json = Encoding.UTF8.GetString(buffer);
return json;
}

Related

Making HTTPS call in C# with the BouncyCastle library

Using C# 4.0, I need to make HTTPS call with the BouncyCastle library (Short story : Windows XP + TLS 1.2).
When using the following code, I get a "HTTP Error 400. The request verb is invalid."
Here is my code :
using (var client = new TcpClient("serverName", 443))
{
var sr = new SecureRandom();
var cl = new MyTlsClient();
var protocol = new TlsClientProtocol(client.GetStream(), sr);
protocol.Connect(new MyTlsClient());
using (var stream = protocol.Stream)
{
var hdr = new StringBuilder();
hdr.AppendLine("GET /Url/WebService.asmx?wsdl HTTP/1.1");
hdr.AppendLine("Host: serverName");
hdr.AppendLine("Content-Type: text/xml; charset=utf-8");
hdr.AppendLine("Connection: close");
hdr.AppendLine();
var dataToSend = Encoding.ASCII.GetBytes(hdr.ToString());
sr.NextBytes(dataToSend);
stream.Write(dataToSend, 0, dataToSend.Length);
int totalRead = 0;
string response = "";
byte[] buff = new byte[1000];
do
{
totalRead = stream.Read(buff, 0, buff.Length);
response += Encoding.ASCII.GetString(buff, 0, totalRead);
} while (totalRead == buff.Length);
}
}
class MyTlsClient : DefaultTlsClient
{
public override TlsAuthentication GetAuthentication()
{
return new MyTlsAuthentication();
}
}
class MyTlsAuthentication : TlsAuthentication
{
public TlsCredentials GetClientCredentials(CertificateRequest certificateRequest) { return null; }
public void NotifyServerCertificate(Certificate serverCertificate) { }
}
What I've already done :
Using WireShark to decrypt the ssl stream and inspect the request send => I've never succeeded to decrypt ssl stream
Using fiddler to decrypt the https stream => No detection by fiddler so I suspect something might be badly encrypted
Any ideas ?
Thanks to PeterDettman who gave me the solution :
I must not use the sr.NextBytes(instructions), so the code becomes :
using (var client = new TcpClient("serverName", 443))
{
var sr = new SecureRandom();
var cl = new MyTlsClient();
var protocol = new TlsClientProtocol(client.GetStream(), sr);
protocol.Connect(new MyTlsClient());
using (var stream = protocol.Stream)
{
var hdr = new StringBuilder();
hdr.AppendLine("GET /Url/WebService.asmx?wsdl HTTP/1.1");
hdr.AppendLine("Host: serverName");
hdr.AppendLine("Content-Type: text/xml; charset=utf-8");
hdr.AppendLine("Connection: close");
hdr.AppendLine();
var dataToSend = Encoding.ASCII.GetBytes(hdr.ToString());
stream.Write(dataToSend, 0, dataToSend.Length);
int totalRead = 0;
string response = "";
byte[] buff = new byte[1000];
do
{
totalRead = stream.Read(buff, 0, buff.Length);
response += Encoding.ASCII.GetString(buff, 0, totalRead);
} while (totalRead == buff.Length);
}
}

Lua decompress gzip from string

I have some trouble with decompress gzip from string in lua. (mb bad understanding)
Response of one web-services is base64-encoded gzip string, for sample I get some code on C#.
public static string Decompress(byte[] value, Encoding Encoding = null)
{
if (value == null) return null;
Encoding = Encoding ?? System.Text.Encoding.Unicode;
using (var inputStream = new MemoryStream(value))
using (var outputStream = new MemoryStream())
{
using (var zip = new GZipStream(inputStream, CompressionMode.Decompress))
{
byte[] bytes = new byte[4096];
int n;
while ((n = zip.Read(bytes, 0, bytes.Length)) != 0)
{
outputStream.Write(bytes, 0, n);
}
zip.Close();
}
return Encoding.GetString(outputStream.ToArray());
}
}
static void Main(string[] args)
{
const string encodedText = "H4sIAAAAAAAEAHMNCvIPUlRwzS0oqVQoLinKzEtXyC9SyCvNyYFxM/OAqKC0RKEgsSgxN7UktQgAwOaxgjUAAAA=";
byte[] decodedBytes = Convert.FromBase64String(encodedText);
var decodedString = Decompress(decodedBytes, Encoding.UTF8);
Console.WriteLine(decodedString);
}
I try to do this with lua (on nginx) and make from base64 string array of byte
local byte_table={};
base64.base64_decode(res_string):gsub(".", function(c){
table.insert(byte_table, string.byte(c))
})
but have some problem with zlib.
Please help me understanding how can I use IO stream in lua and decompress gzip.
I try to do this with lua (on nginx) and make from base64 string array of byte
No, you are making Lua table with bunch of numbers.
Decode base64 and feed to zlib the whole resulting string.
I'm use http://luaforge.net/projects/lzlib/ for gzip decompressing:
local result = zlib.decompress(str,31)

PCLCrypto exception 'CryptographicException: Bad PKCS7 padding. Invalid length'

I'm struggling with the PCLCryptho libraby, I can't get it working without retrieving the exception 'CryptographicException: Bad PKCS7 padding. Invalid length'. Running the code once is working, but running it multiple times after each other fails (with different input strings). The decryption takes place after a new instance of the program. I'm running this code on iOS with Xamarin Forms. Here's my code (I'm using the same VI each time and save the salt in the Settinsg for now):
public static string EncryptAnswer(string answer, string passWord)
{
try
{
var keyMaterial = CreateKey(passWord);
var cipherTextBuffer = GetBytes(answer);
var symmetricAlgorithm = WinRTCrypto.SymmetricKeyAlgorithmProvider.OpenAlgorithm(SymmetricAlgorithm.AesCbcPkcs7);
var symmetricKey = symmetricAlgorithm.CreateSymmetricKey(keyMaterial);
using (var encryptor = WinRTCrypto.CryptographicEngine.CreateEncryptor(symmetricKey, GetBytes("vivivivivivivivi")))
{
using (var ms = new MemoryStream())
{
using (var cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write))
{
using (var bWriter = new BinaryWriter(cs))
{
bWriter.Write(cipherTextBuffer, 0, cipherTextBuffer.Length);
cs.FlushFinalBlock();
}
}
return GetString(ms.ToArray());
}
}
}
catch (Exception e)
{
return string.Empty;
}
}
public static string DecryptAnswer(string encryptedAnswer, string passWord)
{
try
{
var cipherTextBuffer = GetBytes(encryptedAnswer);
var keyMaterial = CreateKey(passWord);
var symmetricAlgorithm = WinRTCrypto.SymmetricKeyAlgorithmProvider.OpenAlgorithm(SymmetricAlgorithm.AesCbcPkcs7);
var symmetricKey = symmetricAlgorithm.CreateSymmetricKey(keyMaterial);
using (var decryptor = WinRTCrypto.CryptographicEngine.CreateDecryptor(symmetricKey, GetBytes("vivivivivivivivi")))
{
using (var ms = new MemoryStream())
{
using (var cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Write))
using (var binWriter = new BinaryWriter(cs))
{
binWriter.Write(cipherTextBuffer, 0, cipherTextBuffer.Length);
}
return GetString(ms.ToArray());
}
}
}
catch (Exception e)
{
}
return string.Empty;
}
public static byte[] CreateSalt()
{
var salt = WinRTCrypto.CryptographicBuffer.GenerateRandom(8);
CrossSettings.Current.AddOrUpdateValue("Salt", GetString(salt));
return salt;
}
private static byte[] GetSalt()
{
var saltString = CrossSettings.Current.GetValueOrDefault<string>("Salt");
var salt = GetBytes(saltString);
return salt;
}
private static byte[] CreateKey(string passWord)
{
var key = NetFxCrypto.DeriveBytes.GetBytes(passWord, GetSalt(), 1000, 32);
return key;
}
private static byte[] GetBytes(string str)
{
return Encoding.Unicode.GetBytes(str);
}
private static string GetString(byte[] bytes)
{
return Encoding.Unicode.GetString(bytes, 0, bytes.Length);
}
This seems to be equal to the answers and examples I found. Can someone tell me what's wrong?

Reactive Extensions + Async vs Threading Performance

Wondering if this is implemented sufficiently and efficiently. Trying to get the most speed out of this I can but I feel that I have not implemented this as best as possible. Here is the code snippet. If I tweak the .Buffer() call and only use 1 (simulating a single thread) it takes longer. If I set the buffer to 9 (based on #of cores + 1) it's much faster. However, I was under the impression that async would be faster than multi-threading.... doesn't seem to be working here.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Data;
using System.Data.Sql;
using System.Data.SqlClient;
using CryptSharp.Utility;
using CryptSharp; // CryptSharp Lib in NuGet
using System.Reactive;
using System.Reactive.Linq;
using System.Reactive.Concurrency;
using System.Reactive.Linq.ObservĪ±ble;
using System.Reactive.PlatformServices;
namespace ConsoleApplication1
{
internal class Program
{
private static void Main(string[] args)
{
const bool dispalyOutput = true;
Console.WriteLine("Main thread id: {0}", Thread.CurrentThread.ManagedThreadId);
// if the bugger is adjusted then it will create threads via .net TPL Parallel foreach
var o = (from num in Enumerable.Range(1, 1000) select num).ToObservable(Scheduler.Default).Buffer(32);
var w = Stopwatch.StartNew();
o.SubscribeOn(Scheduler.Default).ObserveOn(Scheduler.Default).Subscribe(n =>
{
Parallel.ForEach(n, chunk =>
{
using (new SimultaneousDelegateCheck(false))
{
Thread.Sleep(0);
var crypto = GetCrypto(chunk.ToString());
if (dispalyOutput)
if (chunk%10 == 0)
Console.WriteLine("Hash|{0} Tid:{1}: {2}", chunk,
Thread.CurrentThread.ManagedThreadId, crypto.Result);
}
});
},
(n) => Console.WriteLine("Error: {0}", n.Message),
() =>
{
Console.WriteLine("Done!");
Console.WriteLine("Total time: {0}", w.Elapsed);
});
Console.WriteLine("Main thread finished in {0}", w.Elapsed);
Console.WriteLine("Finished (called before anon delegate runs above)");
Console.ReadLine();
}
static async Task<string> GetCrypto(string s)
{
//var crypto = Task.Run(() => s.CryptSharpSHA256Hash()); // requires CryptSharp lib from Nuget
var crypto = Task.Run(() => s.Hash(new SHA256Managed()));
//var crypto = Task.Run(() => s.Hash(new SHA256CryptoServiceProvider()));
//var crypto = Task.Factory.StartNew(() => s.CryptSharpSHA256Hash()); // requires CryptSharp lib from Nuget
//var crypto = Task.Run(() => s.Hash(new SHA256Managed()));
//var crypto = Task.Run(() => s.Hash(new SHA256CryptoServiceProvider()));
return await crypto;
}
}
public static class HashExtension
{
public static string CryptSharpScryptHash(this string input)
{
throw new NotImplementedException("don't do that");
}
public static string CryptSharpSHA256Hash(this string input)
{
return Crypter.Sha256.Crypt(input);
}
public static string Hash(this string input, HashAlgorithm algorithm = null)
{
if(algorithm==null)
algorithm=new SHA256Managed();
var inputBytes = Encoding.UTF8.GetBytes(input);
// Combine salt and input bytes
var salt = "hashsalt".GetBytes();
var saltedInput = new Byte[salt.Length + inputBytes.Length];
salt.CopyTo(saltedInput, 0);
inputBytes.CopyTo(saltedInput, salt.Length);
var hashedBytes = algorithm.ComputeHash(inputBytes);
return BitConverter.ToString(hashedBytes).Replace("-", "");
//return Convert.ToBase64String(hashedBytes);
}
public static byte[] GetBytes(this string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
public static string GetString(this byte[] bytes)
{
char[] chars = new char[bytes.Length / sizeof(char)];
System.Buffer.BlockCopy(bytes, 0, chars, 0, bytes.Length);
return new string(chars);
}
}
public class SimultaneousDelegateCheck : IDisposable
{
private static int _delegateCount = 0;
public void Dispose()
{
Interlocked.Decrement(ref _delegateCount);
}
public SimultaneousDelegateCheck(bool display = true)
{
var usage = 0;
if ((usage = Interlocked.Increment(ref _delegateCount)) == 1) return;
if(display)
Console.WriteLine("threads:{0}", _delegateCount);
}
}
}

Trying to download all URL's in html

Can anybody help me with this code?
I am trying to download all the URL's in this html http://mises.org/books/ (they are all pdf's )
I understand the basic logic, I think I just am messing up the regular expression. This is what I have so far:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Text.RegularExpressions;
namespace DownloadPdfs
{
class Program
{
static void Main(string[] args)
{
StringBuilder sb = new StringBuilder();
byte[] buf = new byte[8192];
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("http://mises.org/books/");
HttpWebResponse response = (HttpWebResponse)
request.GetResponse();
Stream resStream = response.GetResponseStream();
string tempString = null;
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
tempString = Encoding.ASCII.GetString(buf, 0, count);
sb.Append(tempString);
}
}
while (count > 0); // any more data to read?
string html = sb.ToString();
List<string> listoflinks = new List<string>();
string input = html;
Regex rx = new Regex(#"(?<="")[^""]+(?="")|[^\s""]\S*");
for (Match match = rx.Match(input); match.Success; match = match.NextMatch())
{
listoflinks.Add(match.ToString());
}
foreach (var v in listoflinks)
{
using (WebClient Client = new WebClient())
{
Client.DownloadFile(v,v);
}
}
}
}
}
Try the code below. The pattern will match the value of HREF attribute for anchors.
Regex rx = new Regex(#"href=""(?<Url>[^.""]+\.pdf)""",RegexOptions.IgnoreCase | RegexOptions.Multiline);
for (Match match = rx.Match(input); match.Success; match = match.NextMatch())
{
var link = match.Groups["Url"].Value;
listoflinks.Add(link);
}
Use a library to parse html like HtmlAgilityPack.
public List<string> GetLinks(string html)
{
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var linkNodes = htmlDoc.DocumentNode.SelectNodes("//a[#href]");
if (linkNodes == null)
{
return new List<string>();
}
var linkNodesWithLink = linkNodes.Where(x => x.Attributes.Contains("href")).ToList();
var links = linkNodesWithLink.Select(x => x.Attributes["href"].Value)
.Where(x => !string.IsNullOrWhiteSpace(x))
.Select(x => x.Trim())
.ToList();
links = links.Distinct().ToList();
return links;
}

Resources