I am trying to read a text from XML node. The text is base 64 encrypted and Gzip compressed. I am trying in the following ways:
1. XmlNodeList nodeDebugs = xmlDoc.GetElementsByTagName("Debugs");
2. if (nodeDebugs != null)
3. {
4. for (int i = 0; i < nodeDebugs.Count; i++)
5. {
6. XmlNodeList childNodes = nodeDebugs.Item(i).ChildNodes;
7. String nodeName = childNodes.Item(i).Name;
8. if (nodeName == "Debug")
9. {
10. byte[] compressed = System.Text.Encoding.UTF8.GetBytes(childNodes.Item(i).InnerText.Trim());
11. using (var uncompressed = new MemoryStream())
12. using (var inStream = new MemoryStream(compressed))
13. using (var outStream = new GZipStream(inStream, CompressionMode.Decompress))
14. {
15. outStream .CopyTo(uncompressed);
16. Console.WriteLine(Encoding.UTF8.GetString(uncompressed.ToArray()));
17. }
Getting error at Line no 15, as "the magic number in gzip header is not correct."
Any help to resolve this issue is much appreciated. Thanks a lot in advance.
This is the problem, I suspect:
byte[] compressed = System.Text.Encoding.UTF8.GetBytes(childNodes.Item(i).InnerText.Trim());
You said that your data is compressed and then base64-encoded... but your code doesn't use base64 anywhere, which is a massive warning sign. I suspect you want:
string text = childNodes.Item(i).InnerText.Trim();
byte[] compressed = Convert.FromBase64String(text);
If that doesn't work, you should check each step of the transformation - you should have the same data during encoding and decoding, just in the reverse order. See my blog post about diagnosing reversible data transformations for more details - but hopefully the change above will fix it anyway.
Related
I have a QRCode image and it's content: P01 V0N S0000000
I tried to regenerate this image with different matchPattern using node-qrcode:
const $qr = require('qrcode')
const text = 'P01 V0N S0000000'
for (let i = 0; i < 8; i++) {
let path = `P01V0NS0000000-${i}.png`
const opt = {
maskPattern: i
}
$qr.toFile(path, text, opt, function (err) {
if (err) {
console.log(err)
} else {
console.log(opt)
}
})
}
but none of generated image has the same pattern as the target:
I also tried with different version paramter, but all version>1 will generate a much dense pattern.
Is this QRCode image generated with some kind of salt to make hard to replicate?
No need to try with another version as your first image clearly is a version 1 (21x21). Instead, it might be the format encoding (given the set of chars, the app might use alphanumeric and your original might well be byte for example)
As per the link you provided, I'd recommand reading this section about modes and try either with:
mode: 'alphanumeric'
or with:
mode: 'byte'
to see if any of those match your original code.
Another parameter that can change the aspect of the QR-code is the error correction level. I'd guess by default node-qrcode uses the highest level possible given version 1, and your original might use a lower one.
With those params, you have a set of 8(maskPattern)*2(mode)*4(errorCorrectionLevel) = 64 possible QR-codes with the same data inside, so maybe one of them will match!
Edit
As your data is 16 characters long, the scope is reduced as error correction level can only have the following values (source):
L,M,Q for alphanumeric mode
L for byte mode
Therefore, you have (3+1)*8 = 32 possible resulting images.
I am asking if I can, but I would also like to know if I should.
Here's my scenario:
I am receiving Avro serialized messages in small batches. I want to store them for later analysis using a Hive table with the Avro SerDe. I'm running in Azure, and I am storing the messages in a blob.
I am trying to avoid having lots of small blobs (because I believe this will have a negative impact on Hive). If I have the Avro header already written to the blob, I believe that can append Avro data blocks with CloudBlockBlob.PutBlockAsync(). (As long, as I know the sync marker.)
However, I've examined two .NET libraries and that don't seem to support my approach. (I have to write the entire Avro container file at once).
http://www.nuget.org/packages/Apache.Avro/
http://www.nuget.org/packages/Microsoft.Hadoop.Avro/
Am I taking the correct approach?
Am I missing something in the libraries?
My question is similiar (but different) to this one:
Can you append data to an existing Avro data file?
The short answer here is that I was trying to do the wrong thing.
First, we decided that Avro is not the appropriate format for the on-the-wire serialization. Primarily, because Avro expects the schema definition to be present in every Avro file. This adds a lot of weight to what is trasmitted. You could still use Avro, but that's not what it's designed for. (It is designed for big files on HDFS.)
Secondly, the existing libraries (for .NET) only support appending to Avro files via a stream. This does not map well to Azure block blobs (you don't want to open a block blob as a stream).
Thirdly, even if these first two could be bypassed, all of the items in a single Avro file are expected to share the same schema. We had a set of heterogenous items flowing in that we wanted to buffer, batch, and write to blob. Trying to segregate the items by type/schema as we were writing them to blob added lots of complication. In the end, we opted to use JSON.
It is possible to do.
First of all, you have to use CloudAppendBlob:
CloudAppendBlob appBlob = container.GetAppendBlobReference(
string.Format("{0}{1}", date.ToString("yyyyMMdd"), ".log"));
appBlob.AppendText(
string.Format(
"{0} | Error: Something went wrong and we had to write to the log!!!\r\n",
dateLogEntry.ToString("o")));
Second step is to tell to avro lib not to write header on append and share the same sync marker between appends:
var avroSerializer = AvroSerializer.Create<Object>();
using (var buffer = new MemoryStream())
{
using (var w = AvroContainer.CreateWriter<Object>(buffer, Codec.Deflate))
{
Console.WriteLine("Init Sample Data Set...");
var headerField = w.GetType().GetField("header", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
var header = headerField.GetValue(w);
var marker = header.GetType().GetField("syncMarker", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
marker.SetValue(header, new byte[16]);
using (var writer = new SequentialWriter<Object>(w, 24))
{
// Serialize the data to stream by using the sequential writer
for (int i = 0; i < 10; i++)
{
writer.Write(new Object());
}
}
}
Console.WriteLine("Append Sample Data Set...");
//Prepare the stream for deserializing the data
using (var w = AvroContainer.CreateWriter<Object>(buffer, Codec.Deflate))
{
var isHeaderWritten = w.GetType().GetField("isHeaderWritten", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
isHeaderWritten.SetValue(w, true);
var headerField = w.GetType().GetField("header", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
var header = headerField.GetValue(w);
var marker = header.GetType().GetField("syncMarker", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
marker.SetValue(header, new byte[16]);
using (var writer = new SequentialWriter<Object>(w, 24))
{
// Serialize the data to stream by using the sequential writer
for (int i = 10; i < 20; i++)
{
writer.Write(new Object());
}
}
}
Console.WriteLine("Deserializing Sample Data Set...");
}
Is there any example which helps in generating Base64 based image from Bitmapdata?
I know I have to use encode in bitmapdata, but I am not sure how to use it?
It takes (rect : flash.geom.Rectangle, compressor : flash.utils.Object, ?byteArray : flash.utils.ByteArray) : flash.utils.ByteArray;
How to fill compressor incase I want to compress using jpeg?
Also, what's the input of byteArray ?
Any help?
There is an example in the AS3 doc of BitmapData.html#encode():
// Compress a BitmapData object as a JPEG file.
var bitmapData:BitmapData = new BitmapData(640,480,false,0x00FF00);
var byteArray:ByteArray = new ByteArray();
bitmapData.encode(new Rectangle(0,0,640,480), new flash.display.JPEGEncoderOptions(), byteArray);
To generate a Base64 String, you can use haxe.crypto.Base64 (added after 3.0.1).
In OpenFL, ByteArray extends Bytes. So you may simply feed it to Base64.encode.
My app needs to generate a hex string to use as a session ID. Java's SecureRandom doesn't seem to be working ("java/lang/NoClassDefFoundError: java/security/SecureRandom: Cannot create class in system package")
I thought of doing something like this:
byte[] resBuf = new byte[50];
new Random().nextBytes(resBuf);
String resStr = new String(Hex.encode(resBuf));
But the method nextBytes(byte[] bytes) isn't available for some strange reason.
Does anyone have a means of generating a random hex number in Java ME/J2ME?
Many thanks.
Edit: The above generator seems to work when using Bouncy Castle lcrypto-j2me-145 (but not lcrypto-j2me-147).
JavaME is a subset of JavaSE, so many classes and methods in the desktop version are not available.
Looks like you are trying to get a random string of a given length. You can do something like this:
private String getRandomHexString(int numchars){
Random r = new Random();
StringBuffer sb = new StringBuffer();
while(sb.length() < numchars){
sb.append(Integer.toHexString(r.nextInt()));
}
return sb.toString().substring(0, numchars);
}
I have a directory OUTPUT where I have the output files from a Map Reduce job. The output files are Text files written with a TextOutputFormat.
Now I want to read the key value pairs from the output file. How can I do so using some existing classes in hadoop. One way I could do it was as follows
FileSystem fs = FileSystem.get(conf);
FileStatus[] files = fs.globStatus(new Path(OUTPUT + "/part-*"));
for(FileStatus file:files){
if(file.getLen() > 0){
FSDataInputStream in = fs.open(file.getPath());
BufferedReader bin = new BufferedReader(new InputStreamReader(
in));
String s = bin.readLine();
while(s!=null){
System.out.println(s);
s = bin.readLine();
}
in.close();
}
}
This approach would work but increases my task to a great deal as I now need to manually parse the key value pairs out of each individual line. I am looking for something more handy that directly lets me read key and value in some variables.
Are you forced to use TextOutputFormat as your output format in the previous job?
If not then consider using SequenceFileOutputFormat, then you can use a SequenceFile.Reader to read back the file in Key / Value pairs. You can also still 'view' the file using hadoop fs -text path/to/output/part-r-00000
EDIT: You can also use the KeyValueLineRecordReader class, you'll just need to pass in a FileSplit to teh constructor.