Creating a JPG file with Node.JS - node.js

In my node.js server I am downloading a file from another server. The downloaded file is a JPG image data encoded with Base64 two times, that means I have to decode it 2 times. Given is my code.
var base64DecodedFileData = new Buffer(file_data, 'base64').toString('binary');
var tmp = base64DecodedFileData.split("base64,");
var base64DecodedFileData = new Buffer(tmp[1], 'base64').toString('binary');
var file = fs.createWriteStream(file_path, stream_options);
file.write(base64DecodedFileData);
file.end();
I know my image data is valid the first time I have decoded it ( I have verified that data in online base64 decoders by decoding it second time and I have got the proper image), but when I decode it second time and create a file with this data. I am not getting a valid JPG file.
I have compared it with the actual image, start and ends of both files seems fine but something is not right in my constructed file. The constructed file is also of bigger in size than the original one.
PS: I am doing the split before decoding second time because the data after the first decoding starts with
data:; base64, DATASTARTS
Any thoughts.
Farrukh Arshad.

I have solve My problem. The problem seems to be in the decoding from the node.js so I have written a C++ addon to do the job. Here is the code. I am pretty much sure the problem will remain if we have image file encoded only once.
.js file
ModUtils.generateImageFromData(file_data, file_path);
c++ addon: This uses the base64 C++ encoder/decoder from
http://www.adp-gmbh.ch/cpp/common/base64.html
#define BUILDING_NODE_EXTENSION
#include <node.h>
#include <iostream>
#include <fstream>
#include "base64.h"
using namespace std;
using namespace v8;
static const std::string decoding_prefix =
"data:;base64,";
// --------------------------------------------------------
// Decode the image data and save it as image
// --------------------------------------------------------
Handle<Value> GenerateImageFromData(const Arguments& args) {
HandleScope scope;
// FIXME: Improve argument checking here.
// FIXME: Add error handling here.
if ( args.Length() < 2) return v8::Undefined();
Handle<Value> fileDataArg = args[0];
Handle<Value> filePathArg = args[1];
String::Utf8Value encodedData(fileDataArg);
String::Utf8Value filePath(filePathArg);
std::string std_FilePath = std::string(*filePath);
// We have received image data which is encoded with Base64 two times
// so we have to decode it twice.
std::string decoderParam = std::string(*encodedData);
std::string decodedString = base64_decode(decoderParam);
// After first decoding the data will also contains a encoding prefix like
// data:;base64,
// We have to remove this prefix to get actual encoded image data.
std::string second_pass = decodedString.substr(decoding_prefix.length(), (decodedString.length() - decoding_prefix.length()));
std::string imageData = base64_decode(second_pass);
// Write image to file
ofstream image;
image.open(std_FilePath.c_str());
image << imageData;
image.close();
return scope.Close(String::New(" "));
//return scope.Close(decoded);
}
void Init(Handle<Object> target) {
// Register all functions here
target->Set(String::NewSymbol("generateImageFromData"),
FunctionTemplate::New(GenerateImageFromData)->GetFunction());
}
NODE_MODULE(modutils, Init);
Hope it will help for someone else.

Related

CEREAL failing to serialise - failed to read from input stream exception

I found a particular 100MB bin file (CarveObj_k5_rgbThreshold10_triangleCameraMatches.bin in minimal example), where cereal fails to load throwing exception "Failed to read 368 bytes from input stream! Read 288"
The respective 900MB XML file (CarveObj_k5_rgbThreshold10_triangleCameraMatches.xml in minimal example), built from the same data, loads normally.
The XML file was produced by
// {
// std::ofstream outFile(base + "_triangleCameraMatches.xml");
// cereal::XMLOutputArchive oarchive(outFile);
// oarchive(m_triangleCameraMatches);
// }
and the binary version was produced by
// {
// std::ofstream outFile(base + "_triangleCameraMatches.bin");
// cereal::BinaryOutputArchive oarchive(outFile);
// oarchive(m_triangleCameraMatches);
// }
Minimal example: https://www.dropbox.com/sh/fu9e8km0mwbhxvu/AAAfrbqn_9Tnokj4BVXB8miea?dl=0
Version of Cereal used: 1.3.0
MSVS 2017
Windows 10
Is this a bug? Am I missing something obvious?
Created a bug report in the meanwhile: https://github.com/USCiLab/cereal/issues/607
In this particular instance, the "failed to read from input stream exception" thrown from line 105 of binary.hpp arises because the ios::binary flag is missing from the ifstream constructor call. (This is needed, otherwise ifstream will attempt to interpret some of the file contents as carriage return and linefeed characters. See this question for more information.)
So the few lines of code in your minimal example that read from the .bin file should look like this:
vector<vector<float>> testInBinary;
{
std::ifstream is("CarveObj_k5_rgbThreshold10_triangleCameraMatches.bin", ios::binary);
cereal::BinaryInputArchive iarchive(is);
iarchive(testInBinary);
}
However, even after this is fixed there does also seem to be another problem with the data in that particular .bin file, as when I try to read it I get a different exception thrown, seemingly arising from an incorrectly encoded size value. I don't know if this is an artefact of copying to/from Dropbox though.
There doesn't seem to be a fundamental 100MB limit on Cereal binary files. The following minimal example creates a binary file of around 256MB and reads it back fine:
#include <iostream>
#include <fstream>
#include <vector>
#include <cereal/types/vector.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/archives/xml.hpp>
#include <cereal/archives/binary.hpp>
using namespace std;
int main(int argc, char* argv[])
{
vector<vector<double>> test;
test.resize(32768, vector<double>(1024, -1.2345));
{
std::ofstream outFile("test.bin");
cereal::BinaryOutputArchive oarchive(outFile, ios::binary);
oarchive(test);
}
vector<vector<double>> testInBinary;
{
std::ifstream is("test.bin", ios::binary);
cereal::BinaryInputArchive iarchive(is);
iarchive(testInBinary);
}
return 0;
}
It might be worth noting that in your example code on Dropbox, you're also missing the ios::binary flag on the ofstream constructor when you're writing the .bin file:
/// Produced by:
// {
// std::ofstream outFile(base + "_triangleCameraMatches.bin");
// cereal::BinaryOutputArchive oarchive(outFile);
// oarchive(m_triangleCameraMatches);
// }
It might be worth trying with the flag set. Hope some of this helps.

Resource file not loading for a DLL

My code for loading the file from resource is given below
void LoadFileInResource(int name, int type, DWORD& size, const char*& data)
{
HMODULE handle = ::GetModuleHandle(NULL);
HRSRC rc = ::FindResource(handle, MAKEINTRESOURCE(name),MAKEINTRESOURCE(type));
HGLOBAL rcData = ::LoadResource(handle, rc);
size = ::SizeofResource(handle, rc);
data = static_cast<const char*>(::LockResource(rcData));
}
This code works perfect if its just an application. When the same code is used as a DLL, I am getting null in rc, which is post the FindResource.
I have defined the symbols in resourceful file as shows below:
#define TEXTFILE 256
#define IDR_MYTEXTFILE 105
Also the file which I need to add is defined in rc file:
IDR_MYTEXTFILE TEXTFILE "C:/Docs/Lib.XML"
As I mentioned earlier, this code is perfectly working when its an application, converting it to DLL is creating issue.
LoadFileInResource function is called as given below:
LoadFileInResource(IDR_MYTEXTFILE, TEXTFILE, size, data);

Pubnub library not compiling on Particle Photon

I am trying to publish a message using pubnub on the Particle Photon. The code snippet below comes straight out of the Pubnub example code.
The code will not compile, with the message from the compiler as follows:
PubNub/PubNub.h:87:47: error: expected class-name before '{' token
class PubSubClient: public PubNub_BASE_CLIENT {
^
PubNub/PubNub.h: In constructor 'PubSubClient::PubSubClient()':
PubNub/PubNub.h:23:28: error: class 'PubSubClient' does not have any field named 'WiFiClient'
#define PubNub_BASE_CLIENT WiFiClient
^
The code for this tiny project is as follows:
// This #include statement was automatically added by the Particle IDE.
#include "PubNub/PubNub.h"
char pubkey[] = "<key here>";
char subkey[] = "<key here>";
char channel[] = "Channel";
void setup() {
Serial.begin(9600);
Particle.publish("Serial set up");
PubNub.begin(pubkey, subkey);
}
void loop() {
TCPClient *client;
char msg[64] = "{\"photon\":\"on\"}";
client = PubNub.publish(channel, msg);
client->stop();
Delay (30000);
}
Has anyone had a similar problem, and if so, can you guide me as to how to fix this.
Thanks.
It looks like the library available in Build IDE was in an older version (0.0.1). Fixed, latest version (0.0.2) has been published.
To update library in your app you need to remove the PubNub library from your app in Apps drawer:
And then go to Libraries drawer, find PubNub library, click Include in App, select your app and confirm:

Linux alternative to _NSGetExecutablePath?

Is it possible to side-step _NSGetExecutablePath on Ubuntu Linux in place of a non-Apple specific approach?
I am trying to compile the following code on Ubuntu: https://github.com/Bohdan-Khomtchouk/HeatmapGenerator/blob/master/HeatmapGenerator2_Macintosh_OSX.cxx
As per this prior question that I asked: fatal error: mach-o/dyld.h: No such file or directory, I decided to comment out line 52 and am wondering if there is a general cross-platform (non-Apple specific) way that I can rewrite the code block of line 567 (the _NSGetExecutablePath block) in a manner that is non-Apple specific.
Alen Stojanov's answer to Programmatically retrieving the absolute path of an OS X command-line app and also How do you determine the full path of the currently running executable in go? gave me some ideas on where to start but I want to make certain that I am on the right track here before I go about doing this.
Is there a way to modify _NSGetExecutablePath to be compatible with Ubuntu Linux?
Currently, I am experiencing the following compiler error:
HeatmapGenerator_Macintosh_OSX.cxx:568:13: error: use of undeclared identifier
'_NSGetExecutablePath'
if (_NSGetExecutablePath(path, &size) == 0)
Basic idea how to do it in a way that should be portable across POSIX systems:
#define _XOPEN_SOURCE 500
#include <stdio.h>
#include <limits.h>
#include <stdlib.h>
static char *path;
const char *appPath(void)
{
return path;
}
static void cleanup()
{
free(path);
}
int main(int argc, char **argv)
{
path = realpath(argv[0], 0);
if (!path)
{
perror("realpath");
return 1;
}
atexit(&cleanup);
printf("App path: %s\n", appPath());
return 0;
}
You can define an own module for it, just pass it argv[0] and export the appPath() function from a header.
edit: replaced exported variable by accessor method

error: cannot allocate an object of abstract type ‘FRONTEND_RFInfo_In_i

Using:
Redhawk 1.9 / CentOS 6.4 (32 bit) / C++ implementation
Creating a new FRONTEND::TUNER device
Using default setting on code generation
The following error message happens when I add the following port required for FRONTEND Digital Tuner and regenerate the code.
<ports>
<provides repid="IDL:FRONTEND/DigitalTuner:1.0" providesname="DigitalTuner"/>
<provides repid="IDL:FRONTEND/RFInfo:1.0" providesname="RFInfo"/>
</ports>
Error message (Problems window):
cannot allocate an object of abstract type
‘FRONTEND_RFInfo_In_i’ TestFrontEndDevice_base.cpp /TestFrontEndDevice/cpp line 50 C/C++ Problem
Error message (console):
port_impl.h:56: note: because the following virtual functions are
pure within ‘FRONTEND_RFInfo_In_i’:
/usr/local/redhawk/core/include/redhawk/FRONTEND/RFInfo.h:323: note:
virtual void FRONTEND::_impl_RFInfo::rf_flow_id(const char*)
/usr/local/redhawk/core/include/redhawk/FRONTEND/RFInfo.h:325: note:
virtual void FRONTEND::_impl_RFInfo::rfinfo_pkt(const
FRONTEND::RFInfoPkt&)
make: * [TestFrontEndDevice-TestFrontEndDevice_base.o] Error 1
There appears to be a bug in the code generation for the RFInfo class. If you compare the signatures of the generated code in the port_impl.h file to those of the "unimplemented" ones above, you'll notice that for the rf_flow_id function in port_impl.h there is no const keyword. The same can be said about the rfinfo_pkt method. It is missing the const keyword and an ampersand in the function declaration.
To fix this, simply add the const keywords and the ampersand in the appropriate places in both the declaration in the port_impl.h file and the definition in the port_impl.cpp file.
This is a known issue that has been fixed for the 1.9.1 release.
The problem is the result of the RFInfo port function signatures in
the generated port_impl.* files being different from those in the
parent/base class, which also happen to be pure virtual. To fix the
issue in your code, you'll need to add "const" to rf_flow_id, and both
"const" and "&" to rfinfo_pkt, as shown below:
In port_impl.h:
- void rf_flow_id( char* data);
+ void rf_flow_id( const char* data);
- void rfinfo_pkt( FRONTEND::RFInfoPkt data);
+ void rfinfo_pkt( const FRONTEND::RFInfoPkt& data);
In port_impl.cpp:
-void FRONTEND_RFInfo_In_i::rf_flow_id( char* data)
+void FRONTEND_RFInfo_In_i::rf_flow_id( const char* data)
-void FRONTEND_RFInfo_In_i::rfinfo_pkt( FRONTEND::RFInfoPkt data)
+void FRONTEND_RFInfo_In_i::rfinfo_pkt( const FRONTEND::RFInfoPkt& data)

Resources