I'm trying to make a simple messaging app. I've encoded the strings in UTF-8 in Android through the provided methods:
String msg = "helloo";
byte[] bytemsg = msg.getBytes("UTF-8");
I'm sending this through MQTT (Paho library) to my server where I receive it using mosca library.
var received = new Buffer(payload).toString('utf8');
However, I'm not receiving what I sent. In the above example, I get ��� Also, when I send "hello", nothing gets displayed.
I read about javascript's problem with utf-8 characters, but I'm not using characters beyond the basic plane. I'd appreciate any help.
Get UTF-8 string from array of bytes
var buff = new Buffer(bytes);
console.log(buff.toString('utf8'));
Related
I am working on a project in which API request & responses are sent after PGP encryption and then Base64 encoding. Similarly, after receiving the request, I have to first decode it with base64 & then PGP decryption is performed.
Whenever I use my own function in .Net to decode the request, PGP decryption gives error of illegal characters or unknown message format. But when I decode the request from an online tool that is https://www.base64decode.org , my PGP decryptor works fine.
I am using DidiSoft.PGP library for PGP encryption/decryption. Below is the code snippet I am using for Base64 decoding:
public static string Base64Decode(string base64EncodedData)
{
//Base64 URL Decode
string converted = base64EncodedData
.Replace('-', '+')
.Replace('_', '/')
.Replace(#"\/", "/");
byte[] base64EncodedBytes = Convert.FromBase64String(converted);
return Encoding.UTF8.GetString(base64EncodedBytes);
}
I wonder what algorithm is used in the online tool, or what I am missing to get the correct string. Any help is appreciated.
EDIT: One additional info I found out that I need to perform Base64 decoding according to RFC4648 Section 5 (RFC 4648 §5: base64url (URL- and filename-safe standard)). Can anyone pls. guide its implementation in .NetCore or is there any NugetPackage to do so
I'm setting up an Express.js endpoint that would support decoding a base64 string that came from the client (camera record), which is basically an encoded webm video.
The problem is that to validate this, I am trying to create a .webm video file from the encoded base64 string on the server. Unfortunately this results in a video file that cannot be played due to an error of "No video with supported format and MIME type found" (Ubuntu 18.04).
I did not have any issue with any other .webm files but also the base64 URL is sure to be correct as I am using it on the cliet as the source of a element to replay what was recorded and works fine.
The issue I suspect is something wrong with the code that generates the file on the server route.
import express from "express";
import multer from "multer";
import fs from "fs";
const router = express.Router();
// ...
router.post("/upload", multer().fields([]), (req, res) => {
const formData = req.body;
// formData.vid_string is the base64 encoded string
fs.writeFileSync(
"./test.webm",
Buffer.from(formData.vid_string.split(",")[1], "base64")
);
res.sendStatus(200);
});
I have also tried to change fs.writeFileSync code and add "base64" as a third argument but it has the same result.
Any input would be appreciated! Thanks!
I am pretty sure the correct way to do this is to remove the portion of the base64 string that is the URL specifications (If this is applicable to your situation being the part string that states base64 data and the "video/webm" portion and the trailing comma,) and call atob() on the resulting string this atob will represent the binary string version of the video. Then simply write that string to your new file!
However, I would recommend parsing the base64URL to binary string from the client side, since I am fairly sure that base64 uses more space to account for a smaller group of available bytes, if you have any control over how the video will be sent to the node server.
I need to post a form that has been set to use windows-1252 charset for urlencoding its data. for simple characters, default encoding (utf8) works but it is the special characters that have to be encoded with the required charset.
the npm "request" package i am using does not allow setting any specific charset and uses utf8 by default underneath. i tried another package "Restler", which allows encoding to be set but it throws exception saying invalid charset when i specify windows-1252 (Node only offers a handful of encoding charsets (Buffer class) and windows-1252 is not one of them).
please let me know whether what i am trying to achieve is even possible in node nor not? for verification purposes, i created a little client in java and used apache's http client library with windows-1252 encoding and my request was successfully accepted by the server. so far, i have not been able to figure this out in node.
Sending HTTP request data in a legacy encoding like Windows-1252 is not straightforward in node, as there is no native support for these encodings.
Support can be added in the form of an iconv library, so it's definitely doable, even if it does not work out of the box.
The following targets restler, because you are using it, but in principle this applies to any client HTTP library.
Notes:
Traditional HTTP POSTs are URL-encoded, we will use qs for this.
Support for encodings other than UTF-8 will be provided by qs-iconv, as documented in qs - Dealing with special character sets.
Restler usually encodes data as UTF-8 if you pass it as a string or plain object, but if you pass a Buffer, Restler will send it as it is.
Setting a proper Content-Type and Content-Length will ensure the data can be interpreted properly at the receiving end. Since we supply our own data here, we need to set those headers manually.
Be aware that any character that is not contained in the target charset (Windows-1252 in this case) will be encoded as ? by iconv (%3F in URL form) and therefore will be lost.
Code:
var rest = require('restler');
var qs = require('qs');
var win1252 = require('qs-iconv/encoder')('win1252');
var requestData = {
key1: "‘value1‘",
key2: "‘value2‘"
};
var requestBody = qs.stringify(requestData, { encoder: win1252 });
// => "key1=%91value1%91&key2=%91value2%91"
var requestBuf = new Buffer(requestBody);
rest.post('your/url', {
data: requestBuf,
headers: {
'Content-Type': 'application/x-www-form-urlencoded; charset=windows-1252',
'Content-Length': requestBuf.length
}
}).on('complete', function(data) {
console.log(data);
});
I have a server running on nodejs, and I have the following piece of code to manage a post request -
form.on('file', function (field, file) {
var RecordingInfo = JSON.parse(file.name);
...
when I tried to upload a file I got the following exception:
undefined:1
"}
SyntaxError: Unexpected end of input
at Object.parse (native)
at IncomingForm.<anonymous> (.../root.js:31:34)
...
searching around the web, I fond that this exception is caused because the data comes in bits, and the event is fired after the first bit arrives, and I don't have all the data. OK. The thing is, after a little testing I fond that from chrome I can upload large files (tried a 1.75gb file) without any problem, while firefox crashes the server with a 6kb file.
My question is - why are they different?
A sample capture can be downloaded form here. The first post is from chrome, the second from firefox.
The complete file.name string before uploading is:
// chrome
"{"subject":"flksajfd","lecturer":"אבישי וינר","path":"/גמרא","fileType":".png"}"
// firefox
"{"subject":"fdsa","lecturer":"אלקס ציקין","path":"/גמרא","fileType":".jpg"}"
(The data submitted is not the same, but I don't think it matters)
Chrome is encoding double-quotes in the JSON-encoded "filename" as %22 while Firefox is encoding them as \".
Your file upload parsing library, Formidable, explicitly truncates the filename from the last \ character. It expects double-quotes to be encoded as %22 although RFC 2616 allows backslash-escaped quotes like Firefox has implemented. You can consider this a bug in Formidable. The result is that the following JSON string:
'{"subject":"fdsa",...,"fileType":".jpg"}'
...is encoded as follows:
'{%22subject%22:%22fdsa",...,%22fileType%22:%22.jpg%22}' // Chrome
'{\"subject\":\"fdsa\",...\"fileType\":\".jpg\"}' // Firefox
...and then decoded by Formidable:
'{"subject":"fdsa",..."fileType":".jpg"}' // Chrome
'"}' // Firefox
To fix the issue you have a few choices:
Raise the issue with Formidable to correctly handle backslash-escaped quoted-value strings (or fix it yourself and submit a pull request).
Send the JSON payload in a separate part of the FormData object, e.g. using a Blob.
Transliterate all double-quote characters in your JSON-format filename to a 'safe' character that will not appear elsewhere in the string (I chose ^ as an example); replace the quote client-side and reinstate it server-side as follows.
Client:
var formData = new FormData();
formData.append('file', $scope.recording, JSON.stringify(RecordingInfo).replace(/"/g, '^');
Server
form.on('file', function (field, file) {
var RecordingInfo = JSON.parse(file.name.replace(/\^/g, '"');
I would like to store some documents in a database as base64 strings. Then when those docs are requested using HTTP, I would like ExpressJS to decode the base64 docs and return them. So something like this:
app.get('/base64', function (req, res) {
//pdf is my base64 encoded string that represents a document
var buffer = new Buffer(pdf, 'base64');
res.send(buffer);
});
The code is simply to give an idea of what I'm trying to accomplish. Do I need to use a stream for this? If so, how would I do that? Or should I be writing these docs to a temp directory and then serving up the file? Would be nice to skip that step if possible. Thanks!
UPDATE: Just to be clear I would like this to work with a typical HTTP request. So the user will click a link in his browser that will take him to a URL that returns a file from the database. Seems like it must be possible, Microsoft SharePoint stores serialized files in a SQL database and returns those files over http requests, and I don't believe it writes all those files to a temp location first. I'm feeling like a nodejs stream may be the answer, but I'm not very familiar with streaming.
Before saving a file representation to the DB you can just use the toString method with base 64 encoding:
var base64pdf = pdf.toString('base64');
After you get the base64 file representation from db use the buffer as follows in order to convert it back to a file:
var decodedFile = new Buffer(base64pdf, 'base64');
More information on Buffer usages can be found here - NodeJS Buffer
As for how to send a buffer from express server to the client, Socket IO should solve this issue.
Using socket.emit -
Emits an event to the socket identified by the string name. Any
other parameters can be included.
All datastructures are supported, including Buffer. JavaScript
functions can’t be serialized/deserialized.
var io = require('socket.io')();
io.on('connection', function(socket){
socket.emit('an event', { some: 'data' });
});
Required documentation on socket.io website.