I have a Node JS application running with Express and mongodb. I have a use case where my device sends data to my server. It might be JSON data or Comma-Separated String(Only String, not CSV file). I need to check which type of data is coming and manipulate that to JSON if request body would be a String. When I was trying to display the data type of data being sent to the server, it's displaying as "object" even after giving the "String" data as input. And the operation is getting successful but data is not inserting into the database. Can anyone help me in resolving this issue?
Sample Payload(request.body) would be,
"heat:22,humidity:36,deviceId:sb-0001"
Expected response is,
{
"heat": "22",
"humidity": "36",
"deviceId": "sb-0001"
}
#Aravind Actually typeof will returns "string" if operand is a string.So please check whether the string is coming or not in body, because if it is null then typeof will return "object".
I need to check which type of data is coming and manipulate that to JSON ...
HTTP is build upon the media-type concept that defines the syntax used in the payload of that type as well as the semantics of the elements used within that payload. You might take a look at how HTML defines forms to get a grip what a media-type specification should include. Through content negotiation client and server agree on a common media-type and thus a representation format to exchange payloads for supported by both. This simply increases interoperability between the participants as they exchange data in well-defined, hopefully standardized, representation formats both understand and support.
Through Accept headers a receiver can state preferences on which types to receive, including a weighting scheme to indicate that a certain representation format is preferred over an other one but the recipient is also fine with the other one, while a Content-Type header will indicate the actual representation format being sent.
RFC 7111 defines text/csv for CSV based representations and RFC 8259 specifies application/json for JSON payload. As the sender hopefully knows what kind of document it sends to the receiver, you can use this information to distinguish the payload on the receiver side. Note however that according to Fielding true REST APIs must use representation formats that support hypertext-driven interaction flows to allow clients to take further actions upon the payload received without having to invoke some external documentation. Both, JSON and CSV, by default don't support such an interaction model, that is abreviated with the term HATEOAS. For your simple scenario the content type negotiation might though be sufficient enough to solve your needs.
In terms of processing CSV and converting the data to JSON this is simply a matter of splitting up the CSV string on the delimiter symbol (, in the sample) to key-value pairs and then splitting the key and values further on the : symbol and adding these to a new JSON object. There is also a csvtojson library available at NPM that you might utilize in such a case.
Related
I'm trying to extract the language from the detection response:
response.full_text_annotation.pages[0].property.detected_languages[0].language_code
but it seems that sometimes the detections are missing the TextProperty (property) field - as specified here: Page
Is it not always guaranteed to be in the detection?
Also, is there a way to receive only the fullTextAnnotation without the singular textAnnotations fields?
I think is not possible to receive only the fullTextAnnotation without the singular textAnnotations, because the response structure is TextAnnotation -> Page -> Block -> Paragraph -> Word -> Symbol and if you look into the TextAnnotation response, there is no way to modify it.
Regarding the missing TextProperty (property) field, you can try to fix this by using “DOCUMENT_TEXT_DETECTION” instead of “TEXT_DETECTION” towards TYPE. According to the documentation, The TEXT_DETECTION endpoint will auto-detect only a subset of supported languages, while the DOCUMENT_TEXT_DETECTION endpoint will auto-detect the full set of supported languages.
I'm sending series of texts to be classified by https://iab.taxonome.org classification API, but since I work in node JS enviroment in a proper, async way, the responses get's out of sync with the requests. I can use recorsive calls, but this might overflow my stack. Any idea how to sync calls with classification responses?
You might use the optional private ID to pass a "cookie" to identify the session.
From https://iab.taxonome.org/api
id optional String Request user defined identifier or empty string.
curl -i https://rest.taxonome.org/v1/taxono?me=this%20is%20a%20text%20to%20be%20classified&token=123e4567-e89b-12d3-a456-426655440000&ver=1&id=MyUserDefinedId
Let's consider a REST endpoint which receives a JSON object. One of the JSON fields is a String, so I want to validate that no malicious text is received.
#ValidateRequest
public interface RestService {
#POST
#Consumes(APPLICATION_JSON)
#Path("endpoint")
void postData (#Valid #NotNull Data data);
}
public class Data {
#ValidString
private String s;
// get,set methods
}
I'm using the bean validation framework via #ValidString to delegate the validation to the ESAPI library.
#Override
public boolean isValid (String value, ConstraintValidatorContext context) {
return ESAPI.validator().isValidInput(
"String validation",
value,
this.constraint.type(),
this.constraint.maxLength(),
this.constraint.nullable(),
true);
}
This method canonicalizes the value (i.e. removes encryption) and then validates against a regular expression provided in the ESAPI config. The regex is not that important to the question, but it mostly whitelists 'safe' characters.
All good so far. However, in a few occasions, I need to accept 'less' safe characters like %, ", <, >, etc. because the incoming text is from an end user's free text input field.
Is there a known pattern for this kind of String sanitization? What kind of text can cause server-side problems if SQL queries are considered safe (e.g. using bind variables)? What if the user wants to store <script>alert("Hello")</script> as his description which at some point will be send back to the client? Do I store that in the DB? Is that a client-side concern?
When dealing with text coming from the user, best practice is to white list only known character sets as you stated. But that is not the whole solution, since there are times when that will not work, again as you pointed out sometimes "dangerous" characters are part of the valid character set.
When this happens you need to be very vigilant in how you handle the data. I, as well as the commenters, recommended is to keep the original data from the user in its original state as long as possible. Dealing with the data securely will be to use proper functions for the target domain/output.
SQL
When putting free format strings into a SQL database, best practice is to use prepared statements (in java this is the PreparedStatement object or using ORM that will automatically parameterizes the data.
To read more on SQL injection attacks and other forms of Injection attacks (XML, LDAP, etc.) I recommended OWASPS Top 10 - A1 Injections
XSS
You also mentioned what to do when outputting this data to client. In this case I you want to make sure you html encode the output for the proper context, aka contextual output encoding. ESAPI has Encoder Class/Interface for this. The important thing to note is which context (HTML Body, HTML Attribute, JavaScript, URL, etc.) will the data be outputted. Each area is going to be encoded differently.
Take for example the input: <script>alert('Hello World');<script>
Sample Encoding Outputs:
HTML: <script>alert('Hello World');<script>
JavaScript: \u003cscript\u003ealert(\u0027Hello World\u0027);\u003cscript\u003e
URL: %3Cscript%3Ealert%28%27Hello%20World%27%29%3B%3Cscript%3E
Form URL:
%3Cscript%3Ealert%28%27Hello+World%27%29%3B%3Cscript%3E
CSS: \00003Cscript\00003Ealert\000028\000027Hello\000020World\000027\000029\00003B\00003Cscript\00003E
XML: <script>alert('Hello World');<script>
For more reading on XSS look at OWASP Top 10 - A3 Cross-Site Scripting (XSS)
Is it possible to have any kind of payload when using JOSE?
I was thinking of doing something like:
{"alg":"ES512",
"cty":"XML"
}
and just make a string out of the XML file, on the server side I check cty and create the XML.
I'm pretty sure its possible, but I didn't see any example so I started thinking that maybe that's not the idea behind jose and that cty is only meant to indicate payload is a JWT or something related to jose.
As you can see from http://www.rfc-editor.org/rfc/rfc7515.txt:
JWS Payload
The sequence of octets to be secured -- a.k.a. the message. The
payload can contain an arbitrary sequence of octets.
So your hunch is right. And as indicated in https://www.rfc-editor.org/rfc/rfc7515#section-4.1.10, the cty element can be used to indicate XML content following http://www.iana.org/assignments/media-types/media-types.xhtml with the ommission of the application/ prefix. So xml would be appropriate in your case but in (the regular) case that both the sender and the receiver know that they're going to exchange XML, the cty element can be omitted.
For JWE's it is the same. There's an example with non-JSON plaintext here: https://www.rfc-editor.org/rfc/rfc7516#appendix-A.3
Here is what RFC 5789 says:
The PATCH method requests that a set of changes described in the request entity be applied to the resource identified by the Request-URI. The set of changes is represented in a format called a "patch document" identified by a media type. If the Request-URI does not point to an existing resource, the server MAY create a new resource, depending on the patch document type (whether it can logically modify a null resource) and permissions, etc.
The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version.
Let's say I have { "login": "x", "enabled": true }, and I want to disable it.
According to post "Please. Don't Patch Like An Idiot.", the proper PATCH request would be
[{ "op": "replace", "path": "/enabled", "value": false }]
However, let's take this request:
{ "enabled": false }
It also 'contains a set of instructions describing how a resource currently residing on the origin server should be modified', the only difference is that JSON property is used instead of JSON object.
It seems less powerful, but array changes can have some other special syntax if required (e.g. {"a":{"add":[], "remove":[]}}), and server logic might not be able to handle anything more powerful anyway.
Is it an improper PATCH request as per RFC? And if so, why?
And, on the other hand, would a { "op": "disable" } be a correct PATCH request?
the only difference is that JSON property is used instead of JSON object.
It's actually a bit deeper than that. The reference to RFC 6902 is important. The first request has a Content-Type of application/json-patch+json, but the second is application/json
The important thing is that you use a 'diff media type,' one that's useful for this purpose. You don't have to use JSON-PATCH, (I'm a big fan of json-merge-patch), but you can't just use anything you want. What you're asking about in the second part is basically 'can I make my own media type' and the answer is 'yes,' just please document it and register it with the IANA.