Why are there hexadecimal numbers included in my view results from CouchDB? How can I get rid of them?
7f
{"total_rows":108,"offset":0,"rows":[
{"id":"5c718dbd01bc0cde8152e08ed6003405","key":"2013-03-19T22:43:27.2683661Z","value":0}
5b
,
...
{"id":"5c718dbd01bc0cde8152e08ed6037404","key":"2013-03-19T23:07:35.5972058Z","value":0}
5b
,
{"id":"5c718dbd01bc0cde8152e08ed60376e5","key":"2013-03-19T23:07:35.6062063Z","value":0}
4
]}
1
0
TL;DR
I am new to CouchDB, and are investigating its use as a database for an event log. I have created a simple map function to view the event log by date:
function(doc)
{
if (doc.DateTime)
{
emit(doc.DateTime, doc);
}
}
When I use Fiddler to test this view with the following request:
GET http://localhost:5984/stuff/_design/eventlog/_view/datetime
Host: localhost:5984
User-Agent: Fiddler
The results returned included hexadecimal numbers that aren't a part of the JSON structure. Hence the JSON returned is invalid. Why are these hexadecimal numbers included in the results, and how can I get rid of them?
I am using Windows (x86) CouchDB version 1.2.1.
The weird hex numbers are used for so-called chuncked transfer-encoding. This is a way for HTTP responses to become available in a streaming format instead of the client having to wait for entire response to be ready. The hex numbers denote the length for the next chunk.
I think the use of chunking is independent of the request's Accept values, but I'm not sure.
To get a pure JSON result you must include the Accept: application/json header in your HTTP request.
If you omit the the Accept header CouchDB will return results in a manner that is more suitable for displaying nicely in web browsers. The results will be in a JSON format, but with a text/plain content-type.
See Apache CouchDB 1.3 Manual Section 2.2.1. Request Headers.
The hexadecimal numbers are a result of Chunked transfer encoding.
Related
I have an API created in Loopback 4 which retrieves data from a database in PostgreSQL 13 encoded with UTF8. Visiting the API explorer (localhost:3000/explorer) and executing the GET requests I realize that even when the database fields contain characters like letters with accents and ñ's; the retrieved JSON only shows blanks in the position where the character must have appeared. For example, if the database has a field with a word like 'piña', the JSON returns 'pi a'.
When I try a POST request, inserting a field like 'ramírez' (note the í), in the database, the field is shown as 'ramφrez' and when I execute a GET of that entry, the JSON now has de correct value, 'ramírez'.
How can I fix that?
I'd recommend using the Buffer class:
var encodedString = Buffer.from('string', 'utf-8');
with this way you will be able to return anything you want. In NodeJS Buffer class already included so you don't need to install any dependencies.
If you don't get what you need you can change 'utf-8' part.
We are currently struggeling with Batch Query,
which seems to ignore the filter expressions on S4 side caused by a wrong URL encoding.
/sap/opu/odata/sap/ZP2M_A_CONTRACT_SEARCH_HDR_CDS/ZP2M_A_CONTRACT_SEARCH_HDR?$filter=PurchaseContractID eq %274600002020%27&$select=*&$format=json
Executing the query using FluentHelperRead.execute(HttpClient)
the returned list of entities contains the expected result with exactly one entity.
Executing the query as Batch Query the following request is logged in console:
GET ZP2M_A_CONTRACT_SEARCH_HDR?%24filter%3DPurchaseContractID+eq+%25274600002020%2527%26%24select%3D*%26%24format%3Djson HTTP/1.1
The collected list from all batch result parts contains all entities.
It seems, that the query URL is encoded in wrong way
and that S4 ignored the filter expressions when encoded in this way.
e.g. $filter is encoded to %24filter which is ignored by S4.
This seems to be a bug in BatchRequestImpl.getRequest(ODataQueryImpl) method,
where URL encoding is done a 2nd time on already encoded URL parts.
if(systemQuery.indexOf("$format=json&$count=true") != -1)
{
systemQuery = systemQuery.substring(0, systemQuery.indexOf("$format=json&$count=true") -1);
keysUrl.append("/$count");
}
systemQuery = URLEncoder.encode(systemQuery, "UTF-8"); // this code line which encodes the query 2nd time
keysUrl.append("?");
The code line systemQuery = URLEncoder.encode(systemQuery, "UTF-8"); located in
BatchRequestImpl(1.38.0) - line 295
BatchRequestImpl(1.42.2) - line 307
encodes the systemQuery string again (including the already encoded parts of FilterExpression as well).
When undoing the changes of this code line in debugger and replacing the scapces by %20 or '+' the Batch Query looks like that
GET ZP2M_A_CONTRACT_SEARCH_HDR?$filter=PurchaseContractID%20eq%20%274600002020%27&$select=*&$format=json HTTP/1.1
GET ZP2M_A_CONTRACT_SEARCH_HDR?$filter=PurchaseContractID+eq+%274600002020%27&$select=*&$format=json HTTP/1.1
and it returns the expected result (exactly 1 entity).
This wrong encoding appears when using these library versions:
sdk-bom: 3.16.1
connectivity: 1.38.0
This issue appears in newest SDK versions as well:
sdk-bom: 3.21.0
connectivity: 1.39.0
This issue appears with connectivity JAR in newest version too:
sdk-bom: 3.21.0
connectivity: 1.40.2
Debugging together with a ABAP/S4 colleague figures out,
that S4 only applies filter expressions, if the keyword $filter is found in request,
%24filter%3D is ignored (the cause why we get all entities running the Batch Query).
My suggestion to solve it would be
// decode query first (to decode the filter expression)
systemQuery = URLDecoder.decode(systemQuery, "UTF_8");
// encode query
systemQuery = org.apache.commons.httpclient.util.URIUtil.encodeQuery(systemQuery, "UTF_8");
My code, how I am calling the batchRequest:
FluentHelperRead<?, MyEntity, ?> queryApi = myService.getAll... // with adding some filter expression
BatchRequestBuilder batchRequestBuilder = BatchRequestBuilder.withService(MyService.DEFAULT_SERVICE_PATH);
ODataQuery query = queryApi.toQuery();
batchRequestBuilder.addQueryRequest(query);
HttpClient httpClient = HttpClientAccessor
.getHttpClient(DefaultErpHttpDestinationAccessor.get());
BatchRequest request = batchRequestBuilder.build();
BatchResult result = request.execute(httpClient);
// ... evaluate response
I think, this is a general issue in the Cloud SDK.
Would is be possible to get this fixed in next Cloud SDK release?
Can you share your code for Batch request? Do you use BatchRequestImpl directly?
The thing is SAP Cloud SDK relies on some dependencies one of which introduces the BatchRequestImpl and if it's called directly the bug is on the dependency side. I have already informed them to investigate this double encoding issue. Unfortunately, we can't directly influence how fast it is resolved and sometimes it takes longer than we'd like.
The good news, we're working on replacing this dependency with our own implementation to solve exactly this kind of problem. The batch is work in progress and should be available in Beta around the end of next month for OData V4 and hopefully around the same time for OData V2 (it's not a hard commitment and depends on other priorities).
From here we have to wait for whatever happens first:
The bug is fixed on the dependency side
Internal OData client implementation is ready together with Batch
I hope it helps and explains current solution path. If you share a bit around your deadlines and the potential impact we'll be happy to consider that.
This has been fixed within the dependency and as of version 3.25.0 the SAP Cloud SDK includes the fix.
I have a Node JS application running with Express and mongodb. I have a use case where my device sends data to my server. It might be JSON data or Comma-Separated String(Only String, not CSV file). I need to check which type of data is coming and manipulate that to JSON if request body would be a String. When I was trying to display the data type of data being sent to the server, it's displaying as "object" even after giving the "String" data as input. And the operation is getting successful but data is not inserting into the database. Can anyone help me in resolving this issue?
Sample Payload(request.body) would be,
"heat:22,humidity:36,deviceId:sb-0001"
Expected response is,
{
"heat": "22",
"humidity": "36",
"deviceId": "sb-0001"
}
#Aravind Actually typeof will returns "string" if operand is a string.So please check whether the string is coming or not in body, because if it is null then typeof will return "object".
I need to check which type of data is coming and manipulate that to JSON ...
HTTP is build upon the media-type concept that defines the syntax used in the payload of that type as well as the semantics of the elements used within that payload. You might take a look at how HTML defines forms to get a grip what a media-type specification should include. Through content negotiation client and server agree on a common media-type and thus a representation format to exchange payloads for supported by both. This simply increases interoperability between the participants as they exchange data in well-defined, hopefully standardized, representation formats both understand and support.
Through Accept headers a receiver can state preferences on which types to receive, including a weighting scheme to indicate that a certain representation format is preferred over an other one but the recipient is also fine with the other one, while a Content-Type header will indicate the actual representation format being sent.
RFC 7111 defines text/csv for CSV based representations and RFC 8259 specifies application/json for JSON payload. As the sender hopefully knows what kind of document it sends to the receiver, you can use this information to distinguish the payload on the receiver side. Note however that according to Fielding true REST APIs must use representation formats that support hypertext-driven interaction flows to allow clients to take further actions upon the payload received without having to invoke some external documentation. Both, JSON and CSV, by default don't support such an interaction model, that is abreviated with the term HATEOAS. For your simple scenario the content type negotiation might though be sufficient enough to solve your needs.
In terms of processing CSV and converting the data to JSON this is simply a matter of splitting up the CSV string on the delimiter symbol (, in the sample) to key-value pairs and then splitting the key and values further on the : symbol and adding these to a new JSON object. There is also a csvtojson library available at NPM that you might utilize in such a case.
I am trying to pull data from a REST API that uses a "similar standard to JSON RPC". The params I am passing look right according to the documentation here and here.
The error I am receiving is ...message:"Header missing request ID".... I am unsure what I am missing that would properly declare the requestID.
I have looked at the documentation provided via the API I am trying to pull data from but it's not very helpful considering it's all in PHP and cURL. I am trying to complete this task using python-requests.
getParams = {'method': 'getCustomers', 'params':{'where':'', 'limit': 2}, 'id': 'getCustomers'}
Result:
{"result":null,"error":{"code":102,"message":"Header missing request ID","data":[]},"id":null}
The return result should contain a list of All Customers and their attributes in JSON format.
Turns out there is nothing wrong with the code I am using. There is an issue with the API I am attempting to call.
In my situation, I was getting the same error back and was required to send a X-Request-ID header. I fixed it by adding the following to my headers:
headers = {
'X-Request-ID': str(uuid.uuid1()) # generate GUID based on host & time
...
Note that (for me) the GUID needed to be of a specific format (e.g. matching the Regex ^[{]?[0-9a-fA-F]{8}-([0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}[}]?$
taken from https://www.geeksforgeeks.org/how-to-validate-guid-globally-unique-identifier-using-regular-expression/). For example, it still gave the same error if I just sent "test".
I'm building a Node.js application on the express.js framework with CouchDB as a database. I'm utilizing CouchDB's session api for maintaining session state, and various databases for different sections of data.
On essentially every request my application code makes a request to Couch and then if there's an error (with Node) I can respond appropriately, by logging the error and redirecting to a 404 page or something like that. But if I get a CouchDB error, Node wouldn't consider it an error, it would consider that data. Now that's totally fine with me as long as CouchDB can only return this format:
{
"error": "illegal_database_name",
"reason": "Only lowercase characters (a-z), digits (0-9), and any of the characters _, $, (, ), +, -, and / are allowed. Must begin with a letter."
}
A JSON doc with two properties, error and reason. That's fine I can parse it and return the appropriate message; quite gracefully actually.
BUT! Is that all I can expect from CouchDB, or is there another way Couch might fail, that wouldn't yield a JSON doc with those two fields (properties)?
dscape's information of relying on the response codes is correct, and in most situations you will get an object with error and reason. The bulk-document errors are the only place I can think of where neither of these will be true. If just one document fails then you'll still get a 200, but you'll get the error/reason within the array element corresponding to the document that failed. See the docs for more info on that.