Connecting Eclipse Hono to Ditto - "description":"Check if all required JSON fields were set."},"status":400}" Error - eclipse-hono

I was successfully able to connect Hono to Ditto using AMQP adapters and I got the following messages in the log. The value sent from the demo device registered in Hono is successfully received and updated in the Ditto thing.
connectivity_1_ad306c4c315b | 2019-07-08 21:12:05,434 INFO [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-35] o.e.d.s.c.m.a.AmqpPublisherActor akka://ditto-cluster/system/sharding/connection/1/Insight-connection-1/pa/$a/c1/amqpPublisherActor2 - Response dropped, missing replyTo address: UnmodifiableExternalMessage [headers={orig_adapter=hono-http, device_id=4716, correlation-id=ID:AMQP_NO_PREFIX:TelemetrySenderImpl-35, content-type=application/vnd.eclipse.ditto+json, etag="hash:18694a24", orig_address=/telemetry, source=nginx:ditto}, response=true, error=false, authorizationContext=null, topicPath=ImmutableTopicPath [namespace=org.eclipse.ditto, id=4716, group=things, channel=twin, criterion=commands, action=modify, subject=null, path=org.eclipse.ditto/4716/things/twin/commands/modify], enforcement=null, headerMapping=null, sourceAddress=null, payloadType=TEXT, textPayload={"topic":"org.eclipse.ditto/4716/things/twin/commands/modify","headers":{"orig_adapter":"hono-http","device_id":"4716","correlation-id":"ID:AMQP_NO_PREFIX:TelemetrySenderImpl-35","content-type":"application/vnd.eclipse.ditto+json","etag":"\"hash:18694a24\"","orig_address":"/telemetry","source":"nginx:ditto"},"path":"/features","value":null,"status":204}, bytePayload=null']
things-search_1_8f2ad3dda4bf | 2019-07-08 21:12:05,593 INFO [] o.e.d.s.t.p.w.s.EnforcementFlow - Updating search index of <1> things
things-search_1_8f2ad3dda4bf | 2019-07-08 21:12:05,598 INFO [] o.e.d.s.t.p.w.s.EnforcementFlow - Got SudoRetrieveThingResponse <1> times
things-search_1_8f2ad3dda4bf | 2019-07-08 21:12:05,725 INFO [] a.s.Materializer akka.stream.Log(akka://ditto-cluster/user/thingsSearchRoot/searchUpdaterRoot/StreamSupervisor-21) - [SearchUpdaterStream/BulkWriteResult] Element: BulkWriteResult[matched=1,upserts=0,inserted=0,modified=1,deleted=0]
But when I tried to make a new connection (Hono - installed in a different server and ditto hosted in same server where the above successful connection is made). Connection is established and also when I try to send the messages from the demo devices registered in Hono to Ditto. I get the following response.
vigkam#srvgal89:~$ curl -X POST -i -u sensor0101#tenantAdapters:mylittle -H 'Content-Type: application/json' -d '{"temp": 23.09, "hum": 45.85}' http://srvgal89.deri.ie:8080/telemetry
HTTP/1.1 202 Accepted
content-length: 0
And when I try to retrieve connection metrices, I can see the increase in the metric count with respect to the number of messages sent from Hono.
But the only problem is the sensor values (temp and Humidity as in the above curl command) are not getting updated in the ditto thing.
I got the below error message in the log which says "description":"Check if all required JSON fields were set."},"status":400}"
connectivity_1_ad306c4c315b | 2019-07-08 21:34:17,640 INFO [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-13] o.e.d.s.c.m.a.AmqpPublisherActor akka://ditto-cluster/system/sharding/connection/23/Gal-Connection-10/pa/$a/c1/amqpPublisherActor2 - Response dropped, missing replyTo address: UnmodifiableExternalMessage [headers={content-type=application/vnd.eclipse.ditto+json, orig_adapter=hono-http, orig_address=/telemetry, device_id=4816, correlation-id=ID:AMQP_NO_PREFIX:TelemetrySenderImpl-13}, response=true, error=true, authorizationContext=null, topicPath=ImmutableTopicPath [namespace=unknown, id=unknown, group=things, channel=twin, criterion=errors, action=null, subject=null, path=unknown/unknown/things/twin/errors], enforcement=null, headerMapping=null, sourceAddress=null, payloadType=TEXT, textPayload={"topic":"unknown/unknown/things/twin/errors","headers":{"content-type":"application/vnd.eclipse.ditto+json","orig_adapter":"hono-http","orig_address":"/telemetry","device_id":"4816","correlation-id":"ID:AMQP_NO_PREFIX:TelemetrySenderImpl-13"},"path":"/","value":{"status":400,"error":"json.field.missing","message":"JSON did not include required </path> field!","description":"Check if all required JSON fields were set."},"status":400}, bytePayload=null']
Please let me know if I am missing something. Thank you in advance.!!
More Information :
The thingId in Ditto is org.eclipse.ditto:4816,
Tenant Id in Hono - tenantAdapters,
Device Registered in Hono - 4816 (tenantAdapters),
Auth Id of the device - sensor0101,
ConnectionId between Hono and Ditto - Gal-Connection-10

probably you are getting this failure since Ditto can't parse non ditto protocol messages. From reading your logs, I think your Ditto thing currently looks like this:
{
"thingId": "org.eclipse.ditto:4716",
"features": null
}
You could verify this by using a GET request to http://<your-ditto-address>:<your-ditto-gateway-port>/api/2/things/org.eclipse.ditto:4716.
Since you probably want to store the temperature and humidity to a feature of your thing, it would be best to not have the features as null, but already provide a feature with an ID for the value. Do this by creating a feature, e.g. with id 'environment' via a PUT to http://<your-ditto-address>:<your-ditto-gateway-port>/api/2/things/org.eclipse.ditto:4716/features/environment and content {}. Afterwards your thing should probably look like this:
{
"thingId": "org.eclipse.ditto:4716",
"features": {
"environment": {}
}
}
Now back to your initial question: Ditto will only understand ditto protocol messages and therefore doesn't know what to do with your JSON object.
To solve this problem you have two options:
1. adding a payload mapping script for incoming messages to your connection.
2. publishing a ditto protocol message instead of the simple JSON object. This would then look something like this:
vigkam#srvgal89:~$ curl -X POST -i -u sensor0101#tenantAdapters:mylittle -H 'Content-Type: application/json' -d '{ "topic": "org.eclipse.ditto/4716/things/twin/commands/modify", "path": "/features/environment", "value": {"temp": 23.09, "hum": 45.85} }' http://srvgal89.deri.ie:8080/telemetry
Note that I have specified the path /features/environment which will update the value of the environment feature of your thing.

Messages processed by Eclipse Ditto via AMQP (e.g. Hono) must be in the so called Ditto Protocol being a JSON based protocol which contains apart from other JSON fields the path field which is missing in your JSON (hence the error message "JSON did not include required </path> field!").
So you have at least 2 options to proceed:
Instead of your JSON format {"temp": 23.09, "hum": 45.85} send a message in Ditto Protocol, e.g. have a look here for an example
Use the Payload mapping feature of Ditto in order to specify a JavaScript function to invoke on all incoming messages from Hono in order to transform them into a valid Ditto Protocol message

Related

Handling of etags in batch request using SAP Cloud SDK

I am trying to carry out a batch request including a create, update and a delete (all are different salesorders). As per this question here which deals with something similar, I have done a get for the items I want to update and delete before I add them to the batch request. I am using the SalesOrder.builder() to prepare the SalesOrder I want to create.
final ErpHttpDestination destination = DestinationAccessor.getDestination(DESTINATION_NAME)
.asHttp().decorate(DefaultErpHttpDestination::new);
final SalesOrderItem salesOrderItem1 = SalesOrderItem.builder().material(material)
.requestedQuantityUnit(requestedQuantityUnit).build();
final SalesOrder salesOrder1 = SalesOrder.builder().distributionChannel(distributionChannel)
.salesOrderType(salesOrderType).salesOrganization(salesOrganization)
.organizationDivision(organizationDivision).soldToParty(soldToParty)
.item(salesOrderItem1).build();
final SalesOrder orderToUpdate = new GetSingleSalesOrderCommand(orderToUpdateID, destination,
new DefaultSalesOrderService()).execute();
orderToUpdate.setSoldToParty(updateSoldToParty);
final SalesOrder orderToDelete = new GetSingleSalesOrderCommand(orderToDeleteID, destination,
new DefaultSalesOrderService()).execute();
SalesOrderServiceBatch service = new DefaultSalesOrderServiceBatch(
new DefaultSalesOrderService());
BatchResponse bRes = service.beginChangeSet().createSalesOrder(salesOrder1).updateSalesOrder(orderToUpdate)
.deleteSalesOrder(orderToDelete).endChangeSet().execute(destination);
I am then logging the BatchResponse and see I am getting a Batch Response Failure:
eTag handling not supported for http method 'POST'
I have searched for this error but can't find any resolution to it. Any ideas?
Thanks.
UPDATE: Increasing the logging to DEBUG I can see the batch request that is being sent and can see that there is an if-match header being added to the create request, which doesn't make sense as it can't match something that doesn't exist yet.
"msg":"--batch_123\r\nContent-Type: multipart/mixed;
boundary=changeset_(changeset number)\r\n\r\n--
changeset_(changeset number)\r\nContent-Type:
application/http\r\nContent-Transfer-Encoding: binary\r\n\r\nPOST
/sap/opu/odata/sap/API_SALES_ORDER_SRV/A_SalesOrder HTTP/1.1\r\nContent-
Length:
193\r\nIf-Match: W/\"datetimeoffset'2020-05-
01T11%3A51%3A16.8631720Z'\"\r\nAccept:
application/json;odata=verbose\r\nContent-Type:......
The I get the error:
Inner Error:
"msg":"batch
responseFailure(com.sap.cloud.sdk.odatav2.connectivity.ODataException:
null: <?xml version=\"1.0\" encoding=\"utf-8\"?><error
xmlns=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\">
<code>/IWFND/CM_MGW/537</code><message xml:lang=\"en\">eTag handling not
supported for http method 'POST'</message><innererror>...
However, what does work is if I wrap each request in its own changeset e.g.
service
.beginChangeSet().createSalesOrder(order).endChangeSet()
.beginChangeSet().updateSalesOrder(orderToUpdate).endChangeSet()
.beginChangeSet().deleteSalesOrder(orderToDelete).endChangeSet()
.execute(destination);
Edit:
This is fixed as of version 3.25.0.
Initial Answer:
This seems to be a bug. I was able to reproduce this with a different service and the behaviour is the same: The if-match header is incorrectly applied to the POST operation as well.
When debugging it seems like the request is build up correctly with the header only being present on update and delete. However, it seems that when the batch request is serialised to JSON it gets added to all requests.
So until this is fixed the workaround is isolating these operations via change sets, as you already pointed out.
Looks like eTag handling is not supported for your endpoint.
Now you can do the following to omit eTag headers:
orderToUpdate.setVersionIdentifier(null);
orderToDelete.setVersionIdentifier(null);
However I'm not sure how 'POST' fits the error description, because update uses PATCH and delete uses DELETE. The only POST that I expect would be coming from create. But we do not add headers for entity version identifiers (eTag) in OData create operation. If the same error still comes up, please try again without running createSalesOrder(salesOrder1).

rsyslog import non standard logs

I'm trying to import an application log to mysql that is not in the standard syslog format.
An example line:
Dec 5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2
When I use the imfile module to import the log (and then subsequently forward it to mysql), it works ok but the entire line all goes into the message field. This also means that the fields ReceivedAt and DeviceReportedTime are the timestamp of when the log is imported, rather than the actual event time in the message.
I think the answer lies with the property replacer, but I can't seem to find an example online about how to actually grab the actual date, and force it into the DeviceReportedTime field.
This is what ends up in the DB:
53052 NULL 2018-12-04 16:17:44 2018-12-04 16:17:44 16 5 server Dec 5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2 5 NULL customtag NULL NULL 0 NULL NULL NULL NULL NULL 3 customtag - NULL NULL
I have the following config on the client side in /etc/rsyslog.d:
module(load="imfile" mode="inotify")
input(type="imfile"
File="/var/log/appname/applog.log"
Tag="customtag")
And this on the server side under /etc/rsysconfig.d:
:syslogtag, contains, "customtag":ommysql:10.255.2.6,rsyslogdb,loganalyzer,password
This is not the complete answer, as it is not a part of rsyslog I have used before, but it should get you close to the final solution.
You can use rsyslog's input parsing library, liblognorm, and module mmnormalize. You may need to install an extra package or two if these are not included with rsyslog. To start with, write a rules file myrules.rb containing a single line describing the fields you have:
rule=:%date:date-rfc3164% %tag:word% %host:char-to:[%[%pid:number%]: %msg:rest%
You can use your example line by providing it as standard input to the test program lognormalizer:
echo 'Dec 5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2' |
lognormalizer -r myrules.rb
You should get the json formatted output:
{ "msg": "Client process timed out: 2", "pid": "10099",
"host": "coova-chilli", "tag": "wifi", "date": "Dec 5 10:50:06" }
You can now add the use of this module to your rsyslog configuration file:
module(load="mmnormalize")
action(type="mmnormalize" rulebase="myrules.rb")
template(name="simple" type="string" string="%$!date:::date-rfc3339% %$!host% %$!msg%\n")
if $parsesuccess=="OK" then action(type="omfile" file="output" template="simple")
The same example input line in the input file should now be parsed and the json keys will be available as variables such as $!host for use in a template. The above should write a line in the output file like:
Dec 5 10:50:06 coova-chilli Client process timed out: 2
There is a lot I still do not understand about the above, so you should probably start a new separate post for each new question on specific points so that others may answer.

How do I get the JMS Header JMSMessageID from my message?

I want to substract the JMSMessageID Header from my message using apache nms. With the message.Properties[] I can only access my message Properties, how do I get the JMSMessageID?
I tried the code below with no success.
ITextMessage message = (ITextMessage)consumer.Receive(TimeSpan.FromMilliseconds(10000));
string JMSMessageID = message.Properties["JMSMessageID"].ToString();
The Message ID in NMS is accessed via the NMSMessageID properties or I think you can use the "NMSMessageID" string in the Message properties and it will be intercepted and the message ID value will be returned.

http://localhost:8529/_api/query/225 - syntax correct?

I am trying to abort a long running query (with id 225) with the new API feature for the first time. But I can't get it killing the request. the API request answers with
{"error":true,"code":404,"errorNum":404,"errorMessage":"not found"}
although the query still is running.
[
{
"id": "225",
"query": [SNIP]
}
]
What am I doing wrong?
Thanks in advance ...
Can only guess as the question does not contain full information on what is actually posted and with which HTTP method.
My guess is that you used HTTP GET when you tried killing the query and not HTTP DELETE. So the URL was probably correct but not the HTTP method. In this case you will also get a 404 error.
There are two ways for terminating a running query:
using the ArangoShell
First of all, the query id needs to be determined. This can be achieved as follows:
require("org/arangodb/aql/queries").current();
Using the returned id value, the command to kill the query is:
require("org/arangodb/aql/queries").kill(id);
using HTTP
When the query is known, it can be used in an HTTP DELETE request:
curl -X DELETE http://127.0.0.1:8529/_api/query/id
Again, id needs to be query's real id.

How to tag a Commit in API using curl command

I am trying to use a curl command to tag a commit. Is this possible? I went through the create-a-tag-object link from GitHub, but it doesn't work.
Creating a tag is a bit complicated, but if you just follow the API docs you should be fine. Note that the API docs say:
Note that creating a tag object does not create the reference that makes a tag in Git. If you want to create an annotated tag in Git, you have to do this call to create the tag object, and then create the refs/tags/[tag] reference. If you want to create a lightweight tag, you simply have to create the reference - this call would be unnecessary.
So, before you continue with creating a tag, you should know what kind of tag you want to create -- annotated or lightweight. Basically, an annotated tag is the same as a lightweight, but it also contains a message of the tag, info about the author of the tag, and the date and time when the tag was created. A lightweight tag is just a named pointer to a specific commit in your history.
Ok, so, what the API docs basically say is: if you want to create an annotated tag - you will have to make 2 API calls, and if you want to create a lightweight tag - you will have to make just 1 call. So, I'll give an example of creating an annotated tag with the 2 API calls, and if you want to create a lightweight tag - just skip the first API call and go to the second.
To create an annotated tag, you have to:
Step 1
Create a tag object using the tags API. The API docs are a bit unclear here how the parameters should be passed. What's missing is an example of the message that you need to send to the server. So, create a file called tag_object_req.json on your local disk, and put the following JSON document in it:
{
"tag": "v0.0.1",
"object": "c5f8759ffd808d4a57ea36c63960f3e2cc6fcc2b",
"message": "creating a tag",
"tagger": {
"name": "Ivan Zuzak",
"email": "izuzak#gmail.com",
"date": "2012-06-17T14:53:35-07:00"
},
"type": "commit"
}
Obviously, you have to replace the information in the document to reflect your situation. The meaning of the parameters are described in the API docs here.
After you have saved the file, you can make an API call using curl to create the tag object:
curl -v -X POST -d #tag_object_req.json --header "Content-Type:application/json" -u izuzak "https://api.github.com/repos/izuzak/test/git/tags"
So, the -v part will force curl to output all HTTP headers, the -X POST part means that an HTTP POST request must be made, -d #tag_object_req.json specifies which file will be used as the content (body) of the POST request, --header "Content-Type:application/json" specifies the media type of the request (JSON message), and -u izuzak specifies your username for authorization (and curl will ask you for your password when you make the request).
The response you get should be a 201 Created HTTP response, with the JSON message structured as this:
{
"sha": "e6d9fb6b9a13cab11923345e2400d5cf8df97267",
"url": "https://api.github.com/repos/izuzak/test/git/tags/e6d9fb6b9a13cab11923345e2400d5cf8df97267",
"tagger": {
"name": "Ivan Zuzak",
"email": "izuzak#gmail.com",
"date": "2012-06-17T21:53:35Z"
},
"object": {
"sha": "c5f8759ffd808d4a57ea36c63960f3e2cc6fcc2b",
"type": "commit",
"url": "https://api.github.com/repos/izuzak/test/git/commits/c5f8759ffd808d4a57ea36c63960f3e2cc6fcc2b"
},
"tag": "v0.0.1",
"message": "creating a tag"
}
Before continuing, notice the sha attribute of the object you just created (e6d9fb6b9a13cab11923345e2400d5cf8df97267) because you will use this value in the next step.
Step 2
Create a tag reference using the references API. The API docs are much clearer here about what the request should look like. So, you first have to create another file on the disk, named tag_ref_req.json, and put this content inside:
{
"ref": "refs/tags/v0.0.1",
"sha": "e6d9fb6b9a13cab11923345e2400d5cf8df97267"
}
However, notice that the value of the sha in this JSON depends on the type of tag you are creating. If you are creating an annotated tag, the value of the sha is the same value you got from the previous step - the sha of the tag object (e6d9fb6b9a13cab11923345e2400d5cf8df97267). However, if you are creating a lightweight tag, the value of the sha is the sha of the commit object you are tagging with the tag, because you have not created a tag object. In my case, you can see in step 1 that the commit object I am tagging is c5f8759ffd808d4a57ea36c63960f3e2cc6fcc2b, and this will be different in your case of course if you are creating a lightweight tag.
Ok, so after you have created this file and defined the sha and the name of the tag, you can make an API request in a similar way as in the previous step, using curl:
curl -v -X POST -d #tag_ref_req.json --header "Content-Type:application/json" -u izuzak "https://api.github.com/repos/izuzak/test/git/refs"
Notice that we are now making a request to https://api.github.com/repos/izuzak/test/git/refs with the second file as the content.
The response should again be a 201 Created HTTP response, and the body will be a JSON document looking like this:
{
"ref": "refs/tags/v0.0.1",
"url": "https://api.github.com/repos/izuzak/test/git/refs/tags/v0.0.1",
"object": {
"sha": "e6d9fb6b9a13cab11923345e2400d5cf8df97267",
"type": "tag",
"url": "https://api.github.com/repos/izuzak/test/git/tags/e6d9fb6b9a13cab11923345e2400d5cf8df97267"
}
}
And now you can navigate to your project on GitHub and go to "Switch branches/tags" and see your tag there.
Hope this helps!
You can also try the new Releases API
http://developer.github.com/v3/repos/releases/#create-a-release
curl \
--user <username> \
--header "Accept: application/vnd.github.manifold-preview" \
--data "tag_name=mytagname" \
"https://api.github.com/repos/<username>/<repository>/releases"
This is curl will create Releases. (But as ChucK mentioned it may create only lightweight tag, I haven't personally verified this )
curl \
--user <Your Github username> \
--header "Accept: application/vnd.github.manifold-preview" \
--data '{"tag_name": "v1.0.0", "target_commitish": "master", "name": "v1.0.0", "body": "Description of the release", "draft": false, "prerelease": false }' \
"https://api.github.com/repos/<OrganizationName>/<RepoName>/releases" -X POST
Adding this here, incase anybody come looking for this, like me.

Resources