I am currently using the following mapping template to pass data sent to an AWS API Gateway endpoint to AWS Kinesis Firehose stream:
{
"DeliveryStreamName": "[STREAMNAME]",
"Record": {
"Data": "$util.base64Encode($input.body)"
}
}
What I would like to do is: adding information to the $input.body that is being encoded like the $context.identity.sourceIp of the client making the request.
How can I go about this when the output being passed to Kinesis Firehose needs to be Base64-encoded? Ideally I would like my data that is being posted to Kinesis Firehose look like this:
{
"x": 1,
"y": 2,
"z": 3,
..., // all the properties from the JSON-request by the client
"clientIp": "x.x.x.x" // property added by API-Gateway into client's object
}
After a little bit more digging I managed to get the following to work:
#set($inputRoot = $input.path('$'))
#set($data = "{
#foreach($key in $inputRoot.keySet())
""$key"": $input.json($key),
#end
""clientIP"": ""$context.identity.sourceIp"",
}")
{
"DeliveryStreamName": "[STREAMNAME]",
"Record": {
"Data": "$util.base64Encode($data)"
}
}
I was not aware that you could do a #foreach inside a #set. Note that you also have to use double-quotes to get this right.
Related
I am currently using SDK version 3.39.0 and version 0004 of the API_MKT_CONTACT service definition to try to create a new Contact with an AdditionalID in Marketing Cloud with the following code:
ODataRequestUpdate contactRequest =
contactService
.updateContactOriginData(contact)
.withHeader("Sap-Cuan-RequestTimestamp", getFormattedTime(System.currentTimeMillis()))
.withHeader("Sap-Cuan-SequenceId", "UpdatePatch")
.withHeader("Sap-Cuan-SourceSystemType", "EXT")
.withHeader("Sap-Cuan-SourceSystemId", "sdk-test")
.toRequest();
var additionalId =
AdditionalID.builder()
.externalContactID(pii.getId().toString())
.originOfContact(origin)
.originOfContact_2("EMAIL") //ContactAdditionalOrigin
.externalContactID_2(pii.getEmail()) //ContactAdditionalID
.build();
var additionalIdRequest = contactService
.updateAdditionalIDs(additionalId)
.replacingEntity()
.withHeader("Sap-Cuan-RequestTimestamp", getFormattedTime(System.currentTimeMillis()))
.withHeader("Sap-Cuan-SourceSystemType", "EXT")
.withHeader("Sap-Cuan-SourceSystemId", "sdk-test")
.toRequest();
// use low level API as a work around for https://github.com/SAP/cloud-sdk/issues/156
ODataRequestBatch requestBatch = new ODataRequestBatch(ContactService.DEFAULT_SERVICE_PATH, ODataProtocol.V2);
requestBatch
.beginChangeset()
.addUpdate(contactRequest)
.addUpdate(additionalIdRequest)
.endChangeset();
HttpClient httpClient = HttpClientAccessor.getHttpClient(contactsDestination);
ODataRequestResultMultipartGeneric batchResult = requestBatch.execute(httpClient);
batchResult.getResult(additionalIdRequest);
This results in the following error:
{
"error": {
"code": "SY/530",
"message": {
"lang": "en",
"value": "Inline component is not defined or not allowed (HTTP PUT)"
},
"innererror": {
"application": {
"component_id": "CEC-MKT-DM-IC",
"service_namespace": "/SAP/",
"service_id": "API_MKT_CONTACT_SRV",
"service_version": "0004"
},
"transactionid": "3B63A2A6CC9205E0E00604E1D31F1CDF",
"timestamp": "20210315142401.8432680",
"Error_Resolution": {
"SAP_Transaction": "For backend administrators: use ADT feed reader \"SAP Gateway Error Log\" or run transaction /IWFND/ERROR_LOG on SAP Gateway hub system and search for entries with the timestamp above for more details",
"SAP_Note": "See SAP Note 1797736 for error analysis (https://service.sap.com/sap/support/notes/1797736)",
"Batch_SAP_Note": "See SAP Note 1869434 for details about working with $batch (https://service.sap.com/sap/support/notes/1869434)"
},
"errordetails": []
}
}
}
I am using this documentation as a guide for building my requests (under the section "Create Contacts with Additional IDs"). When I run the example code in Postman it works as expected. Note that the payload for the AdditionalIDs is an empty JSON object.
So I enabled HTTP wire logs and noticed that the SDK seems to be including the following payload:
PUT AdditionalIDs(ContactAdditionalOrigin='EMAIL',ContactAdditionalID='wade.watts#theoasis.com',ContactID='ae46e174-52a3-4de6-8caa-57213151b295',ContactOrigin='<CONTACT_ORIGIN>') HTTP/1.1
Sap-Cuan-SourceSystemId: sdk-test
Accept: application/json
Sap-Cuan-SourceSystemType: EXT
Content-Type: application/json
Sap-Cuan-RequestTimestamp: '2021-03-15T14:24:00.828'
{"ContactOrigin":"<CONTACT_ORIGIN>","ContactID":"ae46e174-52a3-4de6-8caa-57213151b295","ContactAdditionalOrigin":"EMAIL","ContactAdditionalID":"wade.watts#theoasis.com","ContactAdditionalIdUUID":null,"ContactUUID":null,"ContactAddlIDIsInvalid":null,"MarketingAreas":[]}
Unfortunately, I can't seem to find a way to omit the payload/inline component while using the SDK so that it matches the example code. Is this an issue with the SDK or am I doing something wrong? Any help would be much appreciated!
Cheers!
UPDATE
Applying the suggested workaround from #matkuhr I changed my additionalIdRequest above to this and it worked:
ODataEntityKey key = new ODataEntityKey(ODataProtocol.V2)
.addKeyProperty(AdditionalID.EXTERNAL_CONTACT_ID.getFieldName(), mcContact.getContactId())
.addKeyProperty(AdditionalID.ORIGIN_OF_CONTACT.getFieldName(), origin)
.addKeyProperty(AdditionalID.ORIGIN_OF_CONTACT_2.getFieldName(), "EMAIL")
.addKeyProperty(AdditionalID.EXTERNAL_CONTACT_I_D_2.getFieldName(), mcContact.getEmailAddress());
var request = new ODataRequestUpdate(
ContactService.DEFAULT_SERVICE_PATH,
"AdditionalIDs",
key,
"{}",
UpdateStrategy.REPLACE_WITH_PUT,
null,
ODataProtocol.V2);
request.addHeader("Sap-Cuan-RequestTimestamp", getFormattedTime(System.currentTimeMillis()));
request.addHeader("Sap-Cuan-SourceSystemType", "EXT");
request.addHeader("Sap-Cuan-SourceSystemId", "sdk-test);
It's not you doing something wrong, it's also not the SDK, it's the service. The service seems to substantially deviate from the OData V2 conventions as well as basic HTTP conventions.
You can work around this by leveraging the low-level APIs of the SDK even more. Create the update request fully manually with the payload the service requires, e.g.:
ODataEntityKey key = new ODataEntityKey(ODataProtocol.V2)
.addKeyProperty(Contact.XYZ.getFieldName(), contact.getXyz())
request = new ODataRequestUpdate(
contactService.getServicePath(),
contact.getEntityCollection(),
key,
"{ }", // this will be the payload
UpdateStrategy.REPLACE_WITH_PUT,
null,
ODataProtocol.V2);
request.addHeader("key", "val");
// add more headers & parameters if needed and execute
In Spring Integration, I have message like following :
{
"name":"House",
"attributeIds": [1,3,5]
}
I need to enrich/transform this message using some Rest Service, which will give me the attribute values.
For example http://restservice.com/attributes?id=1,3,5 will answer me with
{"attributes": [
{"id": 1, "value":"Waterproof"},
{"id": 3, "value":"SoundProof"},
{"id": 5, "value":"Concrete"}
]}
And the final object should look like this:
{
"name":"House",
"attributes": [
{"id": 1, "value":"Waterproof"},
{"id": 3, "value":"SoundProof"},
{"id": 5, "value":"Concrete"}
]
}
How can this be achieved?
Should it be like this? https://www.youtube.com/watch?time_continue=273&v=DHPsWDgEUXg
InboundAdapter -> Enricher -> Request Channel -> Service Activator -> Enricher -> Outbound Adapter?
This is indeed a typical task for Content Enricher.
So, what you need is to deserialize that incoming JSON into a plain Map. Use a request-payload-expression="payload.attributeIds" to have that list of ids as a payload for sub-flow request.
A subscriber on the request-channel could be just simple Spring Integration HTTP Outbound Gateway to call that REST service and get an attributes message back.
This gateway can just come without an output-channel to produce its result back into a content-enricher via replyChannel header.
When this reply message comes to the content-enricher, a simple <int:property name="attributes"> can be used to populate that new option in the request Map.
Afterwards you can remove an attributeIds key from that map and serialize it back to JSON if needed.
UPDATE
Here is a sample how it could be possible with Java DSL and Spring Boot: https://github.com/artembilan/sandbox/tree/master/spring-integration-enricher
I'm writing a simple function that gets triggered when a new message comes to the Service Bus queue. The input for that function is a DocumentDB document. I need to access a value in the document and type it into the console log.
When you create a Service Bus queue trigger binding, it is automatically created as this:
public static void Run(string myQueueItem, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
}
Now to access the correct document in my database, according to this page it is possible to set your bindings so that you get a document ID from your queue message:
id supports bindings similar to {queueTrigger}, which uses the string value of the queue message as the document Id.
That means that you can send a message to the queue and then bind to your document like this:
{
"name": "inputDocument",
"type": "documentDB",
"databaseName": "MyDatabase",
"collectionName": "MyCollection",
"id" : "{myQueueItem}",//<<<
"connection": "MyAccount_DOCUMENTDB",
"direction": "in"
}
So far, everything seems to work as intended. However, in order to send a message to the Service Bus queue, you have to send it as a BrokeredMessage type (link). And when you do that, you can't access the message as string anymore, making the automatically created function useless.
It is possible to work with BrokeredMessages through this:
project.json:
{
"frameworks": {
"net46":{
"dependencies": {
"WindowsAzure.ServiceBus": "2.7.6"
}
}
}
}
run.csx:
using Microsoft.ServiceBus.Messaging;
....
log.Info($"C# ID: {queueItem.GetBody<string>()})
But I haven't been able to find out how to pass {queueItem.GetBody<string>()} into the id attribute of the input document binding. So I can no longer use the aforementioned method of binding my documentdb input through "id" : "{myQueueItem}" and thus am unable to read any values from any documents.
Is there a way of sending a raw string to the Bus Queue or is BrokeredMessage really the only possibility? (from what I've been able to find out, raw string is not possible)
If BrokeredMessage is indeed the only way, is it possible to still get the string value from the message and use it as an ID for a DocumentDB document?
If neither of the above are possible, could you please point me in the right direction with this?
A more complete explanation of my function:
An application periodically sends new documents into the DocumentDB database. Together with the document (after waiting for the insert to complete) it sends a message with the new document's ID into the Service Bus queue. Then the trigger connects to the new document and checks an integer value inside it. If the value is larger than 10, it sends an email notification about it.
I have successfully created this and tested it through the web-based editor's manual Run with a string id of one of the documents as the test input. However, when I tried to use it with the app that sends the documents and messages automatically, I ran into the BrokeredMessage problem as described above.
Below is the whole code for the wórking function without BrokeredMessage:
using System;
using System.Net;
using System.Net.Mail;
public static void Run(string queueItem, dynamic inputDocument, TraceWriter log)
{
log.Info($"C# ID: {queueItem}, Value: {inputDocument.Value}");
if (inputDocument.Value > 10)
{
var fromAddress = new MailAddress("---#gmail.com", "From ---");
var toAddress = new MailAddress("---#gmail.com", "To ---");
const string fromPassword = "---";
const string subject = "Notification";
const string body = "Temperature too high!";
var smtp = new SmtpClient
{
Host = "smtp.gmail.com",
Port = 587,
EnableSsl = true,
DeliveryMethod = SmtpDeliveryMethod.Network,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(fromAddress.Address, fromPassword)
};
using (var message = new MailMessage(fromAddress, toAddress)
{
Subject = subject,
Body = body
})
{
smtp.Send(message);
}
}
}
Here's a working example showing how to receive a ServiceBus message as a POCO and bind to it's properties.
If you send a message to the queue with the correct application/json content type, the function will deserialize into the POCO for you, and the Document DB input binding will bind to the DocumentId property and retrieve the document for you. You don't need to add anything in the project.json for this to work.
The function.json file:
{
"bindings": [
{
"name": "input",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "<your-queue.",
"connection": "<your-connection>",
"accessRights": "Manage"
},
{
"type": "documentDB",
"name": "document",
"databaseName": "<your-db>",
"collectionName": "<your-collection>",
"id": "{DocumentId}",
"connection": "<your-connection>",
"direction": "in"
}
]
}
The function code:
using System;
using System.Threading.Tasks;
public class Input
{
public string DocumentId { get; set; }
public int Value { get; set; }
}
public static void Run(Input input, dynamic document, TraceWriter log)
{
log.Info($"Message received (DocumentId: {input.DocumentId}, Value {input.Value})");
log.Info($"Document read {document.id}");
}
For C# functions, to use binding parameters (e.g. the {DocumentId} parameter) the trigger input must be bound to a POCO object defining those properties.
For sending email messages, you might also look into our SendGrid output binding. We have a complete sample "SendGrid-CSharp" available in the portal under "Samples" :)
I created a basic GET url in API Gateway; with a path parameter called "name".
How can I access this name param? I don't see it in neither event or context.
Am I mistaken at the purpose of the parameter path?
Let me use my play app for example:
GET /api/v1/person #controllers.PersonController.list(limit: Int ?= 50, offset: Long ?= 0)
GET /api/v1/person/$id<[0-9]+> #controllers.PersonController.getById(id: Long)
GET /api/v1/person/$id<[0-9]+>/email #controllers.PersonController.getEmailByPersonId(id: Long)
Is this achievable using AWS API Gateway?
According to the docs you should be able to do this with a mapping template: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
Modifying their example("Example Request and Response" near the bottom) to fit your URI it would look like this
//Resource: /api/v1/person/{id}
//With input template(this is what your lambda will see in the event):
{
"id" : "$input.params('id')"
"person" : $input.json(‘$.person')
}
//POST /api/v1/person/{id}
{
"person" : {
"name": "bob"
}
}
I am struggling in order to bind the kendo grid with Web API. Web API is hosted in the Azure worker role (not yet published). And the Web API is decorated in order to work with the cross domains i.e Enabled CORS for the Web API.
Let us look the code I have written.
Action method in the Web API look like the following.
public JToken Get()
{
JToken json = JObject.Parse(
"{ \"firstName\": \"John\",
\"lastName\": \"Smith\",
\"isAlive\": true,
\"age\": 25,
\"height_cm\": 167.6,
\"address\":
{
\"streetAddress\": \"21 2nd Street\",
\"city\": \"New York\",
\"state\": \"NY\",
\"postalCode\": \"10021-3100\"
},
\"phoneNumbers\":
[{
\"type\": \"home\",
\"number\": \"212 555-1234\"
},
{
\"type\": \"office\",
\"number\": \"646 555-4567\"
}]
}"
);
return json;
}
When I execute this web api we will get the emulator with the IP address as "127.0.0.1" and I used the controller "TestController".
When I execute the above api in the browser, I am getting the JSON data perfectly, but when I use the same to bind the grid, all is vain.
The code used to bind the data is
$("#dw_report_container1").kendoGrid({
dataSource: {
type: "json",
transport: {
read: {
url: "http://127.0.0.1/test",
dataType: "json"
}
},
},
height: 500,
scrollable: true,
selectable: true
});
I am getting no exception.
Finally, after a long struggle i got the solution :(. I find a strange thing when working with the kendo grid. Whenever we work with the models inside our project and returned the model to the datasource of the kendogrid, it internally creates a model for it. As i worked with sample json as above, it (kendogrid) doesn't create the model for the grid to bind. Hence the problem occurs. Eventually, what i concluded that it is better to have a schema/model in the datasource (kendo grid). Past i didn't :( Thank you.