I am triggering a step function through an Express route in a Node app. This step function interacts with the Ethereum blockchain, and is thus highly asynchronous. There is also a possibility of transactions failing if a multiple attempts are made at once.
As such, I want to queue up executions of this step function, but oddly there doesn't seem a straightforward way to do so.
What is the best way to do this?
You can go with Map in Step Functions.
Step Functions provide inbuilt way execute in parallel or on execution at a time for given number of items. Below is an example:
"Validate-All": {
"Type": "Map",
"InputPath": "$.detail",
"ItemsPath": "$.shipped",
"MaxConcurrency": 0,
"Iterator": {
"StartAt": "Validate",
"States": {
"Validate": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:ship-val",
"End": true
}
}
},
"ResultPath": "$.detail.shipped",
"End": true
}
You need to change the value of MaxConcurrency=1 so that only one execution will happen once and it will keep going until your items in InputPath is completely not exhausted. InputPath is supposed to be a list, you can queue up your items in InputPath and start the State Machine.
You can read more here.
Related
So I'm using an testing tool called ReadyAPI and for scripting it uses the Groovy language. I'm not familiar with the language and the fact that it's based on Java it somehow makes it even worse.
Now I'm trying to validate a REST response in JSON with an assertion that checks that certain elements exist in the response.
This is the code that I have now:
import groovy.json.*
def response = context.expand( 'RequestResponseHere' )
def object = new JsonSlurper().parseText(response)
assert response.contains("CODE1")
assert response.contains("CODE2")
assert response.contains("CODE3")
assert response.contains("CODE4")
The assertion seems to work but I was wondering if there is maybe a simpler way to do it than to have to write so many lines and making it less 'bulky'?
Any help is greatly appreciated.
Thank you!
Added an example of the json data that I have to parse:
What I need is to check that the value of "code" is always part of a list of acceptable values e.i. CODE1, CODE2, etc.
{
"_embedded": {
"taskList": [
{
"code": "CODE1",
"key": 123
},
{
"code": "CODE2",
"key": "234"
},
{
"code": "CODE3",
"key": "2323"
},
{
"code": "CODE4",
"key": "7829"
},
{
"code": "CODE5",
"key": "8992"
}
]
}
}
If you want to check for certain things to be there, you can DRY that
code with:
["code1","code2"].each{ assert response.contains(it) }
And as stated in the comments, if you want to make sure, "all are there,
but I don't care for the order", extracting the values and comparing it
as results can shorten this:
assert response._embeded.taskList*.code.toSet() == ["code1", ...].toSet()
Note the use of *., which is the "spread operator". It is basically
the same as ...collect{ it.code }. Then comparing the string sets
does the rest (this is fine if you are comparing not to many items since
the power assert will print all the items along the failed assertion; if
you have to find the proverbial needle in the haystack, you are better
off writing something smarter to check the two sets).
The assertion seems to work but I was wondering if there is maybe a
simpler way to do it than to have to write so many lines and making it
less 'bulky'?
Unclear what it is that is bulky but if what you want is to lessen the number of statements, one thing you could do is use a single assert.
Instead of this:
assert response.contains("CODE1")
assert response.contains("CODE2")
assert response.contains("CODE3")
assert response.contains("CODE4")
You could have this:
assert response.contains("CODE1") &&
response.contains("CODE2") &&
response.contains("CODE3") &&
response.contains("CODE4") &&
So I'm trying to understand how I can create a unit test that tests a function, which calls other functions. I want to be able to test this function, and override one of the functions that it calls because I don't want it to actually go out and make a call, I want to provide a return_value for that function.
So far I've seen this is mainly doable if you're using a class, but this is an API endpoint and I'm not using a class in this case. Here is the actual endpoint (simplified):
def _get_mock_request(self, queryStringParameters):
return {
"headers": {"x-api-key": "SOME-KEY"},
"requestContext": {
"resourceId": "t89kib",
"authorizer": {
"x-authorizer-key": "SOME KEY",
"principalId": "9de3f415a97e410386dbef146e88744e",
"integrationLatency": 572,
},
},
"pathParameters": {"proxy": "unittest"},
"resource": "/{proxy }",
"httpMethod": "GET",
"queryStringParameters": queryStringParameters,
"body": None,
}
def get(request, response):
"""ApiGateway GET function"""
if validator.valid_raw_data_request(request, response):
handle_query(request, response)
return response
So what I am trying to do, is simply just override the handle_query function, and tell it what return value I want. Here is what I've currently tried (and many different variations/flavors of this):
#mock.patch('requests.get', side_effect=mock_response.requests_get)
#mock.patch('application.v1.logic.common.raw_data_request_handler.handle_query')
#mock.patch.dict(os.environ, {'DEFAULT_ATHENA_MAX_RESULTS': '100'})
def test_get_pass(self, mock_handle_query, mock_get):
mock_handle_query.return_value = raw_fields_json_mock
Except I still see this function being called, and then also going out and making API requests which I don't want. I know this is definitely something I'm not quite understanding about Pytest (fairly new to Python in general, came from Node). But I just don't know what im missing here. Based on everything I've googled, this should work.
I am building an azure durable function which has may activities/azure functions to be called as a part of Job execution.
I have a requirement to view the total execution time taken for completing an orchestration instance. Is there any way to read the total execution time took to run an instance of durable function orchestration?
You can calculate the total execution time for an orchestration by calling the statusQueryGetUri returned when the durable function is first created. The call URI should look like this:
http://<functionappname>.azurewebsites.net/runtime/webhooks/durabletask/instances/<instanceId>?taskHub=<taskHub>&connection=<connectionName>&code=<systemKey>&showHistory=true
The response should look like this:
{
"createdTime": "2018-02-28T05:18:49Z",
"historyEvents": [
...
],
"input": null,
"customStatus": { "nextActions": ["A", "B", "C"], "foo": 2 },
"lastUpdatedTime": "2018-02-28T05:18:54Z",
"output": [
...
],
"runtimeStatus": "Completed"
}
The duration can be determined by polling the status URI until the runtimeStatus reaches any of the terminal states (Failed, Cancelled, Terminated, or Completed), and then subtracting createdTime from lastUpdatedTime.
The following Typescript snippet shows how a the above JSON response (parsed into the status variable) could be processed to show duration as hh:mm:ss:
formatDuration(status: DurableFunctionJob): string {
const diff:number = (new Date(status.lastUpdatedTime).getTime()
- new Date(status.createdTime).getTime());
return new Date(diff).toISOString().substr(11,8);
}
For the example response, this function outputs 00:00:05.
I'm having a hard time tracking down a casting error in Azure Stream Analytics. The input data is coming from an Azure IoT Hub. Here's my query code:
-- Create average data from raw data
WITH
AverageSensorData AS
(
SELECT
[Node],
[SensorType],
udf.round(AVG([Value]), 2) AS [Value],
MAX(TRY_CAST([Timestamp] AS DateTime)) AS [Timestamp]
FROM [SensorData]
WHERE TRY_CAST([Timestamp] AS DateTime) IS NOT NULL
GROUP BY
[Node],
[SensorType],
TumblingWindow(minute, 2)
)
-- Insert average data into PowerBI-Output
SELECT
[Node],
[SensorType]
[Value],
[Timestamp],
DATETIMEFROMPARTS(
DATEPART(yy, [Timestamp]),
DATEPART(mm, [Timestamp]),
DATEPART(dd, [Timestamp]),
0, 0, 0, 0) AS [Date]
INTO [SensorDataAveragePowerBI]
FROM [AverageSensorData]
While this runs fine most of the time (at least for a couple of hundreds or thousands of input entities) it will eventually fail. After having turned on Diagnostic logs I was able to find the following eror message in the corresponding execution log (in reality it was in JSON format, I cleaned it up a little for readability):
Message: Runtime exception occurred while processing events, - Specified cast is not valid. OutputSourceAlias: averagesensordata; Type: SqlRuntimeError, Correlation ID: 49938626-c1a3-4f19-b18d-ee2c5a5332f9
And here's some JSON input that probably caused the error:
[
{
"Date": "2017-04-27T00:00:00.0000000",
"EventEnqueuedUtcTime": "2017-04-27T07:53:52.3900000Z",
"EventProcessedUtcTime": "2017-04-27T07:53:50.6877268Z",
"IoTHub": {
/* Some more data that is not being used */
},
"Node": "IoT_Lab",
"PartitionId": 0,
"SensorType": "temperature",
"Timestamp": "2017-04-27T09:53:50.0000000",
"Value": 21.06
},
{
"Date": "2017-04-27T00:00:00.0000000",
"EventEnqueuedUtcTime": "2017-04-27T07:53:53.6300000Z",
"EventProcessedUtcTime": "2017-04-27T07:53:52.0157515Z",
"IoTHub": {
/* Some more data that is not being used */
},
"Node": "IT_Services",
"PartitionId": 2,
"SensorType": "temperature",
"Timestamp": "2017-04-27T09:53:52.0000000",
"Value": 27.0
}
]
The first entry was the last one to go through, so the second one might have been the one breaking everything. I'm not sure, though, and do not see any suspicious values here. If I upload this as test data within the Azure portal then no errors are being raised.
The query above explicitely uses casting for the [Timestamp] column. But since I'm using TRY_CAST I wouldn't expect any casting errors:
Returns a value cast to the specified data type if the cast succeeds; otherwise, returns null.
As I said, the error only appears once in a while (sometimes after 20 minutes and sometimes after a couple of hours) and cannot be reproduced explicitely. Does anyone have an idea about where the error originates or if there is a chance of getting more detailed error information?
Thanks a lot in advance.
UPDATE: Here's the source of the udf.round function:
function main(value, decimalPlaces) {
if (typeof(value) === 'number'){
var decimalMultiplier = 1;
if (decimalPlaces){
decimalMultiplier = Math.pow(10, decimalPlaces);
}
return Math.round(value * decimalMultiplier) / decimalMultiplier
}
return value;
}
Unfortunately, it's been a while since I wrote this, so I'm not a hundred percent sure why I wrote exactly this code. One thing I do remember, though, is, that I all the analyzed messages always contained a valid number in the respective field. I still think that there's a good chance of this function being responsible for my problem.
my controller returns data like this:
{
"success":true,
"data":{
"35":{
"msg":{
"32":{
"module_id":"35",
"alert_id":"32",
"alert_datetime":"2012-11-28 16:19:19",
"param1_type":"imo",
"param1_value":"453465",
"param2_type":"",
"param2_value":"0",
"param3_type":"",
"param3_value":"0",
"msg":"triiiis dve",
"count":1
},
"33":{
"module_id":"35",
"alert_id":"33",
"alert_datetime":"2012-10-28 00:00:00",
"param1_type":"imo",
"param1_value":"54984",
"param2_type":"",
"param2_value":"0",
"param3_type":"",
"param3_value":"0",
"msg":"triis tri",
"count":1
}
}
},
"42":{
"msg":{
"1":{
"module_id":"42",
"alert_id":"1",
"alert_datetime":"2012-10-28 16:19:19",
"param1_type":"imo",
"param1_value":"9281906",
"param2_type":"",
"param2_value":"0",
"param3_type":"",
"param3_value":"0",
"msg":"",
"count":1
}
}
},
"39":{
"msg":{
"2":{
"module_id":"39",
"alert_id":"2",
"alert_datetime":"2012-10-28 12:36:31",
"param1_type":"imo",
"param1_value":"65464546",
"param2_type":"",
"param2_value":"0",
"param3_type":"",
"param3_value":"0",
"msg":"",
"count":1
}
}
}
}
}
After that I do this
that.tpl.overwrite(that.el, Ext.decode(response).data);
The problem is that I can't loop through the result object keys... I know how to loop through objects with pre-defined key names, but mine are dynamicaly generated...
Will appreciate some help, thanks!
I am assuming you have an idea of the depth of nesting (4 levels below the "data" element in this case):
You could loop through the data with Ext.Object.each (maybe there are some query methods for this too, not sure), looping through each element's children too. In case you use Ext.data.Model instances, you can use the Ext.data.association links to loop through the data.
In that case you could make a different template for each level and insert the result of each template in the template of the level above.
It sounds more difficult than it actually is I think.
foreach in templates is currently indeed only available for support subscribers.