I'm running python v3 function app and it contains multiple functions with different bindings(cosmos, blob, http etc). I'm trying to get the details of this function app in application insights like no of request, exception raised during execution or number of request per function app and per function etc.
I'm able to run and get few details like request count. Now I'm trying to map request details with other tables like exceptions, request etc but not able to map and drill down to the particular function.
For e.g Let suppose I have 10 function in function app and they run one after another based on output of previous function. Let say in any case flow got failed at any function. Now I want at which step/function my function app failed, details of error, successful and unsuccessful flow completion of function app
Below are the some query I have used for monitoring purpose.
Request on first function to get the total number of request counts for function app.
requests
| where timestamp > ago(1d)
| where operation_Name =~ "function name"
| summarize RequestsCount=sum(itemCount) by cloud_RoleName,bin(timestamp,1d)
Request and Average Duration of functions
requests
| summarize RequestsCount=sum(itemCount), AverageDuration=avg(duration) by operation_Name
| order by RequestsCount desc
You can check the exception per function with:
exceptions
| extend OperationName = iff(operation_Name == "","[No operation name]",operation_Name)
| summarize Count = count() by cloud_RoleName, OperationName, type, method
To join with requests:
requests
| where timestamp > ago(24h) and success == false
| join kind= inner (
exceptions
| where timestamp > ago(24h)
) on operation_Id
| project exceptionType = type, failedMethod = method, requestName = name, requestDuration = duration, success
Keep it mind, if you catch an error yourself, the result of the function will be success.
You could also work with custom error logs in your functions where you maybe create a json object which will end up in the message column of the traces table. You can query further then:
traces
| where message contains "the error i am searching for"
| extend json = parse_json(message)
| project
timestamp,
errorSource = json.error_source,
step = json.step,
errors = json.errors,
url = json.url
Related
these are the docs
https://docs.chain.link/docs/jobs/types/cron/
type = "cron"
schemaVersion = 1
schedule = "CRON_TZ=UTC * */20 * * * *"
externalJobID = "0EEC7E1D-D0D2-476C-A1A8-72DFB6633F01"
observationSource = """
fetch [type="http" method=GET url="https://chain.link/ETH-USD"]
parse [type="jsonparse" path="data,price"]
multiply [type="multiply" times=100]
fetch -> parse -> multiply
"""
But what I am wondering is how the job connects to the Oracle contract. How does it connect with the user contract to get paid. Where and how do we send the data once job is complete at specified increment.
Does the job start when the job is posted on the node side. Or does it start the clock once a user contract calls it?
Any help would be much appreciated. I trying to run through the types of jobs to familarize myself with the capbilities of a chainlink node.
A Chron job executes a job based on a chron defined schedule. This means it's triggered based on some condition that the Chainlink node evaluates, and not triggered externally via a smart contract. Because of this, the node isn't paid in LINK tokens for processing the request like it does for API calls, because it's initiating a request itself as opposed to receiving a request (and payment) from on-chain. A node can't initiate a job on its own and then expect to receive payment from a consuming contract, if you require such functionality then you can try to have some logic in the function called to withdraw some LINK. But be careful of who can call this function.
If you want to send data on-chain to a smart contract once the job is completed, you need to manually define an ethtx task at the end of the cron job.
Here's an extended version of your job that sends the result back to a function called someFunction at the contract deployed at address 0xa36085F69e2889c224210F603D836748e7dC0088. The data can be in any format, as long as the abi of the function matches what's being encoded in the job. ie if the function expects a bytes param, you need to ensure your encoding a bytes param, if it expects a uint param, then you need to encode a uint param. In this example, a bytes parameter is used
type = "cron"
schemaVersion = 1
name = "GET > bytes32 (cron)"
schedule = "CRON_TZ=UTC #every 1m"
observationSource = """
fetch [type="http" method=GET url="https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=USD"]
parse [type="jsonparse" path="USD"]
multiply [type="multiply" times=100]
encode_response [type="ethabiencode"
abi="(uint256 data)"
data="{\\"data\\": $(multiply) }"]
encode_tx [type="ethabiencode"
abi="someFunction(bytes32 data)"
data="{ \\"data\\": $(encode_response) }"]
submit_tx [type="ethtx"
to="0x6495C9684Cc5702522A87adFd29517857FC99f45"
data="$(encode_tx)"]
fetch -> parse -> multiply -> encode_response -> encode_tx -> submit_tx
"""
And here's the consuming contract for the cron job above:
// SPDX-Lincense-Identifier: MIT
pragma solidity ^0.8.7;
contract Cron {
bytes32 public currentPrice;
function someFunction(bytes32 _price) public {
currentPrice = _price;
}
}
To answer your other question, the is active as soon as it's created on the node, and will start evaluating the triggering conditions for it to active a run, based on the schedule defined
I’m new to python and I need to get connected to “Kibana” via python. we’re using Kibana 7.4.1. The requirement is to get them just the count (hits).
Due to some restrictions, I need to use Python 3.6 only. I’ve added the “ElasticSearch” & “ElasticSearch-dsl” library.
I’m able to get connected to the Kibana via the client, but I’m getting the wrong hits count.
Code:
from elasticsearch import Elasticsearch
from elasticsearch_dsl import MultiSearch, Search
from elasticsearch_dsl.query import QueryString, Range, SimpleQueryString
es = Elasticsearch(['host2', 'host2'], http_auth=('usr', 'pass'), port=9200)
s = Search(using=es, index='c*')
s.filter(SimpleQueryString(query="tags:prod AND severity:INFO AND service: finder AND msg:* is processed"))
s.filter(Range(** {'#timestamp': {'gte': 'now-5m', 'lt': 'now'}}))
response = s.execute()
print("Got %d Hits:" % response['hits']['total']['value']) # Always coming as 1000 so this is wrong
Can I get some help with this, please?
First of all a little clarification. You are connecting to Elasticsearch and not Kibana (Kibana is a client, like the program you are writing).
You are receiving always 10000 as result, because your index has more than 10000 hits. It is a documented feature. Indeed, since the count computation is expensive in the general case it is performed only when needed. In order to obtain the right number of results you have two possibilities
to set the query parameter track_total_hits to true
use the count API.
track_total_hits
You can add this extra parameter to the search object as reported here as follows:
s = Search(using=es, index='c*')
s = s.extra(track_total_hits=True)
<the-rest of your code>
Count API approach
Instead of invoking the execute() function, you can simply use the count() function:
s = Search(using=es, index='c*')
s.filter(SimpleQueryString(query="tags:prod AND severity:INFO AND service: finder AND msg:* is processed"))
s.filter(Range(** {'#timestamp': {'gte': 'now-5m', 'lt': 'now'}}))
response = s.cpunt()
print("Got %d Hits:" % response)
Kind regards
I have been trying to set up alerts of a .NET Core App Service hosted in Azure to fire an event if X% of the requests are failing in the past 24 hours. I have also tried setting up an alert from the Service's AppInsights resource using the following metrics: Exception rate, Server exceptions, or Failed request.
However, none of these have the ability to capture a % (failure rate), all of them are using count as a metric.
Does anyone know a workaround for this?
Please try the query-based alert:
1.Go to application insights analytics, in the query editor, input below scripts:
exceptions
| where timestamp >ago(24h)
| summarize exceptionsCount = sum(itemCount) | extend t = ""| join
(requests
| where timestamp >ago(24h)
| summarize requestsCount = sum(itemCount) | extend t = "") on t
| project isFail = 1.0 * exceptionsCount / requestsCount > 0.5 // if fail rate is greater than 50%, fail
| project rr = iff(isFail, "Fail", "Pass")
| where rr == "Fail"
2.Then click the "New alert rule" on the upper right corner:
3.In the Create rule page, set as following:
I was looking for a way to avoid writing queries using something that is already built-in in app insights but in the end i also came up with something like yours solution using the requests instead:
requests
| summarize count()
| extend a = "a"
| join
(
requests
| summarize count() by resultCode
| extend a = "a"
)
on a
| extend percentage = (todouble(count_1)*100/todouble(count_))
| where resultCode == 200
| where percentage < 90 //percentage of success is less than 90%
| project percentage_of_failures = round(100- percentage,2), total_successful_req = count_, total_failing_req = count_ - count_1 , total_req = count_1
I'd like to use Azure Log Analytics to create a monitoring alert for possible brute-force attempts on my users' accounts. That is to say, I'd like to be notified by Azure (or, at the very least, be able to manually run the script to obtain the data) when a user's account is successfully authenticated into O365 following a number of failed attempts.
I know how to parse the logs to, for example, obtain the number of unsuccessful sign-in attempts by all users during a defined period (see the example below):
SigninLogs
| where TimeGenerated between(datetime("2018-11-19 00:00:00") .. datetime("2018-11-19 23:59:59"))
| where ResultType == "50074"
| summarize FailedSigninCount = count() by UserDisplayName
| sort by FailedSigninCount desc
But I don't know how to script the following:
A user has created 9 unsuccessful sign-in attempts (type 50074) and
created a successful sign-in attempt.
Within a 60-second period.
Any help would be gratefully received.
Check out the Azure Sentinel community GitHub and see if the queries there help. Specifically I added https://github.com/Azure/Azure-Sentinel/blob/master/Detections/SigninLogs/SigninBruteForce-AzurePortal.txt which I think more or less does what you are after - also repasted below. Hope that helps.
// Evidence of Azure Portal brute force attack in SigninLogs:
// This query returns results if there are more than 5 authentication failures and a successful authentication
// within a 20-minute window.
let failureCountThreshold = 5;
let successCountThreshold = 1;
let timeRange = ago(1d);
let authenticationWindow = 20m;
SigninLogs
| where TimeGenerated >= timeRange
| extend OS = DeviceDetail.operatingSystem, Browser = DeviceDetail.browser
| extend StatusCode = tostring(Status.errorCode), StatusDetails = tostring(Status.additionalDetails)
| extend State = tostring(LocationDetails.state), City = tostring(LocationDetails.city)
| where AppDisplayName contains "Azure Portal"
// Split out failure versus non-failure types
| extend FailureOrSuccess = iff(ResultType in ("0", "50125", "50140"), "Success", "Failure")
| summarize StartTimeUtc = min(TimeGenerated), EndTimeUtc = max(TimeGenerated),
makeset(IPAddress), makeset(OS), makeset(Browser), makeset(City), makeset(ResultType),
FailureCount=countif(FailureOrSuccess=="Failure"),
SuccessCount = countif(FailureOrSuccess=="Success")
by bin(TimeGenerated, authenticationWindow), UserDisplayName, UserPrincipalName, AppDisplayName
| where FailureCount>=failureCountThreshold and SuccessCount>=successCountThreshold
Scenario
I want to send data to an MQTT Broker (Cloud) by querying measurements from InfluxDB.
I have a field in the schema which is called status. It can either be 1 or 0. status=0 indicated that series has not been sent to the cloud. If I get an acknowlegdment from the MQTT Broker then I wish to rewrite the query back into the database with status=1.
As mentioned in FAQs for InfluxDB regarding Duplicate data If the information has the same timestamp as the previous query but with a different field value => then the update field will be shown.
In order to test this I created the following:
CREATE DATABASE dummy
USE dummy
INSERT meas_1, type=t1, status=0,value=123 1536157064275338300
query:
SELECT * FROM meas_1
provides
time status type value
1536157064275338300 0 t1 234
now if I want to overwrite the series I do the following:
INSERT meas_1, type=t1, status=1,value=123 1536157064275338300
which will overwrite the series
time status type value
1536157064275338300 1 t1 234
(Note: this is not possible via Tags currently in InfluxDB)
Usage
Query some information using the client with "status"=0.
Restructure JSON to be sent to the cloud
Send the information to cloud
If successful then write the output from Step 1. back into the DB but with status=1.
I am using the InfluxDBClient Python3 to create the Application (MQTT + InfluxDB)
Within the write_points API there is a parameter which mentions batch_size which require int as input.
I am not sure how can I use this with the Application that I want. Can someone guide me with this or with the Schema of the DB so that I can upload actual and non-redundant information to the cloud ?
The batch_size is actually the length of the list of the measurements that needs to passed to write_points.
Steps
Create client and query from measurement (here, we query gps information)
client = InfluxDBClient(database='dummy')
op = client.query('SELECT * FROM gps WHERE "status"=0', epoch='ns')
Make the ResultSet into a list:
batch = list(op.get_points('gps'))
create an empty list for update
updated_batch = []
parse through each measurement and change the status flag to 1. Note, default values in InfluxDB are float
for each in batch:
new_mes = {
'measurement': 'gps',
'tags': {
'type': 'gps'
},
'time': each['time'],
'fields': {
'lat': float(each['lat']),
'lon': float(each['lon']),
'alt': float(each['alt']),
'status': float(1)
}
}
updated_batch.append(new_mes)
Finally dump the points back via the client with batch_size as the length of the updated_batch
client.write_points(updated_batch, batch_size=len(updated_batch))
This overwrites the series because it contains the same timestamps with status field set to 1