Get real-time distance between running cars on Azure Stream Analytics - azure

We will get streaming data of many cars on a particular Stream Analytics. Each row will have vehicleId, latitude and longitude of vehicle. I need to raise an alarm whenever distance between ANY two cars is less than suppose x meters.
Right now we can consider radial distance to keep it simple. Hence, we need to calculate distance of cars NOT from a fix point but from other cars ( near by cars can keep changing with time ). Hence, we cannot hard-code vehicle id in the query for sure.
We do have Geo-spatial functions support https://learn.microsoft.com/en-us/stream-analytics-query/geospatial-functions ..
I am not sure if this can even be done by Stream Analytics query directly.

I created a small example of the potential solution, not perfect one perhaps, but it resolves the problem in the ASA job.
Essentially I have re-used javascript function that expects simple latitude and longitude and gives the distance in meters. You can use potentially the geospatial embedded function - I haven't tried to play with that.
So, idea is to cross join input, for all input messages(unfortunately yes, you get duplicated result but it works), and then you apply distance function and filter to the output only those that have a distance less than a threshold value. The following example propagates to the output only if the distance is not zero(it means it compared with itself) and if it is less than 5 meters:
with inputData as (select * from input i1 inner join input i2 ON DATEDIFF(second,i1,i2) BETWEEN 0 AND 5),
distances as (select *, udf.getGeoDistance(i1.lat,i1.long,i2.lat,i2.long) as distance from inputData)
select *
into output
from distances
where distance <> 0 and distance < 5
UDF function:
// Sample UDF which returns sum of two values.
function getDistanceFromLatLonInKm(lat1, lon1, lat2, lon2) {
'use strict';
var R = 6371; // Radius of the earth in km
var dLat = deg2rad(lat2 - lat1); // deg2rad below
var dLon = deg2rad(lon2 - lon1);
var a =
Math.sin(dLat / 2) * Math.sin(dLat / 2) +
Math.cos(deg2rad(lat1)) * Math.cos(deg2rad(lat2)) *
Math.sin(dLon / 2) * Math.sin(dLon / 2)
;
var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
var d = R * c * 1000; // Distance in m
return d;
}
function deg2rad(deg) {
return deg * (Math.PI / 180);
}
Input:
[
{
"name" : "car1",
"lat" : 59.3293371,
"long" : 13.4877472
},
{
"name" : "car2",
"lat" : 59.293371,
"long" : 13.2619422
},
{
"name" : "car3",
"lat" : 59.3293371,
"long" : 13.4877040
}
]
And result(car1 and car3 are close to each other):

Follow your latest comment,you use tumbling-window and set 5 seconds timeunit to get slice of data.Per my knowledge, you still could not calculate distance of cars each other by sql and Geo-spatial functions directly.Not to mention a warning.
I came up with an idea that you may could use Azure Function as the output of ASA job.Collect the slice of data and send them into Azure Function as json parameter.Inside function,you could write code to calculate the distances between cars each other,even alert warnings to other destinations.

Related

Polyline encode gets wrong lat/lng after decoding

We are using Google's Polyline decoding algorithm to decode our coordinates. But in our case the most coordinates are wrong after decoding it. We have also tested the process with a deeper precision.
This is our code and also our logs to test that the coordinates are wrong:
let coordinates = [ [lat, lng], [...], ...];
console.log(coordinates[13347]); // Output: [ 13.44668, 52.47429 ]
let encoded = Polyline.encode(coordinates);
let decoded = Polyline.decode(encoded);
console.log(decoded[13347]); // Output: [ 13.44671, 52.47445 ]
console.log(coordinates.length == decoded.length)// true
In this case the distance is 20 meters which is a lot. Other points have distances like 150 meter or even more.
In my coordinates array are around 250.000 coordinates which we want to decode.
Am I missing something so the decode/encode process fails this hard ?
TL;DR
Add the following lines after the declaration of the coordinates variable:
coordinates = coordinates.map(
pair => { return [pair[0].toFixed(5), pair[1].toFixed(5)]; }
);
Full answer
It looks like you're dealing with floating point rounding errors. Probably the library you use has incorrect implementation of the Polyline encoding algorithm.
In the description of the algorithm we read that the encoded string generated by the algorithm stores the differences between consecutive coordinates using fixed-precision numbers (with 5 decimal places). Therefore it is important to round the latitude and longitude to 5 decimal places before computing the differences. Without that step, the rounding errors may accumulate. In the worst case error may increase by about 0.000005 deg for each subsequent item in the encoded list.
The official implementation of the algorithm does not introduce accumulated rounding errors. However, the implementation found in NPM (package polyline) gives incorrect results that indicate the invalid rounding of numbers.
Please look at the examples bellow:
Example 1. Encoding a polyline using official implementation of the algorithm
(using google.maps.geometry.encoding.encodePath from the Google Maps JavaScript API)
originalList = [];
for (var i = 0; i < 100; ++i)
originalList.push(
new google.maps.LatLng(6 * i / 1000000, 0)
);
// originalList looks like: [[0.000000,0],[0.000006,0],[0.000012,0],[0.000018,0], ..., [0.000594,0]];
// (but with LatLng objects instead of 2-element arrays)
console.log(originalList[99].lat()) // 0.000594
var encodedList = google.maps.geometry.encoding.encodePath(originalList)
var decodedList = google.maps.geometry.encoding.decodePath(encodedList)
console.log(decodedList[99].lat()) // 0.00059
Example 2. Encoding a polyline using package polyline from NPM
let Polyline = require('polyline');
var originalList = [];
for (var i = 0; i < 100; ++i)
originalList.push(
[6 * i / 1000000, 0]
);
// again: originalList == [[0.000000,0],[0.000006,0],[0.000012,0],[0.000018,0], ..., [0.000594,0]];
console.log(originalList[99][0]) // 0.000594
var encodedList = Polyline.encode(originalList);
var decodedList = Polyline.decode(encodedList);
console.log(decodedList[99][0]) // 0.00099
Invalid result: the values 0.000594 and 0.00099 differ by more than 0.000005.
Possible fix
The library that you're using probably doesn't round the coordinates before computing the differences.
For example when two consecutive points have latitudes 0.000000 and 0.000006, the difference is 0.000006 and it is rounded to 0.00001 giving error of 0.000004.
You may want to round the coordinates manually, before passing them to Polyline.encode(), eg. using the function .toFixed(5):
let Polyline = require('polyline');
var originalList = [];
for (var i = 0; i < 100; ++i)
originalList.push(
[(6 * i / 1000000).toFixed(5), 0]
);
// before rounding: [[ 0.000000,0],[ 0.000006,0],[ 0.000012,0],[ 0.000018,0], ..., [ 0.000594,0]];
// after rounding: [['0.00000',0],['0.00001',0],['0.00001',0],['0.00002',0], ..., ['0.00059',0]];
console.log(originalList[99][0]) // 0.00059
var encodedList = Polyline.encode(originalList);
var decodedList = Polyline.decode(encodedList);
console.log(decodedList[99][0]) // 0.00059
Polyline encoding is lossy:
https://developers.google.com/maps/documentation/utilities/polylinealgorithm (Polyline encoding is a *lossy* compression algorithm that allows you to store a series of coordinates as a single string
How about using your own encoding scheme? The page above also shows the encoding scheme used by Google. Perhaps you can look for a trade-off between space and accuracy.

Billing calculation of rates

Azure Rate Card API returns a MeterRates field (see documentation).
Azure UsageAggregate gives a quantity (see documentation).
According to the azure support page. This is the forum to ask questions.
So, how are meter rates applied?
Example meter rates:
{"0":20, "100":15, "200":10}
If I have a quantity of 175 is the amount 100*20 + 75*15 or 175*15?
Why specify an included quantity?
Example: rates:{"0":23} with included quantitiy 10 could be expressed as rates:
{"0":0, "10":23}
example meter rates: {"0":20, "100":15, "200":10}
if I have a quantity of 175 is the amount 100*20 + 75*15 or 175*15 ?
The pricing is tiered pricing. So when you get the rates it essentially tells you that:
from 0 - 99 units, the unit rate is 20
from 100 - 199 units, the unit rate is 15
from 200 units and above, the unit rate is 10
Based on this logic, your calculation should be:
99 * 20 + 75 * 15 = 3105
One thing which confuses me though is the upper limit. Above calculation is based on the information I received from Azure Billing team. What confused me is what would happen if the consumption is say 99.5 units? For the first 99 units it is fine but I am not sure how the additional 0.5 units will be calculated.
Guarav gets at the core of the issue and is why I marked it as the answer. Based on that I devised the following code to implement the logic. It falls in two parts:
Creating a bucket list from the meter rates
Processing a quantity with the bucket list to determine an amount
The following function creates a list of buckets (each bucket object is a simple POCO with Min, Max and Rate properties). The list is attached to a meter object that has the other properties from the rate card api.
private Dictionary<int, RateBucket> ParseRateBuckets(string rates)
{
dynamic dRates = JsonConvert.DeserializeObject(rates);
var rateContainer = (JContainer)dRates;
var buckets = new Dictionary<int, RateBucket>();
var bucketNumber = 0;
foreach (var jToken in rateContainer.Children())
{
var jProperty = jToken as JProperty;
if (jProperty != null)
{
var bucket = new RateBucket
{
Min = Convert.ToDouble(jProperty.Name),
Rate = Convert.ToDouble(jProperty.Value.ToString())
};
if (bucketNumber > 0)
buckets[bucketNumber - 1].Max = bucket.Min;
buckets.Add(bucketNumber, bucket);
}
bucketNumber++;
}
return buckets;
}
The second function uses the meter object with two useful properties: the bucket list, and the included quantity. According to the rate card documentation (as I read it) you don't start counting billable quantity until after you surpass the included quantity. I'm sure there's some refactoring that could be done here, but the ordered processing of the buckets is the key point.
I think I've addressed issue on the quantity by recognizing that it's a double and not an integer. Therefore the quantity associated with any single bucket is the difference between the bucket max and the bucket min (unless we've only filled a partial bucket).
private double CalculateUsageCost(RateCardMeter meter, double quantity)
{
var amount = 0.0;
quantity -= meter.IncludedQuantity;
if (quantity > 0)
{
for (var i = 0; i < meter.RateBuckets.Count; i++)
{
var bucket = meter.RateBuckets[i];
if (quantity > bucket.Min)
{
if (bucket.Max.HasValue && quantity > bucket.Max)
amount += (bucket.Max.Value - bucket.Min)*bucket.Rate;
else
amount += (quantity - bucket.Min)*bucket.Rate;
}
}
}
return amount;
}
Finally, the documentation is unclear about the time scope for the tiers. If I get a discounted price based on quantity, over what time scope do I aggregate quantity? The usage api allows me to pull data either daily or hourly. I want to pull my data hourly so I can correlate my costs by time of day. But when is it appropriate to actually calculate the bill? Seems like hourly is wrong, daily may work, but it might only be appropriate over the entire month.
Recently I just did this similar task. Following is my example (I think you can use regex to remove those characters rather than like me using replace). The first function parse the rate info string to generate a key-value pair collection, and the second function is used to calculate the total price.
private Dictionary<float, double> GetRatesDetail(string r)
{
Dictionary<float, double> pairs = null;
if(string.IsNullOrEmpty(r) || r.Length <=2)
{
pairs = new Dictionary<float, double>();
pairs.Add(0, 0);
}
else
{
pairs = r.Replace("{", "").Replace("}", "").Split(',')
.Select(value => value.Split(':'))
.ToDictionary(pair => float.Parse(pair[0].Replace("\"", "")), pair => double.Parse(pair[1]));
}
return pairs;
}
public decimal Process(Dictionary<float, double> rates, decimal quantity)
{
double ret = 0;
foreach (int key in rates.Keys.OrderByDescending(k => k))
{
if (quantity >= key)
{
ret += ((double)quantity - key) * rates[key];
quantity = key;
}
}
return (decimal)ret;
}

Moving from SQLite to Core Data

I'm moving my SQLite database over to Core Data. My database table looks like this:
CREATE TABLE IF NOT EXISTS stops (id integer primary key autoincrement, type text, lat real, lon real, stop_id integer unique, stop_code integer, title text, subtitle text, url text, lastupdate text
My Entity looks like this:
I'm not worries about moving the data over, it's really just a local cache that gets updated from time to time. If its empty, it will just repopulate.
My issue is I have a SQLite custom function:
static void distanceFunc(sqlite3_context *context, int argc, sqlite3_value **argv)
{
// check that we have four arguments (lat1, lon1, lat2, lon2)
assert(argc == 4);
// check that all four arguments are non-null
if (sqlite3_value_type(argv[0]) == SQLITE_NULL || sqlite3_value_type(argv[1]) == SQLITE_NULL || sqlite3_value_type(argv[2]) == SQLITE_NULL || sqlite3_value_type(argv[3]) == SQLITE_NULL) {
sqlite3_result_null(context);
return;
}
// get the four argument values
double lat1 = sqlite3_value_double(argv[0]);
double lon1 = sqlite3_value_double(argv[1]);
double lat2 = sqlite3_value_double(argv[2]);
double lon2 = sqlite3_value_double(argv[3]);
// convert lat1 and lat2 into radians now, to avoid doing it twice below
double lat1rad = DEG2RAD(lat1);
double lat2rad = DEG2RAD(lat2);
// apply the spherical law of cosines to our latitudes and longitudes, and set the result appropriately
// 6378.1 is the approximate radius of the earth in kilometres
sqlite3_result_double(context, acos(sin(lat1rad) * sin(lat2rad) + cos(lat1rad) * cos(lat2rad) * cos(DEG2RAD(lon2) - DEG2RAD(lon1))) * 6378.1);
}
The function given a 2 latitudes and 2 longitudes would return the distance. This would let me do something like:
SELECT *, distance(lat, lon, %f, %f) as dist FROM stops WHERE dist < 1 ORDER BY dist
Now, I've got all my data in Core Data, but no idea how to do something like the SQL above with a NSFetchRequest. So how would I go about fetching entities in this manner?

Spectrogram - Calculating is wrong

Ok, so basically, I am implementing the following algorithm:
1) Slice signal of size 256 with an overlap of 128
2) Multiply each chunk with the Hanning window
3) Get DFT
4) Compute the abs value sqrt(re*re+im*im)
Plotting these values, as a imshow I get the following result:
This looks ok, it's clearly showing some difference, i.e. the spike where the signal has most amplitude shows. However, in Python I get this result:
I know that I'm doing something right, but, also doing something wrong. I just can't seem to find out where which is making me not think I have done it correctly.
Any rough ideas to where I could be going wrong here? I mean, is plotting the abs value the right way here or not?
Thanks
EDIT:
Result after clamping..
UPDATE:
Code:
for(unsigned j=0; (j < stft_temp[i].size()/2); j++)
{
double v = 10 * log10(stft_temp[i][j].re * stft_temp[i][j].re + stft_temp[i][j].im * stft_temp[i][j].im);
double pixe = 1.5 * (v + 100);
STFT[i][j] = (int) pixe;
}
Typically you might want to use a log magnitude and then scale to the required range, which would usually be 0..255. In pseudo-code:
mag_dB = 10 * log10(re * re + im * im); // get log magnitude (dB)
pixel_intensity = 1.5 * (mag_dB + 100); // offset and scale
pixel_intensity = min(pixel_intensity, 255); // clamp to 0..255
pixel_intensity = max(pixel_intensity, 0);

Fuzzy logic on price calculation

I am having one price calculation code in C#.There is particular sequence of calculating all the prices and taxes.how to apply fuzzy logic so that I will get related search of that particular price.That means if one of tax name is 'cost/kg'.So I should get all the names of taxes or prices which start from C or CO.
A simple way would be getting prices within some min/max bounds of what has been searched
double k = 0.1;
double min = searchedAmount - (searchedAmount * k);
double max = searchedAmount + (searchedAmount * k);
var results = products.Where(p => p.Price >= min && p.Price <= max);

Resources