Transforms to ArcGIS API geojson output and geopandas series/dataframe - python-3.x

The following GET yields a result you can view via Postman and/or geojson.io (NOTE: you need cookies, which are pasted at the bottom of this post):
url_geojson = "https://services6.arcgis.com/GklOjOaok2jR6aKf/ArcGIS/rest/services/NM_OG_ROWs_Linear_031417/FeatureServer/0/query?f=geojson&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&orderByFields=FID%20ASC&outSR=102100&resultOffset=0&resultRecordCount=4000&cacheHint=true&quantizationParameters=%7B%22mode%22%3A%22edit%22%7D"
In the following GET call, another GIS API call yields just a json response with additional info in regards to the transformations that need to be applied to the geometry objects generated from the geojson call:
url_json = "https://services6.arcgis.com/GklOjOaok2jR6aKf/ArcGIS/rest/services/NM_OG_ROWs_Linear_031417/FeatureServer/0/query?f=json&where=1%3D1&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*&orderByFields=FID%20ASC&outSR=102100&resultOffset=0&resultRecordCount=4000&cacheHint=true&quantizationParameters=%7B%22mode%22%3A%22edit%22%7D"
Note that only difference is the f parameter (json and geojson).
The output of the json request has a section with the following output:
{
"objectIdFieldName": "FID",
"uniqueIdField": {
"name": "FID",
"isSystemMaintained": true
},
"globalIdFieldName": "",
"geometryProperties": {
"shapeLengthFieldName": "Shape__Length",
"units": "esriMeters"
},
"geometryType": "esriGeometryPolyline",
"spatialReference": {
"wkid": 102100,
"latestWkid": 3857
},
"transform": {
"originPosition": "upperLeft",
"scale": [
0.0001,
0.0001,
0,
0
],
"translate": [
-20037700,
-30241100,
0,
0
]
},...
I assume these are the parameters that I need to use to change the output coordinates of the geojson request, which are (as a single example) this:
{
"type": "FeatureCollection",
"crs": {
"type": "name",
"properties": {
"name": "EPSG:3857"
}
},
"properties": {
"exceededTransferLimit": true
},
"features": [
{
"type": "Feature",
"id": 1,
"geometry": {
"type": "LineString",
"coordinates": [
[
-11533842.1198518,
3857288.84408179
],
[
-11534147.0371623,
3857067.64072161
]
]
},...
I have managed to take the geojson output and assign to a pandas dataframe and apply the scaling transform with the following command:
main_gdf_clean['geometry'] = GeoSeries.scale(main_gdf_clean['geometry'],
xfact=0.00001, yfact=0.00001,
origin=(0,0))
At this point, I'm lost on how to apply - as listed above in the json output - the translation parameters. I've tried the following command, and it yields wildly incorrect result to geometry objects:
GeoSeries.translate(main_gdf_clean['geometry'], xoff=-20037700.0, yoff=-30241100.0)
Based on what I've presented here, can someone suggest a way to apply the translation transform properly or go about this process in a different way?
header = {'Cookie': 'ASP.NET_SessionId=umzghudgzvz22wpo3a0bgeoq; OCDUserPreference=AAEAAAD/////AQAAAAAAAAAMAgAAAEtFTU5SRC5PQ0QuUGVybWl0dGluZywgVmVyc2lvbj0xLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPW51bGwFAQAAACVOTUVNTlJELk9DRC5QZXJtaXR0aW5nLlVzZXJQcmVmZXJlbmNlCQAAAAhQYWdlU2l6ZRJXZWxsU29ydFBhcmFtZXRlcnMWRmFjaWxpdHlTb3J0UGFyYW1ldGVycxxGZWVBcHBsaWNhdGlvblNvcnRQYXJhbWV0ZXJzFkluY2lkZW50U29ydFBhcmFtZXRlcnMRUGl0U29ydFBhcmFtZXRlcnMSVGFua1NvcnRQYXJhbWV0ZXJzGkdlbmVyYXRlZEFwaVNvcnRQYXJhbWV0ZXJzFk9wZXJhdG9yU29ydFBhcmFtZXRlcnMABAQEBAQEBAQIKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAKE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlckxpc3QCAAAAAgAAAGQAAAAJAwAAAAkEAAAACQUAAAAJBgAAAAkHAAAACQgAAAAJCQAAAAkKAAAABQMAAAAoTk1FTU5SRC5PQ0QuUGVybWl0dGluZy5Tb3J0UGFyYW1ldGVyTGlzdAMAAAANTGlzdGAxK19pdGVtcwxMaXN0YDErX3NpemUPTGlzdGAxK192ZXJzaW9uBAAAJk5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlcltdAgAAAAgIAgAAAAkLAAAAAgAAAAIAAAABBAAAAAMAAAAJDAAAAAIAAAACAAAAAQUAAAADAAAACQ0AAAABAAAAAQAAAAEGAAAAAwAAAAkOAAAAAgAAAAIAAAABBwAAAAMAAAAJDwAAAAIAAAACAAAAAQgAAAADAAAACRAAAAACAAAAAgAAAAEJAAAAAwAAAAkRAAAAAwAAAAMAAAABCgAAAAMAAAAJEgAAAAEAAAABAAAABwsAAAAAAQAAAAQAAAAEJE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlcgIAAAAJEwAAAAkUAAAADQIHDAAAAAABAAAABAAAAAQkTk1FTU5SRC5PQ0QuUGVybWl0dGluZy5Tb3J0UGFyYW1ldGVyAgAAAAkVAAAACRYAAAANAgcNAAAAAAEAAAAEAAAABCROTUVNTlJELk9DRC5QZXJtaXR0aW5nLlNvcnRQYXJhbWV0ZXICAAAACRcAAAANAwcOAAAAAAEAAAAEAAAABCROTUVNTlJELk9DRC5QZXJtaXR0aW5nLlNvcnRQYXJhbWV0ZXICAAAACRgAAAAJGQAAAA0CBw8AAAAAAQAAAAQAAAAEJE5NRU1OUkQuT0NELlBlcm1pdHRpbmcuU29ydFBhcmFtZXRlcgIAAAAJGgAAAAkbAAAADQIHEAAAAAABAAAABAAAAAQkTk1FTU5SRC5PQ0QuUGVybWl0dGluZy5Tb3J0UGFyYW1ldGVyAgAAAAkcAAAACR0AAAANAgcRAAAAAAEAAAAEAAAABCROTUVNTlJELk9DRC5QZXJtaXR0aW5nLlNvcnRQYXJhbWV0ZXICAAAACR4AAAAJHwAAAAkgAAAACgcSAAAAAAEAAAAEAAAABCROTUVNTlJELk9DRC5QZXJtaXR0aW5nLlNvcnRQYXJhbWV0ZXICAAAACSEAAAANAwwiAAAATVN5c3RlbS5XZWIsIFZlcnNpb249NC4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj1iMDNmNWY3ZjExZDUwYTNhBRMAAAAkTk1FTU5SRC5PQ0QuUGVybWl0dGluZy5Tb3J0UGFyYW1ldGVyAwAAAAVfbmFtZQtfZXhwcmVzc2lvbgpfZGlyZWN0aW9uAQEEJ1N5c3RlbS5XZWIuVUkuV2ViQ29udHJvbHMuU29ydERpcmVjdGlvbiIAAAACAAAABiMAAAANV2VsbCBPcGVyYXRvcgYkAAAACm9ncmlkX25hbWUF2////ydTeXN0ZW0uV2ViLlVJLldlYkNvbnRyb2xzLlNvcnREaXJlY3Rpb24BAAAAB3ZhbHVlX18ACCIAAAAAAAAAARQAAAATAAAABiYAAAAJV2VsbCBOYW1lBicAAAAId2VsbG5hbWUB2P///9v///8AAAAAARUAAAATAAAABikAAAAFT2dyaWQGKgAAAAVvZ3JpZAHV////2////wAAAAABFgAAABMAAAAGLAAAAAtGYWNpbGl0eSBJZAYtAAAAAmlkAdL////b////AQAAAAEXAAAAEwAAAAYvAAAACkNyZWF0ZWQgT24GMAAAAAljcmVhdGVkT24Bz////9v///8AAAAAARgAAAATAAAACSkAAAAJKgAAAAHM////2////wAAAAABGQAAABMAAAAGNQAAAAtJbmNpZGVudCBJZAktAAAAAcn////b////AQAAAAEaAAAAEwAAAAkpAAAACSoAAAABxv///9v///8AAAAAARsAAAATAAAABjsAAAAGUGl0IElkBjwAAAAGcGl0X2lkAcP////b////AQAAAAEcAAAAEwAAAAkpAAAACSoAAAABwP///9v///8AAAAAAR0AAAATAAAABkEAAAAHVGFuayBJZAZCAAAAB3RhbmtfaWQBvf///9v///8BAAAAAR4AAAATAAAACSMAAAAJJAAAAAG6////2////wAAAAABHwAAABMAAAAJJgAAAAZIAAAADXByb3BlcnR5X25hbWUBt////9v///8AAAAAASAAAAATAAAABkoAAAALV2VsbCBOdW1iZXIGSwAAAAt3ZWxsX251bWJlcgG0////2////wAAAAABIQAAABMAAAAGTQAAAA1PcGVyYXRvciBOYW1lCSQAAAABsf///9v///8AAAAACw==',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
}

Related

Is there any way to map two different types (string and array of string) in a single field in C#?

First json data coming from request body.
"Operations": [
{
"op": "Replace",
"path": "displayName",
"value": "AZGrp333"
}
]
Second request body
"Operations": [
{
"op": "Remove",
"path": "members",
"value": [
{
"value": "bb05229b-7813-4f43-bcf9-f7430b5639a8"
}
]
}
]
In the value field, I'm getting 2 different types of data (string and array of string). In POST request one is getting de-serialized properly and one is not (showing Bad Request). Is it possible to handle both request using single DTO?
I can't use Newtonsoft JsonConverter as this is not working with DataContractSerializer.

Can I index an array in a composite index in Azure Cosmos DB?

I have a problem indexing an array in Azure Cosmos DB
I am trying to save this indexing policy via the portal
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*"
}
],
"excludedPaths": [
{
"path": "/\"_etag\"/?"
}
],
"compositeIndexes": [
[
{
"path": "/DeviceId",
"order": "ascending"
},
{
"path": "/TimeStamp",
"order": "ascending"
},
{
"path": "/Items/[]/Name/?",
"order": "ascending"
},
{
"path": "/Items/[]/DoubleValue/?",
"order": "ascending"
}
]
]
}
I get the error "Failed to update container DeviceEvents:
Message: {"code":"BadRequest","message":"Message: {"Errors":["The indexing path '\/Items\/[]\/Name\/?' could not be accepted, failed near position '8'."
This seems to be the array [] syntax that is giving an error.
On a side note I am not sure what I am doing makes sense at all but I have a query that looks like this
SELECT SUM(de0["DoubleValue"])
FROM root JOIN de0 IN root["Items"]
WHERE root["ApplicationId"] = 57 AND root["DeviceId"] = 126 AND root["TimeStamp"] >= "2021-02-21T17:55:29.7389397Z" AND de0["Name"] = "Use Case"
Where ApplicationId is the partition key and the item saved looks like this
{
"id": "59ab9323-26ca-436f-8d29-e1ddd826f025",
"DeviceId": 3,
"ApplicationId": 3,
"RawData": "640F7A000A00E30142000000",
"TimeStamp": "2021-02-20T18:36:52.833174Z",
"Items": [
{
"Name": "Battery Status",
"StringValue": "Full",
"DoubleValue": null
},
{
"Name": "Use Case",
"StringValue": null,
"DoubleValue": 12
},
{
"Name": "Battery Voltage",
"StringValue": null,
"DoubleValue": 3.962
},
{
"Name": "Rain Gauge Count",
"StringValue": null,
"DoubleValue": 10
}
],
"_rid": "CgdVAO7B0DNkAAAAAAAAAA==",
"_self": "dbs/CgdVAA==/colls/CgdVAO7B0DM=/docs/CgdVAO7B0DNkAAAAAAAAAA==/",
"_etag": "\"61008771-0000-0d00-0000-603156c50000\"",
"_attachments": "attachments/",
"_ts": 1613846213
}
I need to aggregate on some of these items in the array like say get MAX on temperature or something like this (using Use Case for test although it doesn't make sense). I reasoned that if all the data in the query is in a single composite index the database would be able to do the aggregation without reading the documents themselves. However I can't seem to add a composite index containing an array at all.
Yes, composite index can't contain an array path. It should be a scalar value.
Unlike with included or excluded paths, you can't create a path with
the /* wildcard. Every composite path has an implicit /? at the end of
the path that you don't need to specify. Composite paths lead to a
scalar value and this is the only value that is included in the
composite index.
Reference:https://learn.microsoft.com/en-us/azure/cosmos-db/index-policy#composite-indexes

Influxdb python 5.3.1 write_points not allowing multiple rows to be written

I am new to using influxdb (v1.7.4), and I am using its python module (influxdb-python).
I want to write data in bulk into influxdb.
But unable to get the correct output when using the write_points method.
point = [
{
"fields": {
"PATH": "/",
"DISK_USED_PERCENT": "10"
},
"measurement": "xxxxxxx"
},
{
"fields": {
"PATH": "/xxxxxxxxx",
"DISK_USED_PERCENT": "0"
},
"measurement": "xxxxxxx"
}
]
client.write_points(point)
I want to add multiple points in a go. As per the documentation, it can be done by providing a list of dictionaries.
But the result I'm seeing is only taking the last item from the list and creating a ResultSet with it.
ResultSet({'('xxxxxxx', None)': [{'time': '2021-05-10T09:38:20.818555Z', 'DISK_USED_PERCENT': '0', 'PATH': '/xxxxxxxxx'}]})
Any leads would be appreciated.

Azure Identity Protection - Risk Detection API - Filter by date

I am trying to filter the RiskDetection data retrieved from Azure Identity Protection by date and so far no success.
For the below sample data below filter by activityDateTime (or any date fields in the sample data) show internal error in the response:
https://graph.microsoft.com/beta/riskDetections?$filter=activityDateTime ge 2020-02-05
{'error': {'code': 'Internal Server Error', 'message': 'There was an internal
server error while processing the request.
Error ID: 0c2de841-9d83-479a-b7f2-ed2c102908f6',
'innerError':
{'request-id': '0c2de841-9d83-479a-b7f2-ed2c102908f6',
'date': '2020-02-07T01:28:17'}}}
From https://learn.microsoft.com/en-us/graph/query-parameters
Note: The following $filter operators are not supported for Azure AD
resources: ne, gt, ge, lt, le, and not. The contains string operator
is currently not supported on any Microsoft Graph resources.
Is there a way to filter by date for RiskDetections? Will appreciate any help.
Below filter with riskType and riskLevel shows data:
risk_detections_api_url = "https://graph.microsoft.com/beta/riskDetections?$filter=riskType eq 'anonymizedIPAddress' or riskLevel eq 'medium'"
Below filter with userPrincipalName shows data:
risk_detections_api_url = "https://graph.microsoft.com/beta/riskDetections?$filter=userPrincipalName eq 'john.doe#example.com'"
Below filter with ipAddress shows data:
risk_detections_api_url = "https://graph.microsoft.com/beta/riskDetections?$filter=ipAddress eq '195.228.45.176'"
Sample data
{
"id": "8901d1fee9bqwqweqwe683a221af3d2ae691736f2e369e0dd530625398",
"requestId": "cc755f41-0313-4cb2-96ce-3a6283fef200",
"correlationId": "c422083d-0e32-4afb-af4e-6ca46e4235b4",
"riskType": "anonymizedIPAddress",
"riskState": "atRisk",
"riskLevel": "medium",
"riskDetail": "none",
"source": "IdentityProtection",
"detectionTimingType": "realtime",
"activity": "signin",
"tokenIssuerType": "AzureAD",
"ipAddress": "195.228.45.176",
"activityDateTime": "2019-12-26T17:40:02.1402381Z",
"detectedDateTime": "2019-12-26T17:40:02.1402381Z",
"lastUpdatedDateTime": "2019-12-26T17:43:21.8931807Z",
"userId": "e3835755-80b0-4b61-a1c0-5ea9ead75300",
"userDisplayName": "John Doe",
"userPrincipalName": "john.doe#example.com",
"additionalInfo": "[{\"Key\":\"userAgent\",\"Value\":\"Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0\"}]",
"location": {
"city": "Budapest",
"state": "Budapest",
"countryOrRegion": "HU",
"geoCoordinates": {
"latitude": 47.45996,
"longitude": 19.14968
}
}
}
Based on Properties, activityDateTime is datetimeoffset type.
So you should use GET https://graph.microsoft.com/beta/riskDetections?$filter=activityDateTime gt 2019-12-25 rather than GET https://graph.microsoft.com/beta/riskDetections?$filter=activityDateTime gt '2019-12-25'.
A similar API document here: List directoryAudits.
But when I test it, it gives 500 error:
{
"error": {
"code": "Internal Server Error",
"message": "There was an internal server error while processing the request. Error ID: d52436f6-073b-4fc8-b3bc-c6a6336d6886",
"innerError": {
"request-id": "d52436f6-073b-4fc8-b3bc-c6a6336d6886",
"date": "2020-02-05T04:10:45"
}
}
}
I believe beta version for this API is still in change. You could contact Microsoft support with your request-id for further investigation.
You will need to provide the date in the UTC format.
Example:
https://graph.microsoft.com/beta/riskDetections?$filter=activityDateTime ge 2020-01-01T22:13:50.843847Z
In python you would do something like below to create the URL with filter:
from datetime import datetime
date_filter = datetime.utcnow().isoformat()+"Z"
request_url = "https://graph.microsoft.com/beta/riskDetections?$filter=activityDateTime ge " + date_filter
The response is now filtered:
[
{
"id": "68f0402c7063a2fbbae5895f2c63598ca3c2b81c44be60145be1a9cd7e20af4b",
"requestId": "181d3817-b4fb-4d2b-a87c-065776f05800",
"correlationId": "6d02786c-0bc7-441f-b303-51430016f955",
"riskType": "unfamiliarFeatures",
"riskState": "atRisk",
"riskLevel": "low",
"riskDetail": "none",
"source": "IdentityProtection",
"detectionTimingType": "realtime",
"activity": "signin",
"tokenIssuerType": "AzureAD",
"ipAddress": "52.185.138.50",
"activityDateTime": "2020-02-07T05:48:07.6322964Z",
"detectedDateTime": "2020-02-07T05:48:07.6322964Z",
"lastUpdatedDateTime": "2020-02-07T05:49:33.3003616Z",
"userId": "e3835755-80b0-4b61-a1c0-5ea9ead75300",
"userDisplayName": "John Doe",
"userPrincipalName": "john.doe#example.com",
"additionalInfo": "[{\"Key\":\"userAgent\",\"Value\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36\"}]",
"location": {
"city": "tokyo",
"state": "tokyo",
"countryOrRegion": "jp",
"geoCoordinates": {
"latitude": 35.69628,
"longitude": 139.7386
}
}
}
]

Marklogic 8 Node.js API - How can I scope a search on a property child of root?

[updated 17:15 on 28/09]
I'm manipulating json data of type:
[
{
"id": 1,
"title": "Sun",
"seeAlso": [
{
"id": 2,
"title": "Rain"
},
{
"id": 3,
"title": "Cloud"
}
]
},
{
"id": 2,
"title": "Rain",
"seeAlso": [
{
"id": 3,
"title": "Cloud"
}
]
},
{
"id": 3,
"title": "Cloud",
"seeAlso": [
{
"id": 1,
"title": "Sun"
}
]
},
];
After inclusion in the database, a node.js search using
db.documents.query(
q.where(
q.collection('test films'),
q.value('title','Sun')
).withOptions({categories: 'none'})
)
.result( function(results) {
console.log(JSON.stringify(results, null,2));
});
will return both the film titled 'Sun' and the films which have a seeAlso/title property (forgive the xpath syntax) = 'Sun'.
I need to find 1/ films with title = 'Sun' 2/ films with seeAlso/title = 'Sun'.
I tried a container query using q.scope() with no success; I don't find how to scope the root object node (first case) and for the second case,
q.where(q.scope(q.property('seeAlso'), q.value('title','Sun')))
returns as first result an item which matches all text inside the root object node
{
"index": 1,
"uri": "/1.json",
"path": "fn:doc(\"/1.json\")",
"score": 137216,
"confidence": 0.6202662,
"fitness": 0.6701325,
"href": "/v1/documents?uri=%2F1.json&database=Documents",
"mimetype": "application/json",
"format": "json",
"matches": [
{
"path": "fn:doc(\"/1.json\")/object-node()",
"match-text": [
"Sun Rain Cloud"
]
}
]
},
which seems crazy.
Any idea about how doing such searches on denormalized json data?
Laurent:
XPaths on JSON are supported by MarkLogic.
In particular, you might consider setting up a path range index to match /title at the root:
http://docs.marklogic.com/guide/admin/range_index#id_54948
Scoped property matching required either filtering or indexed positions to be accurate. An alternative is to set up another path range index on /seeAlso/title
For the match issue it would be useful to know the MarkLogic version and to see the entire query.
Hoping that helps,

Resources