I am using the below code to extract satellite image using sentinelsat
from sentinelsat import SentinelAPI, read_geojson, geojson_to_wkt
from datetime import date
api = SentinelAPI('userid', 'password', 'https://scihub.copernicus.eu/dhus')
footprint = geojson_to_wkt(read_geojson('./map.geojson'))
products = api.query(footprint,
date = ('20180101', '20191010'),
platformname = 'Sentinel-2',
processinglevel = 'Level-2A',
cloudcoverpercentage = (0,3)
)
products_gdf = api.to_geodataframe(products)
products_gdf_sorted = products_gdf.sort_values(['cloudcoverpercentage'], ascending=[True])
products_gdf_sorted.index
api.download('fed1003b-effa-41f5-9079-5d017af0eea2')
I have ensured that my geojson is pointing to the correct position.
{
"type": "Polygon",
"coordinates": [
[
[
80.265872,13.064500
],
[
80.265526,13.064076
],
[
80.266435,13.064190
],
[
80.265872,13.064500
]
]
]
}
I am expecting building to be part of my tiff file. But I am just getting green patch as the output.
I am using the following code to create the tiff file.
import rasterio as rio
R10 = '.\S2A_MSIL2A_20190511T045701_N0212_R119_T44PMV_20190511T104102.SAFE\GRANULE\L2A_T44PMV_A020279_20190511T050605\IMG_DATA\R10m'
b4 = rio.open(R10+'\T44PMV_20190511T045701_B04_10m.jp2' , driver='JP2OpenJPEG')
b3 = rio.open(R10+'\T44PMV_20190511T045701_B03_10m.jp2' , driver='JP2OpenJPEG')
b2 = rio.open(R10+'\T44PMV_20190511T045701_B02_10m.jp2', driver='JP2OpenJPEG')
with rio.open('RGB.tiff','w',driver='Gtiff', width=b4.width, height=b4.height,
count=3,crs=b4.crs,transform=b4.transform, dtype=b4.dtypes[0]) as rgb:
rgb.write(b2.read(1),1)
rgb.write(b3.read(1),2)
rgb.write(b4.read(1),3)
rgb.close()
Where am I going wrong, whether I am missing any parameter while doing the extraction or my expectation is wrong. Kindly clarify. Thanks in advance.
Related
I'm looking for a library to convert a text file to JSON.
Do you know which one has the following behavior?
I already test some libraries but without success.
The source files contains a list of key=value pairs, one key per line.
Converting to correct data type is important, my files has:
string keys
number keys
boolean keys
object (JSON) keys
arrays (of simple strings or of JSON objects)
Example
name = "test"
version = 3
enabled = true
fruit = {"type":"orange","color":"orange"}
consumers = ["kids", "adults"]
years = [2014, 2015]
fruits = [{"type":"orange","color":"orange"},{"type":"apples","method":"red"}]
Expected Result after conversion: Valid JSON (don't need style/identation)
{
"name": "test",
"version": 3,
"enabled": true,
"fruit": {
"type": "orange",
"color": "orange"
},
"consumers": [
"kids",
"adults"
],
"years": [
2014,
2015
],
"fruits": [
{
"type": "orange",
"color": "orange"
},
{
"type": "apples",
"method": "red"
}
]
}
The format you're using isn't standardized so I'm doubtful you'll find a package that can parse it out of the box. Your values do look to be valid JSON primitives so you can leverage JSON.parse to parse the right hand side. With that you'd just need a parser to robustly pull out all the raw [key, value] pairs, but most parsers probably try to do more than just that which might not be what you want.
If you know the input will always be clean and don't need a completely robust parser, it's not difficult to roll this yourself:
const data = fs.readFileSync('./data.txt', {encoding: 'utf8'}).split('\n').filter(Boolean)
const obj = {}
for (const line of data) {
const [key, val] = line.split(/\s*=\s*(.+)/)
obj[key] = JSON.parse(val)
}
Writing a utility method for Jenkins pipeline using groovy and noticed its not easy to get value from a deeply nested map using externalised object retrieval notation as string.
Question: If you see the example below, you will notice that its not possible to retrieve object in the form datas."bankctrl.deployment.clientConfigMap.enabled". You definitely can do datas."bankctrl"."deployment"."clientConfigMap"."enabled". I have a workaround, but question is if there is any known best approach to do this?
datas = [
"ao":[
deployment:[
clientConfigMap:[
enabled:"true", name:"volume-ao-custom"
], image:[
name:"test/mit/ao",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
], "bankctrl":[
deployment:[
clientConfigMap:[
enabled:"false",
name:"volume-bankctrl-custom"
], image:[
name:"test/mit/bankctrl",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
]
]
String name = "bankctrl"
datas."${name}".deployment.clientConfigMap.enabled = true
println datas
//following does't work as dots within string are not evaluated.
String elementNameToUpdate = "bankctrl.deployment.clientConfigMap.enabled"
datas."${elementNameToUpdate}" = true
Following code works when you need to access values using externalised string:
String s = "bankctrl.deployment.clientConfigMap.enabled"
def q = s.split( /\./ ).inject( datas ) { obj1, prop -> obj1?."$prop" }
println q
Even this works when you need to access values using externalised string:
String s = "bankctrl.deployment.clientConfigMap.enabled"
Eval.x(datas,"x.${s}")
def gpathSet = { obj,path,value -> Eval.xy(obj, value, "x.${path}=y") }
datas = [
"bankctrl":[
deployment:[
clientConfigMap:[
enabled:"false",
name:"volume-bankctrl-custom"
], image:[
name:"test/mit/bankctrl",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
]
]
String elementNameToUpdate = "bankctrl.deployment.clientConfigMap.enabled"
gpathSet(datas, elementNameToUpdate, true)
One way of workaround.
String s = "bankctrl.deployment.clientConfigMap.enabled"
def setDeepProps(s,datas, value) {
s.split(/\./).inject(datas) {
obj, prop ->
if (obj."$prop" instanceof Map)
obj?."$prop"
else
obj."$prop" = value
}
}
setDeepProps(s,datas,true)
println datas
Groovy doesn't have this functionality out-of-box, so you have to either go wild and use Eval :) which for this certain case is a bit fragile (if the path-string is misspelled it can produce RuntimeExceptions) and not the best performer (due to run-time compilation).
You can solve your problem with a simple recursion like so:
def datas = [
"ao":[
deployment:[
clientConfigMap:[
enabled:"true", name:"volume-ao-custom"
], image:[
name:"test/mit/ao",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
], "bankctrl":[
deployment:[
clientConfigMap:[
enabled:"false",
name:"volume-bankctrl-custom"
], image:[
name:"test/mit/bankctrl",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
]
]
def traverser
traverser = { map, path ->
int ix = path.indexOf '.'
if( -1 == ix ) return map[ path ]
def val = map[ path[ 0..<ix ] ]
if( val in Map )
traverser val, path.substring( ix + 1 )
else
null
}
traverser datas, "bankctrl.deployment.clientConfigMap.enabled"
Below workaround worked for me. The reason I preferred this over any other solution is cause Jenkins is very picky about what script you invoke and hence wanted to have a work around that would not force me to accept script execution in Jenkins while getting around this requirement.
On a side note, would have loved if Jenkins would let me explore full power of Groovy and unfortunately it doesn't because of security reasons.
datas = [
"ao":[
deployment:[
clientConfigMap:[
enabled:"false",
name:"volume-ao-custom"
], image:[
name:"test/mit/ao",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
], "bankctrl":[
deployment:[
clientConfigMap:[
enabled:"false",
name:"volume-bankctrl-custom"
], image:[
name:"test/mit/bankctrl",
tag:"10.0.0-3.0.0",
pullPolicy:"IfNotPresent"
]
]
]
]
boolean customConfigFolderExists = true
String elementNameToUpdate = "bankctrl.deployment.clientConfigMap.enabled"
String[] elementNameToUpdateArray = "${elementNameToUpdate}".split("\\.")
def tempDatas = datas
//get property like datas.bankctrl.deployment.clientConfigMap (note that we are leaving enabled out which would be invoked in the following steps)
for (int i=0; i<elementNameToUpdateArray.length-1; i++) {
println "elementNameToUpdateArray[i] -> ${elementNameToUpdateArray[i]}"
tempDatas = tempDatas?."${elementNameToUpdateArray[i]}"
}
//we are invoking property returned from for loop above using the last element from elementNameToUpdateArray
tempDatas?."${elementNameToUpdateArray[elementNameToUpdateArray.length-1]}" = true
println "datas = ${datas}"
Thanks for taking the time out to read this. I want to find a way of parsing the json below. I'm really struggling to get the correct values out. I am getting this info from an API, and want to save this data into a database.
I am really struggling to parse info_per_type because I first need to get the available_types. This can change depending on the info available (i.e. I might get 2 different types in the next call, there's a total of 4) so my code needs to be flexible enough to deal with this
```
{
"data": [
{
"home_team": "Ravenna",
"id": 82676,
"available_types": [
"type_a",
"type_b"
],
"info_per_type": {
"type_a": {
"options": {
"X": 0.302,
"X2": 0.61,
"X3": 0.692,
"X4": 0.698,
"X5": 0.39,
"X6": 0.308
},
"status": "pending",
"output": "12",
"option_values": {
"X": 3.026,
"X2": 1.347,
"X3": 1.516,
"X4": 1.316,
"X5": 2.936,
"X6": 2.339
}
},
"type_b": {
"options": {
"yes": 0.428,
"no": 0.572
},
"status": "pending",
"output": "no",
"option_values": {
"yes": null,
"no": null
}
}
}
}
]
}```
So far, I can get the available_types out. But after that, I'm stuck. I have tried eval and exec but I can't seem to get that working either.
```
r = requests.get(url, headers=headers).text
arrDetails = json.loads(r)
arrDetails = arrDetails['data']
x = arrDetails[0]['available_types']
print(x[1]) #I get the correct value here
y = exec("y = arrDetails[0]['info_per_type']['" + x[1] + "']")
print(y)```
When I print out y I get None. What I want is some way to reference that part of the json file, as the results within that node are what I need. Any help would be HIGHLY appreciated!
Something like this should work :
for row in arrDetails['data']:
for available_type in row['available_types']:
print(row['info_per_type'][available_type])
I am new to Python. I am writhing a code to generate a excel file having the data sourced by calling API and correlate those to get desired result.
basically taking input from one database and search that in others and fetch related information.
The 4 databases have below data :
EEp
---------------------
{u'data': [{u'_id': u'5c30702c8ca9f51da8178df4',
u'encap': u'vlan-24',
u'ip': u'7.12.12.16',
u'mac': u'5B:P9:01:9E:42:08'}]}
PathEp
-----------
{u'data': [{u'_id': u'5c54a81a8ca9f51da84ae08e',
u'paths': u'paths-1507',
u'endpoint': u'eth1/10',
u'cep': u'5B:P9:01:9E:42:08',
u'tenant': u'ESX'}]}
ip4_address
-----------------------
{u'data': [{u'Allocation': u'Build_Reserved',
u'address': u'7.12.12.16',
u'name': u'fecitrix-1',
u'state': u'RESERVED'}]}
asset
---------------
{u'data': [{u'_id': u'57ccce8110dd54f02881fedc',
u'client': u'CES',
u'hostname': u'fecitrix-1'
u'os_team': u'Window'}]}
Logic:
If "mac" of EEp and "cep" of PathEp is same than take "encap","ip" ,"mac"
"paths" ,'endpoint","cep" and "tenant" (these values need to be exported
to excel)
Take ip of EEp and search in "ip4_address"
and get the "name" from ip4_address ( name need to be exported to excel).
If "name" of ip4_address is equal to "hostname" of database "asset" then take
"client" and "os_team" ( export that to excel)
I have written the script but not getting the desired result.
def get_host_details(self):
data = {
"find": {
"hostname": self.controller
},
"projection":{
"tenant": 1,
"paths": 1,
"endpoint":1
}
}
host_details = self.post("https://database.app.com/api/data/devices/PathEp/find", data)
#print host_details
hosts = []
for record in host_details:
if "mig" not in record["endpoint"]:
hosts.append(record)
return hosts
def get_ipaddress(self, controller):
host_record = {"tenant": "UNKNOWN",
"paths": "UNKNOWN",
"endpoint": "UNKNOWN",
"ip": "UNKNOWN",
"mac": "UNKNOWN",
"encap": "UNKNOWN"}
data = {
"find": {
"hostname": controller,
"ip": {
"$ne": "0.0.0.0"
}
},
"projection": {
"ip": 1,
"mac":1,
"encap":1,
}
}
endpoints = self.post("https://database.app.com/api/data/devices/EEp/find", data)
IPAM = self.get_dns()
print endpoints
host_details = self.get_host_details()
host_details_record = []
for record in endpoints:
for host in host_details:
if record["mac"] == host["cep"]:
host_record = {"tenant": host["tenant"],
"paths": host["paths"],
"endpoint": host["endpoint"],
"ip": record["ip"],
"mac": record["mac"],
"encap": record["encap"]}
host_details_record.append(host_record)
self.get_excel(host_details_record)
def get_dns(self, endpoints):
ip_dns_record = []
for each_endpoint in endpoints:
data = {
"find":
{
"address": {
"$eq": each_endpoint["ip"]
},
},
"projection":
{
"name": 1
}
}
dns_record = {"client":"UNKNOWN",
"os_team":"UNKNOWN",
ipam_record = self.post("https://database.app.com/api/data/"
"internal/ip4_address/find", data)
if ipam_record:
dns_record["ip_address"] = each_endpoint["ip"]
dns_record["hostname"] = ipam_record[0]["name"]
dns_record = self.get_remedy_details(ipam_record[0]["name"],
dns_record)
ip_dns_record.append(dns_record)
else:
dns_record["ip_address"] = each_endpoint["ip"]
dns_record["hostname"] = "UNKNOWN"
ip_dns_record.append(dns_record)
self.get_excel(ip_dns_record)
def get_remedy_details(self, hostname, dns_record):
data = {
"find":
{
"hostname": hostname.upper(),
}
}
remedy_data = self.post("https://database.app.com/api/data/internal/asset/find", data)
print(remedy_data)
#remedy_data = remedy_data["data"]
if remedy_data:
dns_record["client"] = remedy_data[0].get("client","UNKNOWN")
dns_record["os_team"] = remedy_data[0].get("os_team", "UNKNOWN")
else:
dns_record["client"] = "UNKNOWN"
dns_record["os_team"] = "UNKNOWN"
return dns_record
def get_excel(self, ip_dns_record):
filename = self.controller + ".xls"
excel_file = xlwt.Workbook()
sheet = excel_file.add_sheet('HOSTLIST')
sheet.write(0, 0, "IP Address")
sheet.write(0, 1, "HostName")
sheet.write(0, 2, "Client")
sheet.write(0, 3, "OS Team")
for count in xrange(1, len(ip_dns_record)+1):
sheet.write(count, 0,ip_dns_record[count - 1]["ip_address"])
sheet.write(count, 1,ip_dns_record[count - 1]["hostname"])
sheet.write(count, 2,ip_dns_record[count - 1]["client"])
sheet.write(count, 3,ip_dns_record[count - 1]["os_team"])
excel_file.save(filename)
if __name__ == "__main__":
controller = sys.argv[1]
OBJ = ACIHostList(controller)
print "SCRIPT COMPLETED"
No idea where I am going wrong and what needs to be done .
Your question leaves too much out. You should include all errors that you get. You should also comment your code as well so we can understand what you are trying to achieve in each step.
This is not an answer but something to try:
Rather than trying to wrap your head around a module like excel, wright your data to a simple CSV file. A CSV file can be opened up in excel and it formats correctly but is a lot easier to create.
import csv
data = [["a", "b"], ["c", "d"]]
with open("file.csv", "w+") as csv_file:
create_csv = csv.writer(csv_file)
create_csv .writerows(data)
simply grab all your data into a 2D list and using the above code dump it into a file so you can easily read it.
check the output of the file and see if you are getting the data you expect.
If you are not getting the desired data into this CSV file then there is an issue with your database queries.
Need some help converting data returned from:
def groupedResults = searchResults.groupBy{it.eventYear}{it.eventMonth}{it.eventStartDate};
The above code currently returns:
[2017: [ July: [ 07/17/2017: [ [eventName: TestEvent1, eventStartDatetime: 2017-07-17 12:00:00Z], [eventName: TestEvent2, eventStartDatetime: 2017-07-17 1:00:00Z] ], [07/18/2017: [ [eventName: TestEvent3, eventStartDatetime: 2017-07-17 12:00:00Z],...],...],...]
I need the following:
[yearList: [yearValue: 2017, monthList: [ monthValue: July, dateList: [ dateValue: 07/21/2017, events: [ eventTitle: TestEvent, eventStartDatetime: 2017-07-17 12:00:00Z],...],...],...]....]
How can I create the data output as described?
So with a bit of work, I got an example input (real example input is always nice in a question btw, as it makes it easier to see what you have. Two things are important in a journey, where you start and where you want to finish):
import java.time.*
import java.time.format.*
def parse = { s ->
LocalDateTime.parse(s, DateTimeFormatter.ofPattern('yyyy-MM-dd HH:mm:ssVV'))
}
def searchResults = [
[xEventStartDateTime:parse('2017-07-17 12:00:00Z'), eventTitle:'National Funeral Directors Association (NFDA) Conference'],
[xEventStartDateTime:parse('2017-07-24 12:00:00Z'), eventTitle:'Gathering of Warlocks'],
[xEventStartDateTime:parse('2018-02-17 12:00:00Z'), eventTitle:'Goose worriers meetup'],
[xEventStartDateTime:parse('2017-01-17 12:00:00Z'), eventTitle:'The new year angering of optometrists']
]
We can then group them as you currently have:
// Group as normal
def grouped = searchResults.groupBy(
{ it.xEventStartDateTime.year },
{ it.xEventStartDateTime.month },
{it.xEventStartDateTime.toLocalDate() }
)
Then, we just need a simple re-organising step to get this grouped map into the format you want:
// Then modify to the format you want
def result = [
yearList: grouped.collect { year, yGroup ->
[ yearValue: year, monthList: yGroup.collect { month, mGroup ->
[ monthValue: month, dateList: mGroup.collect { date, dGroup ->
[ dateValue: date, events: dGroup ]
} ]
} ]
}
]
Hope this helps!
I may have made a small mistake some where in here but I think this what you want:
def yearList = []
groupedResults.each{ year , month ->
def dateList = []
month.each{ date , events
def eventList = []
events.each{ event ->
eventList.add([ eventTitle: event.evenName, eventStartDatetime: event.eventStartDateTime])
}
def eventMap = [:]
eventMap.put("dateValue": date)
eventMap.put ("events": eventList)
dateList.add(eventMap)
}
dateMap =[:]
dateMap.put("dateList": dateList)
dateMap.put("monthValue": month)
yearList.add(["yearValue": year, "monthList": dateMap])
}
yearMap = [:]
yearMap.put("yearList": yearList)
I didn't get a chance to test so like I said I may have made a small mistake but you should get the idea.