Creating Azure storage authorization header using python - azure

I am trying to create the Authorization header for using Azure storage REST APIs. What a nightmare. The reason I am trying to do this is because I am trying to use a workflow builder (Alteryx) to call the API so my only programmatic options are Alteryx, python, or command line.
I think I'm close, but I just don't understand these last three lines of code, following this article - https://learn.microsoft.com/en-us/azure/storage/common/storage-rest-api-auth?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
// Now turn it into a byte array.
byte[] SignatureBytes = Encoding.UTF8.GetBytes(MessageSignature);
// Create the HMACSHA256 version of the storage key.
HMACSHA256 SHA256 = new HMACSHA256(Convert.FromBase64String(storageAccountKey));
// Compute the hash of the SignatureBytes and convert it to a base64 string.
string signature = Convert.ToBase64String(SHA256.ComputeHash(SignatureBytes));
So if I follow this correctly, I have to create a SHA256 version of the storage key but then I make a SHA256 hash of the SHA256 hash of the signaturebytes?
I'm current googling and not getting far, but basically trying to do the same thing above in .net using python.

In python, you can just use this line of code:
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
Here is the complete code of using List blobs api:
import requests
import datetime
import hmac
import hashlib
import base64
storage_account_name = 'xx'
storage_account_key = 'xxx'
container_name='aa1'
api_version = '2017-07-29'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
string_params = {
'verb': 'GET',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
'CanonicalizedResource': '/' + storage_account_name +'/'+container_name+ '\ncomp:list\nrestype:container'
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
headers = {
'x-ms-date' : request_time,
'x-ms-version' : api_version,
'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.blob.core.windows.net/'+container_name+'?restype=container&comp=list')
r = requests.get(url, headers = headers)
print(r.status_code)
print('\n\n'+r.text)
Test result:

Related

API in python Error 400 "invalid input parameters" - how to fix

I am programming an API in python to query a server if it has endpoint agents in it. The server and the endpoint belong to Apex central SaaS trend micro.
The error I get is that I'm putting the wrong parameters but I don't think the problem is there.
The code I have for the query is as follows:
import base64
import jwt
import hashlib
import requests
import time
import json
import urllib.parse
def create_checksum(http_method, raw_url, headers, request_body):
string_to_hash = http_method.upper() + '|' + raw_url.lower() + '|' + headers + '|' + request_body
base64_string = base64.b64encode(hashlib.sha256(str.encode(string_to_hash)).digest()).decode('utf-8')
return base64_string
def create_jwt_token(appication_id, api_key, http_method, raw_url, headers, request_body,
iat=time.time(), algorithm='HS256', version='V1'):
payload = {'appid': appication_id,
'iat': iat,
'version': version,
'checksum': create_checksum(http_method, raw_url, headers, request_body)}
token = jwt.encode(payload, api_key, algorithm=algorithm)
return token
# Use this region to setup the call info of the Apex Central server (server url, application id, api key)
# server info
use_url_base = 'https://arct3w.manage.trendmicro.com'
use_application_id = '52EB0005-B6DA-4249-9764-62AE3BFCDBB1'
use_api_key = 'B3FE1D91-5D05-490C-B45C-26A9EFF6C363'
productAgentAPIPath = '/WebApp/API/AgentResource/ProductAgents'
canonicalRequestHeaders = ''
useQueryString=''
payload = {
'computerId':'e34a13a1-1d0f-47bc-96e0-ae4db4288940'
}
useRequestBody = json.dumps(payload)
jwt_token = create_jwt_token(use_application_id, use_api_key, 'POST',
productAgentAPIPath + useQueryString,
canonicalRequestHeaders, useRequestBody, iat=time.time())
headers = {'Authorization': 'Bearer ' + jwt_token , 'Content-Type': 'application/json;charset=utf-8'}
#Choose by call type.
r = requests.post(use_url_base + productAgentAPIPath + useQueryString, data=useRequestBody, headers=headers, verify=False)
print(r.status_code)
if 'application/json' in r.headers.get('Content-Type', '') and len(r.content):
print(json.dumps(r.json(), indent=4))
else:
print(r.text)
I tried to do something similar to an api belonging to Vision One but it didn't work:
https://automation.trendmicro.com/xdr/api-v2#tag/Search/paths/~1v2.0~1xdr~1eiqs~1query~1endpointInfo/post
the original code of the query is :
https://automation.trendmicro.com/apex-central/api#tag/Security-Agents/operation/AgentResource_GetProductAgentsV2

Huobi working python 3.6 example create url (including signature)

Working example to generate a valid url (including signature) for the Huobi API.
In the Huobi API documenation there is no explicit example that allows you to verify your signature creation method step by step.
My intention is to create that here, but I need help, because I haven't managed yet.
The following is supposed to be the recipe.
Note that once you have this working, substitute valid values for your API key + secret and timestamp:
import hmac
import hashlib
import base64
from urllib.parse import urlencode
API_KEY = 'dummy-key'
API_SECRET = 'dummy-secret'
timestamp = '2021-03-04T11:36:39'
params_dict = {
'AccessKeyId': API_KEY,
'SignatureMethod': 'HmacSHA256',
'SignatureVersion': '2',
'Timestamp': timestamp
}
params_url_enc = urlencode(sorted(params_dict.items()))
pre_signed = 'GET\n'
pre_signed += 'api.huobi.pro\n'
pre_signed += '/v1/account/accounts\n'
pre_signed += params_url_enc
sig_bytes = hmac.new(
API_SECRET.encode(),
pre_signed.encode(),
hashlib.sha256).hexdigest().encode()
sig_b64_bytes = base64.b64encode(sig_bytes)
sig_b64_str = sig_b64_bytes.decode()
sig_url = urlencode({'Signature': sig_b64_str})
url = 'https://api.huobi.pro/v1/account/accounts?'
url += params_url_enc + '&'
url += sig_url
print('API_KEY={}'.format(API_KEY))
print('API_SECRET={}'.format(API_SECRET))
print('timestamp={}'.format(timestamp))
print('params_dict={}'.format(params_dict))
print('params_url_enc={}'.format(params_url_enc))
print('pre_signed:\n{}'.format(pre_signed))
print('sig_bytes={}'.format(sig_bytes))
print('sig_b64_bytes={}'.format(sig_b64_bytes))
print('sig_b64_str={}'.format(sig_b64_str))
print('sig_url={}'.format(sig_url))
print('url={}'.format(url))
Gives:
API_KEY=dummy-key
API_SECRET=dummy-secret
timestamp=2021-03-04T11:36:39
params_dict={'AccessKeyId': 'dummy-key', 'SignatureMethod': 'HmacSHA256', 'SignatureVersion': '2', 'Timestamp': '2021-03-04T11:36:39'}
params_url_enc=AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T11%3A36%3A39
pre_signed:
GET
api.huobi.pro
/v1/account/accounts
AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T11%3A36%3A39
sig_bytes=b'1921de9f42284bc0449c5580f52a9f7e7e3a54a6e8befc0d320992e757517a6b'
sig_b64_bytes=b'MTkyMWRlOWY0MjI4NGJjMDQ0OWM1NTgwZjUyYTlmN2U3ZTNhNTRhNmU4YmVmYzBkMzIwOTkyZTc1NzUxN2E2Yg=='
sig_b64_str=MTkyMWRlOWY0MjI4NGJjMDQ0OWM1NTgwZjUyYTlmN2U3ZTNhNTRhNmU4YmVmYzBkMzIwOTkyZTc1NzUxN2E2Yg==
sig_url=Signature=MTkyMWRlOWY0MjI4NGJjMDQ0OWM1NTgwZjUyYTlmN2U3ZTNhNTRhNmU4YmVmYzBkMzIwOTkyZTc1NzUxN2E2Yg%3D%3D
url=https://api.huobi.pro/v1/account/accounts?AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T11%3A36%3A39&Signature=MTkyMWRlOWY0MjI4NGJjMDQ0OWM1NTgwZjUyYTlmN2U3ZTNhNTRhNmU4YmVmYzBkMzIwOTkyZTc1NzUxN2E2Yg%3D%3D
Also add header in sending:
{"Content-Type": "application/x-www-form-urlencoded"}
Unfortunately, when I substitute my own valid API key + secret and a proper UTC time stamp, I invariably receive:
{"status":"error","err-code":"api-signature-not-valid","err-msg":"Signature not valid: Verification failure [校验失败]","data":null}
So what is going wrong here?
Huobi API documentation is
https://huobiapi.github.io/docs/spot/v1/en/#introduction
To get all accounts, use endpoint GET /v1/account/accounts
from datetime import datetime
import requests
import json
import hmac
import hashlib
import base64
from urllib.parse import urlencode
#Get all Accounts of the Current User
AccessKeyId = 'xxxxx-xxxxx-xxxxx-xxxxx'
SecretKey = 'xxxxx-xxxxx-xxxxx-xxxxx'
timestamp = str(datetime.utcnow().isoformat())[0:19]
params = urlencode({'AccessKeyId': AccessKeyId,
'SignatureMethod': 'HmacSHA256',
'SignatureVersion': '2',
'Timestamp': timestamp
})
method = 'GET'
endpoint = '/v1/account/accounts'
base_uri = 'api.huobi.pro'
pre_signed_text = method + '\n' + base_uri + '\n' + endpoint + '\n' + params
hash_code = hmac.new(SecretKey.encode(), pre_signed_text.encode(), hashlib.sha256).digest()
signature = urlencode({'Signature': base64.b64encode(hash_code).decode()})
url = 'https://' + base_uri + endpoint + '?' + params + '&' + signature
response = requests.request(method, url)
accts = json.loads(response.text)
print(accts)
Subsequently, if you need to run another endpoint (note timestamp allowance is ±5 minutes),
example, to get account balance, use GET /v1/account/accounts/{account_id}/balance
#Get Account Balance of a Specific Account
account_id = accts['data'][0]['id']
method = 'GET'
endpoint = '/v1/account/accounts/{}/balance'.format(account_id)
pre_signed_text = method + '\n' + base_uri + '\n' + endpoint + '\n' + params
hash_code = hmac.new(SecretKey.encode(), pre_signed_text.encode(), hashlib.sha256).digest()
signature = urlencode({'Signature': base64.b64encode(hash_code).decode()})
url = 'https://' + base_uri + endpoint + '?' + params + '&' + signature
response = requests.request(method, url)
r = json.loads(response.text)
print(r)
The mistake was that I took the hexidigest of the hash, whereas the digest was needed.
Working recipe here that you can check numerically to validate your code:
import hmac
import hashlib
import base64
from urllib.parse import urlencode
API_KEY = 'dummy-key'
API_SECRET = 'dummy-secret'
timestamp = '2021-03-04T12:54:56'
params_dict = {
'AccessKeyId': API_KEY,
'SignatureMethod': 'HmacSHA256',
'SignatureVersion': '2',
'Timestamp': timestamp
}
params_url_enc = urlencode(
sorted(params_dict.items(), key=lambda tup: tup[0]))
pre_signed = 'GET\n'
pre_signed += 'api.huobi.pro\n'
pre_signed += '/v1/account/accounts\n'
pre_signed += params_url_enc
sig_bin = hmac.new(
API_SECRET.encode(),
pre_signed.encode(),
hashlib.sha256).digest()
sig_b64_bytes = base64.b64encode(sig_bin)
sig_b64_str = sig_b64_bytes.decode()
sig_url = urlencode({'Signature': sig_b64_str})
url = 'https://api.huobi.pro/v1/account/accounts?'
url += params_url_enc + '&'
url += sig_url
print('API_KEY={}'.format(API_KEY))
print('API_SECRET={}'.format(API_SECRET))
print('timestamp={}'.format(timestamp))
print('params_dict={}'.format(params_dict))
print('params_url_enc={}'.format(params_url_enc))
print('pre_signed:\n{}'.format(pre_signed))
print('sig_bin={}'.format(sig_bin))
print('sig_b64_bytes={}'.format(sig_b64_bytes))
print('sig_b64_str={}'.format(sig_b64_str))
print('sig_url={}'.format(sig_url))
print('url={}'.format(url))
Result:
$ python test_huobi_so.py
API_KEY=dummy-key
API_SECRET=dummy-secret
timestamp=2021-03-04T12:54:56
params_dict={'AccessKeyId': 'dummy-key', 'SignatureMethod': 'HmacSHA256', 'SignatureVersion': '2', 'Timestamp': '2021-03-04T12:54:56'}
params_url_enc=AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T12%3A54%3A56
pre_signed:
GET
api.huobi.pro
/v1/account/accounts
AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T12%3A54%3A56
sig_bin=b'_\xb9k\x82!\xb4B%A\xfe\x0c \xff\x07%JE\xbe\x82\x8b-<^\xb7\xfc\x06\x85G\xb5$\x81\xd7'
sig_b64_bytes=b'X7lrgiG0QiVB/gwg/wclSkW+gostPF63/AaFR7Ukgdc='
sig_b64_str=X7lrgiG0QiVB/gwg/wclSkW+gostPF63/AaFR7Ukgdc=
sig_url=Signature=X7lrgiG0QiVB%2Fgwg%2FwclSkW%2BgostPF63%2FAaFR7Ukgdc%3D
url=https://api.huobi.pro/v1/account/accounts?AccessKeyId=dummy-key&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2021-03-04T12%3A54%3A56&Signature=X7lrgiG0QiVB%2Fgwg%2FwclSkW%2BgostPF63%2FAaFR7Ukgdc%3D

getting 415 error while posting in requests in rest api , trying to run cms query

Hi I am new to python and REST API,
I am getting 415 error while trying to run a query in cms using requests.post
I am not able to pass content-type and Accept along with the logon token.
I am able to run this in talend along with these 2 headers.
Can you please help me in how to add these 2 headers in requests.post at the end.
Below is my code
import requests
from lxml import etree
import xml.etree.ElementTree as ET
import pandas as pd
import openpyxl as x
from bs4 import BeautifulSoup
import xmltodict
protocol='http'
host='HOST'
port='6405'
content_type='application/xml'
base_url = protocol + '://' + host + ':' + port
bip_url = base_url + '/biprws'
webi_url = bip_url + '/raylight/v1'
sl_url = bip_url + '/sl/v1'
headers_auth = {
'Content-Type' : content_type,'Accept':'application/xml'
}
headers = {
}
username = 'user'
password = 'pass'
auth = requests.get(bip_url + '/logon/long', headers=headers)
root = etree.fromstring(auth.text)
root[3].text = username
root[0].text = password
etree.tostring(root)
send = requests.post(bip_url + '/logon/long',
headers=headers_auth,
data=etree.tostring(root))
tokenresp = etree.fromstring(send.content)
headers['X-SAP-LogonToken'] = tokenresp[3][0][0].text
folder_get = requests.get(bip_url + '/v1/cmsquery', headers=headers)
folder_root = etree.fromstring(folder_get.text)
Query_var = 'SELECT SI_ID,SI_NAME FROM CI_INFOOBJECTS WHERE SI_ANCESTOR = 12141'
folder_root[0].text = Query_var
data1 = etree.tostring(folder_root)
folder_post = requests.post(bip_url + '/v1/cmsquery', headers = headers, data = data1)
folder_post.status_code
I think 415 means that you're passing a content type that the API doesn't accept. You need to configure your headers correctly. Try this:
headers = {
'Content-Type' : 'application/xml'
}
auth = requests.get(bip_url + 'logon/long', headers=headers)
print(auth.status_code)
It looks like your problem is that you set headers to a blank dictionary.

Python upload files and folder to sharepoint

I'm trying to upload some files (multiple files and folder) to sharepoint and when I run the script i dont have any errors but i'm unable so see my files in sharepoint.
import requests
from shareplum import Office365
# get data from configuration
username = 'last.surname#tenant.com'
password = 'mypassword'
site_name = 'BI_odair'
base_path = 'https://tenant.sharepoint.com'
doc_library = 'data'
file_name = "links.txt"
# Obtain auth cookie
authcookie = Office365(base_path, username=username,
password=password).GetCookies()
session = requests.Session()
session.cookies = authcookie
session.headers.update({'user-agent': 'python_bite/v1'})
session.headers.update({'accept': 'application/json;odata=verbose'})
# perform the actual upload
with open(file_name, 'rb') as file_input:
try:
response = session.post(
url=base_path + "/sites/" + site_name +
"/_api/web/GetFolderByServerRelativeUrl('Shared%20Documents/" +
doc_library+"')/Files/add(url='"
+ file_name + "',overwrite=true)",
data=file_input)
except Exception as err:
print("Some error occurred: " + str(err))
URL for my sharepoint: https://tenant.sharepoint.com/sites/BI_odair/Documents%20partages/Forms/AllItems.aspx?viewid=4e7fdfb9%2De84a%2D42cd%2Db537%2D0d2837ca92cc
Theres already a folder named Data, but i wanted to upload my files in the root folder "/Documents%20partages"
I've already added this to my code: https://stackoverflow.com/a/59083429/6754555
Thanks in advance.
Here's the solution that worked perfectly on my end:
pip install SharePlum and then use the code below
import requests
from shareplum import Office365
# Set Login Info
username = '<username>'
password = '<password>'
site_name = '<site_name>'
base_path = 'https://<domain_name>.sharepoint.com'
doc_library = 'Shared%20Documents'
nested_folder = 'Shared%20Documents/<folder1>/<folder2>' #if you want to upload in nested folders else nested_folder = doc_library
file_name = "my_file.zip" #when your file in the same directory
# Obtain auth cookie
authcookie = Office365(base_path, username=username, password=password).GetCookies()
session = requests.Session()
session.cookies = authcookie
session.headers.update({'user-agent': 'python_bite/v1'})
session.headers.update({'accept': 'application/json;odata=verbose'})
session.headers.update({'X-RequestDigest': 'FormDigestValue'})
response = session.post(url=base_path + "/sites/" + site_name + "/_api/web/GetFolderByServerRelativeUrl('" + doc_library + "')/Files/add(url='a.txt',overwrite=true)",
data="")
session.headers.update({'X-RequestDigest': response.headers['X-RequestDigest']})
# perform the actual upload
with open(file_name, 'rb') as file_input:
try:
response = session.post(
url=base_path + "/sites/" + site_name + f"/_api/web/GetFolderByServerRelativeUrl('" + nested_folder + "')/Files/add(url='"
+ file_name + "',overwrite=true)",
data=file_input)
print("response: ", response.status_code) #it returns 200
if response.status_code == '200':
print("File uploaded successfully")
except Exception as err:
print("Something went wrong: " + str(err))
print('File Uploaded Successfully')
"Shared Documents" is the default document library. If you want to upload file to a custom library, please modify the path as below:
response = session.post(
url=base_path + "/sites/" + site_name +
"/_api/web/GetFolderByServerRelativeUrl("Documents%20partages")/Files/add(url='"
+ file_name + "',overwrite=true)",
data=file_input)
Also you can have a look at below blog:Uploading files to SharePoint
//////// Updated
I created a document library and tested the demo in above blog, it works well.
Below is my code:
import requests
from shareplum import Office365
from config import config
# get data from configuration
username = config['sp_user']
password = config['sp_password']
site_name = config['sp_site_name']
base_path = config['sp_base_path']
doc_library = config['sp_doc_library']
file_name = "test.csv"
# Obtain auth cookie
authcookie = Office365(base_path, username=username, password=password).GetCookies()
session = requests.Session()
session.cookies = authcookie
session.headers.update({'user-agent': 'python_bite/v1'})
session.headers.update({'accept': 'application/json;odata=verbose'})
# dirty workaround.... I'm getting the X-RequestDigest from the first failed call
session.headers.update({'X-RequestDigest': 'FormDigestValue'})
response = session.post( url=base_path + "/sites/" + site_name + "/_api/web/GetFolderByServerRelativeUrl('" + doc_library + "')/Files/add(url='a.txt',overwrite=true)",
data="")
session.headers.update({'X-RequestDigest': response.headers['X-RequestDigest']})
# perform the actual upload
with open( r'C:\Users\xxx\Documents\test.csv', 'rb+') as file_input:
try:
response = session.post(
url=base_path + "/sites/" + site_name + "/_api/web/GetFolderByServerRelativeUrl('" + doc_library + "')/Files/add(url='"
+ file_name + "',overwrite=true)",
data=file_input)
except Exception as err:
print("Some error occurred: " + str(err))
print('end...')

Audit Logs Retrieval from Azure Data Lake Storage (Gen 2)

I am trying to retrieve audit logs from Azure Data Lake Storage (Gen 2)..
So far I have tried using AZCOPY, REST API (unsupported for now) in Gen 2 to retrieve (connect) the audit logs and looking for an alternative solution for retrieving the logs
When connected using AZCOPY it uses nothing but API based calls and when I tried to retrieve log I got the error that API calls are not supported for hierarchical namespace accounts. Image added for reference.
Snapshot of AZCOPY error
Is there any workaround for this use case or any other approach which I can try to retrieve logs?
Update:
I can get the file content from the ADLS GEN2 with read api. I can provide you an example written by python code(you can change to any other language as per my code). From the code below, you can directly get the file content, or get the Authorization which can be used in postman.
Python 3.7 code like below:
import requests
import datetime
import hmac
import hashlib
import base64
storage_account_name = 'xxx'
storage_account_key = 'xxx'
api_version = '2018-11-09'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
#the file path on adls gen2
FILE_SYSTEM_NAME='dd1/myfile.txt'
string_params = {
'verb': 'GET',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version,
'CanonicalizedResource': '/' + storage_account_name+'/'+FILE_SYSTEM_NAME
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']+'\n'
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
#print out the datetime
print(request_time)
#print out the Authorization
print('SharedKey ' + storage_account_name + ':' + signed_string)
headers = {
'x-ms-date' : request_time,
'x-ms-version' : api_version,
'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.dfs.core.windows.net/'+FILE_SYSTEM_NAME)
#print out the url
print(url)
r = requests.get(url, headers = headers)
#print out the file content
print(r.text)
After run the code, the file content is fetched:
And you can also use the generated values like authorization / date in the above code, in the postman:
As you may know that the SDK is not ready for azure data lake gen 2, so as of now, the solution is using ADLS Gen2 Read api.
After retrieving the content of the file, you can save it.
And you may do your own work for the authentication. If you have any issues about how to read using ADLS Gen 2 api, please feel free to let me know.
ADLS Gen2 $logs are now available when you sign up for Multi Protocol Access in ADLS Gen2. A blog describing Multi Protocol Access can be found at http://aka.ms/mpaadls. You can sign up for access here.
Enabling logs in the Azure portal is not currently supported. Here's an example of how to enable the logs by using PowerShell.
$storageAccount = Get-AzStorageAccount -ResourceGroupName <resourceGroup> -Name <storageAccountName>
Set-AzStorageServiceLoggingProperty -Context $storageAccount.Context -ServiceType Blob -LoggingOperations read,write,delete -RetentionDays <days>.
To consume logs, you can use AzCopy and SDKs today. You cannot view $logs in Azure Storage Explorer for the time being.
With November 2019 (Version 1.11.1) release of Azure Storage Explorer, it is now possible to view hidden containers such as $logs

Resources