Audit Logs Retrieval from Azure Data Lake Storage (Gen 2) - azure

Question

I am trying to retrieve audit logs from Azure Data Lake Storage (Gen 2)..
So far I have tried using AZCOPY, REST API (unsupported for now) in Gen 2 to retrieve (connect) the audit logs and looking for an alternative solution for retrieving the logs
When connected using AZCOPY it uses nothing but API based calls and when I tried to retrieve log I got the error that API calls are not supported for hierarchical namespace accounts. Image added for reference.
Snapshot of AZCOPY error
Is there any workaround for this use case or any other approach which I can try to retrieve logs?

Answer 1

Update:
I can get the file content from the ADLS GEN2 with read api. I can provide you an example written by python code(you can change to any other language as per my code). From the code below, you can directly get the file content, or get the Authorization which can be used in postman.
Python 3.7 code like below:
import requests
import datetime
import hmac
import hashlib
import base64
storage_account_name = 'xxx'
storage_account_key = 'xxx'
api_version = '2018-11-09'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
#the file path on adls gen2
FILE_SYSTEM_NAME='dd1/myfile.txt'
string_params = {
'verb': 'GET',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version,
'CanonicalizedResource': '/' + storage_account_name+'/'+FILE_SYSTEM_NAME
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']+'\n'
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
#print out the datetime
print(request_time)
#print out the Authorization
print('SharedKey ' + storage_account_name + ':' + signed_string)
headers = {
'x-ms-date' : request_time,
'x-ms-version' : api_version,
'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.dfs.core.windows.net/'+FILE_SYSTEM_NAME)
#print out the url
print(url)
r = requests.get(url, headers = headers)
#print out the file content
print(r.text)
After run the code, the file content is fetched:
And you can also use the generated values like authorization / date in the above code, in the postman:
As you may know that the SDK is not ready for azure data lake gen 2, so as of now, the solution is using ADLS Gen2 Read api.
After retrieving the content of the file, you can save it.
And you may do your own work for the authentication. If you have any issues about how to read using ADLS Gen 2 api, please feel free to let me know.

Answer 2

ADLS Gen2 $logs are now available when you sign up for Multi Protocol Access in ADLS Gen2. A blog describing Multi Protocol Access can be found at http://aka.ms/mpaadls. You can sign up for access here.
Enabling logs in the Azure portal is not currently supported. Here's an example of how to enable the logs by using PowerShell.
$storageAccount = Get-AzStorageAccount -ResourceGroupName <resourceGroup> -Name <storageAccountName>
Set-AzStorageServiceLoggingProperty -Context $storageAccount.Context -ServiceType Blob -LoggingOperations read,write,delete -RetentionDays <days>.
To consume logs, you can use AzCopy and SDKs today. You cannot view $logs in Azure Storage Explorer for the time being.

Answer 3

With November 2019 (Version 1.11.1) release of Azure Storage Explorer, it is now possible to view hidden containers such as $logs