Do azure team plan on introducing new query feature like skip,contains etc in future, where can i find what are the new feature/functionality that will be available in future.
Thanks
I would recommend following Azure Storage blog for announcements. In addition, you can also share your ideas or features you would like to see at Azure Feedback Storage forum.
Related
I am working on something, where I need to replicate only few tables instead of entire database from the leader cluster. How should I do it in the Azure Portal using Azure data share? I can see from azure documentation, that they are using C# or some other language for it, can we do it directly via Azure Portal?
As of this writing, table-level sharing isn't yet available through Azure Data Share, but should become available in the next few weeks (follow this doc for updates: https://learn.microsoft.com/en-us/azure/data-explorer/data-share)
As you mentioned correctly, it is already available programmatically using the management API (documented here: https://learn.microsoft.com/en-us/azure/data-explorer/follower#table-level-sharing)
I'm very new to the Azure Data Lake Storage and currently training on Data Factory. I have a developer background so right away I'm not a fan of the 'tools' approach for development. I really don't like how there's all these settings to set and objects you have to create everywhere. I much prefer a code approach which allows us to detach the logic from the service (don't like the publishing thing to save), see everything by scrolling or navigate to different objects in a project, see differences easier in source control and etc. So I found this Micrososft's Filesystem SDK that seems to be an alternative to Data Factory:
https://azure.microsoft.com/en-us/blog/filesystem-sdks-for-azure-data-lake-storage-gen2-now-generally-available/
What has been your experience using this approach? Is this a good alternative? Is there a way to run SDK code in data factory? that way we can leverage scheduling and triggers? I guess i'm looking for Pros/cons.
thank you
Well, the docs refer to several SDKs, one of them being the .Net SDK and the title is
Use .NET (or Python or Java etc.) to manage directories, files, and ACLs in Azure Data Lake Storage Gen2
So, the SDK lets you manage the filesystem only. No support for triggers, pipelines, dataflows and the lot. You will have to stick to the Azure Data Factory for that.
Regarding this:
I'm not a fan of the 'tools' approach for development
I hate to tell you but the world is moving that way whether you like it or not. Take Logic Apps for example. Azure Data Factory isn't aimed at the hardcore developer but fulfils a need for people working with large sets of data like Data Engineers. I am already glad it integrates with git very well. Yes, there is some overhead in defining sinks and sources but they are reusable across pipelines.
If you really want to use code try Azure Databricks. Take a look at this Q&A as well.
TL;DR:
The FileSystem SDK is not an alternative.
The code-centric alternative to Azure Data Factory for building and managing your Azure Data Lake is Spark. Typically either Azure Databricks or Azure Synapse Spark.
Is there an inbuilt way to notify an email when a new entry is added to the Table?
I am asking for anything programatically just within their own UI
Not currently but you could put it on an Azure Storage Queue and process it to Table Storage and send an Email with Azure Functions.
Check out this page what is possible - https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings
Okay, so the easiest way to see the data is to use their desktop app
Use https://azure.microsoft.com/en-us/features/storage-explorer/
The lastest Azure product updates covering event-driven applications suggest to adopt these application patterns to react to events published by Azure Storage. These docs/resources might help to explore the application pattern and current platform/framework support.
Azure Storage
Reacting to Blob storage events (preview)
Did not found anything similar for Azure Tables - what about suggesting this on UserVoice?
CosmosDB
Working with the change feed support in Azure Cosmos DB
I have an account on azure and build a cluster on it (Chemalivethermotest). It has a storage account and I just by mistake erase all data on it (I misanderstood something on the online azure interface). Is there a way to recover the datas? I see there are no files anymore but there are still blob. Are they allowing to recover things?
Thanks you very much for your help.
This is an issue you will have to take up directly with Azure support, there is no technical answer. Good luck!
I have been trying to set up an import job as described here; the problem is that we do not have "Classic" storage, rather we are trying to set it up with "New" storage. Using the new portal I cannot find the place where one is meant to create a new job. The linked article shows how to do this for classic storage on the old portal only.
I have tried using the second approach they mention, which is to use the API, but that is turning out to be more of a pain than I though.
Does anyone know where I can add an import/export job in the new portal? Is this possible with "new" storage? If I manage to get the API way to work, can it be applied to "new" storage or is it only for "classic"?
Unfortunately, Import/Export is not available in the Preview Portal, and does not work at this time with v2 storage accounts. Can you use a Classic storage account instead?
We may be able to provide a sample code to unblock your scenario. Can you please send an email with your detailed requirements to waimportexport#microsoft.com so that we can set up a call to discuss further.
Thanks,
Aung