I am trying to do a tutorial for DataBricks and am trying to create a single node cluster, however whatever I do it just will not work. I'm pretty sure I'm on a pay-as-you-go subscription and the limit should be 10.
Whichever compute I select, I get this error:
with this message:
This account may not have enough CPU cores to satisfy this request
Estimated available: 0, requested: 4
Learn more about CPU quota.
When I've gone to up my limit it says that the limit is 0/10 so I should have 10 avaliable?
Please can someone else me with this. Thanks
First confirm your subscription is pay-as-you-go or not. If not upgrade, it
Sign in to the Azure portal
Navigate to Subscriptions and then select your subscription.
In overview under Offer you will find pay-as-you-go.
After this if it is pay as you go subscription you can follow below procedure to Increase Azure Databricks Cluster vCPU Cores Limits
Go to Azure Portal > Clik in Search Resource Field> Type in “help” > Choose “Help + support”
At the top of the page , select “Create a support request” and fill the below information.
Summary: Services and subscription limits (quotas)
Issue type : Technical
Subscription: choose your subscription
Service: My services
Service Type: Databricks
Resource: select your resource
Problem Type: cluster
Proble subtype: select “Compute-VM (cores-vCPUs) subscription limit increases”
After selecting all the required fields, click “Next”.
In the “Additional Details” tab, click the link “Provide details” to bring the “Quota details” blade window to the right. Then, in the window:
Deployment model: Select “Resource Manager”.
Location: Select your location(s). Please note that you can request quota increases for multiple locations at one time.
Quotas: Select the VM series that you want to increase the quota. Please note that you can request quota increases for multiple VM series at one time
New vCPU Limit: Input the new quota that you want to increase up to
After filling in all the required information, click “Save and continue”.
after this select the Support format you want
Now Next: Review + create >>.
Related
I ran into this issue below, when trying to run a simple pyspark script in Azure:
%%pyspark
df = spark.read.load('abfss://products#xyzabcstorageaccount.dfs.core.windows.net/userdata1.parquet', format='parquet')
display(df.limit(10))
InvalidHttpRequestToLivy: Your Spark job requested 24 vcores. However, the workspace has a 12 core limit. Try reducing the numbers of vcores requested or increasing your vcore quota. HTTP status code: 400. Trace ID: 3308513f-be78-408b-981b-cd6c81eea8b0.
I am new to Azure and using the free trial now. Do you know how to reduce the numbers of vcores requested?
Thanks a lot
I tried to reproduce the same in my environment and got the below results:
If I create 4 vCores, It's working fine for me.
As per the above error, If you want 24 vCores as your pre-request. Please follow below steps:
First of all, you need to raise a request. To Increase capacity via the azure portal by creating a new support ticket.
Step 1:
Go to Azure Synapse -> Create support ticket -> +Add Issue type as service and subscription limits quotes and +Add Quota type as Azure Synapse
Step2:
Go to additional details -> Select Enter details add azure synapse quota type, resource and request quota.
I am very new to Azure. I have an Azure for Students subscription and I'm trying to create an Apache Kafka cluster using Azure HDInsight. I selected West Europe as my region. I'm using this resource as a guide: https://lenadroid.github.io/posts/kafka-hdinsight-and-spark-databricks.html
The first step was to create a resource group, which I managed to do. The next step is to deploy is, which consistently fails. The error is there are not enough cores: 26 cores are required, but 0 are available.
{
"status":"Failed",
"error":
{
"code":"DeploymentFailed",
"message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details":[
{
"code":"BadRequest",
"message":"{\r\n \"code\": \"BadRequest\",\r\n \"message\": \"User SubscriptionId '29bb2797-a6e9-4412-90da-52863c7579ad' does not have cores left to create resource 'kafka-kafkaclustertest1'. Required: 26, Available: 0.\"\r\n}"}
]
}
}
Looking at Usage + quotas, I'm at 100% usage of Network Watchers (1 out of 1). I tried disabling the Network Watcher to no avail.
My subscription is not eligible for a quota increase.
My questions are:
Does Azure for Students, with its $100 credit, support the services I will need: Resource Group, Virtual Networks, HDInsight?
If yes, do I have to configure/activate my subscription credit somehow?
Could the issue be related to the region? How do I make sure West Europe has enough available cores? Is it reasonable to try different regions?
Can I configure my Kafka resource group to requires less cores?
Are there easier ways to deploy a Kafka cluster, perhaps on other cloud providers?
I would very much appreciate some guidance -- thanks!
Unfortunately, you cannot create the Azure HDInsight cluster with the Student subscription.
Reason: With an Azure Student subscription, you will initially be able to access only Azure services that are available with a free tier of service use.
For more limitations, refer to the below links:
https://azure.microsoft.com/en-us/offers/ms-azr-0144p/
https://azure.microsoft.com/en-us/free/students/
You may want to look at a pay-as-you-go subscription as this should allow you to deploy a HDInsight cluster, however, may not allow you to utilize the benefits of a student subscription.
https://azure.microsoft.com/en-us/offers/ms-azr-0003p/
https://azure.microsoft.com/en-us/offers/ms-azr-0023p/
https://azure.microsoft.com/en-us/pricing/details/hdinsight/
When I tried to create Azure Synapse Workspace I get below error:
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"ReachedPerSubscriptionWorkspaceLimit","message":"Reached the maximum number of Synapse workspaces allowed for this subscription. Please contact Microsoft support to request an exception. Subscription: xxxx. Subscription offer type: PayAsYouGo. Current maximum: 0."}]}
I see that the limit is 20 workspaces, but as in error I don't have any created now. I even created a ticket to increate the quota from 20 to 30, but still have the same error.
Is there something I am missing?
The issue is resolved now from Microsoft.
Context : app-service in Azure with enabled auto-scale 2 to 8 instances. Usually workload fluctuates between 2..4 instances, and only on rare occasions scaling maxes out to 8 instances - for such cases I want to set up an alert. Let's say, I am interested in all scaling operation above 4 instance count.
Problem : cannot find an alert "scope + condition signal" combination where instance count of auto-scale would be possible to select. Does such data is accessible at all?
And "no" - I do not want to use OOB "Scale out - Notify" functionality, because this feature sends out emails about all scaling operations. Since I am not interested 2..4 instances and only >4 instances, conditioning must be possible.
You can create the alert mechanism for the autoscale operation in web app by projecting the autoscale operation logs to log analytic workspace & followed by creating custom alert.
Here are the steps you need to follow:
Go to Scale-out (App service plan) option in the portal under that Navigate to Diagnostic settings.
Create Diagnostic settings for the autoscale operations & project those logs to log analytics workspace.
Based on the requirement shared above, use the below query to pull scale out operation of a web app with instance count is greater.
AutoscaleScaleActionsLog | where OperationName == 'InstanceScaleAction' and ResultType == "Succeeded"
| where NewInstanceCount >4 and ScaleDirection == 'Increase'
Use the New alert rule option in the log analytics space to create a custom alert & using the above query as signal as shown in below picture.
Here is the sample image of the alert rule that was created using the above query
The above alert query run for every thirty minutes , if there any autoscale operation got recorded it will trigger an email to mentioned recipients.
click on save , enable the alert rule
Here is the sample email output triggered by alert rule
I am creating Azure QnA service and I am getting below error message "The scale operation is not allowed for this subscription in this region. Try selecting different region or scale option."
I have Selected below options
SKU :F0
Location:westus
SKU2: Free
Search hosting mode : Default
Microsoft.Web/serverfarms
Status: Unauthorized
Provisioning: StateFailed
I am new to azure, appreciate your help