I have been able to successfully create a Google Container Cluster in the developers console and have deployed my app to it. This all starts up fine, however I find that I can't connect to Cloud SQL, I get;
"Error: Handshake inactivity timeout"
After a bit of digging, I hadn't had any trouble connecting to the Database from App Engine or my local machine so I thought this was a little strange. It was then I noticed the cluster permissions...
When I select my cluster I see the following;
Permissions
User info Disabled
Compute Read Write
Storage Read Only
Task queue Disabled
BigQuery Disabled
Cloud SQL Disabled
Cloud Datastore Disabled
Cloud Logging Write Only
Cloud Platform Disabled
I was really hoping to use both Cloud Storage and Cloud SQL in my Container Engine Nodes. I have allowed access to each of these API's in my project settings and my Cloud SQL instance is accepting connections from any IP (I've been running Node in a Managed VM on App Engine previously), so my thinking is that Google is Explicitly disabling these API's.
So my two part question is;
Is there any way that I can modify these permissions?
Is there any good reason why these API's are disabled? (I assume there must be)
Any help much appreciated!
With Node Pools, you can sort of add scopes to a running cluster by creating a new node pool with the scopes you want (and then deleting the old one):
gcloud container node-pools create np1 --cluster $CLUSTER --scopes $SCOPES
gcloud container node-pools delete default-pool --cluster $CLUSTER
The permissions are defined by the service accounts attached to your node VMs during cluster creation (service accounts can't be changed after a VM is instantiated, so this the only time you can pick the permissions).
If you use the cloud console, click the "More" link on the create cluster page and you will see a list of permissions that you can add to the nodes in your cluster (all defaulting to off). Toggle any on that you'd like and you should see the appropriate permissions after your cluster is created.
If you use the command line to create your cluster, pass the --scopes command to gcloud container clusters create to set the appropriate service account scopes on your node VMs.
Hmm, I've found a couple of things, that maybe would be interested:
Permissions belong to a service account (so-called Compute Engine default service account, looks like 12345566788-compute#developer.gserviceaccount.com)
Any VM by default works using this service account. And its permissions do not let us Cloud SQL, buckets and so on. But...
But you can change this behavior using another service account with the right perms. Just create it manually and set only needed perms. Switch it out using gcloud auth activate-service-account --key-file=/new-service-account-cred.json
That's it.
For the cloudsql there's the possibility to connect from containers specifying a proxy as explained here https://cloud.google.com/sql/docs/postgres/connect-container-engine
Related
Given that I have a 24x7 AKS Cluster on AZURE, for which afaik Kubernetes cannot stop/pause a pod and then resume it standardly,
with, in my case, a small Container in a Pod, and for that Pod it can be sidelined via --replicas=0,
then, how can I, on-demand, best kick off a LINIX script packaged in that Pod/Container which may be not running,
from an AZURE Web App?
I thought using ssh should work, after first upscaling the pod to 1 replica. Is this correct?
I am curious if there are simple http calls in AZURE to do this. I see CLI and Powershell to start/stop AKS cluster, but that is different of course.
You can interact remotely with AKS by different methods. The key here is to use the control plane API to deploy your kubernetes resource programmatically (https://kubernetes.io/docs/concepts/overview/kubernetes-api/) .
In order to do that, you should use client libraries that enable that kind of access. Many examples can be found here for different programming languages:
https://github.com/kubernetes-client
ssh is not really recommended since that is sort of a god access to the cluster and its usage is not meant for your purpose.
I am struggling to get my Azure batch nodes to start within a Pool that is configured to use a virtual network. The virtual network has been configured with a service endpoint policy that has a "Microsoft.Storage" policy definition and it points at a single storage account. Without the service endpoints defined on the virtual network the Azure batch pool works as expected, but with it the following error occurs and the node never starts.
I have tried creating the Batch account in both Pool allocation modes. This did not seem to make a difference, the pool resizes successfully and then the nodes are stuck in "Starting" mode. In the "User Subscription" mode I found the start-up error because I can see the VM instance in my account:
VM has reported a failure when processing extension 'batchNodeExtension'. Error message: "Enable failed: processing file downloads failed: failed to download file[0]: failed to download file: unexpected status code: actual=403 expected=200" More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot
From what I can determine this is an Azure VM extension that is running to configure the VM for Azure Batch. My base image is Canonical, ubuntuserver, 18.04-lts (batch.node.ubuntu 18.04). I can see that the extensions is attempting to download from:
https://a52a7f3c745c443e8c2cac69.blob.core.windows.net/nodeagentpackage-version9-22-0-2/Ubuntu-18.04/batch_init-ubuntu-18.04-1.8.7.tar.gz (note I removed the SAS token from this URL for posting here)
there are 8 further files that are downloaded and it looks like this is configuring the Batch agent on the node.
The 403 error indicates that the node cannot connect to this storage account, which makes sense given the service endpoint policy. It does not include this storage account within it and this storage account is external to my Azure subscription. I thought that I might be able to add it to the service endpoint policy, but I have no way of determining what Azure subscription it is part of it. If I knew this I thought I could add it like:
Endpoint policy allows you to add specific Azure Storage accounts to allow list, using the resourceID format. You can restrict access to all storage accounts in a subscription
E.g. /subscriptions/subscriptionId (from https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoint-policies-overview)
I tried adding security group rules using service tags for Azure storage, but this did not help. The node still cannot connect and this makes sense given the description of service endpoint policies.
The reason for my interest in this is the following post:
[https://github.com/Azure/Batch/issues/66][1]
I am trying to minimise the bandwidth charges from my storage account by using service endpoints.
I have also tried to create my own VM, but I am not sure whether the "batchNodeExtension" script is run automatically for VMs that you're using with Batch.
I would really appreciate any pointers because I am running out of ideas to try!
Batch requires a generic rule for all of Storage (can be regional variant) as specified at https://learn.microsoft.com/en-us/azure/batch/batch-virtual-network#network-security-groups-specifying-subnet-level-rules. Currently it is mainly used to download our agent and maintain state/get information needed to run tasks.
I am facing the same problem with Azure Machine Learning. We are trying to fight data exfiltration by using the SP Policies in order to prevent sending the data to any non-subscription storage accounts.
Since Azure ML Computes depends on the Batch service, we were unable to run any ML compute if the SP policy is associated to the compute subnet.
Microsoft stated the follwoing:
Filtering traffic on Azure services deployed into Virtual Networks: At this time, Azure Service Endpoint Policies are not supported for any managed Azure services that are deployed into your virtual network.
https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoint-policies-overview#scenarios
I understand from this kind of restriction, that any service that use Azure Batch (which almost all services in Azure?) cannot use the SP Policy which make it useless freature...
Finally we endup by removing the SP policy completly from our network architecture and considered it only for scenarios where you to want to restrict customers to access specific storage accounts.
I have enabled Kubernetes RBAC authorization in all my Azure kubernetes clusters. Now I need to give permissions for viewing live data in containers tab
How can I do it? Which is the minimum premission needed?
Thanks
As far as I understand from my investigation, if you want to do it using Azure Built-in roles, you need the following three roles at the very least:
Reader role assignment scoped to the AKS cluster to be able to discover the cluster
Azure Kubernetes Service Cluster User Role role assignment scoped to the AKS cluster as mentioned in the note atop the page in the docs. This is needed to allow access to Microsoft.ContainerService/managedClusters/listClusterUserCredential/action API call. This API call lists the cluster user credentials.
Log Analytics Contributor role assignment scoped to the Log Analytics workspace associated to the AKS cluster. This is needed to execute an Analytics query for data, i.e., perform a /workspaces/{workspaceId}/query API call. More here.
This should let one pull up the live data for containers. If not comfortable with this approach, you might also create a Custom Role allowing only those exact actions.
Hope this helps!
When submitting an aml compute job, I want to access the nodes where the compute happens for debugging purposes. The portal gives me the IP address, the port and the nodeID, but no password seems to exist within the portal.
How am I supposed to connect to the machine?
I am running on NC6 machines for a single node training. I have already tried to run the command given through the portal, but the node is (hopefully so) protected by a password.
In order to be able to log in to the nodes of an AML Compute cluster, you have to provide a username and password and ssh key (ssh key is optional), when you create the cluster. You can only do that at creation time.
When you are setting up your cluster, you can define a username/password/ssh key that can be used to login to the cluster. If you did not define these at the time of creation, you would need to recreate the cluster unfortunately.
We are working on better documenting this, and also update our messaging a bit.
Does Azure Container Service integrate with Azure Monitor?
Wondering what the best way is to do logging/monitoring of kubernetes cluster?
If you are looking for monitoring tools on Azure, you may want to use Azure OMS (Opertation Management Suite). This gives you the ability to monitor the container inventory, performance, and logs in a single location. To my understanding, the stats of the container is only available for Linux nodes now, if you are deploying your k8s cluster on the Azure Portal.
To do this, you need to first create an OMS account. By this time, you should have the Workspace ID and the key available. The next step would
be to create the oms pod on each node using a DaemonSet.
For the detailed setup, take a look at https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-oms.
For third party tools, Grafana+influxdb is one of the ways I have tried before. Basically it provides you with the metrics on two levels: POD and NODE respectively. The displayed metrics included CPU Usage, Memory Usage, Network Usage and Filesystem Usage, etc. Of course, you can always alter your query to add extra metrics.
For the implementation of this approach, you can refer to https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/monitoring.md.
Hope this helps :)
you can use this CLI command to browse through kubernetes cluster deployed using azure container service.
az acs kubernetes browse -g -n
This way you can see kubernetes webui
also you can use kubectl proxy command.