I am using Data thirst AzDo task to create bearer token in my yaml pipeline. Reference to the task: https://marketplace.visualstudio.com/items?itemName=DataThirstLtd.databricksDeployScriptsTasks.
The SPN secret I have has special characters in it and is being passed from Azure DevOps library. The sample secret is something like this SP=X$fg#ab*a].
When I pass this value to the Azure DevOps yaml pipeline, I am getting a 403 error. My guess is, the secret value is not being parsed the right way and hence the 403. How should one handle special characters in an Azure DevOps yaml pipeline ?
My AzDo task is something like this:
- task: DataThirstLtd.databricksDeployScriptsTasks.databricksDeployCreateBearer.databricksDeployCreateBearer#0
displayName: 'Get Databricks Bearer Token'
inputs:
applicationId: '$(client_id)'
spSecret: '$(client_secret)' #this is where the secret value is passed from AzDo library
resourceGroup: '$(rg_name)'
workspace: '$(adb_workspace)'
subscriptionId: '$(subscription_id)'
tenantId: '$(tenant_id)'
region: '$(location)'
Based on my research, the variable seems been passed correctly:
trigger:
- none
pool:
vmImage: windows-latest
variables:
- group: xxx
# The below setting can help check what happened.
- name: system.debug
value: true
steps:
- task: databricksDeployCreateBearer#0
inputs:
applicationId: 'xxx'
spSecret: '$(SP)'
resourceGroup: 'xxx'
workspace: 'xxx'
subscriptionId: 'xxx'
tenantId: 'xxx'
region: 'westeurope'
Result:
In fact, the returned 403 code also proves that the incoming secret is correct from another aspect. Code 403 often means that the server side knows who initiated the request but rejected the request (403 Forbidden). The problem may come from insufficient AAD app permissions or restricted ip.
I suggest you take a look at this case:
Error 403 User not authorized when trying to access Azure Databricks API through Active Directory
Related
I try to get a secret from azure key vault in my ansible 4 playbook using azcollection 1.9.0.
- name: Get secret value
azure_rm_keyvaultsecret_info:
vault_uri: https://my-vault.vault.azure.net/
register: kvSecret
According to the docs the result should contain a list of secrets with a property called secret containing the secret value.
However, this property is not present on the result set. This is the result I get:
{
"changed": False,
"secrets": [
{
"sid": "https: //my-vault.vault.azure.net/secrets/ssh-user-username",
"version": "",
"tags": {},
"attributes": {
"enabled": True,
"not_before": None,
"expires": None,
"created": "2021-09-05T14:32:10+00:00",
"updated": "2021-09-05T14:32:10+00:00",
"recovery_level": "Recoverable+Purgeable"
}
}
],
"failed": False
}
If I try to get this exact secret using the name option I get an empty result set.
My vault contains this secret, it has a value and the service principal has access to my key vault through IAM with the roles Key Vault Reader and Key Vault Secrets User.
I tested it on my environment and my service principal is having Key vault reader and Key vault secrets user with the below yml code.
---
- hosts: localhost
connection: local
collections:
- azure.azcollection
vars:
vault_name: Testansumankeyvault01
secret_name: adminPassword
tasks:
- name: Get Key Vault by name
azure_rm_keyvault_info:
resource_group: test-rg
name: "{{ vault_name }}"
register: keyvault
- name: Set key vault URI fact
set_fact: keyvaulturi="{{ keyvault['keyvaults'][0]['vault_uri'] }}"
- name: Get secret value
azure_rm_keyvaultsecret_info:
vault_uri: "{{ keyvaulturi }}"
name: "{{ secret_name }}"
register: kvSecret
- name: set secret fact
set_fact: secretValue="{{ kvSecret['secrets'][0]['secret'] }}"
- name: Output key vault secret
debug:
msg="{{ secretValue }}"
Reference:
Azure built-in roles - Azure RBAC | Microsoft Docs
Use Azure Key Vault to store VM secrets with Ansible | Microsoft Docs
Turns out that this was an issue with the authentication. Ansible is connecting to my remote machine via ssh and therefore I needed to set the authentication for azure. I was doing this with environment variables in my ansible playbook but it turns out that they are not set when the playbook runs it's tasks. Passing them explicitly to the command does the trick.
Error
When calling admin.auth().createCustomToken() I am getting the following error:
Error: The caller does not have permission; Please refer to https://firebase.google.com/docs/auth/admin/create-custom-tokens for more details on how to use and troubleshoot this feature.
The provided documentation leads me to believe that the service account I am initializing the Firebase Admin SDK with does not have sufficient permissions. I don't believe this to be the case, so I want to ask and see if I've missed anything.
Configuration
Firebase Admin SDK is initialized in the backend like so:
admin.initializeApp({
serviceAccountId: 'firebase-adminsdk-xxxxx#my-project-id.iam.gserviceaccount.com'
});
Technically the value is referenced from an env var, but I have confirmed this value to be correct.
The service account being used has the following roles:
roles/firebase.sdkAdminServiceAgent
roles/iam.serviceAccountTokenCreator
Per the documentation, the required permission for creating custom tokens is iam.serviceAccounts.signBlob. This permission is part of the iam.serviceAccountTokenCreator role as per this output:
❯ gcloud beta iam roles describe roles/iam.serviceAccountTokenCreator
description: Impersonate service accounts (create OAuth2 access tokens, sign blobs
or JWTs, etc).
etag: AA==
includedPermissions:
- iam.serviceAccounts.get
- iam.serviceAccounts.getAccessToken
- iam.serviceAccounts.getOpenIdToken
- iam.serviceAccounts.implicitDelegation
- iam.serviceAccounts.list
- iam.serviceAccounts.signBlob
- iam.serviceAccounts.signJwt
- resourcemanager.projects.get
- resourcemanager.projects.list
name: roles/iam.serviceAccountTokenCreator
stage: GA
title: Service Account Token Creator
Lastly, the code in question that is erroring out is as follows:
try {
const loginToken = await admin.auth().createCustomToken(uid);
return response(200).json({ loginToken });
} catch (err) {
...
}
The uid comes from signing in a user via a GoogleUser credential - the provided uid is confirmed to be accurate, and this flow works locally when referencing a JSON key file for the same service account.
Server is running on GKE, in case it could be a cluster permission error.
Any help would be greatly appreciated!
EDIT - RESOLVED
Hiranya's answer did the trick - the K8s deployment had been configured with a service account whose original intent was only to enable Cloud SQL Proxy. Giving this service account the serviceAccountTokenCreator role solved the issue.
You need to make sure the service account that the SDK is authorized with (not the one specified as serviceAccountId) has the token creator role. This is the service account auto-discovered by Google Application Default Credentials. In case of Cloud Functions this is the service account named {project-name}#appspot.gserviceaccount.com. You need to figure out the equivalent service account for GKE and grant it the token creator role.
I'm writing a cloud function in Nodejs (10), and trying to access a secret like so:
const [secret] = await new SecretManagerServiceClient().accessSecretVersion({
name: `projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest`
})
I created the secret in the web console and the name used in code matches that of the existing secret. On the page for the cloud function details, it states that the service account is PROJECT_ID#appspot.gserviceaccount,com, so I added the secretmanager.secretAccessor role to it. However, I'm still getting the same error every time:
Error: 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest' (or it may not exist).
It makes no difference if I specify a concrete version or just use latest.
HTTP cloud function code:
const { SecretManagerServiceClient } = require('#google-cloud/secret-manager');
const secretManagerServiceClient = new SecretManagerServiceClient();
const name = 'projects/shadowsocks-218808/secrets/workflow/versions/latest';
exports.testSecretManager = async (req, res) => {
const [version] = await secretManagerServiceClient.accessSecretVersion({ name });
const payload = version.payload.data.toString();
console.debug(`Payload: ${payload}`);
res.sendStatus(200);
};
Deploy:
gcloud functions deploy testSecretManager --runtime nodejs10 --trigger-http --allow-unauthenticated
Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 256
entryPoint: testSecretManager
httpsTrigger:
url: https://us-central1-shadowsocks-218808.cloudfunctions.net/testSecretManager
ingressSettings: ALLOW_ALL
labels:
deployment-tool: cli-gcloud
name: projects/shadowsocks-218808/locations/us-central1/functions/testSecretManager
runtime: nodejs10
serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-43476143-b555-4cb2-8f6f-1b2d1952a2d7/42c4cda4-98a8-4994-a3be-d2203b9e646a.zip?GoogleAccessId=service-16536262744#gcf-admin-robot.iam.gserviceaccount.com&Expires=1596513795&Signature=kbLw5teN8EoYmj4fEweKKiIaakxcrhlUg2GGHV4jWJjvmeEfXePpRNOn9yz2zLn%2Fba0UqM9qdJMXujs5afBk%2BVBmywPEiptAZe2qgmldpr%2BsYejFu0woNgsPHVqtJ0NoWDo6W2dq4CuNNwO%2BaQ89mnhahUUQTInkJ55Y3wCIe9smk%2BqWtcvta3zICiToA7RQvPKY5MS6NViyj5mLxuJtDlTY9IKPL%2BqG6JAaQJSFYKYVgLyb6JfirXk8Q7%2FMvnHPpXPlhvsBLQksbF6jDPeefp2HyW4%2FSIQYprfpwKV3hlEIQyRQllz5J9yF83%2FxDPh%2BQPc5QmswKP5XAvYaszJPEw%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2020-08-04T03:34:32.665Z'
versionId: '2'
Test:
gcloud functions call testSecretManager --data '{}'
Got error same as you:
error: |-
Error: function terminated. Recommended action: inspect logs for termination reason. Details:
7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/shadowsocks-218808/secrets/workflow/versions/latest' (or it may not exist).
solution:
You can find the serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com from the deployment information details of cloud function.
go to IAM & Admin web UI, click ADD ANOTHER ROLE button, add Secret Manager Secret Accessor role to this service account.
Test again:
> gcloud functions call testSecretManager --data '{}'
executionId: 1tsatxl6fndw
result: OK
Read the logs for testSecretManager cloud function:
gcloud functions logs read testSecretManager
You will see the logs for the secret payload string.
I had the same issue and to solve it, I just had to:
Find the Service Account under General of my Google Cloud Function.
It looked like <project-name>#appspot.gserviceaccount.com
In IAM Admin, Add Secret Manager Secret Accessor Role to this Service Account.
After this, everything worked!
I have had similar issues working with secretmanager and the python google-cloud-secretmanager library (2.4). Specifically, after creating a secret and giving my service account the secretmanager.secretAccessor role on this secret (and nothing else, following the principle of least privilege), I was getting the following error when trying to access it:
details = "Permission 'secretmanager.versions.access' denied for resource 'projects/projectid/secrets/keyname/versions/latest' (or it may not exist)."
I could only make it work by also adding the secretmanager.viewer role at the project level, which as far as I can tell is not described in the documentation.
I had similar problem using terraform under gitlab.
I must add two authorizations to the service account which runs the pipeline:
resource "google_project_iam_policy" "gitlab" {
project = "secret_owner_project_id"
policy_data = data.google_iam_policy.iam.policy_data
}
data "google_iam_policy" "iam" {
binding {
role = "roles/secretmanager.secretAccessor"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
binding {
role = "roles/viewer"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
}
A bit late, but maybe this answer could be useful for future users. I encountered the same behavior only with Python. I tried lots of things but only thing that worked was creating new service account with zero roles(if I granted it secretmanager.secretAccessor role immediately, I got the same error). Then when empty service account is created, in IAM tab I press +Add, copy my empty service account adress and ONLY then I add secretmanager.secretAccessor role to it. Then I use this account as the account that will execute particular function. You of course may need to add other roles depending on what your function is intended to accomplish.
OAuth scope plays an important role here and please make sure the scope is defined correctly.
To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. If you receive an error with the following message, it means the instance or node was not provisioned with the correct OAuth scopes.
Request had insufficient authentication scopes
The required OAuth scope to use Secret Manager is:
https://www.googleapis.com/auth/cloud-platform
Example gcloud command to create dataproc with scope
gcloud dataproc clusters create xyz-pqr --region asia-south1 --subnet projects/xyz-pqr/regions/asia-south1/subnetworks/abc-serverless-vpc --zone asia-south1-b --master-machine-type n1-standard-4 --master-boot-disk-size 100 --num-workers 2 --worker-machine-type n1-standard-4 --worker-boot-disk-size 100 --image-version 2.0-debian10 --project xyz-development -scopes https://www.googleapis.com/auth/cloud-platform
Currently, I am accessing AWS parameter store value as environment variable. It is defined in serverless yml like so:
environment:
XYZ_CREDS: ${ssm:xyzCreds}
In code, I access this like so process.env.XYZ_CREDS
I need to move this value to AWS secret manager and access the xyzCreds in the same way.
Based on the serverless document I tried like so -
custom:
xyzsecret: ${ssm:/aws/reference/secretsmanager/XYZ_CREDS_SECRET_MANAGERa~true}
environment:
XYZ_CREDS: ${self:custom.xyzsecret}}
But it's not working. Please help!
After struggling with this issue by myself I found the solution that worked for me.
Assume that we have a secret XYZ_CREDS where we store user and password ket-value pairs. AWS Secrets manager stores them in JSON format: {"user": "test", "password": "xxxx"}
Here is how to put user and password into Lambda function environment variables:
custom:
xyzsecret: ${ssm:/aws/reference/secretsmanager/XYZ_CREDS~true}
myService:
handler: index.handler
environment:
username: ${self:custom.xyzsecret.user}
password: ${self:custom.xyzsecret.password}
I'm using serverless 1.73.1 for deploying to cloudformation.
Hope this helps others.
Given that the name of your secret in secrets manager is correct. I think you might have an "a" after manager before the decryption.
Secret manager stores in key value/json format.So specify the variables individually
Eg.
environment:
user_name: ${self:custom.xyzsecret}.username
password: ${self:custom.xyzsecret}.password
otherwise pass secret manager name and decrypt using aws-sdk in the code
environment:
secretkey_name:XYZ_CREDS_SECRET_MANAGERa
TL;DR: Is there a way to set app client custom scopes via cli or sdk?
I'm trying to automate my Cognito deployment with CloudFormation. I've already made some custom resources since not everything is supported. For this I'm using the AWS JS SDK. I want to set 'Allowed Custom Scopes' for the app clients in a specific user pool. However, I am unable to find how to do this in any documentation AWS provides. The CLI docs say only this on there docs here Cognito-user-identity docs:
AllowedOAuthScopes
A list of allowed OAuth scopes. Currently supported values are "phone", "email", "openid", and "Cognito".
The scopes mentioned there are default scopes that are always available in user pool. But I also use custom scopes that are provided by a Custom Resource Server I've defined. Those look like: resourceServer.com/scope. I can't find any docs about setting those scopes.
So, is there a way to set custom scopes via cli or sdk?
Custom Scope is supported on AllowedOAuthScopes field.
Documentation: https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_CreateUserPoolClient.html#CognitoUserPools-CreateUserPoolClient-request-AllowedOAuthScopes
To update userpool client via CLI: https://docs.aws.amazon.com/cli/latest/reference/cognito-idp/update-user-pool-client.html
(check out the --allowed-o-auth-scopes option)
See example cloudformation below
UserPoolResourceServer:
Type: AWS::Cognito::UserPoolResourceServer
Properties:
Identifier: users
Name: User API
UserPoolId: !Ref UserPool
Scopes:
- ScopeName: "write"
ScopeDescription: "Write access"
- ScopeName: "read"
ScopeDescription: "Read access"
UserPoolClientAdmin:
Type: "AWS::Cognito::UserPoolClient"
Properties:
AllowedOAuthFlows:
- client_credentials
AllowedOAuthFlowsUserPoolClient: true
AllowedOAuthScopes:
- users/read
- users/write
For anyone coming here looking for a solution, please follow #JohnPauloRodriguez's sample template. But you might need to add DependsOn attribute key in the UserPoolClient template for it work.
The reason being, first the Resource Server with these custom scopes should exist, then only we can refer to them in the client. As per the Cloud Formation Docs:
With the DependsOn attribute you can specify that the creation of a
specific resource follows another. When you add a DependsOn attribute
to a resource, that resource is created only after the creation of the
resource specified in the DependsOn attribute.
So the template for UserPoolClient will become:
CognitoUserPoolClient:
Type: AWS::Cognito::UserPoolClient
DependsOn: UserPoolResourceServer
Properties:
UserPoolId: !Ref UserPool
AllowedOAuthFlowsUserPoolClient: true
AllowedOAuthFlows:
- code
AllowedOAuthScopes:
- users/read
- users/write