SQL Server Column Encryption using Azure Key Vault and Spring Boot - azure

I need to save the data in SQL server having column encryption using the Azure Key vault
#Bean
#Primary
public DataSource dataSource() throws SQLException {
KeyVaultClient client = new KeyVaultClient(keyVaultCredentialService);
String userName = client.getSecret(vaultURL, "spring-datasource-username").value();
String password = client.getSecret(vaultURL, "spring-datasource-password").value();
String url = "jdbc:sqlserver://test.database.windows.net;databaseName=encryption_demo;columnEncryptionSetting=Enabled;";
String driverClass = client.getSecret(vaultURL, "spring-datasource-driverClassName").value();
DataSource dataSource = DataSourceBuilder
.create()
.username(userName)
.password(password)
.url(url)
.driverClassName(driverClass)
.build();
SQLServerColumnEncryptionAzureKeyVaultProvider akvProvider = new SQLServerColumnEncryptionAzureKeyVaultProvider(clientId, clientKey);
Map<String, SQLServerColumnEncryptionKeyStoreProvider> keyStoreMap = new HashMap<String, SQLServerColumnEncryptionKeyStoreProvider>();
keyStoreMap.put(akvProvider.getName(), akvProvider); SQLServerConnection.registerColumnEncryptionKeyStoreProviders(keyStoreMap);
return dataSource;
}
application.properties
azure.keyvault.uri= ....
azure.keyvault.client-id= ...
azure.keyvault.client-key= ...
SQLServer table
CREATE TABLE [dbo].[Patients](
[id] [int] PRIMARY KEY NOT NULL,
[ssn] [varchar](max) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NOT NULL,
[first_name] [varchar](max) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
[last_name] [varchar](max) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
)
GO
While saving the data in DB getting the error:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Internal error while encryption: Illegal key size

Download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. Be sure to read the Readme included in the zip file for installation instructions and relevant details on possible export/import issues.
If using the mssql-jdbc-X.X.X.jre7.jar or sqljdbc41.jar, the policy files can be downloaded from Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 Download.
If using the mssql-jdbc-X.X.X.jre8.jar or sqljdbc42.jar, the policy files can be downloaded from Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8 Download.
If using the mssql-jdbc-X.X.X.jre9.jar, no policy file needs to be downloaded. The jurisdiction policy in Java 9 defaults to unlimited strength encryption.
For more details, you could refer to this article.

Related

Generating ASC files with Bouncy Castle C#

I have this piece of code where its been generated a public and private key using Bouncy Castle API:
RsaKeyPairGenerator g = new RsaKeyPairGenerator();
g.Init(new KeyGenerationParameters(new SecureRandom(), 2048));
AsymmetricCipherKeyPair keyPair = g.GenerateKeyPair();
RsaKeyParameters privateKey = (RsaKeyParameters)keyPair.Private;
RsaKeyParameters publicKey = (RsaKeyParameters)keyPair.Public;
I need to export my keys for an ASC file, for example:
PublicKey.asc
PrivateKey.asc
At the moment, I'm using the code below to create the ASC files:
TextWriter tw = new StringWriter();
PemWriter pw = new PemWriter(tw);
pw.WriteObject(publicKey);
pw.Writer.Flush();
string printPublicKey = tw.ToString();
Console.WriteLine(printPublicKey);
byte[] pbkasc = Encoding.ASCII.GetBytes(printPublicKey);
File.WriteAllBytes("c:\\temp\\PublicKey.asc", pbkasc);
pw.WriteObject(privateKey);
pw.Writer.Flush();
string printPrivateKey = tw.ToString();
Console.WriteLine(printPrivateKey);
byte[] pvkasc = Encoding.ASCII.GetBytes(printPrivateKey);
File.WriteAllBytes("c:\\temp\\PublicKey.asc", pvkasc);
But when I try to import them, to a third party software, Kleopatra for example, I'm receiving the error below:
I'm beginner with all this encryption stuff. So, I need to know if I'm doing that in the correct way or if there is any other to generate those asc files. I did some research and I find this certificate generator:
//// Create PFX (PKCS #12) with private key
//File.WriteAllBytes("c:\\temp\\capronizera.pfx", cert.Export(X509ContentType.Pfx, "senha"));
//// Create Base 64 encoded CER (public key only)
//File.WriteAllText("c:\\temp\\mycert.cer",
// "-----BEGIN CERTIFICATE-----\r\n"
// + Convert.ToBase64String(cert.Export(X509ContentType.Cert), Base64FormattingOptions.InsertLineBreaks)
// + "\r\n-----END CERTIFICATE-----");
But unfortunately, it doesn't apply for my issue/question/doubt.
Thanks. :-)

Azure UserDelegation Key Exception

I am trying to generate UserDelegationKey for my Azure Storage Blob but i am getting an exception:
Can not instantiate Stax reader for XML source type class org.codehaus.stax2.io.Stax2ByteArraySource (unrecognized type)
This Exception occurs when i call getUserDelegationKey on BlobContainerAsyncClient
Version of my azure-storage-blob library is 12.12.0
Below is the code snippet
private void uploadNextWeekReportToAzure() {
BlobServiceAsyncClient blobServiceAsyncClient = blobServiceClientBuilder
.credential(new DefaultAzureCredentialBuilder().build())
.buildAsyncClient();
BlobContainerAsyncClient blobContainerAsyncClient = blobServiceAsyncClient.getBlobContainerAsyncClient("container name");
BlobAsyncClient blobAsyncClient = blobContainerAsyncClient.getBlobAsyncClient("blob name");
OffsetDateTime keyStart = OffsetDateTime.now();
OffsetDateTime keyExpiry = OffsetDateTime.now().plusDays(7);
blobServiceAsyncClient.getUserDelegationKey(keyStart,keyExpiry)
.doOnError(throwable -> log.error("Exception occurred:{}",throwable.getMessage()))
.doOnSuccess(userDelegationKey -> {
log.info("UserDelegationKey:{}",userDelegationKey.getValue());
String saSToken = generateSaSToken(blobAsyncClient, userDelegationKey);
log.info("SAS TOKEN:{}",saSToken);
})
.subscribe();
}
private String generateSaSToken(BlobAsyncClient blobAsyncClient,
UserDelegationKey userDelegationKey) {
BlobContainerSasPermission blobContainerSasPermission = new BlobContainerSasPermission()
.setReadPermission(true);
BlobServiceSasSignatureValues builder = new BlobServiceSasSignatureValues(
OffsetDateTime.now().plusDays(1), blobContainerSasPermission)
.setProtocol(SasProtocol.HTTPS_ONLY);
return String
.format("https://%s.blob.core.windows.net/%s/%s?%s", blobAsyncClient.getAccountName(),
blobAsyncClient.getContainerName(),
blobAsyncClient.getBlobName(),
blobAsyncClient.generateUserDelegationSas(builder, userDelegationKey));
}
Complete Exception Trace:
CallbackNotImplemented: java.lang.IllegalArgumentException: Can not instantiate Stax reader for XML source type class org.codehaus.stax2.io.Stax2ByteArraySource (unrecognized type)
Caused by: java.lang.IllegalArgumentException: Can not instantiate Stax reader for XML source type class org.codehaus.stax2.io.Stax2ByteArraySource (unrecognized type)
at io.strati.libs.forklift.com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:770)
at io.strati.libs.forklift.com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:345)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:631)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:29)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1124)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3643)
at com.azure.core.util.serializer.JacksonAdapter.deserialize(JacksonAdapter.java:281)
at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.deserializeBody(HttpResponseBodyDecoder.java:169)
at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.lambda$decodeByteArray$1(HttpResponseBodyDecoder.java:105)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2397)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:110)
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onNext(FluxDoOnEach.java:173)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.java:188)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:179)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:284)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:173)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2663)
at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:145)
at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:212)
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:269)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:401)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:416)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:470)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
Try this code to generate user delegation key and SAS
String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net", "accountName");,
blobServiceClient = new BlobServiceClientBuilder().endpoint(endpoint).credential(new DefaultAzureCredentialBuilder().build()).buildClient();
keyStart = OffsetDateTime.now();
keyExpiry = OffsetDateTime.now().plusDays(7);
userDelegationKey = blobServiceClient.getUserDelegationKey(keyStart, keyExpiry);
BlobContainerSasPermission blobContainerSas = new BlobContainerSasPermission();
blobContainerSas.setReadPermission(true);
BlobServiceSasSignatureValues blobServiceSasSignatureValues = new BlobServiceSasSignatureValues(keyExpiry,blobContainerSas);
BlobContainerClient blobContainerClient=blobServiceClient.getBlobContainerClient("containerName");
if (!blobContainerClient.exists())
blobContainerClient.create();
String sas = blobContainerClient.generateUserDelegationSas(blobServiceSasSignatureValues, userDelegationKey);
For more information refer this link
You may also check this method to Generate Key
The Get User Delegation Key operation gets a key that can be used to sign a user delegation SAS (shared access signature). A user delegation SAS grants access to resources in the Blob service using Azure Active Directory (Azure AD) credentials.
For more information refer this link

What are the Object Attributes needed to generate KeyPairs from YubiKey with PKCS11?

My application wants to sign documents using YubiKey with PIV (PKCS11). I was able to Login to my YubiKey but When I try to generate KeyPairs, it throws this error.
"Method C_GenerateKeyPair returned CKR_ATTRIBUTE_VALUE_INVALID"
This is the code I have right now (C# .NET)
byte[] ckaId = session.GenerateRandom(20);
// Prepare attribute template of new public key
List<IObjectAttribute> publicKeyAttributes = new List<IObjectAttribute>();
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_CLASS, CKO.CKO_PUBLIC_KEY));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_TOKEN, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_PRIVATE, false));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_LABEL, #"TestApp"));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ID, ckaId));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ENCRYPT, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_VERIFY, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_WRAP, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_MODIFIABLE, false));
// Prepare attribute template of new private key
List<IObjectAttribute> privateKeyAttributes = new List<IObjectAttribute>();
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_CLASS, CKO.CKO_PRIVATE_KEY));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_TOKEN, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_PRIVATE, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_LABEL, #"TestApp"));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ID, ckaId));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_SENSITIVE, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_DECRYPT, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_SIGN, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_UNWRAP, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_MODIFIABLE, false));
// Specify key generation mechanism
IMechanism mechanism = session.Factories.MechanismFactory.Create(CKM.CKM_RSA_PKCS_KEY_PAIR_GEN);
// Generate key pair
IObjectHandle publicKeyHandle = null;
IObjectHandle privateKeyHandle = null;
session.GenerateKeyPair(mechanism, publicKeyAttributes, privateKeyAttributes, out publicKeyHandle, out privateKeyHandle);
There are a few changes that need to be made to get past all the errors. To help anyone that hits similar vague error messages in the future the thing that helped me the most here was setting the YKCS11_DBG environment variable so that error messages from the libykcs11 library get printed to stderr (the Yubico docs here aren't up to date in all sections but they mention the variable and a few other relevant things here https://developers.yubico.com/yubico-piv-tool/YKCS11/)
As for the changes in order:
CKA_ID should be the same for both the private and public key template, but has a finite set of allowed values (covered by the Yubico docs page, and in the source) so you can't just use 20 random bytes
CKA_MODIFIABLE is not a valid attribute for the public or private key templates (although the Yubico docs show it as a supported attribute)
CKA_MODULUS_BITS must be set to 2048 or 1024 for the public key in RSA mode source
So something like this ended up working for me:
// Prepare attribute template of new public key
List<IObjectAttribute> publicKeyAttributes = new List<IObjectAttribute>();
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_CLASS, CKO.CKO_PUBLIC_KEY));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_TOKEN, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_PRIVATE, false));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_LABEL, #"TestApp"));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ID, 2)); // 2 == Digital Signature cert
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ENCRYPT, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_VERIFY, true));
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_WRAP, true));
//publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_MODIFIABLE, false)); -- not allowed
publicKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_MODULUS_BITS, 2048)); // required to be 1024 or 2048 for RSA
// Prepare attribute template of new private key
List<IObjectAttribute> privateKeyAttributes = new List<IObjectAttribute>();
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_CLASS, CKO.CKO_PRIVATE_KEY));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_TOKEN, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_PRIVATE, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_LABEL, #"TestApp"));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_ID, 2)); // 2 == Digital Signature cert
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_SENSITIVE, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_DECRYPT, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_SIGN, true));
privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_UNWRAP, true));
//privateKeyAttributes.Add(session.Factories.ObjectAttributeFactory.Create(CKA.CKA_MODIFIABLE, false)); -- not allowed
Also since it looks like you're roughly following the pkcs11interop sample, note that if you call session.DestroyObject on the key handles then the cert gets deleted

Performing ECIES operation using Bouncy castle with KeyPair in JKS

I have requirement to perform ECIES encryt/decrypt using secp256r1 with BC as provider.
I have need reference of (1) how to store ECIES private-public key pair in JKS Keystore (2) retrieve public key from JKS.
I have provision key-pair using keytool command as per https://zombiesecured.com/html/tutorials/Keytool/ECC-JK.html.
Signature algorithm name: SHA256withECDSA
Subject Public Key Algorithm: 256-bit EC key
(2) Can you help how to retrieve key in ECDSAPublicKey format as input to encryption .
I have found reference of decoding key to ECPublicKey as below. What should be my 1st argument? How to retrieve encoded key from JKS?
public static ECPublicKey decodePublicKey(byte[] encoded, String namedCurve) throws InvalidKeySpecException, NoSuchAlgorithmException, NoSuchProviderException, IOException
{
KeyFactory fact = KeyFactory.getInstance("ECDSA", BouncyCastleProvider.PROVIDER_NAME);
ECNamedCurveParameterSpec params = ECNamedCurveTable.getParameterSpec(namedCurve);
java.security.spec.EllipticCurve ellipticCurve = EC5Util.convertCurve(params.getCurve(), params.getSeed());
java.security.spec.ECPublicKeySpec keySpec = new java.security.spec.ECPublicKeySpec(ECPointUtil.decodePoint(ellipticCurve,encoded),EC5Util.convertSpec(ellipticCurve, params));
return (ECPublicKey) fact.generatePublic(keySpec);
}
how to use IESCipher, IESParameterSpec, engineInit to perform ECIES encrypt/decrypt.

Kusto data ingestion from an Azure Function App ends with a 403

I try to ingest data from azure function app into a ADX database. I followed the instruction found in the the article here.
The difference is, I'd like to insert data into the table. I struggle with a 403 error "Principal 'aadapp=;' is not authorized to access table"
What I did:
I have created a AAD App with the following API permissions:
AAD App configured permission
I configured the database via Kusto Explorer:
.add database myDB ingestors ('aadapp=;')
'theAADAppname'
.add table PressureRecords ingestors ('aadapp=;') 'theAADAppname'
.add table TemperatureRecords ingestors ('aadapp=;') 'theAADAppname'
My code:
var kcsbDM = new KustoConnectionStringBuilder($"https://ingest-{serviceNameAndRegion}.kusto.windows.net:443/").WithAadApplicationKeyAuthentication(
applicationClientId: "<my AD app Id>",
applicationKey: "<my App Secret from Certificates & secrets>",
authority: "<my tenant Id>");
using (var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(kcsbDM))
{
var ingestProps = new KustoQueuedIngestionProperties(databaseName, tableName);
ingestProps.ReportLevel = IngestionReportLevel.FailuresAndSuccesses;
ingestProps.ReportMethod = IngestionReportMethod.Queue;
ingestProps.JSONMappingReference = mappingName;
ingestProps.Format = DataSourceFormat.json;
using (var memStream = new MemoryStream())
using (var writer = new StreamWriter(memStream))
{
var messageString = JsonConvert.SerializeObject(myObject); // maps to the table / mapping
writer.WriteLine(messageString);
writer.Flush();
memStream.Seek(0, SeekOrigin.Begin);
// Post ingestion message
ingestClient.IngestFromStream(memStream, ingestProps, leaveOpen: true);
}
The issue is that the mapping you are using in this ingestion command does not match the existing table schema (it has additional columns). In these cases Azure Data Explorer (Kusto) attempts to add the additional columns it finds in the mappings. Since the permission that the app has is 'ingestor', it cannot modify the table structure and thus the ingestion fails.
In your specific case, your table has a column that is written in a specific casing and in the ingestion mapping the same column has a different casing (for one character) so it is treated as a new column.
We will look into providing a better error message in this case.
Update: the issue is fixed in the system and now it works as expected.
Avnera thanks for your hint, potential it is an issue because of the Real vs double translation. In one of my first try I used double in the table and that worked. That is not longer possible, looks the supported data types changed.
My current configuration:
.create table PressureRecords ( Timestamp:datetime, DeviceId:guid, Pressure:real )
.create-or-alter table PressureRecords ingestion json mapping "PressureRecords"
'['
'{"column":"TimeStamp","path":"$.DateTime","datatype":"datetime","transform":null},'
'{"column":"DeviceId","path":"$.DeviceId","datatype":"guid","transform":null},'
'{"column":"Pressure","path":"$.Pressure","datatype":"real","transform":null}'
']'
public class PressureRecord
{
[JsonProperty(PropertyName = "Pressure")]
public double Pressure { get; set; }
[JsonProperty(PropertyName = "DateTime")]
public DateTime DateTime { get; set; } = DateTime.Now;
[JsonProperty(PropertyName = "DeviceId")]
[Key]
public Guid DeviceId { get; set; }
}

Resources