Databricks not able to create table in minio bucket - apache-spark

Question

Trying to create table in minio bucket using databricks.
spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");
I am passing the s3 configurations using spark context.
access_key = 'XXXX'
secret_key = 'XXXXXXX'
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", access_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", secret_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "http://my-ip:9000")
Can anyone please point out the configs lacking here for table creation?
With this config I am able to write data in s3 using
df.write.format("parquet").save("s3a://my-bucket/file-path");
But it's throwing exception when I m trying to create table/database;
spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://my-bucket/my-database: getFileStatus on s3a://test2/minio_db_1: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://test2.s3.us-east-1.amazonaws.com minio_db_1 {} Hadoop 3.3.4, aws-sdk-java/1.12.189 Linux/5.4.0-1093-aws OpenJDK_64-Bit_Server_VM/25.345-b01 java/1.8.0_345 scala/2.12.14 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 6YBEAZY59EYGAEVB, Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=, Cloud Provider: AWS, Instance ID: i-072d1969af3c17cb6 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 6YBEAZY59EYGAEVB; S3 Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=; Proxy: null), S3 Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=:403 Forbidden)
The request should routed to the s3a endpoint, but it's routing to the generic s3 endpoint. Somehow spar.sql not honouring the spark context configurations.