How to use custom jars in spark-submit --packages - apache-spark

Question

I have a PySpark project which is doing spark structured streaming, to get the query metrics I have created a java project which listens to the microbatch events and logs the data in a log files. The logging works when I pass the jar as a file and using --jar to read it in spark-submit.
But, this process has manual work involved after where jar has to be uploaded manually. To solve it
uploaded the jar in JFrog repository. Now when running the spark-submit command I have added --repositories and also added the coordinate in --packages which already includes few packages like kafka, avro etc. All the packages downloads from the jfrog but when it reaches myjar it throws below error, but the repo url from log if if I try from browser it actually downloads the jar and pom as well!!!
:: problems summary ::
:::: WARNINGS
module not found: <myjar>;<version>
==== central: tried
https://<repo>/myjar.pom
-- artifact <myjar>.jar:
https://<repo>/myjar.jar
==== repo-1: tried
https://<repo>/myjar.pom
-- artifact <myjar>.jar:
https://<repo>/myjar.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: com.spark.extension#<myjar>;<version>: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.spark.extension#<myjar>;<verion>: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1428)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:902)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1038)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1047)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Command exiting with ret '1'
EDIT-
Taken from EMR (some of the url/names are omitted)-
spark-submit --name "A Adapter" --deploy-mode cluster --master yarn --repositories https://<jfrog repo>/artifactory/all/ --packages com.spark.extension:spark-listeners:0.3.8,org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.1,org.postgresql:postgresql:42.2.22,software.amazon.cloudwatchlogs:aws-embedded-metrics:2.0.0-beta-1 --driver-cores 2 --driver-memory 12g --executor-memory 12g --num-executors 1 --executor-cores 2 --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.minExecutors=1 --conf spark.dynamicAllocation.maxExecutors=6 --files s3://<url>/log4j.properties,s3://<url>/logging.json --conf spark.yarn.dist.archives=s3://<url>/libs.zip#app-site-packages --conf spark.yarn.appMasterEnv.PYTHONPATH=app-site-packages --conf "spark.yarn.appMasterEnv.SPARK_APP_NAME=A Adapter" --conf spark.yarn.appMasterEnv.CLUSTER_STUB=dev-b1 --conf "spark.yarn.appMasterEnv.AWS_EMF_SERVICE_NAME=A Adapter" --conf spark.yarn.appMasterEnv.AWS_EMF_SERVICE_TYPE=dev-b1-emr --conf spark.yarn.appMasterEnv.AWS_EMF_LOG_GROUP_NAME=dev-b1-spark-structured-stream-logs --conf spark.yarn.appMasterEnv.AWS_EMF_LOG_STREAM_NAME=dev-b1-spark-structured-stream-logs --conf spark.yarn.appMasterEnv.AWS_EMF_AGENT_ENDPOINT=udp://127.0.0.1:25888 --conf spark.driver.extraJavaOptions= --conf spark.executor.extraJavaOptions= --conf spark.executorEnv.PYTHONPATH=app-site-packages --py-files s3://<url>/libs.zip,s3://<url>/jobs.zip,s3://<url>/.env s3://<url>/main.py --job acc