Getting java.io.IOException: Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/ Error - apache-spark

Question

I am able to fetch the data from Bigquery using gcs-connector and spark-bigquery-in Spark application. But getting below error while trying to load data into Bigquery in GCP by using spark application.
Exception in thread "main" java.io.IOException: Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
Code:
df_bigquery.write.format("bigquery").option("credentialsFile", "D://input/absolute-vertex-321015-a78e81ae77a0.json").option("parentProject", "absolute-vertex-321015").option("temporaryGcsBucket","emp_demo_1").save("absolute-vertex-321015.org.employee_loaded")
Any help is deeply appreciated.
Thanks in Advance.

Answer 1

May I help you with this?
This Metadata endpoint /computeMetadata/v1/instance/service-accounts/default/token Returns the auth token that can be used to authenticate your application to other Google Cloud APIs.
As the error is: Exception in thread "main" java.io.IOException: Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
Use curl to request an access token and send a request to an API:
On the instance where your application runs, query the metadata server for an access token by running the following command:
$ curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"
The request returns a response similar to:
{
"access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
"expires_in":3599,
"token_type":"Bearer"
}
Copy the value of the access_token property from the response and use it to send requests to the API. For example, the following request prints a list of instances in your project from a certain zone:
$ curl https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances \
-H "Authorization":"Bearer [ACCESS_TOKEN]"
where:
[PROJECT_ID] is the project ID for this request.
[ZONE] is the zone to list instances from.
[ACCESS_TOKEN] is the access token value you got from step 1.
Note:
You can use the access token only for scopes that you specified when you created the instance. For example, if the instance has been granted only the https://www.googleapis.com/auth/storage-full scope for Cloud Storage, then it can't use the access token to make a request to BigQuery.
For information about the parameters that you can set in your request, see the parameters documentation.
If you want to Authenticating applications with a client library use this reference.
I am happy to help, let me know if these steps resolve your issue.