Collect performance metrics using Google Cloud Ops Agent and send to Google Cloud Monitoring - google-cloud-monitoring

Question

I'm looking for a general way to collect performance metrics on several Linux VM instances (Azure, GCP, other) and monitor the metrics in GCP.
On an Ubuntu VM in Azure, I have installed Google Cloud Ops Agent, which uses fluentd (to collect logs) and OpenTelemetry (to collect performance metrics) behind the scenes.
I added overrides for the two services to set environment variables so that they pick up the service account JSON credentials file, as follows:
google-cloud-ops-agent-fluent-bit.service 🠒 GOOGLE_SERVICE_CREDENTIALS
google-cloud-ops-agent-opentelemetry-collector.service 🠒 GOOGLE_APPLICATION_CREDENTIALS
See this post for more details on authentication.
I could see log messages appearing in Google Cloud Logging, which must have been scraped and sent by google-cloud-ops-agent-fluent-bit.service. However, I couldn't find any performance metrics from google-cloud-ops-agent-opentelemetry-collector. Where should I expect to find these in GCP? I'm convinced that there is some additional configuration I need to get this working, but the documentation seems to be about getting Ops Agent running on GCP Compute Engine instances.
Update 1:
I can see that the service is running (sudo systemctl status google-cloud-ops-agent-opentelemetry-collector.service), but I now notice errors that I hadn't noticed before which might suggest why metrics are not making it to Google Cloud,
exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "googlecloud", "error": "rpc error: code = InvalidArgument desc = One or more TimeSeries could no
t be written: No matching retention policy was found for one or more points.: timeSeries[0]\nerror details: name = Unknown desc = total_point_count:1 errors:{sta
tus:{code:9} point_count:1}", "interval": "5.52330144s"}
I don't know where to find the logs for the service other than the excerpt printed by systemctl status.
The commandline for the service is /opt/google-cloud-ops-agent/subagents/opentelemetry-collector/otelopscol --config=/run/google-cloud-ops-agent-opentelemetry-collector/otel.yaml. I took a look in the config file and see a few mentions of googlecloud as an exporter, e.g.
exporters:
googlecloud:
metric:
prefix: ""
user_agent: Google-Cloud-Ops-Agent-Metrics/2.11.0 (BuildDistro=focal;Platform=linux;ShortName=ubuntu;ShortVersion=20.04)
Update 2: Output of service status
● google-cloud-ops-agent-opentelemetry-collector.service - Google Cloud Ops Agent - Metrics Agent
Loaded: loaded (/lib/systemd/system/google-cloud-ops-agent-opentelemetry-collector.service; static; vendor preset: enabled)
Drop-In: /etc/systemd/system/google-cloud-ops-agent-opentelemetry-collector.service.d
└─override.conf
Active: active (running) since Tue 2022-03-15 06:36:44 UTC; 1 day 17h ago
Process: 1053790 ExecStartPre=/opt/google-cloud-ops-agent/libexec/google_cloud_ops_agent_engine -service=otel -in /etc/google-cloud-ops-agent/config.yaml -logs ${LOGS_DIRECTORY} (code=exited, status=0/SUCCESS)
Main PID: 1053796 (otelopscol)
Tasks: 10 (limit: 19198)
Memory: 381.2M
CGroup: /system.slice/google-cloud-ops-agent-opentelemetry-collector.service
└─1053796 /opt/google-cloud-ops-agent/subagents/opentelemetry-collector/otelopscol --config=/run/google-cloud-ops-agent-opentelemetry-collector/otel.yaml
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: /root/go/pkg/mod/go.opentelemetry.io/collector#v0.44.0/exporter/exporterhelper/metrics.go:134
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: /root/go/pkg/mod/go.opentelemetry.io/collector#v0.44.0/exporter/exporterhelper/queued_retry_inmemory.go:105
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: /root/go/pkg/mod/go.opentelemetry.io/collector#v0.44.0/exporter/exporterhelper/internal/bounded_memory_queue.go:99
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: /root/go/pkg/mod/go.opentelemetry.io/collector#v0.44.0/exporter/exporterhelper/internal/bounded_memory_queue.go:78
Mar 16 23:47:37 HOSTNAME otelopscol[1053796]: 2022-03-16T23:47:37.980Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "googlecloud", "error": "[rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: No matching retention policy was found for one or more points.: timeSeries[0-199]\nerror details: name = Unknown desc = total_point_count:200 errors:{status:{code:9} point_count:200}; rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: No matching retention policy was found for one or more points.: timeSeries[0-199]\nerror details: name = Unknown desc = total_point_count:200 errors:{status:{code:9} point_count:200}; rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: No matching retention policy was found for one or more points.: timeSeries[0-199]\nerror details: name = Unknown desc = total_point_count:200 errors:{status:{code:9} point_count:200}; rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: No matching retention policy was found for one or more points.: timeSeries[0-111]\nerror details: name = Unknown desc = total_point_count:112 errors:{status:{code:9} point_count:112}]", "interval": "10.435795045s"}
Mar 16 23:47:49 HOSTNAME otelopscol[1053796]: 2022-03-16T23:47:49.299Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "googlecloud", "error": "rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: No matching retention policy was found for one or more points.: timeSeries[0-4]\nerror details: name = Unknown desc = total_point_count:5 errors:{status:{code:9} point_count:5}", "interval": "44.913550864s"}