As for using a single URI please see the "Using a single Identifier for each GCP IO’s MonitoringInfo Labels section.

As for calling multiple APIs, I was planning on focusing on the calls specifically to do batch write and batch reads. As looking at the throughput, errors over time, etc and latency are most meaningful for those.

I do see your point though, and this is a good question. Should we create separate metrics for those separate calls?

One approach I have seen is to have a method label on the metric. So if you had 3 calls open, write, and flush. Then the method could be added as a label.

But that would mean a system building a UI would need to deal with 3 metrics. It may be better to just make sure the errors from the open and flush are exposed well through other means. Like as visible error logs

Had a thought. For a case like that. Maybe we should only count the request counts of those separate but related API requests for read/writes. But only collect latency for the batchRead/batchWrite API calls?

@millsd@google.com WDTY about this one?

I think every service call should have a distinct count + latency metric

Adding Method as a label on both count and latency seems like a reasonable approach for these sorts of IOs

As for the question about using GCP Resource Names. I took a look into and don't think its a good idea. Please see the reasoning in "Using the GCP Resource Name..." section.

I've added the method label for each of the APIs.

I also added a BIGQUERY_QUERY_NAME label and suggested an update to the BigQueryIO.read() to allow specifying the query label.

PTAL at current proposal. I am suggesting to use a single resource label, by either concatenating the others. Or using whatever standard exists today as params to the API.

I am saying to use the human readable string names. So that these can be shown to a user. Plus gRPC based APIs don't offer the HTTP code. so we should avoid that.

API_REQUEST_COUNT is fine, DONE

I have the same comment just above where we should be using a resource URI + method name as labels. Most services have well defined resource URIs for things like s3 paths/bigquery tables/...

I added a section about this to the Alternatives Considered. PTAL and let me know what you think. There was some thought around this in the past.

Thanks for adding the section. Added some comments there. I still prefer consolidation.

I don't really recommend it, as we are struggling to find a common single identifier convention to use for all of these services. I looked into a couple of approaches.

Most service of them have something we could use, but in some cases we will be making one up.
And what they are doing now may be replaced in the future with GCP Resource Names, but that's not widely adopted in GCP.

Additionally, Google Monitoring will require parsed names to match other GCP metric formats. So, RHs will need to parse these out anyways. In order to report them to metric collection services such as Google Monitoring, to get the proper metric breakdowns.

I did add a GENERAL section, where you can use a single resource label, which would be good for a custom/non GCP IO, such as Kafka IO to record metrics on these RPCs.

Adding a GENERAL that is a sibling to BIGQUERY, DATASTORE, ... but represents many possible sources as distinct as the GCP ones doesn't feel like the right design. Kafka should be a sibling of PubSub, not a "nephew."

In general, I think the most flexible is a system that allows one to forward, query, filter, aggregate metrics as generically as possible, and a system that knows/cares about the particular differences between the various IO metrics can interact with the various subtypes individually. (One way to phrase this is that there's a generic interface here, and various concrete subtypes.) I do no think it makes sense to try to parse a "resource identifier" to determine, e.g., if one is looking at a bigtable or datastore metric (e.g. the "type" of IO in question could be a label). But it would be useful to be able to query "what resources are the slowest/returning the most errors/etc" without writing a query (let's imagine one can write SQL against the set of counters) that is a bunch of "union" statements of nearly, but not quite, identical selects, and needs to get modified every time a new source should be considered.

Maybe the set of precise labels could vary from source to source, but the common labels would remain the same.

@robertwb@google.com @lcwik@google.com

PTAL and LMK what you think :). I am proposing using a single resource label always. But having optional ones. So a RH could choose whether or not it just wants to deal with the single label. Or to use the parsed labels.

Yes, process metric is the same thing as a non-bundle metric. Any preference for the terminology here :)? I can update the doc to try and use the same term everywhere.

The SDKs should not be picking any label format. I don't mean to imply this. I only wish to prevent RunnerHarnesses from writing string parsing code. So I am requesting we use the RESOURCE as a fully qualified name. And additional labels which have the same information parsed out.

LMK if you think this is okay. If the redundancy is an issue, then I think we can remove all the additional labels and only use a RESOURCE label.

Yes, this is intentionally redundant, but having to write this string parsing can be burdensome and has been an issue in the past before dataflow built a structured metric format. So we have avoided that approach. In the past metrics were named with very large strings that alway needed parsing to make sense of their parts.

String parsing is an issue if this happens a lot to cause perf issue or it is not well defined leading to parsing complexity. These resource strings in the current format should be able to be done with any regex library that can pull out groups.

Also, duplicating the information leads to different perf problems (additional storage and more bytes being passed around).

Now that we got a bit more clarification. I' updated this section to include the recommended formats if available. And for the remainder I invented some similar to that format. I think we can use those.

BigQuery View: "//bigquery.googleapis.com/projects/{projectId}/datasets/{datasetId}/views/{viewId}"BigQuery Query: "//bigquery.googleapis.com/projects/{projectId}/queries/{queryName}"Datastore Namespace: "//bigtable.googleapis.com/projects/{projectId}/namespaces/{namespaceId}"Spanner Database: "//spanner.googleapis.com/projects/{projectId}/databases/{databaseId}"

It would be good to use a made up namespace to prevent any future conflicts if an official one is used in the future

Possible alternative: we could add explicit methods for the new metric types like GcpMetrics.BigQueryRequestCounts(String status, String table, String dataset, String project)

I think creating a "service call" object that supports metric makes sense and you can have it be a try with resources kind of thing:
try (ServiceCall call : serviceMetric.newCall()) {
status = MakeAnRpc();
call.withStatus(status);
}

This would wrap the count/timing metrics into one.

I like this, although it may need to support direct access too for IOs that use async rpcs

Yeah, async will have to follow another pattern.

Incorporated this into the example. Still an open question as to what might happen inside ServiceCall, which may need to take a counter, and create other counters from it (by only changing the status label).

HarnessMonitoringInfosRequest sounds good to me

I encorporated this idea into the proposal. Suggesting we have non bundle metrics, which we will use for these GCP IOs.

Correct. The auto generated subscription is not the most meaningful. Though it will have a substring with the topic id.

Its important to collect both, as they can be quite different, when specified. Even in different GCP projects.