TrawlerTrawlerPrivateFetchReplyData

What's the post data size (in bytes) if it's a post request.

Number of times we drop the content of a stream reply or the final reply, which can only be caused by REJECTED_NO_RPC_BUFFERS now.

If we do not have Endpoints in FetchReplyData (e.g., url rejected due to hostload limit), do we have a guess of the server IPAddress (e.g., from robots fetch)? This helps us classify URLs based on country code, etc. The field is filled with IPAddress::ToPackedString().

RPC deadline left at the start of url control flow. Can be useful for debugging rpc deadline exceeded error received by clients, this field is only recorded if RpcEndDeadlineLeftMs is small enough.

Set to the hit location (CNS filename) if cache comes from large store.

Dependent fetch type

Set if the fetch goes through the virtual private cloud path so we can track the VPC traffic.

Stores the HTTP version we used in the last hop.

If we fetched using BotFetchAgent, what is the BotGroupName?

Whether this is a bidirectional streaming fetch.

Stores the OAuth authentication method.

Log the loas username in trawler private to help with debugging. Store the username in trawler private so clients won't see it from FetchReply. To reduce disk usage, we only log the loas username if the requestorid being used does not have ClientUsernameRestrictions.

Only set if the fetch uses cache content (is_cache_fetch is true).

Store the original client information.

Was this an internally-initiated robots.txt fetch?

If the requestor shares resource bucket with other requestorids, we will store the resource bucket name in these fields.

Corresponds to AcceptableAge field in FetchParams.

Note TrawlerPrivateFetchReplyData is never sent back to clients. The following field is just for Trawler and Multiverse internal tracking, and clients should not look at this field at all.

If set, this fetch was done through a proxy (e.g., fetchproxy).

How many concurrent streams are on the connection when the request finishes (including this request). Export this value to monitor the stream multiplexing for HTTP/2.

Corresponds to AcceptableAfterDate field in FetchParams.

Log the credential id

The number of bytes we sent back to the client.

If the response header contains Content-Disposition header "attachment; filename="google.zip": the download_file_name would be "google.zip"

Whether or not this is a Floonet fetch request. Floonet requests have inherent lower availability (due to HOPE rejections when HOPE is in degraded mode, and other Floonet specific reasons). Therefore, it is important for debugging and for our availability SLO to know whether of not it is a floonet fetch. IMPORTANT NOTE: This field is only currently set for traffic that explicitly requires Floonet and can not failover to use Googlebot (i.e. "transparent" or "implicit" Floonet fetches).

Multiverse client information

Which Trawler cell was this response fetched in? (e.g. "HR" or "YQ")

HTTP Strict-Transport-Security (RFC6797) header value. We log this so we can generate a list of hosts that prefer HTTPS over HTTP.

Service tier info will be used in traffic grapher for ploting per tier graph.

Represents if the HostId belongs to HostId set in 5xx url patterns, it can work as a tag when emitting requestor minute summary, this helps us to aggregate traffic affected by 5xx patterns, and test if there are any fetching changes.

The useragent string sent to the remote webserver. It corresponds to UserAgentToSend field in FetchParams.

We check if Google-Extended is allowed to crawl this URL, wildcard rules are obeyed, this is for internal analysis. Check RobotsTxtClient::RobotsStatus for the meaning of number.

If this was a robots.txt fetch (IsRobotsFetch above), this may contain the robots.txt body. (It may not, for instance, 404s are omitted; current policy is URL_CRAWLED + partially crawled) This includes http headers + body.

The fp2011 of useragent sent to the remote webserver, note it corresponds to UserAgentToSend field in FetchParams

Log the prod region (only for regional harpoon requestor ids)

RPC deadline left at the end of url control flow. Can be useful for debugging rpc deadline exceeded error received by clients, this field is only recorded if it's small enough.

Whether or not this response is sent from gRPC proxy service.

An arbitrary string signature identifying the remote server type/version. In the case of HTTP, this would be the contents of the "Server:" header.

We check if Google-Extended is allowed to crawl this URL and store the result here, wildcard rules are not obeyed, this is for internal analysis. Check RobotsTxtClient::RobotsStatus for the meaning of number.

This is the HOPE server that we sent the url to. We log the HOPE backend cell and hope server shard number (e.g., 'qf:6'). This allows us to understand how we are balancing our load to the HOPE servers.

The following are vpc information that's only set if is_vpc_traffic is true.

Cache hit for this url, bypassed host_overfull error.

Present if the reply is from the trawler cache. This is the requestorid of the trawler client that populated the cache with the data we are reusing.

Which Trawler fetcher task fetched this URL.