TrawlerFetchReplyDataFetchStats

TrawlerCrawling

GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchReplyDataFetchStats

6
out of 10
Medium
SEO Impact
The fetcher keeps track of various time intervals spent in the states of the fetcher url control flow. Sometimes flows branch out into more than one flow (next flow), and we aggregate all the time intervals spent in a specific state for all the flows to get the time interval for the state. The time interval for a state is therefore not the time interval of an individual flow, if the WaitNextFlow time is non zero. The WaitNextFlow interval included here is the time spent by one flow waiting for another flow. Although the WaitNextFlow time is the time spent by a flow in the WaitNextFlow state, its also equivalent to the entire timeline of another flow. It is the time spent in the various states of another flow. In computing the total time spent in the primary flow, one must omit the WaitNextFlow time, since it is already included in the form of slices of individual state time intervals in the aggregated time intervals for other states, and will result in double counting.

SEO Analysis

AI Generated

Part of Google's web crawling infrastructure (Trawler is Google's internal name for their web crawler). This model governs how Googlebot fetches and processes web pages, affecting crawl efficiency and frequency. Crawl management directly impacts how quickly new content is discovered and how often existing content is refreshed in the index.

Actionable Insights for SEOs

  • Optimize crawl budget by fixing broken links and reducing redirect chains
  • Use robots.txt and sitemap.xml effectively to guide crawling
  • Monitor Google Search Console for crawl errors and indexing issues

Attributes

9
Sort:|Filter:
BotOverheadMSinteger(
Default: nil

Overhead spent RPCing with the Bot/proxy.

ClientControlflowStatsTrawlerFetchReplyDataFetchStatsClientStateStats →
Default: nilFull type: GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchReplyDataFetchStatsClientStateStats.t
ConnectTimeMsinteger(
Default: nil

Report only with first request on connection, so that we keep track of the connect time with a host. Sometimes a connection is initiated by a prior request that times out before the connection is established. Another request can get scheduled on a connection that is already in the process of being established but has no request scheduled onto it. We want to keep track of the entire connect time even if a request didn't need to wait for the entire connection establishment time. Sometimes a connection may get established before the first request uses it. We tag along the connect time with the first request using the connection. ConnectTimeMs also includes SSL negotiation time.

ControlflowStatsTrawlerFetchReplyDataFetchStatsStateStats →
Default: nilFull type: GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchReplyDataFetchStatsStateStats.t
DownLoadTimeinteger(
Default: nil

DownLoadTime = Share of connect time + ServerResponseTimeMs + TransferTimeMs (see below) in ms

EdgeEgressOverheadMsinteger(
Default: nil

Overhead spent routing the request from HOPE to edge egress nodes, which open connection to webservers. This is only set for edge fetches (e.g., through Floonet egress nodes).

SSLConnectTimeMsinteger(
Default: nil

ConnectTimeMs includes TCP connect time + SSL time, whereas SSLConnectTimeMs includes only the latter.

ServerResponseTimeMsinteger(
Default: nil

Time between the request send and the receipt of the first fragment of the response. For HTTP responses the first fragment is the first fragment of the response payload (the headers are ignored).

TransferTimeMsinteger(
Default: nil

Time to receive the entire response payload starting the clock on receiving the first fragment.