TrawlerFetchReplyDataRedirects

TrawlerCrawling

GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchReplyDataRedirects

10
out of 10
Critical
SEO Impact
The sequence of redirects fetched, if applicable. This includes url plus stats for each hop after the first hop. NOTE: This can be one redirect longer than the chain of redirects followed, in the case where there was a redirect at the end of the chain that the fetcher detected but did not follow.

SEO Analysis

AI Generated

Part of Google's web crawling infrastructure (Trawler is Google's internal name for their web crawler). This model governs how Googlebot fetches and processes web pages, affecting crawl efficiency and frequency. Crawl management directly impacts how quickly new content is discovered and how often existing content is refreshed in the index.

Actionable Insights for SEOs

  • Monitor for changes in rankings that may correlate with updates to this system
  • Consider how your content strategy aligns with what this signal evaluates
  • Optimize crawl budget by fixing broken links and reducing redirect chains
  • Use robots.txt and sitemap.xml effectively to guide crawling
  • Monitor Google Search Console for crawl errors and indexing issues

Attributes

18
Sort:|Filter:
BadSSLCertificatestring
Default: nilFull type: String.t

The server SSL certificate chain in SSLCertificateInfo protobuf format. See this field in FetchReplyData (i.e., the initial hop) for more description on when it will be populated.

CrawlTimesTrawlerCrawlTimes →
Default: nilFull type: GoogleApi.ContentWarehouse.V1.Model.TrawlerCrawlTimes.t

Per redirect hop timestamps. This

DownloadTimeinteger(
Default: nil

Download time of this fetch (ms)

EndpointsTrawlerTCPIPInfo →
Default: nilFull type: GoogleApi.ContentWarehouse.V1.Model.TrawlerTCPIPInfo.t

## stats If fetched, ip info.

HSTSInfostring
Default: nilFull type: String.t

This specifies if the url in a redirect was rewritten to HTTPS because of an HSTS policy for the domain. See comments on FetchReplyData.HSTSInfo for how this field's values. A redirect that was rewritten with HSTS will have HSTS_STATUS_REWRITTEN ## here.

HTTPResponseCodeinteger(
Default: nil

The HTTP response code for this hop. We need this since multiple response codes may have the same redirect type (e.g., 302 and 307 are both REDIRECT_TEMPORARILY), but clients may want to know which one was received. Note this is set only for the hops that are followed (i.e., TargetUrl is present). If the last redirect hop was not followed the fetch status will be URL_NOT_FOLLOWED, and the response code will be in the top level ProtocolResponse field.

HopPageNoIndexInfointeger(
Default: nil

Extra trawler::PageNoIndexInfo for this hop. Integer: ORed together bits from trawler::PageNoIndexInfo. The information specified by this field comes from the http header or content of the source url, not the "TargetUrl" in this Redirects group.

HopReuseInfostring
Default: nilFull type: String.t

trawler::ReuseInfo with status of IMS/IMF/cache query, for this hop.

HopRobotsInfointeger(
Default: nil

Extra trawler::RobotsInfo for this hop. Integer: ORed together bits from trawler::RobotsInfo

HostIdstring
Default: nilFull type: String.t

If known, the hostid for this hop

HttpRequestHeadersstring
Default: nilFull type: String.t

The http headers we sent for fetching this redirect hop. Not normally filled in, unless FetchParams.WantSentHeaders is set.

HttpResponseHeadersstring
Default: nilFull type: String.t

The http headers we received from this redirect hop. Trawler does not fill this in; this is intended as a placeholder for crawls like webmirror that fill in and want to track this across redirect hops.

RawTargetUrlstring
Default: nilFull type: String.t

bytes: can contain bad encoding.

RefreshTimeinteger(
Default: nil

Refresh time in meta redirect tag

RobotsTxtstring
Default: nilFull type: String.t

The robots.txt we used for this fetch. Not normally filled in unless WantRobotsBody is set.

SourceBodyTrawlerFetchBodyData →
Default: nilFull type: GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchBodyData.t

For meta-redirects, this field may contain the body of the source document. Currently only filled client side and not implemented (yet) for server-side redirects.

TargetUrlstring
Default: nilFull type: String.t

Difference between the following two fields: TargetUrl is set when we have followed the redirect target, and the url is canonicalized. RawTargetUrl is set in either of the following two cases: (1) The url has not be been followed. For example, the redirect is intended to be handled by the client. In the fetch reply response, you will see the url's status as URL_NOT_FOLLOWED-NOT_FOLLOWED. (2) The extracted redirect url is different from its canonicalized* form. For example, if the target url contains fragments, then this RawTargetUrl will have the fragments. Redirect target

Typestring
Default: nilFull type: String.t

URL and redirect type