AnchorsAnchor

used for history - the first and last time we have seen this anchor. creation_date also used for Freshdocs Twitter indexing, a retweet is an anchor of the original tweet. This field records the time when a retweet is created.

Original text, including capitalization and punctuation. Runs of whitespace are collapsed into a single space.

This is a hash of terms near the anchor. (This is a second-generation hash replacing the value stored in the 'context' field.)

If true, the anchor is for experimental purposes and should not be used in serving.

The URL fragment for this anchor (the foo in http://www.google.com#foo)

is to record the quality of the anchor's source page and is correlated with but not identical to the index tier of the source page. In the docjoins built by the indexing pipeline (Alexandria), - Anchors marked TYPE_HIGH_QUALITY are from base documents. - Anchors marked TYPE_MEDIUM_QUALITY are from documents of medium quality (roughly but not exactly supplemental tier documents). - Anchors marked TYPE_LOW_QUALITY are from documents of low quality (roughly but not exactly blackhole documents). Note that the source_type can also be used as an importance indicator of an anchor (a lower source_type value indicates a more important anchor), so it is important to enforce that TYPE_HIGH_QUALITY < TYPE_MEDIUM_QUALITY < TYPE_LOW_QUALITY To add a new source type in future, please maintain the proper relationship among the types as well. TYPE_FRESHDOCS, only available in freshdocs indexing, is a special case and is considered the same type as TYPE_HIGH_QUALITY for the purpose of anchor importance in duplicate anchor removal.

Weight to be stored in linkmaps for pageranker

The bit ~roughly~ indicates whether an anchor's source and target pages are on the same domain. Note: this plays no role in determining whether an anchor is onsite, ondomain, or offdomain in mustang (i.e., the bit above).

The docid of the anchor's original target. This field is available if and only if the anchor is forwarded.

The full context. These are not written out in the linklogs.

true iff exp domain

CATfish tags attached to a link. These are similar to link tags, except the values are created on the fly within Cookbook. See: http://sites/cookbook/exporting/indexing

Contains info on link type, source page, etc.

How the anchor is forwarded to the canonical, available only for forwarded anchors (i.e., the field is set). The forwarding types are defined in URLForwardingUtil (segindexer/segment-indexer-util.h). Always use URLForwardingUtil to access this field and use URLForwardingUtil::GetAnchorForwardingReason to get the explanation how the anchor is forwarded to the canonical. NOTE: Use with caution as it is only set for docjoins generated using the urlmap from repository/updater.

DEPRECATED. It used to be set if firstseen_date is not set. It's to indicate that the anchor is possibly old, but we don't have enough information to tell until the linkage map is updated. TODO(hxu) rename it to possibly_old_firstseen_date_DEPRECATED after clean up other dependencies.

For ranking purposes, the quality of an anchor is measured by its "locality" and "bucket". See quality/anchors/definitions.h for more information.

DEPRECATED

The number of additional links from the same source page to the same target domain. Not populated if is_local is true.

Space-delimited anchor words. Text that needs segmentation (like CJK or Thai) is unsegmented, since we set FLAGS_segment_during_lexing to false in mr-linkextractor.cc .

A given target URL may be found in different encodings in different documents. We store the URL encoding with each source anchor so that we can count them later to find the encoding most likely to be expected by the Web site. Around 0.7% of target URLs are expected to require a non-default value here. The default value 0 is referenced in C++ as webutil::kDefaultUrlEncoding. See also webutil/urlencoding.

The anchor's original target url, compressed. Available only in Alexandria docjoins when the anchor is forwarded.

# days past Dec 31, 1994, 23:00:00 UTC (Unix time @788914800) that this link was first seen. Should never occupy more than 15 bits. NOTE: this is NOT the same as creation_date; firstseen_date is filled during link extraction

TEMPORARY

Additional information related to the anchor, such as additional anchor text or scores.

DEPRECATED: Now in link_tags

true if we think 'firstseen_date' is an accurate estimate of when the link was actually added to the source page. false if it may have existed for some time before we saw it.

Used for history and freshness tracking - the timestamp this anchor is updated in indexing.

This is the offset for the first term in the anchor - it can be used as a unique ID for the anchor within the document and compared against all per-tag data. This is measured in bytes from the start of the document. We write this out to the linklogs to recover the original order of links after source/target forwarding. This is necessary for computing the global related data.

weights are 0-127

Encoded data containing information about newsiness of anchor. Populated only if anchor is classified as coming from a newsy, high quality site. Encoded data for anchor sources are being stored in googledata/quality/freshness/news_anchors/encoded_news_anchors_data.txt Scores are being computed with quality/freshness/news_anchors/ routines.

If the anchor contained images, these image urls are stored here in compressed form.

This field is DEPRECATED and no longer filled. For source page crawl timestamp, use Source.crawl_timestamp. Next tag id should be 62.