The indexing system, Caffeine, does multiple things:
1. ingests fetchlogs,
2. renders and converts fetched data,
3. extracts links, meta and structured data,
4. extracts and computes some signals,
5. schedules new crawls,
6. and builds the index that is pushed to serving.
If something goes wrong with most of the things that it's supposed to do, that will show downstream in some way. If scheduling goes awry, crawling may slow down. If rendering goes wrong, we may misunderstand the pages. If index building goes bad, ranking & serving may be affected
Don't oversimplify search for it's not simple at all: thousands of interconnected systems working together to provide users high quality and relevant results. Throw a grain of sand in the machinery and we have an outage like yesterday.
before someone jumps in with a but, but, but, this is not an exhaustive list
You can follow @methode.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: