Google 抓取工具 会递归的追踪网页中含有 href 属性的 <a> 标签的链接

flowchart TB
  capture[定时抓取]
  index[编入索引到数据库]
  show[呈现]

	subgraph pages[页面]
		direction TB
		网页1
		网页2
		...
	end
	capture --> pages-->index --> show
  

参考文档:

https://developers.google.com/search/docs/advanced/guidelines/get-started

https://moz.com/learn/seo/robotstxt