Category · 33 articles

Crawling

Crawling, URLs, Indexing, and Ranking: The Keys to SEO

Click here to learn what Crawling means for SEO.

Categories 9

Articles

Crawling 3 min read

A paywall for AI is not viable yet

Suganthan Mohanadasan built a site that charges AI agents a penny per page via x402. Only the AIs he had configured paid; the rest stopped at the 402.

Crawling 20 min read

Reduce Google’s crawling of our website

Google ignores Crawl-delay. How to slow down its crawling: block fake Googlebots by IP and return 503 or 429 with Retry-After on Apache, LiteSpeed and nginx.

Crawling 2 min read

Block LLM in a website

Why should we block an LLM on our website? Can’t it help us generate more leads?

Crawling 22 min read

Sitemaps

Everything you need to know about how to create and interpret a sitemap

Crawling 8 min read

SEO rendering

How Google renders a website: HTML, CSS and JS all matter

SEO Crawling

Understanding crawling is essential for improving the online ranking of a professional website, as it is one of the processes that Google carries out before proceeding with indexing.

Crawling can be described as the process that bots perform on a website, where they navigate through all its URLs, storing content, and eventually rendering (understanding) it before indexing.

If a page is not indexed, we can’t even begin to think about competing in search engine rankings. Therefore, crawling is one of the fundamental pillars of this discipline.

Controlling the crawling of a web project will give you greater flexibility to achieve the desired results.

Training
Interested in Advanced SEO Training?

I currently offer advanced SEO training in Spanish. Would you like me to create an English version? Let me know!

Tell me you're interested

Recommended Posts

Metatags 10 min read

Meta Refresh & http-equiv

The meta refresh is an HTML tag that can be used to automatically refresh a webpage or redirect to another page. How does it affect SEO?

Technologies 9 min read

How Does a Website Work?

Discover the basic functioning of a website and the technologies used, and how to apply them to SEO