Before we get into details of the five technical SEO techniques, I want to clarify the difference between crawling and indexing a webpage. The two terms are used interchangeably but they are distinctively different.

• Crawling is the act of finding a page on your website to analyze the content.

• Indexing is the storage of that page and information related to it within search engine’s database (e.g. Google’s database).

In the case of a Googlebot crawling and indexing a webpage, crawling refers to the act of finding a webpage on a website and analyzing its content while indexing is downloading and storage the webpage’s content in Google’s database. When a user conducts a search on Google search engine, Google performs a query in its database to find relevant webpages using its algorithm and lists the webpages on Google search results. Ensuring the crawlability of your website is critical to your SEO because it is the first step in optimizing your website’s search ranking.

To increase the crawlability of your website, use these following five techniques:

1. Limit JavaScript Usage for Critical Content

   According to Google’s Senior Webmaster Trends Analyst, John Mueller, Googlebots have a difficult time crawling content that is in JavaScript.

   When Google crawls and indexes content, it does two passes. The first pass looks at HTML only and a second pass looks at the whole site. The second pass could be weeks after the first pass and it may not be able to crawl all content that is in JavaScript.

   It is highly recommended to write critical content (important content related to SEO and your company such as products or services you offer) in HTML and not in JavaScript so Google can crawl the content in the first pass without missing any critical content.

2. Optimize Robots.txt File

   “A robots.txt file is a text file which is read by search engine spiders and follows a strict syntax. These spiders are also called robots – hence the name – and the syntax of the file is strict simply because it has to be computer readable. That means there’s no room for error here – something is either 1, or 0.

   Also called the “Robots Exclusion Protocol”, the robots.txt file is the result of a consensus among early search engine spider developers. It’s not an official standard set by any standards organization, but all major search engines adhere to it.”

   – Yoast SEO

   Use robots.txt to block bots from crawling low-value sections on your website such as the WodrPress login or directory page. The reason why you want to block bots from crawling such pages is because bots are given a “crawl budget” which is the limited amount of time and resources they are given to crawl a website. If bots spend time crawling low-value pages, they may not have the time or resources to crawl critical pages. You want bots to spend time and resources crawling high-value pages such as your services and product pages.

3. Optimize Website Structure

   A website structure refers to the organization of the website content. Generally, a website is organized by grouping similar content together. For example, all product pages would be linked under the product category page and team bios would be linked under the About Us section.

   By using taxonomies such as categories and tags, internal links, navigation bars, and breadcrumb, you help bots crawl webpages on your website more easily because Googlebots can navigate from one page to another on your website logically.

4. Generate and Submit XML Sitemap

   A XML sitemap is a file where you can list the web pages of your site to tell Google and other search engines about the organization of your site content. Search engine web crawlers like Googlebot read this file to more intelligently crawl your site. Missing a XML sitemap prevents a website from achieving the benefits above.

   If you have a WordPress website, you can install a SEO plugin called Yoast and the plugin will generate an XML sitemap for you. Once the XML sitemap is generated, add it to your Google Search Console account (if you have any questions submitting an XML sitemap to your Google Search Console, email me at raywang@rwdigital.ca).

5. Redirect Soft 404 Pages

   A soft 404 is a URL has little or no content (a completely empty page or a page with only the navigation menu). The page provides no value for website visitors and it wastes Googlebots’ time and resources to crawl an essentially non-existent URL instead of crawling webpages with valuable and unique content. The page may also be indexed by Googlebots and appear on search results. If a user clicks on the soft 404 error page on Google SERP, s/he will be disappointed by the result which will hurt the user-experience.

   To fix soft 404 page errors, first discover these pages in your Google Search Console account (the Google tool provides such data) and redirect the pages to respective webpages. For example, if product A URL has a soft 404 error page, redirect product A page to the product category page that product A falls under.

   The crawlability of your website is crucial to your business and website because if Googlebots and other prominent bots are not able to crawl your webpages properly, how can the bots index the pages and list them on search engine results page? Missing some pages search engine pages means a loss of organic traffic, brand exposure, and more importantly, customers.

   If you have any questions about the crawlability of your website or digital marketing in general or want an audit of your website, please book a complimentary discovery call!