There are a few reasons you may be looking to get your webpage or website indexed, or reindexed on Google. Maybe you just posted a new site or page and want to notify Google that it is ready to be crawled and indexed. Maybe for some reason your page dropped off and you cannot find it on Google so you need their bot to re-crawl and index. Whatever the reason may be, it is always important to use Google Search Console (Webmasters) to monitor the health of your website. Search console will do many things including showing you how many pages on your site Google has indexed, if it has found your robots.txt file and sitemap, if it has found any errors while crawling and more. So make sure to keep your eye on this to always ensure your site is set up for success on Google! I’m going to share a quick story on what led me to write about this, but if you want to dig right into the actionable items on how to index your pages skip down to the bold ‘How to Index Your Pages’ section below.
What got me thinking about this topic? Well, as a digital marketer and project manager for e-commerce websites I am always concerned about SEO and Google Search presence. I ensure that I set up any new site we launch or takeover on Google Webmasters and go through all the steps to get each page indexed. So even though I am always checking these stats for my clients, it is not what brought me back to check indexing.
The issue in fact was found by my team and I while doing an end of year review on the website. After checking Google Analytics we noticed that our paid search had taken an unexpected leap in spend (which wasn’t bad, we had the budgets for it), but our organic search had seen a steady decline. Now this wasn’t necessarily reason for panic but definitely worth a little research. After some digging we started noticing that on the homepage of Google, for our exact brand search where we should always be top if not top 3 we were nowhere to be found. What was showing up? Our different country locations. So we were seeing www.site.ca, www.site.co.uk, www.site.br etc. But for any searches we did ours did not show up. This led to more investigating and me digging further to find out why our pages were delisted. Thoughts came up around full site SSL changes, code releases, bad sitemap structure and more. My job was to make sure webmasters was still properly configured which it ended up being. In the end we found out that during a code release, somehow the metatag in our header for every page had changed from <meta name=”robots” content=”index,follow“> to <meta name=”robots” content=”noindex,follow”>. From this code snippet we see that our site is telling Googlebot to not Index our site.. Not what we want! So all we had to do was let the developers know to change this back to index,follow. Who made this change? Still a mystery. Now into how to index your pages or request a reindex if you have noticed any pages drop off. Always make sure to check different possibilities as to why the page may not be indexed, your robots metatag being a big one!
How to Index Your Pages
To have Googlebot crawl your website all you need to do is make sure that your website is live and the robots metatag in your site head is set to index,follow. If this is set up, Google should naturally starting finding your web pages and crawling / indexing them shortly after. It is never a bad idea though to check to see which pages have been indexed successfully or if you want to manually ask the Googlebot to crawl a webpage. To view which pages of your site Google has indexed, first log into the Google Search Console https://www.google.com/webmasters/.
This is the dashboard. If you have used Webmasters before and registered a domain you will see them listed here. If not, you will need to select ‘Add a Property’ in the top right and verify your ownership. They will give you various options for this but I recommend verifying using the Alternative Methods > Google Analytics method if you have set up Google Analytics. It will recognize your email and that you have it set up and confirm ownership immediately. Once you click into the domain you will see this screen:
This is your main Dashboard for your domain and will give you a snapshot of errors, analytics, sitemaps and news and updates. From here, click on Google Index > Index Status. This will show you which pages have been indexed.
To manually show Google your pages, navigate to Crawl > Fetch as Google.
This allows you to enter the specific URL you wish for Google to crawl. Once entered hit fetch. After being fetched a button will show up that says request indexing. Click this.
Verify you are not a robot using the Captcha, and then I recommend hitting Crawl this URL and it’s direct links in order to have Google crawl any URL’s linked on that page as well as the page itself.
Now you have requested for Google to crawl and index this page! Indexing can take a few days or longer so I recommend checking back a few days after to make sure the page has been picked up.
I hope this helps! Feel free to shoot me any questions you may have about indexing on Google.
Keep on learning.