Site Audit allows you to notify search engines about important content changes on your website using the IndexNow protocol. This article explains how to set up and use IndexNow within Site Audit.
1. What is IndexNow?
IndexNow is a free protocol that allows website owners to inform search engines about the latest content updates, additions, or removals on their sites. By notifying search engines about these changes, you can ensure that they are aware of the updates instantly, rather than waiting for their bots to crawl and discover the changes themselves. IndexNow is currently supported by Bing, Yep, and other search engines. Learn more about IndexNow on the official website.
2. How IndexNow works
To use IndexNow, you need to prove website ownership by generating an API key and hosting a text file containing this key at the root of your website.
The API key should be between 8-128 characters long and can only contain letters (a-z, A-Z), numbers (0-9), and dashes (-). You need to create a UTF-8 text file at the root of your website with the API key as the file name and its content.
For example, if your API key is ab1405d3928e453188a66e13d632abb5, you should create a UTF-8 file at the following location: https://www.example.com/ab1405d3928e453188a66e13d632abb5.txt, and the file content should be ab1405d3928e453188a66e13d632abb5.
Once the key file is hosted, you can use this key to submit URLs to IndexNow. You can do this manually by sending HTTP requests to one of the IndexNow endpoints, as described in the official documentation. Alternatively, Site Audit can handle this process for you.
3. Setting up IndexNow in Site Audit
You can set up IndexNow in Site Audit by navigating to Project settings > Site Audit > Crawl settings, and then locating the IndexNow section.
If you don't have an API key, you can generate a new one directly in this section. However, don't forget to host the key file at the root of your website, as mentioned above. If you already have a key, simply paste it into the input box.
Please ensure that your website is accessible to AhrefsBot, so we can crawl and verify the key file.
There are two ways to submit URLs to IndexNow via Site Audit:
You can submit the necessary pages manually from the Site Audit reports. We suggest using the pre-configured "Pages to submit to IndexNow" issue that selects pages with content changes recommended to submission.
Another way is to enable auto-submission, and Site Audit will automatically submit the changes you select when it crawls your site. Please note that your project needs an upgrade boost that supports the auto-submission to IndexNow feature to enable it.
After everything is set, don't forget to save the settings.
4. Manual submission to IndexNow
To submit multiple URLs in bulk, open Page Explorer, filter the URLs you want to submit, and click the "Submit to IndexNow" button. This will submit all the URLs in the current report to IndexNow.
To avoid wasting your crawl budget, we recommend ensuring that all submitted pages have relevant and positive changes. You can review the pages recommended for submission in the pre-configured "Pages to submit to IndexNow" issue. This issue automatically selects pages with content changes, as well as newly redirected or removed pages. You can also create your own custom issue with specific filter conditions or exclude certain pages from the report.
To submit an individual URL to IndexNow, open the URL details panel and click the "Submit to IndexNow" button.
After submitting the URLs, you'll receive a success notification. If there's an issue, such as a missing key file on your site, Site Audit will inform you about the error.
5. Auto-submission to IndexNow
When auto-submission enabled in Project settings, Site Audit will automatically submit specified changes to IndexNow during each site crawl.
You can choose from four auto-submission options:
Indexable pages with changes in content: Pages that existed previously but have undergone content changes.
New indexable pages: Newly discovered indexable pages or known pages that have become indexable since the last crawl.
Newly redirected pages: Previously indexable pages that now redirect.
Removed page: Previously indexable pages that have been removed from the site.
With auto-submission enabled, manual submission becomes unnecessary, and the "Pages to submit to IndexNow" issue will not be reported.
To track auto-submissions, four new fields have been added to Page Explorer.
These fields are:
IndexNow auto-submission status: The status of IndexNow auto-submission. Possible values:
Success: The page was successfully submitted to IndexNow.
No changes detected: No changes were detected on the page; submission was not required.
Not eligible: The URL isn't eligible for submission, e.g., it's not an indexable HTML page.
Invalid API key: IndexNow submission failed due to an invalid API key.
Failed: Submission to IndexNow failed; see details for the reason.
Auto-submission is off: Automatic submission is disabled in Crawl settings.
IndexNow auto-submission time: The date and time when the URL was auto-submitted to IndexNow.
IndexNow auto-submission reason: The reason the page was considered for auto-submission to IndexNow. Reasons correspond to the four options in Project settings:
Indexable pages with changes in content
New indexable pages
Newly redirected pages
Removed pages
IndexNow auto-submission error message: The error description for a failed auto-submission.
By following these steps, you can efficiently notify search engines about important content changes on your website using the IndexNow protocol within Site Audit.