Knowledge Base
SEO Spider Guide
SEO Tester Online
18 November 2019

SEO Spider Guide

While SEO Checker analyzes the degree of optimization of a page, the SEO Spider scans an entire site, just like the crawler of a search engine.

To scan a site, simply select SEO Spider in the menu on the left, then click on the “make a new crawl” button at the top right: a tab will appear.

seo spider - make a new crawl

All you have to do is enter the URL of the site you want to scan, click on the “+ new scan” button and wait a couple of minutes for the crawler to analyze all the pages.

seo spider - new scan

Once the project status is “completed” you can click anywhere on the tab to view the details.

seo spider - new scan


This is the Overview screen: you will see the home from both mobile and desktop and you will be able to evaluate the overall SEO performance score of the domain. Thanks to the graph, you can see at a glance which aspects need to be improved.

seo spider - report - overview


Browsing the menu above, we can switch to Base. Here we find more in-depth analysis and, above all, advice on how to improve aspects such as the Snippet, meta descriptions, Title tags, and canonical and hreflang attributes.

seo spider - report - base


Moving on to Structure, we can verify if the server has the file compression function enabled, which allows a much faster loading of our site; the presence of the robots.txt file, of the sitemap, of the https protocol; if the server has returned 404 errors; if the images have a correct alt attribute or if the ratio between text and code is optimal. Also, the SEO Spider will help you to identify the critical points and will provide you with suggestions on how to solve them.


In this section, you will analyze the textual content (paragraphs and titles) of the pages of your domain. In this case, the SEO Spider is telling us that 125 pages of our site do not have enough text content. Just click on “Open details” to view the related pages.

seo spider - report - content


We have already mentioned the importance of duplicate contents management. By clicking on “Open details” you will see in detail which content on our site is repeated.


We have already discussed this topic in the overview. In this section, we will see in detail what to optimize our OpenGraph and TwitterCard.

In the menu on the left, you will be able to navigate the menu to view the individual components of each page grouped by section: therefore, we will have a section that groups all the site’s H1’s, all the meta descriptions, etc.


Under the heading “Title” we have the sub-categories Title, H1, and H2. The Title is that portion of the text contained in the homonymous tag inside the HTML code of the page and is the heading that will appear, among other things, in the page snippet on the search engine results page.

Here we will be able to filter the titles, for example, to identify those that exceed the maximum size of 70 characters (beyond which the title is cut) and export them to files: in this case, we decide to export titles both with more than 70 characters and duplicate.

seo spider - title - filter

seo spider - title - export

H1 and H2

H1 and H2 instead, are titles of different importance within the text and are used to give a hierarchy to the paragraphs.

In this case, we can filter titles by excessive size, missing duplicates. Furthermore, only in the case of H1s, we can filter them by multiple titles. This is because the rule would like there to be only one H1 per page.

We can decide to export the data we are interested in a file by clicking on the orange “Export” button as well.


Regarding the content, you can browse through the description – that is, the descriptive text of the snippet – and check its size. For greater convenience, we can filter the titles that exceed the allowed dimensions.

seo spider - content - filter

Furthermore, we can export to a file the pages we want. For example, those with descriptions that are too long and too short:

seo spider - content - export

Furthermore, we can export to a file the pages we want. For example, those with descriptions that are too long and too short:


In the Text section, we can check the number of words and the number of these in relation to the code and filter the pages according to these criteria. We can also export the data we are interested in, for example, the pages that have a number of words which is low or too low and with a bad text/code ratio.


In the Images tab you can control which of these are provided with the Alt attribute, using the filter to see which ones do not have it. Here too, we can decide to export the results using the Export button.


Under the Preview macro category, we find the entries Snippets, Opengraph and TwitterCard.


Snippets are the page previews that we find on the Google results page and include the title, the URL and a description. Generally, search engines reward titles and descriptions of a certain length.

seo spider - preview

Opengraph and TwitterCard

The Opengraph and the TwitterCard are the equivalents of the snippets on Facebook and on Twitter, namely the structured previews of the page that appears when we link it to these social networks.

Structure is crucial from the SEO point of view. The easier a site is to navigate, with simple paths for the crawler and the user, the more it will be rewarded by search engines.



From this point of view, the Taxonomies provide us with a graphic representation of the distribution of the pages – and the connections between them – within the domain.

seo spider - structure - taxonomies


Https is a security protocol whose absence, for some time, is considered penalizing by search engines. To implement it on your site, follow this guide.

In this section, you can filter the pages according to two criteria – with and without https –and to export the list.


The canonical attribute is very useful if our site has multiple pages with the same content. When we use this attribute on a page, we tell the search engine that this is the page that must be considered as the one with the original contents. If we do not use it, the search engine crawler considers all pages with the same contents as duplicates, penalizing the entire site.

We can filter the list by choosing to display only the pages with a canonical attribute, or those without.

We can export the list by clicking on the orange “Export” button:

seo spider - structure - canonical - export


The status is what the client (i.e., the user) receives from the server, to which it has requested to view a certain web page via the HTTP communication protocol. These statuses can have different codes depending on the type of response. You will surely know the code “404”, which corresponds to “resource not found” and the code “301” which signals a redirect to another page. We can use the filter in the upper left to view which pages each of these three codes return.

In this case, too, we can also export one or more lists (for example codes 200 and 404) with the Export button.


Under “Language”, thanks to the “filter” function, we can view which pages do not have the hreflang or x-default attributes and export them to a file. When we have a multilingual site, on each page, it is good practice to insert the link to its equivalents in each language using the <link hreflang> attribute. The x-default attribute instead, indicates which “version” should be considered by default when the visitor of the site comes from a region whose language is not present on the site.


The robots.txt file on the site contains the “instructions” to be given to the crawler about the pages it has to explore, how to do it, and which to ignore.

With the “filter” function you will be able to view which pages in the robots.txt file have the noindex attribute (an attribute which blocks the indexing of that page in search engines), noodp and noydir – which prevent search engine robots to use the site information in the Yahoo and Open Directory Project directories respectively. In any case, both of these directories no longer exist, so they are obsolete attributes.

In this case, too, you can export the pages according to one or more of these filters:

Export xlxs

You can also export data in Excel format to the individual sections that interest you. For example, let’s assume we want to extract information about the textual contents of the site.  seo-spider-export-to-Excel

In the single section, look for the Export orange button above.

So choose which elements you want to filter. In fact, you can choose to export only some elements, with certain characteristics.


In our example, we export both pages that have a text that exceeds 1000 words and pages that have an excellent text-to-code ratio.

We just have to click on Export and then on Download.


Page details

You can also view a single page summary. To access the summary it is sufficient to click on the corresponding line from any section. A preview of the snippet, Opengraph and TwitterCard will appear.

seo spider - page details - select

On the left, you can open the page, open it on Google, or analyze it with SEO Checker.


But they are not the only information you can access! Clicking on the “filter” drop-down menu on the top right you can get more data. In fact, you will have an overview of the text contained in the page and the various titles (h1 and h2); how many and what links that page has as input and output; a detail on the distribution of the keywords of one, two, three and four words; which languages were set with the hreflang attribute and which images (and alt attributes) are present.seo spider - page details - keyword detailseo spider - page details - links detail

Export PDF report

The SEO Spider is a very powerful tool, but how to take a snapshot of the state of the SEO and send it to the client? Taking screenshots of each section could be a solution, but not the optimal one.

Luckily, you can also export the entire report in PDF format of the site analyzed with SEO Spider. In this way, you will have both a useful document to consult and compare with the results of your work; and an elegant report that you can send to your client. And you can do it with a simple click.

All you have to do is, in any screen of the SEO Spider, click on the CREATE REPORT button at the top right.

sto - create report

You can either choose to create a standard report or a customized white-label report with your logo and information.

If you choose the standard report, just click on the download button. It will take a few seconds to prepare it. Once ready, you can download it by clicking on the DOWNLOAD button.

sto - create report - create standard report

What you will download will be a pdf document where you will find all the information you would find navigating between the various sections of the tool (including aspects to improve and advice on how to improve them), sorted and collected in one place.

seo spider - export to pdf - pdf report


If you choose the personalized white label report, you’ll have to add a few information first. Click on the “NEXT” button.

sto - create report - create white label report

First of all, upload your company logo. Click on the blue UPLOAD button.

sto - create report - create white label report - upload logo

You can either drag and drop your image or click on the blue upload button.

sto - create report - create white label report - upload logo

sto - create report - create white label report - upload logo


Then, add any info you want to appear on the report’s heading.sto - create report - create white label report - fill in information

You can also manage your appearance preferences.

sto - create report - create white label report - manage categoriessto - create report - create white label report - manage exams







Finally, add notes using the simple, built-in text editor and click on the download button.

sto - create report - create white label report - add some notes and download

You’ll download will start soon and also you’ll receive the download link to the report in your mailbox.

sto - create report - create white label report - report ready

sto - create report - create white label report - download email

Now you can send your very own customized report to your clients.

Leave a Reply

Your email address will not be published. Required fields are marked *