Site Auditor

Open Site Auditor

Site Auditor, the PirateSERP technical SEO crawler, reads the public pages of one website, flags page-level and site-wide issues, and runs Lighthouse on any page you choose. Site Auditor needs no API key.

Start an audit

1. In the header, select the project.
2. Review Site to audit, or enter a domain or URL such as example.com.
3. Optional: To add page signals to every page's details, enter a keyword in Target keyword (optional).
4. In the Language list, select the language of the site's content.
5. In the Max pages list, select 25, 50, 100, or 200 pages.
6. Click Start Audit.

Stay on the page while the audit runs. The progress bar reports scanned and queued pages. Leaving the page or closing the tab stops the crawl, and the audit saves with the pages scanned so far and a stopped early label.

Site Auditor starts at the URL you enter, follows links to other pages on the same site, and then adds pages listed in the site's XML sitemap until it reaches Max pages. A start URL that redirects is crawled once. URLs that robots.txt disallows are skipped. A finished audit can cover only part of a larger website.

A link to another website is not crawled. When a link on your site redirects to another website, the audit records it as a Request failed issue with the detail "Redirects outside the audited website" and the destination host, and the crawl continues.

Review page issues

Start with the Issues card. It counts Errors, Warnings, and Notices, and lists each issue type under its group as a sentence, such as "3 pages have no meta description", with a fix instruction.

When an earlier audit of the same site exists in the project, the Since the previous audit line reports how many issues are new, resolved, and reopened since that audit. An issue is new when it appears, resolved when it was on a page in the earlier audit and is gone now, and reopened when it had been resolved and is back. An issue row also shows how many of its pages are new since last crawl, which counts new and reopened pages. A page crawled for the first time counts as new for every issue it has.

Click Show resolved to list each fixed issue with its URL and the evidence from the earlier audit.

A resolved issue needs a fair comparison. Both audits must cover the same site and language, use the same Max pages value, finish without a stop, and read robots.txt and the sitemap. A crawl that reached its page limit cannot show that an issue is fixed, because a page it did not reach may still have the issue. When the two audits do not meet these rules, the line says Not comparable with the previous audit with the reason, and nothing is marked resolved. Audits saved before this feature cannot be compared for resolved issues.

1. Click an issue sentence to list the affected URLs.
2. Click Filter table to show only those pages in the results table.
3. Click Clear to remove the filter.

The groups guide your review order:

• Errors include failed requests, server errors, broken pages, redirect loops, missing titles, mixed content, and broken internal links.
• Warnings include noindex pages, duplicate titles and descriptions, missing H1 headings, redirect chains, short pages, images missing alt text, orphan pages, hreflang problems, sitemap URLs that search engines can't index, and AI crawlers that are blocked.
• Notices include deep pages that are more than three clicks from the start page, titles and descriptions too wide to show in full in search results, a missing Open Graph title, links to redirected URLs, and hreflang tags without an x-default entry. Width depends on the letters used, not only the character count: wide letters such as W and M take more room than narrow ones such as i and l.

Hreflang and sitemap issues

If your pages declare language versions with hreflang link tags, the audit checks each page's tags:

• Each code is a two-letter language code, optionally followed by a script or a two-letter region, such as en, en-GB, or zh-Hant. A common mistake is en-UK; the correct code is en-GB. A country code on its own, such as us, is invalid.
• The page lists itself, and the tags include an x-default entry.
• Each language version the audit also crawled returns a 200 status, is indexable, names itself as canonical, and links back to the page.
• The page's canonical tag does not point to another language version.

The audit judges only language versions it crawled. A version on another domain, or one the page limit left out, is not checked and is never reported as broken. The audit reads hreflang tags in page HTML only, not in HTTP headers or sitemaps.

The Non-indexable URL in sitemap issue lists sitemap URLs that the audit crawled and found redirected, broken, marked noindex, or canonical to a different URL. A sitemap should list only canonical, indexable URLs. Sitemap URLs the audit did not crawl are not checked.

AI crawler issues

Two issues appear on the home page:

• AI search crawlers blocked: robots.txt blocks the crawlers that build AI search answers along with the AI training crawlers. This usually happens when a rule meant to stop model training also blocks AI search, which removes the site from AI answers and their citations.
• AI crawlers blocked by the server: robots.txt allows GPTBot, but the server refused a test request that identified itself as GPTBot (HTTP 403 or a challenge page) while it served the page to normal visitors. Many firewalls verify crawlers by IP address and refuse look-alike requests, so the real GPTBot may still get through. First check whether your firewall or CDN bot-management setting allows verified AI crawlers, and change it only if it blocks them.

Some pages use redirects or noindex on purpose. Confirm the intended behavior before you change them. A linked PDF or image file is listed with its status but receives no title, heading, or description issues. A page with no issues passed the checks Site Auditor runs, which is not a complete assessment of the page.

Read the results table

The results table lists every crawled page with its URL, Status, Title, Issues, and Lighthouse scores. A Blocked status means bot protection or a challenge page answered instead of the content.

Click a URL to open its full details, including headings, canonical tags, Open Graph fields, structured data types, images, links, redirects, response time, and internal link authority. Click depth is the fewest clicks it takes to reach the page by following links from the start page. A page that no chain of links among the crawled pages reaches, such as one only the sitemap lists, shows not reached by links among crawled pages. Link to it from a related page or from the navigation. With a target keyword, the details also show page signals for that page. These are descriptive measurements from the crawl, not ranking targets.

Review site-wide checks

The Site-wide checks panel reports the following:

• The robots.txt and XML sitemap status, with the disallow rule and sitemap URL counts.

• The discovered URLs that robots.txt blocks, and the sitemap URLs that the page limit left uncrawled.

• Orphan pages, which no other crawled page links to.

• The pages with the most internal link authority and the best internal link hubs.

• The security headers of the start URL, including the HTTP to HTTPS redirect and HSTS.

• AI crawler access: whether the site's robots.txt allows each AI crawler to crawl the home page, grouped by what the crawler does:

  • AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Meta-WebIndexer build AI search answers. Blocking them keeps your pages out of those answers and their citations.
  • AI model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, and Meta-ExternalAgent collect content that can train AI models. Blocking Google-Extended or Applebot-Extended does not remove the site from Google Search or Apple's search features.
  • Fetches when a user asks: ChatGPT-User, Claude-User, and Perplexity-User open a page when a person asks about it. OpenAI and Perplexity say robots.txt may not apply to these requests, so the panel shows "robots.txt may not apply" for them instead of allowed or blocked.
  • Search engine: Googlebot, for reference.

  A missing robots.txt allows every crawler. If robots.txt does not respond, the panel says the check did not run. Robots.txt is a request, not a lock: a firewall can still refuse a crawler that robots.txt allows, which the AI crawlers blocked by the server issue reports.

Review site structure

Report	What to do next
Topical Cannibalization	Review pages that share one subject and decide whether their purposes differ.
Near-duplicate Pages	Compare paired pages before you merge, redirect, or rewrite them. Pairs where a blog post overlaps a service or product page list first.
Internal Link Suggestions	Add relevant links where they help readers navigate. Click Download CSV to export the list.
Topical Focus	Review the pages that sit furthest from the site's main subject.
Site Topics	Check which subjects the crawled pages cover. Click View graph to open the topics in Entity Graph Viewer.

These reports identify candidates for review. Related pages do not necessarily compete, and similarity alone does not justify deleting a page.

Use Internal Linking for a focused link-suggestion list. Open Entity Graph Viewer to inspect page connections and site topics.

Run a Lighthouse check

In a page's Lighthouse column, click Run. The check covers that one page, not every page in the audit.

The column then shows four scores: P (Performance), A (Accessibility), BP (Best Practices), and SEO. The page details add lab Core Web Vitals. These are lab measurements from one test run, not a report of every visitor's experience. A Lighthouse run takes longer than a page crawl, so keep the page open until the scores appear.

Save, export, and recheck

Every audit saves in Past audits in the active project, with its page count and a running or stopped early label when it did not finish. To reopen an audit with its saved Lighthouse results, click View. To leave the audit, click Close. To remove an audit and every page in it, click its trash icon (Delete audit).

To export the page table, click Export CSV. To print the report or save it as a PDF, click Print.

A finished audit cannot crawl more pages. Make your changes in your website editor, then start another audit. PirateSERP does not change the audited website.

Resolve incomplete or blocked results

If a page shows Blocked, check whether it requires sign-in or presents a crawler challenge. If the Site-wide checks panel lists uncrawled sitemap URLs, select a larger Max pages value for the next audit.

Some readability checks run in English only. Read Languages before you interpret non-English results.

Explore saved audits

Saved audit exploration opens with a completed audit and offers three views:

• Link review shows source and destination metadata, similarity, and shared entities. To record a decision, click Shortlist, Dismiss, or Reset. To download the list, click Export link review CSV.
• Compare audits compares two saved runs. Select an audit in the Earlier audit list, review Crawl coverage details, and sort the changes. To download the differences, click Export comparison CSV.
• Content Opportunity Map connects observed pages with extracted topics and lists candidates for review. To download them, click Export opportunity review CSV.

Your review decisions stay in your browser for the account, project, and audit. They do not sync across devices, and they do not add links to your website.

Compare audits requires two different audits from the same project, site origin, and language. It uses the stored results and does not crawl the site again. A page missing from one crawl is not proof of deletion, so compare the page limits, languages, and completion status of both runs.

The opportunity map uses observed topics and title or H1 matches. These signals do not measure search demand or confirm a page's primary entity.

Optional Atlas industry context adds reference concepts from a researched Service Atlas industry. It does not confirm that your business offers those services.

Graphs and tables state their display limits. Each CSV export includes the rows a table omits.