Soft 404 detection: separating real empty pages from legitimate inventory gaps

Google's Search Console reports soft 404s when a URL returns a success response but the rendered page behaves like an error or has no main content. Common causes include missing server-side includes, failed data requests, empty internal-search results, and pagination beyond the end of a collection. Google's troubleshooting guide recommends checking both the response and what Google rendered.

The problem is that not all empty-looking pages are errors. A seasonal product category that is temporarily empty can still explain the category, expected availability, and useful alternatives. A no-results search page may be functioning correctly but still have little reason to appear in search. Decide from the URL's purpose and content instead of assuming every empty state deserves the same response.

How Google detects soft 404s

Check the exact URL in URL Inspection and compare its rendered main content with a healthy page from the same template. A word count or phrase match can help triage a large export, but Google does not publish a phrase list or percentage formula. A short page can be legitimate; a long custom error page is still an error.

Classifying soft 404s with Python

A conservative script can label review candidates without pretending to reproduce Google's classifier:

import requests
from bs4 import BeautifulSoup

ERROR_PHRASES = ('not found', 'no results', 'nothing here', 'no matches')

def triage_empty_response(url):
    try:
        response = requests.get(url, timeout=10)
        text = BeautifulSoup(response.text, 'html.parser').get_text(' ', strip=True)
        return {
            'url': url,
            'status': response.status_code,
            'words': len(text.split()),
            'error_language': [p for p in ERROR_PHRASES if p in text.lower()],
        }
    except requests.RequestException as error:
        return {'url': url, 'fetch_error': str(error)}

Use the output alongside the route type and rendered screenshot. It is a review queue, not an indexability verdict.

Match the response to the URL state

In Laravel, keep a no-results search page distinct from a missing route:

public function search(Request $request)
{
    $query = $request->get('q');
    $results = Product::search($query)->paginate(20);

    if ($results->isEmpty()) {
        return response()->view('search.empty', [
            'query' => $query,
            'suggestions' => $this->fallbackSuggestions($query),
        ], 200);
    }

    return view('search.results', compact('results', 'query'));
}

Apply noindex to internal-search results through one template or response-header owner if they should not appear in Search. Do not block those URLs in robots.txt before Google can read the directive.

For an existing category, distinguish "no products exist" from "products exist but none are currently available":

public function show($slug)
{
    $category = Category::where('slug', $slug)->firstOrFail();
    $products = $category->products()->where('in_stock', true)->paginate(24);

    if ($products->isEmpty()) {
        // Check if this is a temporary or permanent state
        if ($category->products()->count() === 0) {
            // Category has no products at all—permanent
            abort(404);
        }

        // Category has products but all out of stock—temporary
        return response()->view('categories.empty', [
            'category' => $category,
            'alternatives' => Category::where('parent_id', $category->parent_id)
                ->where('id', '!=', $category->id)
                ->take(3)
                ->get(),
        ], 200);
    }

    return view('categories.show', compact('category', 'products'));
}

Verify pagination at the boundary

Check zero, negative, malformed, and over-limit page numbers. A valid framework-level not-found response must produce a real HTTP 404, not a 200 app shell that changes into an error after JavaScript runs:

export async function getStaticProps({ params }: { params: { page: string } }) {
  const pageNum = parseInt(params.page, 10);
  const { posts, totalPages } = await getPostsPage(pageNum);

  if (pageNum > totalPages) {
    return {
      notFound: true, // Returns 404
    };
  }

  return {
    props: { posts, page: pageNum, totalPages },
  };
}

Track affected templates and counts against your own baseline rather than inventing a sitewide threshold. The durable fix is at the route boundary: return success for a real, useful resource; not-found for a missing resource; and a direct redirect only when a clear replacement exists.