Blog

How to Make a 404 Page for SEO, Usability (404 FAQs & More)

February 17, 2021
by
Tory Gray
Tory Gray

Originally published in 2011. Updated as recently as April, 2021.

404 Error Pages play a very important role in search engine optimization and website usability. Here's a quick Q&A of the most commonly asked questions I get from clients about 404 pages, things that commonly go wrong, plus advice for how to set up & monitor 404 errors properly.

404 “PAGE NOT FOUND” ERRORS FAQ:

What Is A 404 Error?

A 404 error, also called a “header response code” or “http status code”, or simply "crawl errors", is the computer equivalent of saying “Not Found” or “Page Not Found.”

404-page-design

Here’s the “tech speak” definition:

The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.”
- via
w3.org

In other (less technical) words, this response code essentially tells search engines - and users - that the resource (or page/URL) being referenced does not exist or literally can’t be found. It’s a robot’s version of a shrug and blank look.

There are many other response error types, and all URLs return response codes of some sort. A correctly functioning page, for example, should return a “200” status code, which means “Found.” Different error types help webmasters diagnose the source of the errors so they can fix it appropriately.

404 errors are generally the most common error type. They are also often handled incorrectly by well-meaning people - hence the purpose of writing this article!

What Is A "Soft 404"?

A soft 404 is essentially a page that doesn’t return a 404 response code, but Google(bot) nevertheless believes the page is experiencing an error.

  • You can find out if Google believes any of the pages of your site are soft 404s via the Coverage Report, in the Excluded section, of Google Search Console. More on that below.
  • Sometimes these pages are good, functional pages and Google made a mistake.
  • More often, however, they are pages that are blank / have no content, but they aren’t issuing a 404 status code like they should be.

What's The Difference Between 404s And Soft 404s?

We can keep this one short and sweet - essentially, a soft 404 does not issue a 404 header status code;  it (perhaps incorrectly) issues a 200 (Okay) status code. A normal 404 page does correctly issue a 404 (Page Not Found) status code.

How And When Do 404 Errors Occur?

  • When pages have been moved (e.g. a non-existent page or missing page) or or deleted (e.g. an old page/old url)
  • When a webmaster, CMS user or software engineer mistypes a URL on a page/page template or has copy-and-paste mistake (e.g. just a wrong url)
  • When broken links or accidentally truncated links occur on web pages, social media posts, or in an email message (etc.)
  • Soft 404s occur when a page issues a 200 (OK) status when it should have issued some other error, because something went wrong. Most commonly the website should have issued a 404 error, but didn’t.
  • Usually this is an indication that 404s aren’t functioning properly. Sometimes, it works in some sections of your site, but not others.

How Are 404 Pages Useful?

A 404 error tells search engines, website visitors and webmasters when page URLs are broken, or never existed in the first place.

When we can see where these errors occur, we can fix the problem for future website visitors (generally via a 301 redirect to the new location of that information), and preserve the power of the page that existed OR the incorrect link to a page that never did. (That’s what we call “broken backlink building".)

If you don’t issue a 404 response code, you won’t inherently know that the page - and therefore the user - is experiencing an error. So YES. A 404 page is useful.

You can’t fix what you don’t know about!

Are 404 Errors Bad for Search Engine Optimization (SEO)? Are 404s Errors Good for SEO?

There is some dissent online debating this issue.

  • In theory at least 404 errors are bad because they represent errors on your website (or on the web, but relating to your website). But (infrequent) mistakes happen, and they are easily forgiven on a small scale.
  • A user that comes across a 404 page is less likely to return to the site. (We’ll cover how to make this less likely, below.)
  • Too many 404 header response errors - or 403s, 500s, or really any other type of 4{xx} or 5{xx} error - across a site can create an overall high error rate vs. success rate.
  • This creates trust issues; if Google sees too many errors vs. functioning pages, they don’t want to send users to your site. Why would they? That user will get lost and mad, and have a bad experience. Then Google has failed at their job of helping the user find the answer to their question.
  • If your website is actively linking to a bunch of non-functional pages, that means you are passing page rank… to nothing. Picture this like a purse with holes in it, with money dripping out. No bueno!
  • That said - 404 errors will happen no matter what. A properly functioning 404 page will notify the webmaster that the error occurred, and where it occurred. Then we can go about fixing it via a 301 (permanent) redirect. The user, and search engines, never need to come across this particular issue again. So let’s not “shoot the messenger” - errors happen. Just fix them.

Key Point: Some teams see 404s as “bad” and so avoid this inherent badness by simply choosing not to issue a 404 status code on a 404 page. The trouble is - the errors can and will occur no matter what.

So let’s not “shoot the messenger.” Errors happen! Issuing a 404 status code means that you can more easily find issues in order to fix them.

So… to state the obvious… don’t forget to fix them! Too many outstanding, unfixed 404s can be actively bad, or can be big missed opportunities. Plus it’s just a bad user experience.

How Should A 404 Page Function? (User Experience Matters!)

  • Part 1: Remain at the URL that was called, (for example: https://www.thegray.company/i-made-this-up/) - e.g. DON’T redirect this to a page like /404.html. The error should load on the URL with the error, so we can see/fix the error properly.
  • Part 2: A 404 http status error should be issued from the server. Use a tool like httpstatus.io to double check that it’s working.
  • Part 3: A 404 message - visible to the website visitor, on the page - should clearly explain what happened, and include resources, links, and ideally the ability to search your site. More on how to make this UX friendly below.
Example results from httpstatus.io
Example results from httpstatus.io

What Are the Most Common Improper Setups For 404 Pages? What Are The Most Common 404 Setup Mistakes/Errors?

  • Redirecting to a 404 page.
  • This hurts everyone. Users are lost, search engines think everything is honky-dory when it's not, and since you don't know when it's happening - you can't fix it.
  • Automatically redirecting to the page you assume search engines and users want.
  • A risky solution, that can easily go wrong.
  • It's best not to assume. Find the issue and fix it.

What Are The SEO Implications of 404 Setup Mistakes?

  • If there are many 404 errors occurring on the website, this page can actually start ranking (an issue that occurs when you combine a redirect to a 404 page URL that doesn't issue a 404 http header response).
  • Users see a 404 page instead of the page they should be seeing, causing them to leave. They are not likely to return:
  • 404 pages have a high bounce rate - i.e., the percentage of visitors who visit this page first (found from a search engine most likely) and immediately leave without going to any other pages.
  • 404 pages have a high exit rate - i.e., the percentage of visitors that find this page from clicking a link on the website, whereupon they immediately leave.
  • Pages on the website that should be benefiting from the link are not, lowering the overall search engine ranking potential. This results in less traffic to the website overall.

How Can I Make My 404 Page More User-Friendly? (E.g. What Are 404 Page Best Practices)?

There are several things you can do to improve your website's 404 page - and the likelihood that website visitors will stick around and check out the rest of your site - including:

  • Explaining, in plain English (or whatever language your site targets, that is), what happened. This is for the user's benefit…. because that matters!
  • Include a link to your contact page so users can attempt to solve the issue with your help.
  • Include a simplified HTML sitemap so the user can find their own way. Track what's being searched most often and make those resources easier to find.
  • Consider adding a search bar to the 404 page template, so the user can look for the resource where ever it’s now located on your site.
  • Make sure this page template’s title includes either “404” or “Page Not Found” etc. - this way you can figure out which pages your users are hitting, and the frequency they are seeing it, in Google Analytics (or your analytics tracking tool of choice.)

How Can I Find 404 Errors On MyWebsite?

There are several good ways to go about this. I’d honestly recommend doing them all since different tools can find issues that other tools don’t see.

  • Crawl your website. There are a host of really great crawlers, my favorite being Screaming Frog for small sites, and DeepCrawl for very large/enterprise sites (both often work in all situations, it’s often a personal preference!) Each will tell you what status code errors are linked on your site.
  • Check Google Search Console (GSC), formerly known as Google Webmaster Tools.
The Google Search Console (GSC) Coverage report. Click the “Excluded” box to see what URLs on your site are excluded.
The Google Search Console (GSC) Coverage report. Click the “Excluded” box to see what URLs on your site are excluded.
This version helps you see URLs that Google believes are Soft 404s.
This version helps you see URLs that Google believes are Soft 404s.
If you have 404 errors that Google has discovered, you will find them by clicking on this header.
If you have 404 errors that Google has discovered, you will find them by clicking on this header.
  • Check Google Analytics (GA)
  • Search for page titles that contain “not found” or “404” - this is a nice list that tells you how many times a particular page has been hit, and therefore can better help you prioritize which errors need fixing first.
Finding 404s in Google Analytics
Finding 404s in Google Analytics
  • Check your Log Files
  • This is my favorite way to check for 404 errors (and other errors!), since you can see exactly who (which bot) is hitting what errors, on what pages, how frequently, and as what percentage of total site hits. Unfortunately, not everyone can get access to log files, depending on your platform and host. Talk to your IT and/or development team to see if this is an option for you. You’ll need a log analyzing tool to do this; my favorite is from Screaming Frog.

How Can I Find 404 Errors On Pages They Are (Allegedly) Linked From?

If you are having issues finding the 4{xx} error on the page it’s allegedly linked from (e.g. from internal links), here are some things you can test. If you can’t find the URL using any of these methods, it’s likely that the issue used to be present but is now currently fixed.

Run a new crawl of the site to validate this either way! (If it’s still there - keep digging!)

Screaming Frog report on 404 error types
Screaming Frog report on 404 error types
  • First check if the link is from an inlink or a redirect. Screaming Frog’s 404 inlink report will tell you which is which; other crawlers like DeepCrawl and Sitebulb can give you similar data.
  • If the answer is “AHREF” it’s linked from the page referenced, and you can proceed to the steps below.
  • If the answer is “HTTP Redirect”, the link source is another URL that redirects to the new page. Check your redirect file (e.g. htaccess) or your admin (e.g. the a Wordpress redirect plugin) for the original URL. Change the redirect location so it’s no longer pointing to a broken page.
Google Analytics Navigation Summary Report can help you find the source of linked 404 errors
Google Analytics Navigation Summary Report can help you find the source of linked 404 errors
  • View Source on the page in question and Control+Find (C+F) for the broken URLs’s path.
  • View Rendered Source - 3rd party browser plugins (like this View Rendered Source plugin for Chrome) will show you source code AFTER the browser has rendered it (e.g after Javascript, CSS has run.) Again, C+F to find the broken URL path.
  • If you are finding the 404s via Google Analytics, once you narrow down to the offending URL, click into the Navigation panel to find the previous pages in that path:
  • Run the URL through the Mobile-Friendly Tool from GSC. Once it’s run, click the HTML tab, and Copy + Paste the contents of the results into a Word doc, text editor, etc. Again, C+F to find the broken URL path.
  • Pro Tip: Using the mobile-friendly tool gives you actual Googlebot-rendered code - an EXCELLENT resource for QAing issues like this.
Googlebot rendered HTML in the Google Mobile Friendly Tool
Googlebot rendered HTML in the Google Mobile Friendly Tool

How Can I Find 404 Errors That Aren't Linked on My Website?

It’s not uncommon for 404 errors to happen on URLs that never even existed on your website. This is generally due to either:

  • Issues with Googlebot crawling JS or CSS incorrectly (e.g. interpreting JS code that aren’t links as links)
  • In this case, you’ll still see your own website as the source of these links in GSC. Fortunately, as Google has improved their JS crawling ability (note that this is an improvement - it’s not perfect by any means!) this happens less and less commonly. I’m honestly not sure if there’s a solution here. This is a good example of a 404 error that I might simply ignore - especially considering that, at some point, Google will eventually “get it” and the problem will disappear.
  • External links to your site, or other sites accidentally linking to your site incorrectly.
  • You can discover these with 3rd party tools. My favorite is the Broken Backlink report in Ahrefs (a paid SEO tool.)
  • You can export this data in a CVS file
  • Then crawl each URL (on your own site) to see what’s currently 404ing, since - like most tools - caching is at play, and it’s possible that it’s not currently 404ing. If it is, and you redirect it again, you might create redirect hops and redirect loops, which you certainly don’t want.
  • Do keep in mind that not every external link is worth redirecting. If the broken link is from a spammy or really low value website, you might be better off ignoring it. You can determine this answer for yourself, again via paid 3rd party tools. Examples include Ahref’s DR (domain rating) and UR (URL rating metrics,) or Moz’s DA (Domain Authority) or Spam scores.

How Do I Fix 404 Errors? Can I Prevent 404 Errors?

  • There are a variety of ways to fix 404 errors, depending on your site’s setup/platform and your software development team’s capabilities/priorities. Here are some notes:
  • The simple answer is implementing a 301 redirect.
  • You might be able to implement this for yourself in your website’s admin center. If so, this is the easiest way to do so! Just pick out the best new URL this page should be redirected too.
  • Sometimes you’ll need to go through your software or IT team. It’s not uncommon for them to push back on doing this work - full stop, or due to how many there are to implement (sometimes there are a lot). They don’t always see the value, and they are spread thin working on other business priorities. It’s generally something I do recommend fighting for, within limits. See more on this below.
  • One note: Allegedly the 404s in your GSC (webmaster tools) account are in priority order, according to Google’s priority.
  • Can you prevent them? Nope. They are just a reality of the world, and ignoring them won’t help.

How Do I Fix Soft 404 Errors?

  • Part 1: Vet your GSC “Soft 404” list to ensure that any issues found are actually issues to fix (they aren’t always.)
  • For items that were errors and you actively want them indexed, submit them for indexing in GSC.
  • For items that aren’t a priority to index, you can just ignore them.
  • Part 2: Share your vetted list of issues with your software development team. In most cases it’s one of the following:
  • It’s not a page that search engines should be accessing or indexing anyway.
  • In this case, you can either block the page / page path via the Robots.txt file, or via the Remove URL tool in GSC.
  • It’s a page that should have issued a 404 error, but didn’t.
  • Your software team will need to dig in to understand why that’s happening & resolve it. Once it’s correctly issuing a 404 again, you can proceed to part 3.
  • Part 3: Follow the steps outlined to fix regular 404s, above.

Can't I Just Let Pages Keep 404ing? Google Says It's Okay!

  • Yes they do say this. But, as is the way of the world, sometimes Google says things that aren’t - strictly speaking - accurate.
  • See: Marketers Say Most of Google’s Public Statements Are False or Misleading
  • Actively linking to 404s is just bad - for everyone.
  • Having too many 404s, or other site errors, can contribute to an overall high error rate on your site, which can cause Google to distrust your site overtime.
  • Plus, you really don’t want Googlebot spending time crawling your 404s instead of your functional, high quality content!
  • The issue with 404s is that Google treats them as temporary. If there are active links to 404 pages - either on your site, or elsewhere on the web - Google can and will keep checking them to see when they will get fixed. They will continue to do this for months - and sometimes years - after the URLs stop working, if you don’t resolve them.
  • You can validate this yourself by viewing your log files, and seeing the quantity/frequency Googlebot hits 404ing pages.
  • If there is a business case for not redirecting some URLs (good examples include: your site got hacked and the URLs in question were spammy / malware issues, or your purchased the site from a 3rd party and there used to be pages for a different business - and you actively don’t want them associated with your new business), consider issuing a 410 “Gone” message instead. Googlebot tends to respect 410s much more quickly than 404s.

How Should I QA My 404 Page to Ensure It's Working As Expected?

Here’s my step-by-step process to confirm if your 404 page is working as expected (aka like it should be!)

  • Find a 404 page.
  • This can be as simple as sticking a random string of letters after the domain / homepage and clicking enter, e.g. at https://thegray.company/i-made-this-up
  • DO NOTE, however, that larger, more complex sites might have functionality differences in different site sections - in other words, the 404 page might work in some places, but not in others.
  • If you have a small site, don’t worry about this.
  • If you have a large site, at minimum, try these QA steps within each major site section, e.g. https://thegray.company/services/i-made-this-up AND https://thegray.company/clients/i-made-this-up.
  • Also monitor GSC more carefully for 404 errors and soft 404 errors, just in case.
  • Confirm that the page is issuing the correct 404 response. I recommend https://httpstatus.io/. Copy + paste the fake URL into the field and click the Check Status button (you can check multiple URLs at one time):
This is where you enter the URL (or list of URLs) to test them in httpstatus.io
This is where you enter the URL (or list of URLs) to test them in httpstatus.io
  • Confirm that it’s issuing a 404 code. (Rinse and repeat as needed for 404s in other site sections)
  • It should not: redirect (301 or 302) to a 404, OR issue a 200 (ok) status.
  • Outside of this, the primary items to pay attention to are around usability: if/when the user hits the page, do you successfully help them find their way again? How can you do that better? Read more about this and best practices for 404 pages. (You can also learn more about QA for SEO.)

Contact us if you need help fixing or improving the 404 page functionality on your website - we’d love to help.

Work With Us
We’ll help teach, mastermind, and carry out SEO roadmaps that check all the boxes.
CONNECT THE DOTS WITH US