Recover Your Old Website Content From the Wayback Machine (Script Included)
If your old website is gone but you still have the right to its content, the Internet Archive’s Wayback Machine may hold a copy of many of its pages. You can list those copies and download them in bulk with a short script, instead of clicking through the web interface. This guide shows how to list what the archive has with its CDX API, download each page as it was originally served, and avoid the problems I ran into: pages that arrive compressed, queries that time out, images that went missing and a pile of pages that should not be recovered at all. The commands are real. I used them to recover an old blog of my own that now returns “410 Gone”, and the numbers below come from that job.
Only recover what you own
The Wayback Machine keeps copies for preservation. A copy there is not a licence to republish anything. Use this to recover your own writing and your own screenshots. Text and images written or made by other people stay under their authors’ rights, even if they appeared on your old site. Check also whether the old pages are worth bringing back, which is the last section of this guide.
Step 1: list what the archive has
The archive offers a CDX server that lists captures. Ask it for the captured pages of a domain that returned a successful status:
https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json
&fl=timestamp,original,mimetype,statuscode&filter=statuscode:200&limit=20000The parameters, as described in the CDX server’s documentation: matchType=domain includes subdomains, fl picks the columns, filter keeps only matching rows, limit caps the number of rows, and output=json gives a JSON array whose first row is the header. There is also a collapse parameter that drops repeated lines. With collapse=urlkey it keeps the first (oldest) capture of each URL, which is the opposite of what you want here, so the script does not use it and picks the newest capture itself. The server returns rows ordered by URL and then by time, so a row cap can also cut the list short in the middle of the alphabet. For my old blog the query returned 1,409 rows: 261 HTML pages and 443 images, plus feeds and other files.
Be careful with very large domains. A query for a famous site can time out because the archive holds an enormous number of captures for it, and a limit that is too low silently cuts the list short. Query a specific path with matchType=prefix, or use the from and to parameters to narrow the years, and compare the number of rows you got with the limit you set. The script below prints a warning if it reaches the limit.
Step 2: choose one good copy of each page
Many pages have several captures, and each one is a separate row, so the script keeps the newest one that returned status 200. Then drop the URLs you do not need as posts: feeds, comment pages, wp- paths, pagination, and search or tag URLs. These filters depend on your old site, so look at what you throw away. A query string such as ?p=123 can be a genuine post on a WordPress site, and a post whose slug begins with feed or amp would be caught by a careless pattern. After filtering by hand, my list went from 261 HTML captures to 123 post-like pages, and I downloaded 98 of them, the English ones from the years I cared about.
Step 3: download the original page, not the replay
An ordinary archive URL shows the page inside the Wayback Machine, with a toolbar and with every link rewritten to point back at the archive. For recovery you want the page as it was originally served. Adding id_ after the timestamp gives you that. The pattern is:
https://web.archive.org/web/20220716111314id_/https://example.com/some-post/This is the documented raw-capture form that tools use, though captures can differ from one another, so check a file or two before you trust the whole batch. Treat the recovered HTML as untrusted: it can still contain the old site’s scripts and links to external sites. Read the source in a text editor first, and do not open the files in a browser where you are signed in to important accounts. Strip scripts before you reuse any of it.
The script
This standard-library Python script does the steps above: it lists the pages, keeps the newest copy of each URL, and downloads it. It waits between requests, backs off when the archive answers with an error (using its Retry-After header if there is one), does not retry a page the archive answers “not found” for, decompresses gzip responses, and writes a manifest.json that maps each saved file back to its original URL and capture time. File names end with a short hash of the URL so that different URLs get different names, and re-running the script in the same folder skips pages it already saved. It prints a list of the pages it could not fetch at the end. Keep your own filters, such as skipping feeds, in the loop if you need them:
import gzip, hashlib, json, os, re, sys, time
import urllib.error, urllib.parse, urllib.request
UA = {"User-Agent": "Mozilla/5.0 (personal archive recovery)"}
def get(url, tries=4, timeout=60):
for attempt in range(tries):
try:
data = urllib.request.urlopen(
urllib.request.Request(url, headers=UA), timeout=timeout).read()
if data[:2] == b"\x1f\x8b": # gzip magic bytes
data = gzip.decompress(data)
return data
except urllib.error.HTTPError as exc:
exc.close()
if exc.code == 404: # not available at this address: do not retry
return None
try:
wait = int(exc.headers.get("Retry-After"))
except (TypeError, ValueError): # missing, or an HTTP date
wait = 10 * (attempt + 1)
print(" HTTP", exc.code, "- waiting", wait, "s")
time.sleep(wait)
except Exception as exc:
print(" retry", attempt + 1, str(exc)[:60])
time.sleep(10 * (attempt + 1))
return None
def list_pages(domain):
cdx = ("https://web.archive.org/cdx/search/cdx?url=%s&matchType=domain&output=json"
"&fl=timestamp,original,mimetype,statuscode&filter=statuscode:200"
"&limit=20000" % urllib.parse.quote(domain))
raw = get(cdx)
if raw is None:
sys.exit("The CDX query failed or timed out; try a narrower query")
rows = json.loads(raw)[1:]
if len(rows) >= 20000:
print("WARNING: the row limit was reached, so the list is incomplete; narrow the query")
pages = {}
for ts, url, mime, _ in rows:
if mime != "text/html":
continue
p = urllib.parse.urlparse(url)
key = (p.netloc.lower(), p.path, p.query) # every distinct URL stays distinct
if key not in pages or ts > pages[key][0]: # keep the newest capture of each URL
pages[key] = (ts, url)
return pages
domain, out = sys.argv[1], sys.argv[2]
os.makedirs(out, exist_ok=True)
mpath = os.path.join(out, "manifest.json")
manifest = json.load(open(mpath)) if os.path.exists(mpath) else {}
failed = []
for key, (ts, url) in sorted(list_pages(domain).items()):
name = re.sub(r"[^a-z0-9-]", "_", (key[1] + "-" + key[2]).strip("/-").lower())[:60] or "home"
dest = "%s-%s.html" % (name, hashlib.sha1(url.encode()).hexdigest()[:10])
if dest in manifest: # already downloaded on an earlier run
continue
data = get("https://web.archive.org/web/%sid_/%s" % (ts, url))
if data:
open(os.path.join(out, dest), "wb").write(data)
manifest[dest] = {"url": url, "timestamp": ts}
json.dump(manifest, open(mpath, "w"), indent=1) # checkpoint after every page
print("saved", dest, len(data))
else:
failed.append(url)
time.sleep(1) # be polite to archive.org
print(len(manifest), "in the manifest,", len(failed), "failed this run")
for u in failed:
print(" failed:", u)Run it with python fetch_wayback.py your-old-domain.com recovered. I tested it on a small public site, where it listed the pages, saved the files and handled an error response from the archive. Keep the pause between requests, and make it longer if you see errors: the archive is a free public service, and hammering it can get you blocked.
Pitfall: files that are not HTML
When I first downloaded 98 pages with a simpler script, 39 of the files turned out to be unreadable. They were gzip-compressed bytes saved as if they were text. A compressed file starts with the two bytes 1f 8b, which is what the script checks. In my case that was the cause, but a corrupt capture or an error page can also produce a file you cannot read, so open a few of them before you trust the batch.
Step 4: images
Images are separate captures. The same idea works with im_ after the timestamp. Take the image addresses from the recovered HTML (look for the uploads path of your old CMS), then check that each one still exists before you plan around it, by requesting only the headers:
curl -sIL "https://web.archive.org/web/2im_/https://example.com/wp-content/uploads/2022/01/screenshot.png"The short timestamp makes the archive redirect to the nearest capture, which is why the command uses -L to follow it. A final 200 with an image content type means the archive has it, and downloading and opening a few images is the surest check. In my job, the posts I tested had most of their screenshots, and one post lost all three. Many “different” images in a WordPress page are really resized copies of one upload, so count unique originals, not image tags. Check, too, that the screenshots still match the software they show before you reuse them.
Step 5: decide what deserves a second life
Recovering pages is the easy part. Deciding which ones to publish again takes more care. Of the 98 pages I read, only about 11 were worth rewriting for my current site. The rest I left behind, and these are the questions that sorted them:
- Is it still true? Commands, menus and prices change. A post that taught a tool that has been replaced needs a fresh version, not a copy.
- Is it thin or repetitive? Posts of 150 words, or several posts that say the same thing, add little. Fold them together or drop them.
- Did you write it? Anything rewritten from other sites’ articles should not come back.
- Does it fit the site now? Off-topic content, anything that helps people get around licences, such as activation-key lists, and how-tos that copy other people’s material do not belong on a new site.
- Does it still serve a reader? A short, accurate post on a real problem is worth more than a long one on nothing.
I treated the recovered pages as a list of topics and screenshots, and wrote new, accurate versions of the good ones. Bringing material back as a fresh, corrected post also avoids repeating out-of-date advice.
Common questions
Can I recover a site’s database or theme?
No. The archive stores what visitors could see: pages, images, CSS and scripts. It cannot give back a database, plugins or private content.
Why are some pages missing?
The archive only holds pages that were captured successfully, by its crawler or by people who saved them. Pages that were never linked, blocked from crawlers or behind a login may never have been captured.
Is it legal to take another site’s content this way?
Not by itself. This guide is for recovering your own work. Copyright stays with whoever wrote or made the content.
