So here’s a bit of a thread for #WebArchiveWednesday on what I’ve been working on for the last month or so…

tl;dr – #webarchives are fun (and very useful for research)

Thanks to @NetPreserve & @anjacks0n, I’ve been working on a set of Jupyter notebooks using data from #webarchives.

But what sort of data can you get from web archives?

Web archives are HUGE, and the original archived web resources often can’t be accessed in bulk because of things like copyright.

But there’s lots of easily accessible data about the contents of web archives that you can explore without downloading terabytes of data.

For example, if you’d like to get a list of all the times @NLAPandora has archived my blog, it’s as easy as this: https://web.archive.org.au/awa/timemap/json/discontents.com.au

This url will retrieve a machine readable list of captures, which includes the capture dates, the file mime type, http status, & more.

This sort of list is called a Timemap, and it’s part of the Memento protocol for getting capture data from web archives. You can do the same thing with @UKWA, @NLNZ, @internetarchive, & any other Memento compliant system.

There are also Timegates in Memento systems! Query a Timegate with a date and a url and it will send you back details of the archives' capture closest to your target date.

Timemaps, Timegates, & Mementos are cool, but there are a few little differences in the way they're implemented across #webarchives systems. I documented the differences I found in a notebook to save you having to puzzle through it yourself. https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/memento.ipynb

Based on what I found, I wrote some code to smooth out the differences between @NLAPandora, @ukwa, @nlnz, & @internetarchive so you can easily grab data from any of these systems for your own research projects. There's a notebook for querying Timegates: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/get_a_memento.ipynb

And another notebook to get all captures of a particular url using Timemaps (& also CDX APIs which I'll mention later): https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/find_all_captures.ipynb

But what could you do with the data from a Timemap? I'm glad you asked...

What about creating a creating a series of annual screenshots, showing how a web page changed over time? There's a notebook that shows you how: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/screenshots_over_time_using_timemaps.ipynb

ooops short intermission for school pick up...

If you want screenshots, I also made an app (Jupyter notebook running in appmode) that lets you compare screenshots from different dates, or different archives. It uses Timegates to find the captures, so works across a number of systems. https://glam-workbench.github.io/web-archives/#create-and-compare-full-page-screenshots-from-archived-web-pages

There are lots of ways we can use Timemaps to explore change in a page. This notebook grabs each web capture in turn from a Timemap and looks at how the text has changed between captures – displaying deletions, additions, & changes over time. https://glam-workbench.github.io/web-archives/#display-changes-in-the-text-of-an-archived-web-page-over-time

That last notebook also runs as an app, and automatically generates a url you can share to load a particular page for analysis. So if you wanted to explore the history of that now notorious bio labs / pandemic blog post by Dominic Cummings, just go here: https://mybinder.org/v2/gh/GLAM-Workbench/web-archives/master?urlpath=/apps/display-text-changes-from-timemap.ipynb%3Furl%3D%22https%3A//dominiccummings.com/2019/03/04/the-most-secure-bio-labs-routinely-make-errors-that-could-cause-a-global-pandemic-are-about-to-re-start-experiments-on-pathogens-engineered-to-make-them-mammalian-airborne-transmissible/%22%26archive%3D%22ia%22

And yeah, thanks to @mybinderteam for being awesome.

If you just want to compare two different captures of a page, there's another app-ified notebook that gets the captures (via Timegates) and then compares them in a number of different ways – metadata, stats, links, cosine similarity, text/code changes: https://glam-workbench.github.io/web-archives/#compare-two-versions-of-an-archived-web-page

Have you ever wanted to find out when a particular word or phrase first appeared in (or disappeared from) a web page? Timemaps to the rescue! This notebook/app searches through all the captures until your phrase appears (or disappears). https://glam-workbench.github.io/web-archives/#find-when-a-piece-of-text-appears-in-an-archived-web-page

In that last notebook you can also choose to see all occurrences of a word or phrase, in which case it works through the entire Timemap and creates a little chart showing its frequency over time.

You might be wondering how I get the actual content of the archived pages without the navigation etc that web archives add in their UI. There's a little trick. Just get a link to a capture from the UI and add 'id_' immediately after the 14 digit timestamp.

In the case of @NLAPandora you also need to change the domain from ' http://webarchive.nla.gov.au ' to ' http://web.archive.org.au '.

As well as Timemaps & Timegates, some web archives make capture data available through CDX APIs. These are big indexes to all captures in the archive and can deliver LOTS of data. But you can easily filter the results.

I documented the @internetarchive's CDX API here: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/exploring_cdx_api.ipynb

I also examined how this differed from other CDX APIs: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/comparing_cdx_apis.ipynb

CDX APIs deliver much the same data as Timemaps, except that you can broaden your query from a single page to a whole domain. You can also filter by date, status, mimetype & more. Here's a notebook that describes harvesting domain-level data: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/harvesting_domain_data.ipynb

Using the CDX API you can also do intersting things like finding all the archived Powerpoint files in a particular domain! Here's a notebook that no only finds PP files, it downloads them, converts them to PDFs, creates screenshots & extracts text! https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/explore_presentations.ipynb

That last notebook also saves all the metadata, screenshots, and text into an SQLite database, and fires up Datasette to explore it. I put a demo on Glitch with all the archived Powerpoint files (many no longer accessible) from http://defence.gov.au : https://defencegovau-powerpoints.glitch.me/ 

You can also use the CDX API to harvest all the text from a selection of web pages. The API accepts regular expressions, so you can look for words in urls. Here's a notebook for generating text datasets from web archives: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/getting_text_from_web_pages.ipynb

So you could of course harvest lots of text across time from a selection of pages/sites and then feed them into your text analysis program of choice. You can choose to get *all* the text (menus & navigation) or just what looks to be the 'main' text.

Most of these examples work with a small subset of the masses of data available from web archives. You can start small and work up as required, rather than grappling with terabytes of data.

But I thought I'd also try something a little more ambitious, so using the IA CDX API I made an attempt to find all of the subdomains that have been captured from the http://gov.au  domain...

it took a while...

time passing...

... eventually I ended up with about 60gb of data, details of more than 189 million captures.

After a bit of manipulation I could find the unique subdomains – there's 26,233 of them. See the notebook: https://glam-workbench.github.io/web-archives/#exploring-subdomains-in-the-whole-of-govau

I wanted to have a go at visualising all those subdomains as a circular dendrogram. The scale means none of the text is readable, but hey, it looks cool. So here it is, subdomains in the http://gov.au  domain as captured by the Internet Archive.

And shifting up to third level http://gov.au  subdomains here's all of the Australian states and territories.

Zooming in on the ACT you can see the text...

So I hope this thread has got you thinking about ways you might start to use web archives in your own research. I've learnt a huge amount in the last few weeks and plan to do more exploration in the future.