Github Archive

From: Zachary Vance Date: Fri, 20 Nov 2015 06:52:14 +0000 (-0800) Subject: Add estimates X-Git-Url: https://git.za3k.com/?a=commitdiff_plain;h=8e943207f4368938ed3005c6d41b98af27f54878;p=za3k.git Add estimates --- diff --git a/github.html b/github.html index d54edd5..c9d39b6 100644 --- a/github.html +++ b/github.html @@ -1,9 +1,10 @@ -Github Archive +Github Backup -

Currently no one has archived github.com. This webpage is about progress toward that.

Currently no one has backed up github.com (aside from Github). This webpage is about progress toward that. If you have 150-200TB of disk space and really good internet, please contact me about getting a copy of github.

-I host the metadata for the repositories: +

List of Respositories

+I host some metadata about github's repositories. This includes a lot of basic data about the repository, but NOT the issues, any wiki, downloads, or the git repository:

Full repository metadata is available in batches of 10,000 at
```
http://za3k.com/github/repos-<X>0000-<X+1>0000.json
 http://za3k.com/github/repos-<X>0000-<X+1>0000.json.gz
```
@@ -13,19 +14,26 @@ for x in {0..100}; do \ wget "http://za3k.com/github/repos-$((x*10000))-$(((x+1)*10000)).json.gz"; \ done - The files are around 10G compressed, 100G uncompressed. + These files are around 10G compressed, 100G uncompressed. The format is explained on the github API. Files are grouped by id; since some repositories are deleted or privated, each one contains significantly less than 10,000 repositories.
You can grab greatly abbreviated metadata (recommended) as JSON.
You can grab greatly abbreviated metadata (recommended) as JSON. This includes the repository name and URL, a short description, whether it is a fork (and what of), and the approximate size of the repository.
Finally, you can get a txt file of just the repo names: txt.
I got data on github repositories using a custom tool I wrote which uses the github API.
This data was downloaded using a custom tool I wrote. My tool gets the data from the github API v3, with as little modification as possible.

List of Gists

Metadata for gists is currently unavailable from github, but I'm working with them to make it public.

-Additional information: +

Github Timeline

The Events Timeline is emphemeral, and being successfully recorded by githubarchive.org. A second person running the same program in case of downtime would be a plus.

+ +

Estimates on archiving repositories

I selected 1000 random respoitories from the above list, removing 427 forks. I then checked out all repositories. The total size was 4.3G, with or without compression. It was around 3 GB for a shallow checkout. If we assume forks take no space, this means an average github repository takes up 4.3M. Omitting the largest repositories may improve this estimate, but I didn't run further tests. I haven't checked, but the issue taken up by metadata like issues should be very small in comparison.

If there are 35,000,000 repositories on github at an average size of 4.3M each, that multiplies out to around 150TB data total.

+ +

Additional information

githubarchive.org archives the public timeline of github.
Archive Team has some information about github, and you probably should talk to them if you want to archive github. We basically just don't have anyone with space.
Archive Team has some information about github, and you probably should talk to them if you want to help back up github. We basically just don't have anyone with space--the technical issues are not too large.