<html>
-<head><title>Github Archive</title></head>
+<head><title>Github Backup</title></head>
<body>
-<p>Currently no one has archived github.com. This webpage is about progress toward that.</p>
+<p>Currently no one has backed up github.com (aside from Github). This webpage is about progress toward that. If you have 150-200TB of disk space and really good internet, please <a href="https://za3k.com">contact me</a> about getting a copy of github.</p>
-I host the metadata for the repositories:
+<h3>List of Respositories</h3>
+I host some metadata about github's repositories. This includes a lot of basic data about the repository, but NOT the issues, any wiki, downloads, or the git repository:
<ul>
<li>Full repository metadata is available in batches of 10,000 at <pre>http://za3k.com/github/repos-<X>0000-<X+1>0000.json
http://za3k.com/github/repos-<X>0000-<X+1>0000.json.gz</pre>
wget "http://za3k.com/github/repos-$((x*10000))-$(((x+1)*10000)).json.gz"; \
done
</pre>
- The files are around 10G compressed, 100G uncompressed.
+ These files are around 10G compressed, 100G uncompressed. The format is explained on the <a href="https://developer.github.com/v3/repos/#list-all-public-repositories">github API</a>. Files are grouped by id; since some repositories are deleted or privated, each one contains significantly less than 10,000 repositories.
</li>
- <li>You can grab greatly abbreviated metadata (recommended) as <a href="https://za3k.com/github/repos.json">JSON</a>.</li>
+ <li>You can grab greatly abbreviated metadata (recommended) as <a href="https://za3k.com/github/repos.json">JSON</a>. This includes the repository name and URL, a short description, whether it is a fork (and what of), and the approximate size of the repository.</li>
<li>Finally, you can get a txt file of just the repo names: <a href="https://za3k.com/github/repos.txt">txt</a>.</li>
- <li>I got data on github repositories using a <a href="https://github.com/za3k/github-backup">custom tool</a> I wrote which uses the github API.</li>
+ <li>This data was downloaded using a <a href="https://github.com/za3k/github-backup">custom tool</a> I wrote. My tool gets the data from the github API v3, with as little modification as possible.</li>
</ul>
+<h3>List of Gists</h3>
<p>Metadata for gists is currently unavailable from github, but I'm working with them to make it public.</p>
-Additional information:
+<h3>Github Timeline</h3>
+<p>The Events Timeline is emphemeral, and being successfully recorded by <a href="https://www.githubarchive.org">githubarchive.org</a>. A second person running the same program in case of downtime would be a plus.</p>
+
+<h3>Estimates on archiving repositories</h3>
+<p>I selected 1000 random respoitories from the above list, removing 427 forks. I then checked out all repositories. The total size was 4.3G, with or without compression. It was around 3 GB for a <a href="https://git-scm.com/docs/git-clone">shallow</a> checkout. If we assume forks take no space, this means an average github repository takes up 4.3M. Omitting the largest repositories may improve this estimate, but I didn't run further tests. I haven't checked, but the issue taken up by metadata like issues should be very small in comparison.</p>
+<p>If there are 35,000,000 repositories on github at an average size of 4.3M each, that multiplies out to around 150TB data total.</p>
+
+<h3>Additional information</h3>
<ul>
- <li><a href="https://www.githubarchive.org">githubarchive.org</a> archives the public timeline of github.</li>
- <li><a href="http://archiveteam.org/index.php?title=Github">Archive Team</a> has some information about github, and you probably should talk to them if you want to archive github. We basically just don't have anyone with space.</li>
+ <li><a href="http://archiveteam.org/index.php?title=Github">Archive Team</a> has some information about github, and you probably should talk to them if you want to help back up github. We basically just don't have anyone with space--the technical issues are not too large.</li>
</ul>
</body>
</html>
projects / za3k.git / commitdiff