From: Zachary Vance <za3k@za3k.com>
Date: Mon, 23 Nov 2015 07:05:21 +0000 (-0800)
Subject: Fill in missing numbers
X-Git-Url: https://git.za3k.com/?a=commitdiff_plain;h=255222dcb296d6f9d5029fa6542db1d4974c003e;p=za3k.git

Fill in missing numbers
---

diff --git a/github.html b/github.html
index 85c473a..8b541d8 100644
--- a/github.html
+++ b/github.html
@@ -4,18 +4,17 @@
 <p>Currently no one has backed up github.com (aside from Github). This webpage is about progress toward that. If you have 150-200TB of disk space and really good internet, please <a href="https://za3k.com">contact me</a> about getting a copy of github.</p>
 
 <h3>List of Respositories</h3>
-I host some metadata about github's repositories. This includes a lot of basic data about the repository, but NOT the issues, any wiki, downloads, or the git repository:
+I host some metadata about github's repositories. This includes a lot of basic data about the repository, but NOT the issues, any wiki, downloads, or the git repository. As of Nov 2015, github has 28 million repositories.
 <ul>
-	<li><p>Full repository metadata is available in JSON format. The format is explained on the <a href="https://developer.github.com/v3/repos/#list-all-public-repositories">github API</a>.</p>
+	<li><p>Full repository metadata is available in JSON format. The format is explained on the <a href="https://developer.github.com/v3/repos/#list-all-public-repositories">github API</a>. These files contain repeat data because of an error on my part, I am fixing the historical data but you may wish to hold off downloading it for now.</p>
     <p>The files are available in batches of 10,000 at <pre>http://za3k.com/github/repos-&lt;X>0000-&lt;X+1>0000.json
 http://za3k.com/github/repos-&lt;X>0000-&lt;X+1>0000.json.gz</pre>
     To download all files, run <pre>
-
-for x in {0..100}; do \
-  wget "http://za3k.com/github/repos-$((x*10000))-$(((x+1)*10000)).json.gz"; \
-done
+    for x in {0..4700}; do \
+      echo "https://za3k.com/github/repos-$((x*10000))-$(((x+1)*10000)).json.gz"; \
+    done | wget -nc -i -
     </pre>
-    The combined size of these files is 10G compressed, 100G uncompressed. Files are grouped by github's internal id; since some repositories are deleted or privated, each file contains less than 10,000 repositories.
+    The combined size of these files is <b>15G compressed</b>, 168G uncompressed. Files are grouped by github's internal id; since some repositories are deleted or privated, each file contains less than 10,000 repositories.
 </li>
 	<li>You can grab greatly abbreviated metadata (recommended) as <a href="https://za3k.com/github/repos.json">JSON</a>. This includes the repository name and URL, a short description, whether it is a fork (and what of), and the approximate size of the repository.</li>
 	<li>Finally, you can get a txt file of just the repo names: <a href="https://za3k.com/github/repos.txt">txt</a>.</li>
@@ -30,7 +29,7 @@ done
 
 <h3>Estimates on archiving repositories</h3>
 <p>I selected 1000 random repositories from the above list, removing 427 forks. I then checked out all repositories. The total size was 4.3G, with or without compression. It was around 3 GB for a <a href="https://git-scm.com/docs/git-clone">shallow</a> checkout. If we assume forks take no space, this means an average github repository takes up 4.3M. Omitting the largest repositories may improve this estimate, but I didn't run further tests.</p>
-<p>If there are 35,000,000 repositories on github at an average size of 4.3M each, that multiplies out to around 150TB data total for the git repositories.</p>
+<p>If there are 28,000,000 repositories on github at an average size of 4.3M each, that multiplies out to around 120TB data total for the git repositories.</p>
 
 <h3>Additional information</h3>
 <ul>