9

I installed media-wiki on ec2 machine and loaded the Wikipedia page-articles dump content and other relevant data into that. I want to update the data regularly(daily) but I didn't get any resource for that. Is there any source from where I can get Wikipedia daily updates?

Note: I can get the list of modified files using bots. But I need to get the modified content also.

Glorfindel
  • 320
  • 1
  • 2
  • 14
vinod
  • 91
  • 3

3 Answers3

6

Right now, there is no good solution. The options I can think of are:

  1. Just download the normal dump whenever it becomes available. This means you won't get daily updates at all. For the English Wikipedia, a new dump is generated about once a month.

  2. Use the adds/changes dumps (already mentioned by ojdo). There are two problems with using this:

    1. It doesn't include all changes. Specifically, information about moves, deletes and some undeletes are not included.
    2. It's an experimental feature and I wouldn't be surprised if it were discontinued in the near future (because a better option is coming, see below).

  3. Use the API (possibly combined with the IRC recent changes feed) to get the text of all new revisions to a wiki. I think this might work for very small wikis, but it's certainly not feasible for huge active wikis like the English Wikipedia.

  4. Use binary incremental dumps (which is a project I built over this summer). This will do exactly what you want: it will allow you to download only changes since the last dump and it should allow creating dumps much more often (the hope is for daily dumps). The only problem is that this is not live yet, so you will have to wait before using this (I have no idea how long, but I would expect it to go live this year).

svick
  • 869
  • 4
  • 9
  • It's strange that Wikimedia hasn't come up yet with an easy way to "track" changes. OpenStreetMap (diffs) seems to be in much better shape in comparison. – ojdo Sep 30 '13 at 11:17
  • Right now only the second solution helps in my project(Not completely..But something is better than nothing.). Dbpedia is getting live updates from wikipedia. I don't know how much effort it takes to make that live feed public to everyone. – vinod Oct 01 '13 at 05:15
  • @vinod I don't think DBPedia does that. For example, when I look at dbpedia-owl:wikiPageRevisionID of Albert Einstein, it points to a revision from April. – svick Oct 01 '13 at 08:05
  • @svick There are 2 versions of dbpedia. dbpedia, dbpedia-live. I was referring to dbpedia-live.

    Try the query SELECT * WHERE { <http://en.wikipedia.org/wiki/Albert_Einstein> ?p ?o } at "http://live.dbpedia.org/sparql" (DBPedia live data has been last updated on aug 29th, 2013. It is down for maintenance purpose. It should be up and running very soon.)

     – vinod Oct 01 '13 at 08:47
  • @vinod It seems you're right. It looks like that uses update feed service, which is not public. – svick Oct 01 '13 at 08:53
4

There exists an experimental Wikimedia adds/changes dump service. Another approach is generating the differential dump yourself using deltapack, discussed about in this thread in the Wikitech mailing list.

ojdo
  • 2,804
  • 14
  • 31
  • Thanks for the references. I will look into them and get back to you. – vinod Sep 30 '13 at 09:43
  • Note that the adds/changes dumps include only new revisions, but not deletions or things like that. – svick Sep 30 '13 at 10:24
  • And generating the diff itself won't help, if there is nothing to diff (the dumps for Wikipedias are not generated every day). – svick Sep 30 '13 at 10:26
3

It wouldn't give you complete updates, but there is also dbpedia live. Dbpedia only covers the stuctured text boxes in the wikipedia entries. It is monitoring the changes on Wikipedia though.

Andra
  • 687
  • 3
  • 10
  • @Andra.. I tried that already but rate of error is more than we expected and they stopped providing updates from aug 29th(They told me that they would start providing them asap). – vinod Oct 14 '13 at 09:52