Scraping High-Res images from the MoMA and the Van Gogh Museum websites

Question

Many art websites allow the user to zoom into an image to see it at high resolution by using image tiles. It is great on the website, but presents problems if one wishes to compile a complete high resolution picture from these tiles. I am currently using the following:

mOMA[name_, catalogueNumber_] := 
With[{src = 
Import[StringJoin["http://www.moma.org/collection/works/", 
  ToString[catalogueNumber], "?locale=en"], "Source"]}, 
With[{aa = 
 StringReplace[
  StringReplace[
   StringDrop[
    StringTake[src, 
     StringPosition[src, "allZoomBounds"][[1, 1]] - 11], 
    StringPosition[src, "allImageTiles"][[1, 2]] + 2], 
   "/media/" -> "http://www.moma.org/media/"], "]]" -> ","]}, 
With[{bb = StringDrop[aa, (Last@StringPosition[aa, "[["])[[2]]]}, 
With[{cc = {Length@StringPosition[bb, "["], Length@bb}}, 
 With[{dd = 
    StringReplace[StringReplace[bb, "[" -> ""], "]" -> ""]}, 
  With[{ee = StringPosition[dd, ","][[All, 1]]}, 
   With[{ccc = {Length@ee/(cc[[1]] + 1), (cc[[1]] + 1)}}, 
    With[{d1 = StringReplace[bb, "[" -> ""]}, 
     With[{d2 = Length@StringPosition[d1, "]"][[All, 1]] + 1}, 
      With[{cccc = {(Length@ee)/d2, d2}}, 
       With[{ff = 
          StringReplace[
             StringReplace[
              StringDrop[StringTake[dd, ee[[#]]], 
               If[# == 1, 0, ee[[# - 1]] + 1]], "\"" -> ""], 
             "," -> ""] & /@ Range@(Times @@ ccc)}, 
        With[{img = {ff, cccc}}, 
         With[{imgim = Import[#] & /@ img[[1]]}, 
          Rasterize[
           ImageAssemble[
            Table[imgim[[#]] & /@ (x + 
                img[[2, 1]] Range[0, img[[2, 2]] - 1]), {x, 1, 
              img[[2, 1]]}]], ImageResolution -> 100, 
           ImageSize -> 600]
          ]]]]]]]]]]]]]

which works with something like mOMA[Picasso_Minotauromachy _ 1935, 60110], but seems like an awful lot of fiddling to isolate & ImageAssemble relevant image tiles (which on the MOMA site are sandwiched between allImageTiles and allZoomBounds in the source code). Is there a more straightforward way, eg with Cases and "XMLObject", or similar?

Update

In response to requested edits below, the following is the basis of the code:

With[{src = Import[StringJoin["http://www.moma.org/collection/works/", 
ToString[60110], "?locale=en"], "Source"]}, 
StringReplace[StringReplace[StringDrop[StringTake[src, StringPosition[src, 
"allZoomBounds"][[1, 1]] - 11], StringPosition[src, "allImageTiles"][[1, 2]] 
+ 2], "/media/" -> "http://www.moma.org/media/"], "]]" -> ","]]

with the rest of the code botched together to isolate each tile (each http string is a tile).

I agree that the code above, as presented is rather unreadable, with a load of nested With arguments that probably don't help. I will do my best to format appropriately, but am giving basic info in update for now, to hopefully clarify things a bit.

Update 2

Using @belisarus' code, thithe following works for the specific image request from @bills in the comments

With[{aa = ToExpression@Most[StringDrop[#, 2] & /@StringCases[
Import["https://vangoghmuseum-assetserver.appspot.com/tiles?id=\
4820945634066432", "Text"], Shortest["url" ~~ a : __ ~~ "},"] -> a]]},
Rasterize[ImageAssemble[Table[Import@# &@aa[[x + 15*#]], {x, 1, 15}] & /@   
Range[0, 9]], ImageResolution -> 100, ImageSize -> 600]]

... still working on generalisation.

Generalisation

doc = d0364V1968;
pollard = With[{num = StringReplace[StringReplace[StringReplace[StringCases[
Import[StringJoin["http://www.vangoghmuseum.nl/en/collection/", 
ToString[doc]], "Text"], Shortest["tiles?id=" ~~ a : __ ~~ "data"] -> a], 
"\"" -> ""], "\n" -> ""], " " -> ""][[1]]},
With[{aa = 
  Import[StringJoin[
    "https://vangoghmuseum-assetserver.appspot.com/tiles?id=", 
    num], "Text"]}, 
With[{x = 
   Max@ToExpression@
     StringReplace[
      StringDrop[#, 3] & /@ (StringTake[#, 5] & /@ 
         StringCases[aa, Shortest["x" ~~ a : __ ~~ "},"]]), 
      "," -> ""]},
With[{y = 
    Max@ToExpression@
      StringReplace[
       StringDrop[#, 3] & /@ (StringTake[#, 5] & /@ 
          StringCases[aa, Shortest["y" ~~ a : __ ~~ "},"]]), 
       "," -> ""]},
With[{bb = 
     ToExpression@
      Most[StringDrop[#, 2] & /@ 
        StringCases[aa, Shortest["url" ~~ a : __ ~~ "},"] -> a]]},
   Rasterize[ImageAssemble[
    Table[Import@# &@bb[[xx + (x + 1)*#]], {xx, 1, x + 1}] & /@ 
     Range[0, y - 1]], ImageResolution -> 100, ImageSize -> 600]
   ]]]]]

Disclaimer

Please read the terms of use for images from each website before assembling large image sizes from tiles (hence the use of Rasterize and setting appropriate ImageResolution).

Martin, do you think you could format the code in such a way that it is human-readable (i.e. add indents, new lines)? As it is now, I cannot even begin to make any sense of it. — MarcoB, Aug 20 '15 at 11:48
Naming the variables "dd", "cccc", and so on is really not helping, either. — Patrick Stevens, Aug 20 '15 at 12:01
@ScottSEA If the painting itself is old enough, then I believe this is fine in the USA: http://photo.stackexchange.com/questions/54787/can-a-photo-of-a-book-page-be-copyrighted But this legal stuff can be complicated, so just take my link as a starting point for research. If a museum make a high quality digital reproduction of a painting, and the painting itself is public domain, then the reproduction cannot be copyrighted in the USA. In other countries it may be different. — Szabolcs, Aug 21 '15 at 08:41
It looks like I'm late to the party, but I find it a shame, that you've deleted so much of your code from previous revisions. It added some specific examples to the problem, as well as a bit of context on where to look. A question in the state it is in now would almost certainly be closed as too-broad/unclear-what-youre-asking. — LLlAMnYP, Aug 21 '15 at 09:43
Please do not remove details from your question again. Without the context and code, it is useless to anyone else. — rm -rf, Aug 22 '15 at 16:06
@TheToad I am concerned about legal implications. Please advise. — martin, Aug 22 '15 at 16:44
Right now the code is just words on a screen. It contains no images that are subject to copyright and the code, I presume, is your own work which is licensed CC by SA 3.0. Legal implications fall on the person executing the code, who would be the person doing the downloading of said images. Belisarius' answer also uses a value of 1 which is allowed, per my reading of the meta post that you asked and is now deleted. Most jurisdictions have exclusions for non-commercial, home use, fair use, etc. so don't think it'd be a problem. I'm not a lawyer, though. — rm -rf, Aug 22 '15 at 17:04
@TheToad I agree with you that it is just words on a page, but it is going against my wishes to keep the text as it currently stands. Could you please suggest a compromise? — martin, Aug 23 '15 at 09:02
@martin What sort of compromise would satisfy you? We try to discourage people asking detailed questions, getting others to spend time on it and answer it, only to then realize that they probably should not have posted it/their teacher might find out that they cheated/it was their company's proprietary data/the question was silly, etc. People have all sorts of reasons, but unless there is a demonstrable case of actual, explicit copyright infringement, we don't remove stuff. And I don't think your question comes even close to actual infringement (enables it, with the right parameters, perhaps). — rm -rf, Aug 23 '15 at 15:11
@martin I suppose you could add something like that to work around it. I haven't read your code in detail and it's hard to follow because of the variable names and deep nesting, but now that you know the issue, the site's stance and what could be done (if it needs to be done), I trust you'll make the necessary (hopefully minor) modifications to the code, mostly for your own peace of mind :) — rm -rf, Aug 23 '15 at 17:04
@TheToad many thanks for your advise on this. I have made the necessary minor adjustments. Please let me know if suitable or not. — martin, Aug 23 '15 at 17:07
Awesome! Thanks for your patience and for making these edits :) — rm -rf, Aug 23 '15 at 17:08
@TheToad no problem - glad we could work together to get a satisfying solution for both parties :) — martin, Aug 23 '15 at 17:09
@R.M. Could you tell please where did you find that only value of scale 1 is allowed for downloading? Seems like they allow to download any image data from their website for non-commercial educational usage. https://www.moma.org/about/about_site/index — Temak, Apr 25 '16 at 19:44

Dr. belisarius · Accepted Answer · 2016-04-26T05:24:49.953

18

MoMA

res= {1,2,3..} is the resolution.Use res = -1 for the max available resolution, but beware of a shipload of lawyers, middle managers, telephone sanitisers and hairdressers that may try to prosecute you if you use a value greater than 1,

moma[catalogueNumber_, res_] := 
  Module[{m = "http://www.moma.org", src, sj, rep, exp, a},
  rep = {"]" -> "}", "[" -> "{"};
  exp = Shortest["allImageTiles =" ~~ a : __ ~~ ";"] -> a;
  src = Import[StringJoin[m, "/collection/works/", ToString@catalogueNumber], "Source"];
  sj = Map[Import[m <> #] &, (ToExpression@
                         StringReplace[StringCases[src, exp], rep])[[1,1,res]], {2}];
  ImageAssemble@Transpose@sj]

moma[60110, 1]

Mathematica graphics

edited Apr 26 '16 at 05:24

answered Aug 20 '15 at 14:49

Dr. belisarius

115,881
13
203
453

thank you - much clearer! & well done for deciphering my awful unreadable code!! :) – martin Aug 20 '15 at 15:13
@martin I didn't :) I just took a look at the input and output ;D – Dr. belisarius Aug 20 '15 at 15:16
much more sensible! ;) – martin Aug 20 '15 at 15:18
The name variable isn't used. So for example, you can get a Van Gogh using just the item number: moma[xxx, 79802, 1] – bill s Aug 20 '15 at 16:20
1

@belisarius -- This is an amazing piece of detective work! Can you see how to modify the calls to import the highest resolution images from the Van Gogh museum... for instance, from here: http://www.vangoghmuseum.nl/en/collection/d0364V1968 – bill s Aug 20 '15 at 16:51
@bills I'm all ears :) – Dr. belisarius Aug 20 '15 at 19:18
@bills tiles for that one can be found here Most[StringDrop[#, 2] & /@ StringCases[ Import["https://vangoghmuseum-assetserver.appspot.com/tiles?id=\ 4820945634066432"], Shortest["url" ~~ a : __ ~~ "},"] -> a]] – martin Aug 20 '15 at 19:51
@belisarius re bill s' request, tiles can be found here @bills tiles for that one can be found here Most[StringDrop[#, 2] & /@ StringCases[ Import["https://vangoghmuseum-assetserver.appspot.com/tiles?id=\ 4820945634066432"], Shortest["url" ~~ a : __ ~~ "},"] -> a]] - still working on putting htem together! – martin Aug 20 '15 at 19:52
1

@martin Working on it. It can be done in a cleaner way – Dr. belisarius Aug 20 '15 at 19:52
@belisarius at one point, I managed to find the full resolution image somewhere, but like George's Marvellous Medicine, I have no idea how I did it! - Can't seem to get it back, but it is hiding there somewhere! – martin Aug 20 '15 at 19:54
@bills See the other answer – Dr. belisarius Aug 20 '15 at 20:31
@belisarius snap! (almost!!) ;) – martin Aug 20 '15 at 20:33
@bills See update to OP – martin Aug 20 '15 at 20:33
Let us continue this discussion in chat. – martin Aug 21 '15 at 18:25
@Dr.belisarius I would appreciate greatly if you could write the same snippet in python:) Or if it is possible, explain please how it works. I don't know Mathematica very well, but I need to scrap a couple of thousands of images from MoMA:) – Temak Apr 25 '16 at 13:09
@Temak That is off topic here.I suggest you as for it on stackoverflow.com – Dr. belisarius Apr 26 '16 at 05:17

Dr. belisarius · Answer 2 · 2015-08-28T17:59:16.200

13

Van Gogh Museum

Here is the same for the tiles structure of the Van Gogh Museum at @bills' request. Note that the structure is completely different. I preserved the same resolution convention:

vanInTheSkyWithLucy[catalog_, res_] := 
 Module[{i, df, zooms, c, maxXY, t}, 
  i = Import["http://www.vangoghmuseum.nl/en/collection/" <> catalog, "XMLObject"];
  df = First@  Cases[i, XMLElement[__, {___, "data-url" -> a__, ___}, ___] -> a, ∞];
  zooms = Reverse["levels" /. Import[df, "JSON"]];
  If [$VersionNumber >= 10,
     c = Cases[#, {"x" -> x_, "y" -> y_, "url" -> u_} :>{x, y,  u},∞],
     c = Cases[#, {"url" -> u_, "x" -> x_, "y" -> y_} :>{x, y,  u},∞]] &@zooms[[res]];
  maxXY = Max /@ (Transpose[c][[;; 2]]) + 1;
  t = ConstantArray[1, Reverse@maxXY];
  (t[[#[[2]] + 1, #[[1]] + 1]] = #[[3]]) & /@ c;
  ImageAssemble@Map[Import, t, {2}]]

vanInTheSkyWithLucy["d0364V1968", 1]

Mathematica graphics

Warning The code works differently on V9 and V10, so I added a version check.Thanks lot to @bills for the debugging! Apparently the JSON import works differently.

edited Aug 28 '15 at 17:59

answered Aug 20 '15 at 20:31