0

I need to fetch some journals issues, in example like this: www.sciencedirect.com/science/journal/00963003/124/1

You may check this part to be constant: 

string = "https://www.sciencedirect.com/science/journal/00963003"

I read here that I can do something like src=Import["https://www.sciencedirect.com/science/journal/00963003/124/1","HTML"]

But this is only for one issue. In this example there are as many as 310 volumes and each volume may have none, 1, 2, 2-3,3 or 4 issues. Is it possible to write a script in matematica to fech all HTML pages of issues at once, or at least at minimal number of fetching trials? Sorry, I cannot make more than 2 links, so some links are without https://

David G. Stork
  • 41,180
  • 3
  • 34
  • 96
iot
  • 619
  • 3
  • 9
  • A simple Import states that I'm "Not entitled to full text". I don't know how to incorporate permissions in such requests. – David G. Stork May 03 '17 at 18:41
  • 11
    Keep in mind that most journals have software that will stop access (for you and often your institution) if you do this systematically. – Dunlop May 03 '17 at 18:48
  • @Dunlop, thank you; probably they do. But what if I need just abstracts or, say, I will r script with random pause to search at random... – iot May 04 '17 at 04:13
  • Sorry I don't know, as it depends a lot on the publisher and their terms of use. This you will have to check out yourself for each publisher. Alternatively if you are only looking for abstracts, try checking databases like pubmed which should be searchable through Mathematica via open APIs – Dunlop May 04 '17 at 06:09
  • 2
    Import the website above your known issue (e.g., https://www.sciencedirect.com/science/journal/00963003/124 or https://www.sciencedirect.com/science/journal/00963003) and parse it in order to determine which issues lie below it. Then use those results to import the desired volumes. – Jack LaVigne May 04 '17 at 12:56

2 Answers2

3

If the sequences can be enumerated in a Table, then try this:

Table[
 Import[
"http://www.sciencedirect.com/science/journal/00963003/124/" <> ToString[mycounter], 
  "HTML"], 
{mycounter, 1, 3}]

or if a list of texts...

... {mylist, {"myjournal1/1","myjournal1/2","myjournal1/May1999","myjournal2/August2011""}

etc.

David G. Stork
  • 41,180
  • 3
  • 34
  • 96
  • I have no idea what "like Descartes 310 on 5" means. If there is a list 1,2,3,5,6 (missing 4), then simply place // Quiet at the end of the function call and any errors will be "silent." – David G. Stork May 04 '17 at 15:45
2

Is it what you are after?

issues = Flatten@
 Table[(ToString[i] <> "/" <> #) & /@ {"", "1", "2", "2-3", "4"}, {i, 1, 310}]

Then one could StringJoin, smth like:

links = StringJoin[{string, "/", #}] & /@ issues;

"XMLObject" is a better option. Fetch first five objects:

Import[#, "XMLObject"] & /@ Take[links, 5] // AbsoluteTiming

{32.3833, {...1...}}

It took me half of a minute for five pages fetching. (BTW, from PC that has no subscription.)

Proof of work:

% [[2]] // Length
(* 5 *)

Disclaimer I would not recommend to use it from institute network.

In no event will we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising out of, or in connection with, the use of this code. ;-))

garej
  • 4,865
  • 2
  • 19
  • 42
  • 1
    I already solved it but thanks anyway. the question is not well-accepted here – iot May 20 '17 at 05:53