How to save a web page in mathematica?

Question

For a given web page http://www.nytimes.com/, is it possible to save it as :

first: only HTML(within css)

second: more elements(include pictures and so on)

just like what a browser can do.

I have tried this, but it only generates a large picture with plain text.

Export["F:\\nytimes.html", Import["http://www.nytimes.com"], "HTML"]

Answer 1

using Mathematica 9 the easiest way is URLSave

URLSave["http://www.nytimes.com" , "C:\\temp\\test9.html"]

you get the output

"C:\temp\test9.html"

and then directly from within Mathematica open your html default browser

SystemOpen[%]

with earlier versions try the following

source = Import["http://www.nytimes.com", "Source"];
Export["C:\\temp\\test8.html", source, "Text"]

you get the output

"C:\temp\test8.html"

SystemOpen[%]

Answer 2

I can understand, if Mathematica does not provide such functionality. It is running on top of an operating system, which delivers all the functionality to do these things, like socket I/O etc.

I don't see the point to do this inside of Mathematica.

What you can do is this:

a) unix plattform

Run["/path/to/wget", "http://www.nytimes.com"];

This is just running wget with the default settings. wget does have a load of options which you can set to modify its result (for instance, if I want to download a webpage and its requisites (css links to other pages) I regularly use wget -E -H -k -K -p).

b) windows plattform

In case you don't want to download wget for windows...

1) write a powershell script (wget.ps1):

    (new-object System.Net.WebClient).
         DownloadFile($args[0],'C:\tmp\index.html')

2) @Bobthechemist found out how to run this on Windows platform:

Run["powershell.exe -ExecutionPolicy ByPass -file c:\\tmp\\wget.ps1 \ http://www.nytimes.com"]

Once the webpage is downloaded, you can start to do all the extractions you want to do.

Edit 1:

Since I've read in your comments I know that you're about to think of a pure Java solution, you might consider this:

import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;

public class WebPageSaver {
//    public static void main(String args[]) throws Exception {
//      saveWebpage("http://www.nytimes.com", "/path/to/your/home/index.html");
//    }

    public static void saveWebpage(String site, String target ) throws Exception {
        OutputStream out = new FileOutputStream(target);
        URL url = new URL(site);
        URLConnection conn = url.openConnection();
        conn.connect();
        InputStream is = conn.getInputStream();

        byte[] buffer = new byte[4096];
        while(true) {
            int numBytes = is.read(buffer);
            if(numBytes == -1) {
                break;
            }
            out.write(buffer, 0, numBytes);
        }
    }
}

And then you can use one of the many approaches to link this into you Mathematica environment. Like:

1) Java-Reloader by Leonid <-- recommended!

2) Hands down approach

P.S.: i wrote the windows part answer just out of my memory and i don't have anything here to verify that this is working...