How can I get Google search results with Mathematica?

Question

I am trying to write a function that, starting from a string, returns the Google search hits on that string (the format and other stuff, I still have to decide).

I have looked around and have found nothing that works (the wsdl-thing for Google search referenced by the docs, with WebServices, is retired and returns 404).

Any hints, except scraping the result page by hand?

Have a look at this answer. Presumably, Google expects you to switch to a newer API. — Leonid Shifrin, Mar 02 '13 at 23:18
Take a look at this answer. You can use the same approach I show there, but that's a deprecated API as well. It works now, but it won't keep working for long. — Szabolcs, Mar 02 '13 at 23:27

mgm · Accepted Answer · 2013-03-05T17:45:36.880

11

After looking better into the comment by Szabolcs, I have chosen this path (which I did try for images, but it didn't cross my mind to use it for web):

result = Import[ "https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=furry%20beasts", "JSON"]

and then

"url" /. ("results" /. ("responseData" /. result))

to get the links to the pages. Or, to get the "preview" of the pages:

"content" /. ("results" /. ("responseData" /. result))

It works like a charm even if it is unsupported and bound to break at some point.

For reference: https://developers.google.com/web-search/docs/

edited Mar 05 '13 at 17:45

answered Mar 05 '13 at 11:00

mgm

1,029
7
15

I would advise setting the User agent string if you do this. – William Nov 24 '13 at 04:20
@mgm How could we launch a browser from mma with a given url programatically? The front-end can do this already for hyperlinks... – M.R. Apr 08 '15 at 17:58

score 8 · Answer 2 · answered Nov 17 '15 at 18:53

Mathematica now supports a native connection to GoogleCustomSearch API.

You can do for example

gs = ServiceConnect["GoogleCustomSearch"]
gs["Search", {"Query" -> "Jennifer Lawrence"}]

You can also use image search

gs["Search",{"Query"-> "Jennifer Lawrence","SearchType"-> "Image"}]

To use GoogleCustomSearch API you need an API Key and a Custom search engine ID.

To get the API Key you first have to go to https://console.developers.google.com and create a project if you don't have one yet. Once you have the project (asuming you are using the new console interface), click the Credentials menu at the left. There you'll see all your credentials. If don't have one, click the Add credentials and, for GoogleCustomSearch, you need the API key Browser key type.

For the Custom search engine ID, you have to go to https://cse.google.com/all and add a search engine. Once you create it, just click on that and you'll get a screen with a lot of options. There's a button there that says Search engine ID. Click it and you'll get the ID you need.

Here is the official Wolfram documentation for Google API.

You may also want to try the BingSearch service also available from Mathematica. The functionality is basically the same. Here is the official documentation for that.

How can I get Google search results with Mathematica?

2 Answers2