Prev: StopKa - desktop search tool, version 1.5_100520
Next: Gordon Clown and freeware
From: AlleyCat on 11 May 2010 01:29

We regret to announce that our Google scraper may have to be permanently retired, thanks to a
change at Google. It depends on whether Google is willing to restore the simple interface that
we've been scraping since Scroogle started five years ago. Actually, we've been using that
interface for scraping since Google-Watch.org began in 2002.

This interface was remarkably stable all that time. During those eight years there were only about
five changes that required some programming adjustments. Also, this interface was available at
every Google data center in exactly the same form, which allowed us to use 700 IP addresses for
Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a
redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were
generic during that entire time. It didn't show the snippets unless you moused-over the links it
produced (they were there for our program, so that was okay), and it has never had any ads. Our
impression was that these results were from Google's basic algorithms, and that extra features and
ads were added on top of these generic results. Three years ago Google launched "Universal
Search," which meant that they added results from other Google services on their pages. But this
simple interface we were using was not affected at all.

Now that interface is gone. It is not possible to continue Scroogle unless we have a simple
interface that is stable. Google's main consumer-oriented interface that they want everyone to use
is too complex, and changes too frequently, to make our scraping operation possible.

Over the next few days we will attempt to contact Google and determine whether the old interface is
gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us
where it is so that we can continue to use it.

Thank you for your support during these past five years. Check back in a week or so; if we don't
hear from Google by next week, I think we can all assume that Google would rather have no Scroogle,
and no privacy for searchers, at all.

� Daniel Brandt, Public Information Research, scroogle AT lavabit.com
From: VanguardLH on 11 May 2010 07:42
AlleyCat wrote:

> We regret to announce that our Google scraper may have to be permanently
> retired, thanks to a change at Google. It depends on whether Google is
> willing to restore the simple interface that we've been scraping since
> Scroogle started five years ago. Actually, we've been using that
> interface for scraping since Google-Watch.org began in 2002.
>
> This interface was remarkably stable all that time. During those eight
> years there were only about five changes that required some programming
> adjustments. Also, this interface was available at every Google data
> center in exactly the same form, which allowed us to use 700 IP addresses
> for Google.
>
> That interface was at www.google.com/ie but on May 10, 2010 they took it
> down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to
> have a search box, and the results it showed were generic during that
> entire time. It didn't show the snippets unless you moused-over the links
> it produced (they were there for our program, so that was okay), and it
> has never had any ads. Our impression was that these results were from
> Google's basic algorithms, and that extra features and ads were added on
> top of these generic results. Three years ago Google launched "Universal
> Search," which meant that they added results from other Google services
> on their pages. But this simple interface we were using was not affected
> at all.
>
> Now that interface is gone. It is not possible to continue Scroogle
> unless we have a simple interface that is stable. Google's main
> consumer-oriented interface that they want everyone to use is too
> complex, and changes too frequently, to make our scraping operation
> possible.
>
> Over the next few days we will attempt to contact Google and determine
> whether the old interface is gone as a matter of policy at Google, or if
> they simply have it hidden somewhere and will tell us where it is so that
> we can continue to use it.
>
> Thank you for your support during these past five years. Check back in a
> week or so; if we don't hear from Google by next week, I think we can all
> assume that Google would rather have no Scroogle, and no privacy for
> searchers, at all.
>
> � Daniel Brandt, Public Information Research, scroogle AT lavabit.com

Tis the problem with screen/URL scrapers: any change to the web pages or the
navigation between them results in the scraper program becoming unusable.
Some enterprise-level screen-scraping utilities (very expensive) allow the
user to remap the screens or their navigation and even provide a flowchart
of screens so the interconnecting logic for the underlying program remains
intact or requires little modification. Hand-built screen-scraping programs
can be labor intensive.

That a simple change can render a screen scraper unusable is why I no longer
use YahooPOPs or the Web proxy extension for Thunderbid or FreePOPs. I'm
not waiting around under the developer gets notified of the change and until
they get around to modifying their program to match the changes at the site.
While e-mail is too important to let sit unusable for any period of time, I
suppose Scroogle isn't considered a critical app or service by users.

I doubt Google cares about any 3rd party service riding on Google works. If
the screens/pages have changed, and there currently exists no toggle option
to switch between new and old interfaces, it is highly unlikely that Google
is going to revert to the old pages or old navigation paths between them.
Change doesn't always mean better or improved but it rarely results in
changing back. Evolution doesn't guarantee a better outcome, just a
different outcome.
From: Slarty on 12 May 2010 12:25
On Tue, 11 May 2010 20:48:35 +0200, Yrrah wrote:

> OT
>
> schr�dinger's cat <sc(a)invalid.invalid>:
>
>> I've switched to Ixquick until this shakes out:
>
> I wish I could say that it's as good as or better than G., but it
> isn't.
> Anyone using Clusty (Vivisimo)?
> http://clusty.com/
>
> Yrrah

Don't bother, Scroogle's back now (Wednesday).

Cheers,

Roy
 | 
Pages: 1
Prev: StopKa - desktop search tool, version 1.5_100520
Next: Gordon Clown and freeware