I've not yet seen recursion-prevention implementations that can provide
unencumbered service to valid customers while effectively denying access
to leisurely yet persistent recursive gets. You can definetely prevent
the aggressive recursion that is default wget behavior, but a
combination of lazy and determined I've yet to see confounded.....
On Mon, 2005-08-29 at 10:18 +0200, Matthijs R. Koot wrote:
Hi folks,
Which preventive or repressive measures could one apply to protect
larger dynamic websites against automated downloading by tools such as
WebCopier and Teleport Pro (or curl, for that matter)?
For a website like Amazon's, I reckon some technical measures would be
in place to protect against 'leakage' of all product information by such
tools (assuming such measures are justified by calculated risk). The
data we publish online are important company gems which we want to be
accessible by any visitor, but to be protected against systematic
download in either non-intentional context (like Internet Explorer's
built-in MSIECrawler) or intentional context (WebCopier, Teleport, ...).
Consider this:
detailpage.html?bid=0000001
detailpage.html?bid=0000002
detailpage.html?bid=0000003
(...)
Or with multiple levels:
detailpage.html?bid=0000001&t=1
detailpage.html?bid=0000001&t=2
detailpage.html?bid=0000002&t=1
detailpage.html?bid=0000002&t=2
(...)
In specific, I was wondering if it's possible and sensible to limit the
allowed number of requests for certain pages per minute/hour. At the
same time, the data displayed by detailpage.html should be indexable by
Google, so the data itself can't be hidden behind a user login and it's
not possible to use any client-side scripting as Google doesn't
interpret it. I'm using Apache 2 on RedHat 4 Enterprise and know about
mod_throttle (which doesn't work with Apache 2) and mod_security (which
also offers some 'throttling' functionality, regression, but is only
able to work with individual requests and can't remember request sequences).
I'd also suppose that dealing with proxy servers of large ISPs, like
AOL, is a big caveat.
Any ideas?
Best regards,
Matthijs
