Hi,
Am Mittwoch, 19. Oktober 2005 09:03 schrieb Nik Cubrilovic:
When you have content of high value at stake, the
'other side' seems to get more sophisticated as opposed to your
standard home user who has downloaded a website scraper from
download.com.
I think this is the root of the problem. You're publishing valuable
content. The word "publish" already implies that your content is
publicly visible. This means that what you're trying to achieve is
actually a paradox: you want to "protect" content that is already
visible to the general public. This in turn means that no solution
to your problem exists.
What your tips are leading towards are ways to
distinguish human visitors from bots, which with some attackers simply
leads to a game of cat-and-mouse as opposed to a solution that can be
handed to the client.
Yup, and that is about the best you can achieve. Since you're already
publishing your valuable content, the best you can do is make it more
expensive for the attacker to "steal" it. The downside is, (as you
found out) that raising the cost for the attacker usually turns away
some of your legitimate users as well.
An upper limit for the attacker's cost could be estimated as the cost
for paying a number of dumb users who actually surf around on your site
through a logging proxy server. I guess that kind of labour is available
for little money in some parts of the world. If your content is more
valuable than that, you're lost - you cannot win the race.
I have contacted a number of appliance vendors to see if they offer a
transparent application-layer firewall that could identify bad bots
and drop them, but surprisingly not one had a solution to offer.
I don't find that surprising. If a company came up with a technical
solution to the problem, an attacker could produce a bot that evades the
specific protection provided by that solution. The more wide-spread
such a solution would be, the more effort could be invested by an attacker
(because the payoff would be higher). Again, this is a race that noone
can win.
Bye,
Peter
--
Peter Conrad Tel: +49 6102 / 80 99 072
[ t]ivano Software GmbH Fax: +49 6102 / 80 99 071
Bahnhofstr. 18 http://www.tivano.de/
63263 Neu-Isenburg
Germany
