lynx-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: lynx-dev user agent when running lynx via crontab entry?


From: mattack
Subject: Re: lynx-dev user agent when running lynx via crontab entry?
Date: Fri, 9 Jul 1999 10:47:43 -0700 (PDT)

On Thu, 8 Jul 1999, David Woolley wrote:

>Date: Thu, 8 Jul 1999 08:29:54 +0100 (BST)
>From: David Woolley <address@hidden>
>Reply-To: address@hidden
>To: address@hidden
>Subject: Re: lynx-dev user agent when running lynx via crontab entry?
>
>> /usr/local/bin/lynx -useragent='Mozilla/4.04 [en]' -nolist -dump 
>> http://www.imdb.com/StudioBrief | mail address@hidden
>> 
>> So, I was getting a "we're not letting you connect" message for the past
>> week or so.. and after actually reading the message, it is being sent back
>> because the useragent is empty.
>> 
>
>Based on public statements by the IMDB operator in the UK, and some 
>followup email, this rejection is almost certainly to prevent bulk
>fetching of pages by web crawlers.  Blocking lynx from automatically
>fetching single pages would probably considered a desirable side effect,
>as that also causes pages to be fetched without the adverts being read.

Strange.. it _did_ work yesterday (my 4pm automated email of StudioBriefing).
..and I had made no further changes after the initial -useragent attempt.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]