Re: [Lynx-dev] Alternative User-Agent header

[Tim Chase]

On 2021-06-24 13:49, Stef Caunter wrote:
> alias lynx='/usr/local/bin/lynx -nopause -tna -useragent=IE . '
> 
> "IE" seems to be the most innocuous choice for the major news
> sites. I have not tried "googlebot" ;)

I'm a big fan of using the googlebot User Agent on the grounds that
I've encountered a number of sites where the paywall allows Google
through to index the full article content (and occasionally doesn't
bother to serve advertisements to the Googlebot) but prevents other
user-agent strings from receiving the article.

-tim