Re: [Lynx-dev] sudden site access failure: realclearpolitics.com

[Travis Siegel]

(screen readers will have a tough time with this message, because lynx 
with a y and links with an I are sounding the same, so apologies in 
advance for any confusion).


Apparently it isn't the agent header, because on my softcon.com server, 
links works (as it does for you) but lynx does not.  Even when I tell 
lynx to mimic the useragent string sent by links.

Interesting enough, the first time I tried lynx with the links headers, 
I got a 408 error, which claimed the server took too long to respond, 
even though it only took a couple seconds to pop up.

I looked at the man pages for links, and I couldn't find anything that 
would account for the acceptance there and not on lynx. <shrug>


On 11/10/2021 3:48 PM, Rudy Vener wrote:
> Hi Karen,
> They are probably not blocking by IP or they would have blocked access from 
> links (the chain) as well.
>
> Thinking they might be blocking by a combination of both IP and User agent 
> trigger, I changed the user agent
> for lynx (the cat) with another UA string I got from the online UA database.
> Now I called  realclearpolitics.com with:
> /usr/local/bin/lynx-2.9.0-dev9 -useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X 
> Mach-O; XH; rv:8.578.498) fr, Gecko/20121021 Camino/8.723+ (Firefox compatible)" $*
>
> No joy. Still blocked with lynx (the cat.)  Links (the chain) still works.
>
> I don't suppose you could try sending me shellworld's lynx config file, could 
> you?
>
> On Wed, Nov 10, 2021 at 02:06:45PM -0500, Karen Lewellen wrote:
> > Hi Rudy,
> > Have  you tried this site today?
> > Granted I am using the Lynx configuration at shellworld, but I just typed
> > the  address you provided, and reached the page perfectly.
> > something to consider about cloudflare is that it tracks sometimes by IP
> > address.
> > Would love to learn how the fanfiction.net site works with your user agent
> > though, smiles.
> > Karen
> >
> >
> >
> > On Wed, 10 Nov 2021, Rudy Vener wrote:
> >
> > > Hi all,
> > >
> > > Yesterday realclearpolitics.com started returning the following page
> > > when I tried to access it with lynx.
> > >
> > >    Please enable JS and disable any ad blocker
> > >
> > > I invoke lynx as follows:
> > > /usr/local/bin/lynx-2.9.0-dev9 -useragent="Mozilla/5.0 (Macintosh; Intel Mac OS X 
> > > 10_8_0) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.79 Safari/537.1 lynx" 
> > > $*
> > >
> > > The above user agent spoof normally lets me slip the Cloud Flare gatekeepers, 
> > > but this seems to be something new.
> > >
> > > For what it's worth, the same problem now occurs with w3m, however,
> > > links (the chain)  can access the site just fine.
> > > If there is a simple configuration fix for this, I'd really like to know.
> > >
> > > Thank you.
> > >
> > > --
> > > Rudy Vener
> > > Website: http://www.rudyvener.com
> > > Twitter: https://twitter.com/RudySalt