Re: [Lynx-dev] sudden site access failure: realclearpolitics.com

[Rudy Vener]

Hi Karen,

I know that realclearpolitics.com uses cloud flare because five days ago  I was 
blocked with a cloud flare denial
message.

This was fixed by the user agent string I included in a previous post.


Then yesterday I was blocked again with this new cookies and ad blocker message.

This time changing the user agent string does nothing. Catlynx is still blocked.
For the moment links the chain still works. But who knows for how long.

I am not a member of realclearpolitics, but I do plan to visit
their contact page and see if I can find their web administrators email 
address. 

Will post to the list with the results.

Rudy
On Wed, Nov 10, 2021 at 09:27:22PM -0500, Karen Lewellen wrote:
> Hi Rudy,
> There is no reference to cloudflare when visiting
> www.realclealpolitics.com
> All cookies are accepted as well.
> While  I went through fine when you first asked, visiting manually, as well
> as via your link creates a 403 error, presenting the js and ad blocker
> reference.
> Writing this reminded me to try with no user agent sent, to see if I get
> through.
> Are you a member, if so do you have a contact?
> Karen
> 
> 
> 
> On Wed, 10 Nov 2021, Rudy Vener wrote:
> 
> > Hi Travis,
> > 
> > First, to address the screen reader issue, in my vi edit buffer, where
> > I read all my email, I did:
> > :s/lynx/catlynx/g
> > 
> > Now as far as the links and catlynx behavior, it may be
> > that links spoofs the presence of the ad  rendering modules that the
> > cloud flare gatekeeper requires, along with active cookies.
> > 
> > Now I normally turn on cookies in catlynx, but they seem to be hit or miss.
> > 
> > Rudy
> > 
> > On Wed, Nov 10, 2021 at 08:20:01PM -0500, Travis Siegel wrote:
> > > (screen readers will have a tough time with this message, because catlynx 
> > > with
> > > a y and links with an I are sounding the same, so apologies in advance for
> > > any confusion).
> > > 
> > > 
> > > Apparently it isn't the agent header, because on my softcon.com server,
> > > links works (as it does for you) but catlynx does not.?? Even when I tell 
> > > catlynx
> > > to mimic the useragent string sent by links.
> > > 
> > > Interesting enough, the first time I tried catlynx with the links 
> > > headers, I
> > > got a 408 error, which claimed the server took too long to respond, even
> > > though it only took a couple seconds to pop up.
> > > 
> > > I looked at the man pages for links, and I couldn't find anything that 
> > > would
> > > account for the acceptance there and not on catlynx. <shrug>
> > > 
> > > 
> > > On 11/10/2021 3:48 PM, Rudy Vener wrote:
> > > > Hi Karen,
> > > > They are probably not blocking by IP or they would have blocked access 
> > > > from links (the chain) as well.
> > > > 
> > > > Thinking they might be blocking by a combination of both IP and User 
> > > > agent trigger, I changed the user agent
> > > > for catlynx (the cat) with another UA string I got from the online UA 
> > > > database.
> > > > Now I called  realclearpolitics.com with:
> > > > /usr/local/bin/catlynx-2.9.0-dev9 -useragent="Mozilla/5.0 (Macintosh; 
> > > > U; PPC Mac OS X Mach-O; XH; rv:8.578.498) fr, Gecko/20121021 
> > > > Camino/8.723+ (Firefox compatible)" $*
> > > > 
> > > > No joy. Still blocked with catlynx (the cat.)  Links (the chain) still 
> > > > works.
> > > > 
> > > > I don't suppose you could try sending me shellworld's catlynx config 
> > > > file, could you?
> > > > 
> > > > On Wed, Nov 10, 2021 at 02:06:45PM -0500, Karen Lewellen wrote:
> > > > > Hi Rudy,
> > > > > Have  you tried this site today?
> > > > > Granted I am using the Lynx configuration at shellworld, but I just 
> > > > > typed
> > > > > the  address you provided, and reached the page perfectly.
> > > > > something to consider about cloudflare is that it tracks sometimes by 
> > > > > IP
> > > > > address.
> > > > > Would love to learn how the fanfiction.net site works with your user 
> > > > > agent
> > > > > though, smiles.
> > > > > Karen
> > > > > 
> > > > > 
> > > > > 
> > > > > On Wed, 10 Nov 2021, Rudy Vener wrote:
> > > > > 
> > > > > > Hi all,
> > > > > > 
> > > > > > Yesterday realclearpolitics.com started returning the following page
> > > > > > when I tried to access it with catlynx.
> > > > > > 
> > > > > >    Please enable JS and disable any ad blocker
> > > > > > 
> > > > > > I invoke catlynx as follows:
> > > > > > /usr/local/bin/catlynx-2.9.0-dev9 -useragent="Mozilla/5.0 
> > > > > > (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/537.1 (KHTML, like 
> > > > > > Gecko) Chrome/21.0.1180.79 Safari/537.1 catlynx" $*
> > > > > > 
> > > > > > The above user agent spoof normally lets me slip the Cloud Flare 
> > > > > > gatekeepers, but this seems to be something new.
> > > > > > 
> > > > > > For what it's worth, the same problem now occurs with w3m, however,
> > > > > > links (the chain)  can access the site just fine.
> > > > > > If there is a simple configuration fix for this, I'd really like to 
> > > > > > know.
> > > > > > 
> > > > > > Thank you.
> > > > > > 
> > > > > > --
> > > > > > Rudy Vener
> > > > > > Website: http://www.rudyvener.com
> > > > > > Twitter: https://twitter.com/RudySalt
> > > > > > 
> > > > > > 
> > 
> > -- 
> > Rudy Vener
> > Website: http://www.rudyvener.com
> > Twitter: https://twitter.com/RudySalt
> > 
> > 

-- 
Rudy Vener
Website: http://www.rudyvener.com
Twitter: https://twitter.com/RudySalt