This week's book giveaway is in the Reactive Progamming forum.
We're giving away four copies of Reactive Streams in Java: Concurrency with RxJava, Reactor, and Akka Streams and have Adam Davis on-line!
See this thread for details.
Win a copy of Reactive Streams in Java: Concurrency with RxJava, Reactor, and Akka Streams this week in the Reactive Progamming forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Junilu Lacar
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Knute Snortum
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Ganesh Patekar

emulating a synchronous html request

 
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hello, i found this forum entry:
https://coderanch.com/t/517587/oa/Web-Crawler-Java

and looked over this link:
http://java-source.net/open-source/html-parsers

I am looking for something unusual. I wish to automatically parse the HTML (and possibly get a screenshot) of some websites that seem to be circumventing asynchronous HTML requests.

Some sites are built so that if you use one of the many asynchronous methods that are provided in many programming examples that you always return a page that doesn't match what is currently there. It is like they are "masking" their page from automated parsing.

(If you attempt to do a GetURL, or LoadStrings or all of the other methods that I have seen, you make an asynchronous request. The sites in question thwart it by rendering content you don't want.)

Is it possible to emulate a synchronous HTML request and still get the benefits of HTML parsing and/or screenshots? Thanks.

 
Fred Scott
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi,

i hope i'm not bumping this thread too soon. However, does anyone have any thoughts about my question?

thanks,
 
You'll never get away with this you overconfident blob! The most you will ever get is this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!