jsoup – Make Me Engineer

Fetch contents(loaded through AJAX call) of a web page

November 5, 2022 by Tarik

Jsoup is a html parser only. Unfortunately it’s not possible to parse any javascript / ajax content, since jsoup can’t execute those. The solution: using a library which can handle Scripts. Here are some examples i know: HtmlUnit Java Script Engine Apache Commons BSF Rhino If such a library doesn’t support parsing or selectors, you … Read more

JSoup UserAgent, how to set it right?

July 26, 2022 by Tarik

You might try setting the referrer header as well: doc = Jsoup.connect(“https://www.facebook.com/”) .userAgent(“Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6”) .referrer(“http://www.google.com”) .get();