web scraping dynamic content with python

by

Instead of trying to reverse engineer it, you can use ghost.py to directly interact with JavaScript on the page.

If you run the following query in a chrome console, you’ll see it returns everything you want.

document.getElementsByClassName('inline-text-org');

Returns

[<div class=​"inline-text-org" title=​"University of Manchester">​University of Manchester​</div>, 
 <div class=​"inline-text-org" title=​"University of California Irvine">​University of California ...​</div>​
  etc...

You can run JavaScript through python in a real life DOM using ghost.py.

This is really cool:

from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://academic.research.microsoft.com/Search?query=lander')
result, resources = ghost.evaluate(
    "document.getElementsByClassName('inline-text-org');")

Leave a Comment

techhipbettruvabetnorabahisbahis forumu