Nope, BeautifulSoup, by itself, does not support XPath expressions. An alternative library, lxml, does support XPath 1.0. It has a BeautifulSoup compatible mode where it’ll try and parse broken HTML the way Soup does. However, the default lxml HTML parser does just as good a job of parsing broken HTML, and I believe is faster. … Read more
Python 2 Using urllib.urlretrieve import urllib urllib.urlretrieve(“http://www.gunnerkrigg.com//comics/00000001.jpg”, “00000001.jpg”) Python 3 Using urllib.request.urlretrieve (part of Python 3’s legacy interface, works exactly the same) import urllib.request urllib.request.urlretrieve(“http://www.gunnerkrigg.com//comics/00000001.jpg”, “00000001.jpg”)
By adding a few more headers I was able to get the data: import urllib2,cookielib site= “http://www.nseindia.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=JPASSOCIAT&fromDate=1-JAN-2012&toDate=1-AUG-2012&datePeriod=unselected&hiddDwnld=true” hdr = {‘User-Agent’: ‘Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11’, ‘Accept’: ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8’, ‘Accept-Charset’: ‘ISO-8859-1,utf-8;q=0.7,*;q=0.3’, ‘Accept-Encoding’: ‘none’, ‘Accept-Language’: ‘en-US,en;q=0.8’, ‘Connection’: ‘keep-alive’} req = urllib2.Request(site, headers=hdr) try: page = urllib2.urlopen(req) except urllib2.HTTPError, e: print e.fp.read() content = … Read more
Python 2 What you’re looking for is urllib.quote_plus: safe_string = urllib.quote_plus(‘string_of_characters_like_these:$#@=?%^Q^$’) #Value: ‘string_of_characters_like_these%3A%24%23%40%3D%3F%25%5EQ%5E%24′ Python 3 In Python 3, the urllib package has been broken into smaller components. You’ll use urllib.parse.quote_plus (note the parse child module) import urllib.parse safe_string = urllib.parse.quote_plus(…)
This is my understanding of what the relations are between the various “urllibs”: In the Python 2 standard library there exist two HTTP libraries side-by-side. Despite the similar name, they are unrelated: they have a different design and a different implementation. urllib was the original Python HTTP client, added to the standard library in Python … Read more
I was getting the same UnicodeEncodeError when saving scraped web content to a file. To fix it I replaced this code: with open(fname, “w”) as f: f.write(html) with this: with open(fname, “w”, encoding=”utf-8″) as f: f.write(html) If you need to support Python 2, then use this: import io with io.open(fname, “w”, encoding=”utf-8″) as f: f.write(html) … Read more
urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error