The error code 10060 means it cannot connect to the remote peer. It might be because of the network problem or mostly your setting issues, such as proxy setting. You could try to connect the same host with other tools(such as ncat) and/or with another PC within your same local network to find out where … Read more
Pandas can do this right out of the box, saving you from having to parse the html yourself. read_html() extracts all tables from your html and puts them in a list of dataframes. to_csv() can be used to convert each dataframe to a csv file. For the web page in your example, the relevant table … Read more
When you download a file with urllib or urllib2, you can find out whether a charset header was transmitted: fp = urllib2.urlopen(request) charset = fp.headers.getparam(‘charset’) You can use BeautifulSoup to locate a meta element in the HTML: soup = BeatifulSoup.BeautifulSoup(data) meta = soup.findAll(‘meta’, {‘http-equiv’:lambda v:v.lower()==’content-type’}) If neither is available, browsers typically fall back to user … Read more
From the docs Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data: data = urllib.parse.urlencode(d).encode(“utf-8”) req = urllib.request.Request(url) with urllib.request.urlopen(req,data=data) as f: resp = f.read() print(resp)
This is how you do it. from urllib import request, parse data = parse.urlencode(<your data dict>).encode() req = request.Request(<your url>, data=data) # this will make the method “POST” resp = request.urlopen(req)
Python 3 from urllib.error import HTTPError Python 2 from urllib2 import HTTPError Just catch HTTPError, handle it, and if it’s not Error 404, simply use raise to re-raise the exception. See the Python tutorial. Here is a complete example for Python 2: import urllib2 from urllib2 import HTTPError try: urllib2.urlopen(“some url”) except HTTPError as err: … Read more
The http.client.HTTPResponse-object you get back from urlopen() has a _io.BufferedReader-object attached, and this object cannot be pickled. pickle.dumps(urllib.request.urlopen(‘http://www.python.org’).fp) Traceback (most recent call last): … pickle.dumps(urllib.request.urlopen(‘http://www.python.org’).fp) TypeError: cannot serialize ‘_io.BufferedReader’ object multiprocessing.Pool will need to pickle (serialize) the results to send it back to the parent process and this fails here. Since dummy uses threads instead … Read more
While socket.setsocketimeout will set the default timeout for new sockets, if you’re not using the sockets directly, the setting can be easily overwritten. In particular, if the library calls socket.setblocking on its socket, it’ll reset the timeout. urllib2.open has a timeout argument, hovewer, there is no timeout in urllib2.Request. As you’re using mechanize, you should … Read more
Do it in stages, and modify the object, like this: # make a string with the request type in it: method = “POST” # create a handler. you can specify different handlers here (file uploads etc) # but we go for the default handler = urllib2.HTTPHandler() # create an openerdirector instance opener = urllib2.build_opener(handler) # … Read more
import requests img_data = requests.get(image_url).content with open(‘image_name.jpg’, ‘wb’) as handler: handler.write(img_data)