Since v1.2.3 Requests added the PreparedRequest object. As per the documentation “it contains the exact bytes that will be sent to the server”. One can use this to pretty print a request, like so: import requests req = requests.Request(‘POST’,’http://stackoverflow.com’,headers={‘X-Custom’:’Test’},data=”a=1&b=2″) prepared = req.prepare() def pretty_print_POST(req): “”” At this point it is completely built and ready to … Read more
You can easily create a persistent session using: s = requests.Session() After that, continue with your requests as you would: s.post(‘https://localhost/login.py’, login_data) # logged in! cookies saved for future requests. r2 = s.get(‘https://localhost/profile_data.json’, …) # cookies sent automatically! # do whatever, s will keep your cookies intact 🙂 For more about Sessions: https://docs.python-requests.org/en/latest/user/advanced/#session-objects
You are looking for the request history. The response.history attribute is a list of responses that led to the final URL, which can be found in response.url. response = requests.get(someurl) if response.history: print(“Request was redirected”) for resp in response.history: print(resp.status_code, resp.url) print(“Final destination:”) print(response.status_code, response.url) else: print(“Request was not redirected”) Demo: >>> import requests >>> … Read more
Starting with Requests version 2.4.2, you can use the json= parameter (which takes a dictionary) instead of data= (which takes a string) in the call: >>> import requests >>> r = requests.post(‘http://httpbin.org/post’, json={“key”: “value”}) >>> r.status_code 200 >>> r.json() {‘args’: {}, ‘data’: ‘{“key”: “value”}’, ‘files’: {}, ‘form’: {}, ‘headers’: {‘Accept’: ‘*/*’, ‘Accept-Encoding’: ‘gzip, deflate’, ‘Connection’: … Read more
Good news: there is now a requests module that supports javascript: https://pypi.org/project/requests-html/ from requests_html import HTMLSession session = HTMLSession() r = session.get(‘http://www.yourjspage.com’) r.html.render() # this call executes the js in the page As a bonus this wraps BeautifulSoup, I think, so you can do things like r.html.find(‘#myElementID’).text which returns the content of the HTML element … Read more
Just use requests’ features: import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(connect=3, backoff_factor=0.5) adapter = HTTPAdapter(max_retries=retry) session.mount(‘http://’, adapter) session.mount(‘https://’, adapter) session.get(url) This will GET the URL and retry 3 times in case of requests.exceptions.ConnectionError. backoff_factor will help to apply delays between attempts to avoid to fail again … Read more
Requests is not a built in module (does not come with the default python installation), so you will have to install it: OSX/Linux Use $ pip install requests (or pip3 install requests for python3) if you have pip installed. If pip is installed but not in your path you can use python -m pip install … Read more
The user-agent should be specified as a field in the header. Here is a list of HTTP header fields, and you’d probably be interested in request-specific fields, which includes User-Agent. If you’re using requests v2.13 and newer The simplest way to do what you want is to create a dictionary and specify your headers directly, … Read more
Since you’re using requests, you should use the response’s json method. import requests response = requests.get(…) data = response.json() It autodetects which decoder to use.
Provide a User-Agent header: import requests url=”http://www.ichangtou.com/#company:data_000008.html” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36’} response = requests.get(url, headers=headers) print(response.content) FYI, here is a list of User-Agent strings for different browsers: List of all Browsers As a side note, there is a pretty useful third-party package called … Read more