Getting started with python requests — get requests

Request and Response Objects¶

Whenever a call is made to requests.*() you are doing two major things. First, you are constructing a object which will be sent of to a server to request or query some resource. Second, a object is generated once gets a response back from the server. The response object contains all of the information returned by the server and also contains the object you created originally. Here is a simple request to get some very important information from Wikipedia’s servers:

>>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')

If we want to access the headers the server sent back to us, we do this:


>>> r.headers
{'content-length': '56170', 'x-content-type-options': 'nosniff', 'x-cache':
'HIT from cp1006.eqiad.wmnet, MISS from cp1010.eqiad.wmnet', 'content-encoding':
'gzip', 'age': '3080', 'content-language': 'en', 'vary': 'Accept-Encoding,Cookie',
'server': 'Apache', 'last-modified': 'Wed, 13 Jun 2012 01:33:50 GMT',
'connection': 'close', 'cache-control': 'private, s-maxage=0, max-age=0,
must-revalidate', 'date': 'Thu, 14 Jun 2012 12:59:39 GMT', 'content-type':
'text/html; charset=utf-8', 'x-cache-lookup': 'HIT from cp1006.eqiad.wmnet:3128,
MISS from cp1010.eqiad.wmnet:80'}

However, if we want to get the headers we sent the server, we simply access the request, and then the request’s headers:

Main Interface¶

All of Requests’ functionality can be accessed by these 7 methods. They all return an instance of the object.

(method, url, **kwargs)

Constructs and sends a .

Parameters:
  • method – method for the new object: , , , , , , or .
  • url – URL for the new object.
  • params – (optional) Dictionary, list of tuples or bytes to send in the query string for the .
  • data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the .
  • json – (optional) A JSON serializable Python object to send in the body of the .
  • headers – (optional) Dictionary of HTTP Headers to send with the .
  • cookies – (optional) Dict or CookieJar object to send with the .
  • files – (optional) Dictionary of (or ) for multipart encoding upload. can be a 2-tuple , 3-tuple or a 4-tuple , where is a string defining the content type of the given file and a dict-like object containing additional headers to add for the file.
  • auth – (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  • timeout ( or ) – (optional) How many seconds to wait for the server to send data before giving up, as a float, or a tuple.
  • allow_redirects () – (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to .
  • proxies – (optional) Dictionary mapping protocol to the URL of the proxy.
  • verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to .
  • stream – (optional) if , the response content will be immediately downloaded.
  • cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.
Returns:

object

Return type:

Usage:

>>> import requests
>>> req = requests.request('GET', 'https://httpbin.org/get')
>>> req
<Response >
(url, **kwargs)

Sends a HEAD request.

Parameters:
  • url – URL for the new object.
  • **kwargs – Optional arguments that takes. If allow_redirects is not provided, it will be set to False (as opposed to the default behavior).
Returns:

object

Return type:
(url, params=None, **kwargs)

Sends a GET request.

Parameters:
  • url – URL for the new object.
  • params – (optional) Dictionary, list of tuples or bytes to send in the query string for the .
  • **kwargs – Optional arguments that takes.
Returns:

object

Return type:
(url, data=None, json=None, **kwargs)

Sends a POST request.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the .
  • json – (optional) json data to send in the body of the .
  • **kwargs – Optional arguments that takes.
Returns:

object

Return type:
(url, data=None, **kwargs)

Sends a PUT request.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the .
  • json – (optional) json data to send in the body of the .
  • **kwargs – Optional arguments that takes.
Returns:

object

Return type:
(url, data=None, **kwargs)

Sends a PATCH request.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the .
  • json – (optional) json data to send in the body of the .
  • **kwargs – Optional arguments that takes.
Returns:

object

Return type:

Internals¶

These items are an internal component to Requests, and should never be seen by the end user (developer). This part of the API documentation exists for those who are extending the functionality of Requests.

Classes

class requests.models.Request(url=None, headers={}, files=None, method=None, data={}, params={}, auth=None, cookiejar=None, timeout=None, redirect=False, allow_redirects=False, proxies=None)

The object. It carries out all functionality of Requests. Recommended interface is with the Requests functions.

allow_redirects = None

Set to True if full redirects are allowed (e.g. re-POST-ing of data at new Location)

auth = None

AuthObject to attach to .

cookiejar = None

CookieJar to attach to .

data = None

Dictionary or byte of request body data to attach to the .

files = None

Dictionary of files to multipart upload ({filename: content}).

headers = None

Dictonary of HTTP Headers to attach to the .

method = None

HTTP Method to use. Available: GET, HEAD, PUT, POST, DELETE.

params = None

Dictionary or byte of querystring data to attach to the .

redirect = None

True if is part of a redirect chain (disables history and HTTPError storage).

response = None

instance, containing content and metadata of HTTP Response, once sent.

send(anyway=False)

Sends the request. Returns True of successful, false if not. If there was an HTTPError during transmission, self.response.status_code will contain the HTTPError code.

Once a request is successfully sent, sent will equal True.

Parameters: anyway – If True, request will be sent, even if it has

already been sent.

sent = None

True if Request has been sent.

timeout = None

Float describ the timeout of the request.

url = None

Request URL.

API and Module Auto-Generated Documentation¶

This module provides the API for .

The module provides a clean and simple API for making requests via a thread pool. The thread pool will use sessions for increased performance.

A simple use-case is:

from requests_toolbelt import threaded

urls_to_get = [{
    'url' 'https://api.github.com/users/sigmavirus24',
    'method' 'GET',
}, {
    'url' 'https://api.github.com/repos/requests/toolbelt',
    'method' 'GET',
}, {
    'url' 'https://google.com',
    'method' 'GET',
}]
responses, errors = threaded.map(urls_to_get)

By default, the threaded submodule will detect the number of CPUs your computer has and use that if no other number of processes is selected. To change this, always use the keyword argument . Using the above example, we would expand it like so:

responses, errors = threaded.map(urls_to_get, num_processes=10)

You can also customize how a is initialized by creating a callback function:

from requests_toolbelt import user_agent

def initialize_session(session):
    session.headers'User-Agent' = user_agent('my-scraper', '0.1')
    session.headers'Accept' = 'application/json'

responses, errors = threaded.map(urls_to_get,
                                 initializer=initialize_session)
(requests, **kwargs)

Simple interface to the threaded Pool object.

This function takes a list of dictionaries representing requests to make using Sessions in threads and returns a tuple where the first item is a generator of successful responses and the second is a generator of exceptions.

Parameters:
  • requests () – Collection of dictionaries representing requests to make with the Pool object.
  • **kwargs – Keyword arguments that are passed to the object.
Returns:

Tuple of responses and exceptions from the pool

Return type:

(, )

Inspiration is blatantly drawn from the standard library’s multiprocessing library. See the following references:

  • multiprocessing’s pool source
  • map and map_async
class (job_queue, initializer=None, auth_generator=None, num_processes=None, session=<class ‘requests.sessions.Session’>)

Pool that manages the threads containing sessions.

Parameters:
  • queue () – The queue you’re expected to use to which you should add items.
  • initializer (collections.Callable) – Function used to initialize an instance of .
  • auth_generator (collections.Callable) – Function used to generate new auth credentials for the session.
  • num_process () – Number of threads to create.
  • session (requests.Session) –
()

Iterate over all the exceptions in the pool.

Returns: Generator of
classmethod (exceptions, **kwargs)

Create a from an s.

Provided an iterable that provides objects, this classmethod will generate a new pool to retry the requests that caused the exceptions.

Parameters:
  • exceptions (iterable) – Iterable that returns
  • kwargs – Keyword arguments passed to the initializer.
Returns:

An initialized object.

Return type:
classmethod (urls, request_kwargs=None, **kwargs)

Create a from an iterable of URLs.

Parameters:
  • urls (iterable) – Iterable that returns URLs with which we create a pool.
  • request_kwargs () – Dictionary of other keyword arguments to provide to the request method.
  • kwargs – Keyword arguments passed to the initializer.
Returns:

An initialized object.

Return type:
()

Get an exception from the pool.

Return type:
()

Get a response from the pool.

Return type:
()

Join all the threads to the master thread.

()

Iterate over all the responses in the pool.

Returns: Generator of
class (request_kwargs, response)

A wrapper around a requests Response object.

This will proxy most attribute access actions to the Response object. For example, if you wanted the parsed JSON from the response, you might do:

thread_response = pool.get_response()
json = thread_response.json()
= None

The original keyword arguments provided to the queue

= None

The wrapped response

class (request_kwargs, exception)

A wrapper around an exception raised during a request.

This will proxy most attribute access actions to the exception object. For example, if you wanted the message from the exception, you might do:

thread_exc = pool.get_exception()
msg = thread_exc.message
= None

The captured and wrapped exception

= None

The original keyword arguments provided to the queue

Using Python requests

We will use the module requests for learning about http request.

pip install requests 

In the below example we see a case of simple GET request annd print out the result of the response. We choose to print only the first 300 characters.

# How to make http request
import requests as req
r = req.get('http://www.tutorialspoint.com/python/')
print(r.text)

When we run the above program, we get the following output −

<!DOCTYPE html>
<!--><html class="ie ie8"> <!-->
<!--><html class="ie ie9"> <!-->
<!--><!-->	<html> <!--<!-->
<head>
<!-- Basic -->
<meta charset="utf-8">
<title>Python Tutorial</title>
<meta name="description" content="Python Tutorial

Previous Page Print Page

Next Page  

Status Codes

The first thing we can do is check the status code. HTTP codes range from the 1XX to 5XX. Common status codes that you have probably seen are 200, 404, and 500.

Here’s a quick overview of what each status code means:

  • 1XX — Information
  • 2XX — Success
  • 3XX — Redirect
  • 4XX — Client Error (you messed up)
  • 5XX — Server Error (they messed up)

Generally, what you’re looking for when you perform your own requests are status codes in the 200s.

Requests recognizes that 4XX and 5XX status codes are errors, so if those status codes get returned, the response object from the request evaluates to False.

You can test if a request responded successfully by simply checking the response for truth. For example:

The message «Response Failed» will only appear if a 400 or 500 status code returns. Try changing the URL to some nonsense to see the response fail with a 404.

You can take a look at the status code directly by doing:

This will show you the status code directly so you can check the number yourself.

Our First Request

To start, let’s use Requests for something simple: requesting the Scotch.io site. Create a file called script.py and add the following code to it. In this article, we won’t have much code to work with, so when something changes you can just update the existing code instead of adding new lines.

So all this code is doing is sending a GET request to Scotch.io. This is the same type of request your browser sent to view this page, but the only difference is that Requests can’t actually render the HTML, so instead you will just get the raw HTML and the other response information.

We’re using the .get() function here, but Requests allows you to use other functions like .post() and .put() to send those requests as well.

You can run it by executing the script.py file.

And here’s what you get in return:

Main Interface¶

All of Request’s functionality can be accessed by these 7 methods. They all return an instance of the object.

(method, url, **kwargs)

Constructs and sends a . Returns object.

Parameters:
  • method – method for the new object.
  • url – URL for the new object.
  • params – (optional) Dictionary or bytes to be sent in the query string for the .
  • data – (optional) Dictionary, bytes, or file-like object to send in the body of the .
  • headers – (optional) Dictionary of HTTP Headers to send with the .
  • cookies – (optional) Dict or CookieJar object to send with the .
  • files – (optional) Dictionary of ‘name’: file-like-objects (or {‘name’: (‘filename’, fileobj)}) for multipart encoding upload.
  • auth – (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  • timeout – (optional) Float describing the timeout of the request.
  • allow_redirects – (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
  • proxies – (optional) Dictionary mapping protocol to the URL of the proxy.
  • verify – (optional) if , the SSL cert will be verified. A CA_BUNDLE path can also be provided.
  • stream – (optional) if , the response content will be immediately downloaded.
  • cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.

Usage:

>>> import requests
>>> req = requests.request('GET', 'http://httpbin.org/get')
<Response >
(url, **kwargs)

Sends a HEAD request. Returns object.

Parameters:
  • url – URL for the new object.
  • **kwargs – Optional arguments that takes.
(url, **kwargs)

Sends a GET request. Returns object.

Parameters:
  • url – URL for the new object.
  • **kwargs – Optional arguments that takes.
(url, data=None, **kwargs)

Sends a POST request. Returns object.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, bytes, or file-like object to send in the body of the .
  • **kwargs – Optional arguments that takes.
(url, data=None, **kwargs)

Sends a PUT request. Returns object.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, bytes, or file-like object to send in the body of the .
  • **kwargs – Optional arguments that takes.
(url, data=None, **kwargs)

Sends a PATCH request. Returns object.

Parameters:
  • url – URL for the new object.
  • data – (optional) Dictionary, bytes, or file-like object to send in the body of the .
  • **kwargs – Optional arguments that takes.
(url, **kwargs)

Sends a DELETE request. Returns object.

Parameters:
  • url – URL for the new object.
  • **kwargs – Optional arguments that takes.

Содержимое ответа

Мы можем читать содержимое ответа сервера. Рассмотрим тайм-лайн GitHub снова:

Requests будет автоматически декодировать содержимое ответа сервера. Большинство Unicode кодировок без проблем декодируются.

Когда вы делаете запрос, Requests делает предположение о кодировке, основанное на заголовках HTTP. Кодировка текста, угаданная Requests, используется при обращение к r.text. Вы можете узнать, какую кодировку использует Requests, и изменить её воспользовавшись свойством r.encoding:

Если вы измените кодировку, Requests будет использовать новое значение r.encoding всякий раз, когда вы будете использовать r.text. Вы можете сделать это в любой ситуации, где нужна более специализированная логика работы с кодировкой содержимого ответа. Например, в HTML и XML есть возможность задавать кодировку прямо в теле документа. В подобных ситуациях вы должны использовать r.content, чтобы найти кодировку, а затем установить r.encoding. Это позволит вам использовать r.text с правильной кодировкой.

Requests может также использовать пользовательские кодировки в случае, если вы в них нуждаетесь. Если вы создали свою собственную кодировку и зарегистрировали её в модуле codecs, вы можете просто использовать название кодека в качестве значения r.encoding, и Requests будет работать с этой кодировкой для вас.

Session Objects¶

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use ‘s connection pooling. So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).

A Session object has all the methods of the main Requests API.

Let’s persist some cookies across requests:

s = requests.Session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

Sessions can also be used to provide default data to the request methods. This is done by providing data to the properties on a Session object:

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test' 'true'})

# both 'x-test' and 'x-test2' are sent
s.get('http://httpbin.org/headers', headers={'x-test2' 'true'})

Any dictionaries that you pass to a request method will be merged with the session-level values that are set. The method-level parameters override session parameters.

Note, however, that method-level parameters will not be persisted across requests, even if using a session. This example will only send the cookies with the first request, but not the second:

s = requests.Session()
r = s.get('http://httpbin.org/cookies', cookies={'from-my' 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'

r = s.get('http://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'

If you want to manually add cookies to your session, use the to manipulate .

Sessions can also be used as context managers:

with requests.Session() as s
    s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')

This will make sure the session is closed as soon as the block is exited, even if unhandled exceptions occurred.

Remove a Value From a Dict Parameter

Sometimes you’ll want to omit session-level keys from a dict parameter. To do this, you simply set that key’s value to in the method-level parameter. It will automatically be omitted.

Обработка исключений в requests

При работе с внешними сервисами никогда не стоит полагаться на их отказоустойчивость. Всё упадёт рано или поздно, поэтому нам, программистам, необходимо быть всегда к этому готовыми, желательно заранее и в спокойной обстановке.

Итак, как у requests дела обстоят с различными факапами в момент сетевых соединений? Для начала определим ряд проблем, которые могут возникнуть:

  • Хост недоступен. Обычно такого рода ошибка происходит из-за проблем конфигурирования DNS. (DNS lookup failure)
  • «Вылет» соединения по таймауту
  • Ошибки HTTP. Подробнее о HTTP кодах можно посмотреть здесь.
  • Ошибки SSL соединений (обычно при наличии проблем с SSL сертификатом: просрочен, не является доверенным и т.д.)

Базовым классом-исключением в requests является RequestException. От него наследуются все остальные

  • HTTPError
  • ConnectionError
  • Timeout
  • SSLError
  • ProxyError

И так далее. Полный список всех исключений можно посмотреть в requests.exceptions.

Timeout

В requests имеется 2 вида таймаут-исключений:

  • ConnectTimeout — таймаут на соединения
  • ReadTimeout — таймаут на чтение

HTTPError

Я перечислил основные виды исключений, которые покрывают, пожалуй, 90% всех проблем, возникающих при работе с http. Главное помнить, что если мы действительно намерены отловить что-то и обработать, то это необходимо явно запрограммировать, если же нам неважен тип конкретного исключения, то можно отлавливать общий базовый класс RequestException и действовать уже от конкретного случая, например, залоггировать исключение и выкинуть его дальше наверх. Кстати, о логгировании я напишу отдельный подробный пост.

У блога появился свой Telegram канал, где я стараюсь делиться интересными находками из сети на тему разработки программного обеспечения. Велком, как говорится 🙂

Полезные «плюшки»

  • httpbin.org очень полезный сервис для тестирования http клиентов, в частности удобен для тестирования нестандартного поведения сервиса
  • httpie консольный http клиент (замена curl) написанный на Python
  • responses mock библиотека для работы с requests
  • HTTPretty mock библиотека для работы с http модулями

Краткий обзор запросов HTTP

Запросы HTTP лежат в основе всемирной сети. Каждый раз, когда вы открываете веб-страницу, ваш браузер направляет множество запросов на сервер этой веб-страницы. Сервер отвечает на них, пересылая все необходимые данные для вывода страницы, и ваш браузер отображает страницу, чтобы вы могли увидеть ее.

В целом этот процесс выглядит так: клиент (например браузер или скрипт Python, использующий библиотеку Requests) отправляет данные на URL, а сервер с этим URL считывает данные, решает, что с ними делать, и отправляет клиенту ответ. После этого клиент может решить, что делать с полученными в ответе данными.

В составе запроса клиент отправляет данные по методу запроса. Наиболее распространенными методами запроса являются GET, POST и PUT. Запросы GET обычно предназначены только для чтения данных без их изменения, а запросы POST и PUT обычно предназначаются для изменения данных на сервере. Например, Stripe API позволяет использовать запросы POST для тарификации, чтобы пользователь мог купить что-нибудь в вашем приложении.

Примечание. В этой статье рассказывается о запросах GET, поскольку мы не собираемся изменять никакие данные на сервере.

При отправке запроса из скрипта Python или веб-приложения вы как разработчик решаете, что отправлять в каждом запросе и что делать с полученными ответами. Для начала отправим запрос на Scotch.io и используем API для перевода.

Headers

Another thing you can get from the response are the headers. You can take a look at them by using the headers dictionary on the response object.

Headers are sent along with the request and returned in the response. Headers are used so both the client and the server know how to interpret the data that is being sent and received in the response/response.

We see the various headers that are returned. A lot of times you won’t need to use the header information directly, but it’s there if you need it.

The content type is usually the one you may need because it reveals the format of the data, for example HTML, JSON, PDF, text, etc. But the content type is normally handled by Requests so you can easily access the data that gets returned.

Response Text

And finally, if we take a look at res.text (this works for textual data, like a HTML page like we are viewing) we can see all the HTML needed to build the home page of Scotch. It won’t be rendered, but we see that it looks like it belongs to Scotch. If you saved this to a file and opened it, you would see something that resembled the Scotch site. In a real situation, multiple requests are made for a single web page to load things like images, scripts and stylesheets, so if you save only the HTML to a file, it won’t look anything like what the Scotch.io page looks like in your browser because only a single request was performed to get the HTML data.

Request Method

The request method indicates the method to be performed on the resource identified by the given Request-URI. The method is case-sensitive and should always be mentioned in uppercase. The following table lists all the supported methods in HTTP/1.1.

S.N. Method and Description
1 GET

The GET method is used to retrieve information from the given server using a given URI. Requests using GET should only retrieve data and should have no other effect on the data.

2 HEAD

Same as GET, but it transfers the status line and the header section only.

3 POST

A POST request is used to send data to the server, for example, customer information, file upload, etc. using HTML forms.

4 PUT

Replaces all the current representations of the target resource with the uploaded content.

5 DELETE

Removes all the current representations of the target resource given by URI.

6 CONNECT

Establishes a tunnel to the server identified by a given URI.

7 OPTIONS

Describe the communication options for the target resource.

8 TRACE

Performs a message loop back test along with the path to the target resource.

SSL Cert Verification¶

Requests can verify SSL certificates for HTTPS requests, just like a web browser. To check a host’s SSL certificate, you can use the argument:

>>> requests.get('https://kennethreitz.com', verify=True)
requests.exceptions.SSLError: hostname 'kennethreitz.com' doesn't match either of '*.herokuapp.com', 'herokuapp.com'

I don’t have SSL setup on this domain, so it fails. Excellent. GitHub does though:

>>> requests.get('https://github.com', verify=True)
<Response >

You can pass the path to a CA_BUNDLE file with certificates of trusted CAs:

>>> requests.get('https://github.com', verify='/path/to/certfile')

This list of trusted CAs can also be specified through the environment variable.

Requests can also ignore verifying the SSL certificate if you set to False.

>>> requests.get('https://kennethreitz.com', verify=False)
<Response >

By default, is set to True. Option only applies to host certs.

You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file’s path:

>>> requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
<Response >

If you specify a wrong path or an invalid cert:

Properties and Methods

Property/Method Description
apparent_encoding Try it Returns the apparent encoding
close() Try it Closes the connection to the server
content Try it Returns the content of the response, in bytes
cookies Try it Returns a CookieJar object with the cookies sent back from the server
elapsed Try it Returns a timedelta object with the time elapsed from sending the request to the arrival of the response
encoding Try it Returns the encoding used to decode r.text
headers Try it Returns a dictionary of response headers
history Try it Returns a list of response objects holding the history of request (url)
is_permanent_redirect Try it Returns True if the response is the permanent redirected url, otherwise False
is_redirect Try it Returns True if the response was redirected, otherwise False
iter_content() Try it Iterates over the response
iter_lines() Try it Iterates over the lines of the response
json() Try it Returns a JSON object of the result (if the result was written in JSON format, if not it raises an error)
links Try it Returns the header links
next Try it Returns a PreparedRequest object for the next request in a redirection
ok Try it Returns True if status_code is less than 200, otherwise False
raise_for_status() Try it If an error occur, this method returns a HTTPError object
reason Try it Returns a text corresponding to the status code
request Try it Returns the request object that requested this response
status_code Try it Returns a number that indicates the status (200 is OK, 404 is Not Found)
text Try it Returns the content of the response, in unicode
url Try it Returns the URL of the response

Registering Responses¶

Responses are registered with the function on the adapter.

>>> adapter.register_uri('GET', 'mock://test.com', text='Success')
>>> resp = session.get('mock://test.com')
>>> resp.text
'Success'

takes the HTTP method, the URI and then information that is used to build the response. This information includes:

status_code: The HTTP status response to return. Defaults to 200.
reason: The reason text that accompanies the Status (e.g. ‘OK’ in ‘200 OK’)
headers: A dictionary of headers to be included in the response.
cookies: A CookieJar containing all the cookies to add to the response.

To specify the body of the response there are a number of options that depend on the format that you wish to return.

json: A python object that will be converted to a JSON string.
text: A unicode string. This is typically what you will want to use for regular textual content.
content: A byte string. This should be used for including binary data in responses.
body: A file like object that contains a .read() function.
raw: A prepopulated to be returned.
exc: An exception that will be raised instead of returning a response.

These options are named to coincide with the parameters on a object. For example:

>>> adapter.register_uri('GET', 'mock://test.com/1', json={'a' 'b'}, status_code=200)
>>> resp = session.get('mock://test.com/1')
>>> resp.json()
{'a': 'b'}

>>> adapter.register_uri('GET', 'mock://test.com/2', text='Not Found', status_code=404)
>>> resp = session.get('mock://test.com/2')
>>> resp.text
'Not Found'
>>> resp.status_code
404

Response Lists¶

Multiple responses can be provided to be returned in order by specifying the keyword parameters in a list. If the list is exhausted then the last response will continue to be returned.

>>> adapter.register_uri('GET', 'mock://test.com/4', )
>>> resp = session.get('mock://test.com/4')
>>> (resp.status_code, resp.text)
(300, 'resp1')
>>> resp = session.get('mock://test.com/4')
>>> (resp.status_code, resp.text)
(200, 'resp2')
>>> resp = session.get('mock://test.com/4')
>>> (resp.status_code, resp.text)
(200, 'resp2')

Callbacks work within response lists in exactly the same way they do normally;

Authentication

Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called , which allows you to pass your credentials.

One example of an API that requires authentication is GitHub’s API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to :

>>>

The request succeeded if the credentials you passed in the tuple to are valid. If you try to make this request with no credentials, you’ll see that the status code is :

>>>

When you pass your username and password in a tuple to the parameter, is applying the credentials using HTTP’s Basic access authentication scheme under the hood.

Therefore, you could make the same request by passing explicit Basic authentication credentials using :

>>>

Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. provides other methods of authentication out of the box such as and .

You can even supply your own authentication mechanism. To do so, you must first create a subclass of . Then, you implement :

Here, your custom mechanism receives a token, then includes that token in the header of your request.

Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth.

While you’re thinking about security, let’s consider dealing with SSL Certificates using .

Migrating to 2.x¶

Compared with the 1.0 release, there were relatively few backwards incompatible changes, but there are still a few issues to be aware of with this major release.

For more details on the changes in this release including new APIs, links to the relevant GitHub issues and some of the bug fixes, read Cory’s blog on the subject.

API Changes

  • There were a couple changes to how Requests handles exceptions. is now a subclass of rather than as that more accurately categorizes the type of error. In addition, an invalid URL escape sequence now raises a subclass of rather than a .

    requests.get('http://%zz/')   # raises requests.exceptions.InvalidURL
    

    Lastly, exceptions caused by incorrect chunked encoding will now raise a Requests instead.

  • The proxy API has changed slightly. The scheme for a proxy URL is now required.

    proxies = {
      "http" "10.10.1.10:3128",    # use http://10.10.1.10:3128 instead
    }
    
    # In requests 1.x, this was legal, in requests 2.x,
    #  this raises requests.exceptions.MissingSchema
    requests.get("http://example.org", proxies=proxies)
    

Behavioural Changes

  • Keys in the dictionary are now native strings on all Python versions, i.e. bytestrings on Python 2 and unicode on Python 3. If the keys are not native strings (unicode on Python 2 or bytestrings on Python 3) they will be converted to the native string type assuming UTF-8 encoding.
  • Values in the dictionary should always be strings. This has been the project’s position since before 1.0 but a recent change (since version 2.11.0) enforces this more strictly. It’s advised to avoid passing header values as unicode when possible.

Conclusion

Here’s what we learned:

  • How HTTP requests work
  • The various status codes possible in a response
  • How to send requests and receive responses using the Python Requests library
  • How to use a language translation API to translate text
  • How to convert application/json content responses to dictionariesThat covers the basics of Requests in Python. Of course you can do so much more, but what I talked about in this article is the foundation of most requests. Things may change slightly depending on the circumstances, but the basic ideas will remain the same.

If you want to do more, check out https://apilist.fun/ to see different APIs that are available, and try to use them with Python Requests.

Handling Cookies¶

Whilst cookies are just headers they are treated in a different way, both in HTTP and the requests library. To work as closely to the requests library as possible there are two ways to provide cookies to requests_mock responses.

The most simple method is to use a dictionary interface. The Key and value of the dictionary are turned directly into the name and value of the cookie. This method does not allow you to set any of the more advanced cookie parameters like expiry or domain.

>>> adapter.register_uri('GET', 'mock://test.com/7', cookies={'foo' 'bar'}),
>>> resp = session.get('mock://test.com/7')
>>> resp.cookies'foo'
'bar'

The more advanced way is to construct and populate a cookie jar that you can add cookies to and pass that to the mocker.

>>> jar = requests_mock.CookieJar()
>>> jar.set('foo', 'bar', domain='.test.com', path='/baz')
>>> adapter.register_uri('GET', 'mock://test.com/8', cookies=jar),
>>> resp = session.get('mock://test.com/8')
>>> resp.cookies'foo'
'bar'
>>> resp.cookies.list_paths()
['/baz']

С этим читают