Comparing Asynchronous HTTP Libraries in Python 2

Posted on In ,

Wrapper libraries

grequests

Summary

An asynchronous HTTP client built on top of gevent by the author of requests.

Project page

https://github.com/kennethreitz/grequests

Example

1
2
3
4
5
6
7
8
9
10
11
12
import grequests

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://fakedomain/',
    'http://kennethreitz.com'
]
rs = (grequests.get(u) for u in urls)
grequests.map(rs)

get takes the same arguments as requests.get(), and post, put, delete, etc. are supported as well. Failed requests return None by default; you can pass an exception_handler callback to map.

Parameters accepted by map:

1
2
3
4
5
:param requests: a collection of Request objects.
:param stream: If True, the content will not be downloaded immediately.
:param size: Specifies the number of requests to make at a time. If None, no throttling occurs.
:param exception_handler: Callback function, called when exception occurred. Params: Request, Exception
:param gtimeout: Gevent joinall timeout in seconds. (Note: unrelated to requests timeout)

If you prefer a generator, use imap. Note that imap defaults size to 2 and does not offer a gtimeout argument.

requests-threads

Summary

Another async client from the requests author, this time built on Twisted.

Project page

https://github.com/requests/requests-threads

Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from twisted.internet.defer import inlineCallbacks
from twisted.internet.task import react
from requests_threads import AsyncSession

session = AsyncSession(n=100)

@inlineCallbacks
def main(reactor):
    responses = []
    for i in range(100):
        responses.append(session.get('http://httpbin.org/get'))

    for response in responses:
        r = yield response
        print(r)

if __name__ == '__main__':
    react(main)

txrequests

Summary

Another requests-style client built on Twisted.

Project page

https://github.com/tardyp/txrequests

Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from txrequests import Session
from twisted.internet import defer

@defer.inlineCallbacks
def main():
    # use with statement to cleanup session's threadpool, and connectionpool after use
    # you can also use session.close() if want to use session for long term use
    with Session() as session:
        # first request is started in background
        d1 = session.get('http://httpbin.org/get')
        # second requests is started immediately
        d2 = session.get('http://httpbin.org/get?foo=bar')
        # wait for the first request to complete, if it hasn't already
        response_one = yield d1
        print('response one status: {0}'.format(response_one.status_code))
        print(response_one.content)
        # wait for the second request to complete, if it hasn't already
        response_two = yield d2
        print('response two status: {0}'.format(response_two.status_code))
        print(response_two.content)

treq

Summary

A Twisted-based HTTP client with an interface modeled after requests. Its documentation is extensive.

Project page

https://github.com/twisted/treq

Example

1
2
3
4
def main(reactor, *args):
    d = treq.get('http://httpbin.org/get')
    d.addCallback(print_response)
    return d

trip

Summary

An asynchronous HTTP client built on tornado.

Project page

https://github.com/littlecodersh/trip

Example

1
2
3
4
5
6
7
import trip

def main():
    r = yield trip.get('https://httpbin.org/get', auth=('user', 'pass'))
    print(r.content)

trip.run(main)

Lower-level libraries

gevent

Summary

The go-to async library in Python 2.

Project page

https://github.com/gevent/gevent

Example

1
2
3
4
5
6
import gevent
from gevent import socket
urls = ['www.google.com', 'www.example.com', 'www.python.org']
jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls
gevent.joinall(jobs, timeout=2)
[job.value for job in jobs]

Monkey patching that turns standard networking into async operations:

1
2
3
from gevent import monkey
monkey.patch_socket()
import urllib2

twisted

pass

tornado

pass

To be continued.