18.6 `urllib2` -- extensible library for opening URLs

The urllib2 module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world -- basic and digest authentication, redirections, cookies and more.

The urllib2 module defines the following functions:

urlopen( url[, data])

Open the URL url, which can be either a string or a Request object.

data may be a string specifying additional data to send to the server, or None if no such data is needed. Currently HTTP requests are the only ones that use data; the HTTP request will be a POST instead of a GET when the data parameter is provided. data should be a buffer in the standard application/x-www-form-urlencoded format. The urllib.urlencode() function takes a mapping or sequence of 2-tuples and returns a string in this format.

This function returns a file-like object with two additional methods:

geturl() -- return the URL of the resource retrieved
info() -- return the meta-information of the page, as a dictionary-like object

Raises URLError on errors.

Note that None may be returned if no handler handles the request (though the default installed global OpenerDirector uses UnknownHandler to ensure this never happens).

install_opener( opener): Install an OpenerDirector instance as the default global opener. Installing an opener is only necessary if you want urlopen to use that opener; otherwise, simply call OpenerDirector.open() instead of urlopen(). The code does not check for a real OpenerDirector, and any class with the appropriate interface will work.

build_opener( [handler, ...])

Return an OpenerDirector instance, which chains the handlers in the order given. handlers can be either instances of BaseHandler, or subclasses of BaseHandler (in which case it must be possible to call the constructor without any parameters). Instances of the following classes will be in front of the handlers, unless the handlers contain them, instances of them or subclasses of them: ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor.

If the Python installation has SSL support (socket.ssl() exists), HTTPSHandler will also be added.

Beginning in Python 2.3, a BaseHandler subclass may also change its handler_order member variable to modify its position in the handlers list.

The following exceptions are raised as appropriate:

exception URLError: The handlers raise this exception (or derived exceptions) when they run into a problem. It is a subclass of IOError.

exception HTTPError: A subclass of URLError, it can also function as a non-exceptional file-like return value (the same thing that urlopen() returns). This is useful when handling exotic HTTP errors, such as requests for authentication.

exception GopherError: A subclass of URLError, this is the error raised by the Gopher handler.

The following classes are provided:

class Request( url[, data][, headers][, origin_req_host][, unverifiable])

This class is an abstraction of a URL request.

url should be a string containing a valid URL.

headers should be a dictionary, and will be treated as if add_header() was called with each key and value as arguments.

The final two arguments are only of interest for correct handling of third-party HTTP cookies:

origin_req_host should be the request-host of the origin transaction, as defined by RFC 2965. It defaults to cookielib.request_host(self). This is the host name or IP address of the original request that was initiated by the user. For example, if the request is for an image in an HTML document, this should be the request-host of the request for the page containing the image.

unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965. It defaults to False. An unverifiable request is one whose URL the user did not have the option to approve. For example, if the request is for an image in an HTML document, and the user had no option to approve the automatic fetching of the image, this should be true.

class OpenerDirector( ): The OpenerDirector class opens URLs via BaseHandlers chained together. It manages the chaining of handlers, and recovery from errors.

class BaseHandler( ): This is the base class for all registered handlers -- and handles only the simple mechanics of registration.

class HTTPDefaultErrorHandler( ): A class which defines a default handler for HTTP error responses; all responses are turned into HTTPError exceptions.

class HTTPRedirectHandler( ): A class to handle redirections.

class HTTPCookieProcessor( [cookiejar]): A class to handle HTTP Cookies.

class ProxyHandler( [proxies]): Cause requests to go through a proxy. If proxies is given, it must be a dictionary mapping protocol names to URLs of proxies. The default is to read the list of proxies from the environment variables <protocol>_proxy.

class HTTPPasswordMgr( ): Keep a database of (realm, uri) -> (user, password) mappings.

class HTTPPasswordMgrWithDefaultRealm( ): Keep a database of (realm, uri) -> (user, password) mappings. A realm of None is considered a catch-all realm, which is searched if no other realm fits.

class AbstractBasicAuthHandler( [password_mgr]): This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class HTTPBasicAuthHandler( [password_mgr]): Handle authentication with the remote host. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class ProxyBasicAuthHandler( [password_mgr]): Handle authentication with the proxy. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class AbstractDigestAuthHandler( [password_mgr]): This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class HTTPDigestAuthHandler( [password_mgr]): Handle authentication with the remote host. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class ProxyDigestAuthHandler( [password_mgr]): Handle authentication with the proxy. password_mgr, if given, should be something that is compatible with HTTPPasswordMgr; refer to section 18.6.7 for information on the interface that must be supported.

class HTTPHandler( ): A class to handle opening of HTTP URLs.

class HTTPSHandler( ): A class to handle opening of HTTPS URLs.

class FileHandler( ): Open local files.

class FTPHandler( ): Open FTP URLs.

class CacheFTPHandler( ): Open FTP URLs, keeping a cache of open FTP connections to minimize delays.

class GopherHandler( ): Open gopher URLs.

class UnknownHandler( ): A catch-all class to handle unknown URLs.

See About this document... for information on suggesting changes.

18.6 urllib2 -- extensible library for opening URLs

18.6 `urllib2` -- extensible library for opening URLs