How Python Imports Work

Posted on In ,

This post focuses on import in Python 3.

What happens during import

An import statement performs two major steps:

  1. Locate the module.
  2. Bind the result into the local namespace.

We will concentrate on the search logic.

Finding a module

Module lookup happens in two phases:

  1. Check sys.modules.
  2. Consult sys.meta_path.

sys.modules

When a module is imported for the first time, Python stores it—and any modules it pulled in—in the dictionary sys.modules. Subsequent imports short-circuit and reuse the cached entry.

If you import using from package import module or import package.module, the parent package also ends up in sys.modules.

You can even tamper with the cache yourself:

1
2
3
4
5
6
import os
import sys

sys.modules['spp'] = os
import spp
print(spp)

sys.meta_path

If the module is not cached, Python loops through sys.meta_path.

sys.meta_path is a list of importer objects. An importer implements the finder and loader protocols. Instead of dwelling on the formal definition, let us inspect the default entries in Python 3.6:

  • class _frozen_importlib.BuiltinImporter
  • class _frozen_importlib.FrozenImporter
  • class _frozen_importlib.PathFinder

Their roles:

  • Locate and load built-in modules.
  • Locate and load frozen modules (precompiled into executables).
  • Locate and load modules found on the import path.

If none of them can provide the module, Python raises ModuleNotFoundError.

Import hooks

You can customize importing with hooks. There are two kinds: meta hooks and path hooks.

Meta hooks

Meta hooks intercept imports by manipulating sys.meta_path.

Importer objects implement the finder protocol (find_spec() is the modern API; find_module() is the legacy one) and the loader protocol. The example below defines two minimal importers (ignoring the loader part) and adds them to sys.meta_path:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class Importer1(object):

    def find_module(self, fullname, path):
        print('find_module Looking for', fullname, path)
        return None

    def find_spec(self, fullname, path, target=None):
        print('find_spec Looking for', fullname, path)
        return None  # Returning None tells Python we cannot handle it


class Importer2(object):

    def find_module(self, fullname, path):
        print('find_module2 Looking for', fullname, path)
        return None

    def find_spec(self, fullname, path, target=None):
        print('find_spec2 Looking for', fullname, path)
        return None

Insert them at the beginning and end of sys.meta_path:

1
2
3
import sys
sys.meta_path.insert(0, Importer1())
sys.meta_path.append(Importer2())

Importing json now prints diagnostic messages:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
>>>import json

find_spec Looking for json None
find_spec Looking for json.decoder ['C:\\Program Files (x86)\\Python36\\lib\\json']
find_spec Looking for json.scanner ['C:\\Program Files (x86)\\Python36\\lib\\json']
find_spec Looking for _json None
find_spec Looking for json.encoder ['C:\\Program Files (x86)\\Python36\\lib\\json']

>>>json

<module 'json' from 'C:\\Program Files (x86)\\Python36\\lib\\json\\__init__.py'>

>>>import jso

find_spec Looking for jso None

find_spec2 Looking for jso None

Path hooks

PathFinder, one of the default meta importers, uses path hooks internally. The relevant portion of CPython looks like this (abridged here for clarity):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
class PathFinder:

    """Meta path finder for sys.path and package __path__ attributes."""

    @classmethod
    def _path_hooks(cls, path):
        """Search sys.path_hooks for a finder for 'path'."""
        if sys.path_hooks is not None and not sys.path_hooks:
            _warnings.warn('sys.path_hooks is empty', ImportWarning)
        for hook in sys.path_hooks:
            try:
                return hook(path)
            except ImportError:
                continue
        else:
            return None

    @classmethod
    def _path_importer_cache(cls, path):
        """Get the finder for the path entry from sys.path_importer_cache."""
        if path == '':
            try:
                path = _os.getcwd()
            except FileNotFoundError:
                return None
        try:
            finder = sys.path_importer_cache[path]
        except KeyError:
            finder = cls._path_hooks(path)
            sys.path_importer_cache[path] = finder
        return finder

    @classmethod
    def _get_spec(cls, fullname, path, target=None):
        """Find the loader or namespace_path for this module/package name."""
        namespace_path = []
        for entry in path:
            if not isinstance(entry, (str, bytes)):
                continue
            finder = cls._path_importer_cache(entry)
            if finder is not None:
                if hasattr(finder, 'find_spec'):
                    spec = finder.find_spec(fullname, target)
                else:
                    spec = cls._legacy_get_spec(fullname, finder)
                if spec is None:
                    continue
                if spec.loader is not None:
                    return spec
                portions = spec.submodule_search_locations
                if portions is None:
                    raise ImportError('spec missing loader')
                namespace_path.extend(portions)
        else:
            spec = _bootstrap.ModuleSpec(fullname, None)
            spec.submodule_search_locations = namespace_path
            return spec

…and later:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@classmethod
def find_spec(cls, fullname, path=None, target=None):
    """Try to find a spec for 'fullname' on sys.path or 'path'."""
    if path is None:
        path = sys.path
    spec = cls._get_spec(fullname, path, target)
    if spec is None:
        return None
    elif spec.loader is None:
        namespace_path = spec.submodule_search_locations
        if namespace_path:
            spec.origin = 'namespace'
            spec.submodule_search_locations = _NamespacePath(fullname, namespace_path, cls._get_spec)
            return spec
        else:
            return None
    else:
        return spec

To add your own path hook, define a function that returns an importer for matching paths and raises ImportError otherwise. The Python Cookbook demonstrates checking for URLs:

1
2
3
4
5
def check_url(path):
    if path.startswith('http://'):
        return Finder()
    else:
        raise ImportError()

If the hook accepts the path it returns an importer object (which must provide find_spec()); otherwise PathFinder tries the next hook.

References: