markdown cell
# Importing Jupyter Notebooks as Modules
markdown cell
It is a common problem that people want to import code from Jupyter Notebooks.
This is made difficult by the fact that Notebooks are not plain Python files,
and thus cannot be imported by the regular Python machinery.
Fortunately, Python provides some fairly sophisticated [hooks](https://www.python.org/dev/peps/pep-0302/) into the import machinery,
so we can actually make Jupyter notebooks importable without much difficulty,
and only using public APIs.
code cell
import io, os, sys, types
code cell
from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell
markdown cell
Import hooks typically take the form of two objects:
1. a Module **Loader**, which takes a module name (e.g. `'IPython.display'`), and returns a Module
2. a Module **Finder**, which figures out whether a module might exist, and tells Python what **Loader** to use
code cell
def find_notebook(fullname, path=None):
"""find a notebook, given its fully qualified name and an optional path
This turns "foo.bar" into "foo/bar.ipynb"
and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
does not exist.
"""
name = fullname.rsplit('.', 1)[-1]
if not path:
path = ['']
for d in path:
nb_path = os.path.join(d, name + ".ipynb")
if os.path.isfile(nb_path):
return nb_path
# let import Notebook_Name find "Notebook Name.ipynb"
nb_path = nb_path.replace("_", " ")
if os.path.isfile(nb_path):
return nb_path
markdown cell
## Notebook Loader
markdown cell
Here we have our Notebook Loader.
It's actually quite simple - once we figure out the filename of the module,
all it does is:
1. load the notebook document into memory
2. create an empty Module
3. execute every cell in the Module namespace
Since IPython cells can have extended syntax,
the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.
If all of your notebook cells are pure-Python,
this step is unnecessary.
code cell
class NotebookLoader(object):
"""Module Loader for Jupyter Notebooks"""
def __init__(self, path=None):
self.shell = InteractiveShell.instance()
self.path = path
def load_module(self, fullname):
"""import a notebook as a module"""
path = find_notebook(fullname, self.path)
print ("importing Jupyter notebook from %s" % path)
# load the notebook object
with io.open(path, 'r', encoding='utf-8') as f:
nb = read(f, 4)
# create the module and add it to sys.modules
# if name in sys.modules:
# return sys.modules[name]
mod = types.ModuleType(fullname)
mod.__file__ = path
mod.__loader__ = self
mod.__dict__['get_ipython'] = get_ipython
sys.modules[fullname] = mod
# extra work to ensure that magics that would affect the user_ns
# actually affect the notebook module's ns
save_user_ns = self.shell.user_ns
self.shell.user_ns = mod.__dict__
try:
for cell in nb.cells:
if cell.cell_type == 'code':
# transform the input to executable Python
code = self.shell.input_transformer_manager.transform_cell(cell.source)
# run the code in themodule
exec(code, mod.__dict__)
finally:
self.shell.user_ns = save_user_ns
return mod
markdown cell
## The Module Finder
markdown cell
The finder is a simple object that tells you whether a name can be imported,
and returns the appropriate loader.
All this one does is check, when you do:
```python
import mynotebook
```
it checks whether `mynotebook.ipynb` exists.
If a notebook is found, then it returns a NotebookLoader.
Any extra logic is just for resolving paths within packages.
code cell
class NotebookFinder(object):
"""Module finder that locates Jupyter Notebooks"""
def __init__(self):
self.loaders = {}
def find_module(self, fullname, path=None):
nb_path = find_notebook(fullname, path)
if not nb_path:
return
key = path
if path:
# lists aren't hashable
key = os.path.sep.join(path)
if key not in self.loaders:
self.loaders[key] = NotebookLoader(path)
return self.loaders[key]
markdown cell
## Register the hook
markdown cell
Now we register the `NotebookFinder` with `sys.meta_path`
code cell
sys.meta_path.append(NotebookFinder())
markdown cell
After this point, my notebooks should be importable.
Let's look at what we have in the CWD:
code cell
markdown cell
So I should be able to `import nbpackage.mynotebook`.
code cell
import nbpackage.mynotebook
markdown cell
### Aside: displaying notebooks
markdown cell
Here is some simple code to display the contents of a notebook
with syntax highlighting, etc.
code cell
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
from IPython.display import display, HTML
formatter = HtmlFormatter()
lexer = PythonLexer()
# publish the CSS for pygments highlighting
display(HTML("""
<style type='text/css'>
%s
</style>
""" % formatter.get_style_defs()
))
code cell
def show_notebook(fname):
"""display a short summary of the cells of a notebook"""
with io.open(fname, 'r', encoding='utf-8') as f:
nb = read(f, 4)
html = []
for cell in nb.cells:
html.append("<h4>%s cell</h4>" % cell.cell_type)
if cell.cell_type == 'code':
html.append(highlight(cell.source, lexer, formatter))
else:
html.append("<pre>%s</pre>" % cell.source)
display(HTML('\n'.join(html)))
show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))
markdown cell
So my notebook has some code cells,
one of which contains some IPython syntax.
Let's see what happens when we import it
code cell
from nbpackage import mynotebook
markdown cell
Hooray, it imported! Does it work?
code cell
markdown cell
Hooray again!
Even the function that contains IPython syntax works:
code cell
mynotebook.has_ip_syntax()
code cell
markdown cell
## Notebooks in packages
markdown cell
We also have a notebook inside the `nb` package,
so let's make sure that works as well.
code cell
markdown cell
Note that the `__init__.py` is necessary for `nb` to be considered a package,
just like usual.
code cell
show_notebook(os.path.join("nbpackage", "nbs", "other.ipynb"))
code cell
from nbpackage.nbs import other
other.bar(5)
markdown cell
So now we have importable notebooks, from both the local directory and inside packages.
I can even put a notebook inside IPython, to further demonstrate that this is working properly:
code cell
import shutil
from IPython.paths import get_ipython_package_dir
utils = os.path.join(get_ipython_package_dir(), 'utils')
shutil.copy(os.path.join("nbpackage", "mynotebook.ipynb"),
os.path.join(utils, "inside_ipython.ipynb")
)
markdown cell
and import the notebook from `IPython.utils`
code cell
from IPython.utils import inside_ipython
inside_ipython.whatsmyname()
markdown cell
This approach can even import functions and classes that are defined in a notebook using the `%%cython` magic.