Read on Twitter
Friends, I was working on a way to make Jupyter Notebooks work *well* as Python modules, and while I did that, I think I came up with something even cooler. But it's a little hard to explain.
The first part of reusing Jupyter Notebooks as modules is to extend the Python import mechanism to load .ipynb files and pull the code out of the cells. That's not super exciting, it's been demonstrated already, and I won't focus on it here.
Just doing that doesn't account for the fact that code in Jupyter Notebooks is written -- for good reason -- differently from code in .py files. You want to make use of cool features in Jupyter, like demonstrating outputs. Plus you tend to put in scratch code, bla bla
So I saw there was a need for a way to exclude certain cells from being imported. In other words, the notebook needs to have multiple views a notebook view and a module view, and some way to define them. So I created "margin notes" and a specific syntax called "margo"
With margo you can put notes like
# :: ignore-cell ::
to tell Python to ignore it the cell during import. This syntax is contained in code comments, so they have no effect when you're using your notebook in a tool like Jupyter Notebook or JupyterLab.
# :: ignore-cell ::
to tell Python to ignore it the cell during import. This syntax is contained in code comments, so they have no effect when you're using your notebook in a tool like Jupyter Notebook or JupyterLab.
If the above seems cool, and you just want to learn how to make modular notebooks this way, check out this live tutorial (on Binder). https://mybinder.org/v2/gh/margo-notebooks/modular-notebook-tutorial/HEAD
If you want to go further down this rabbit hole, read on....
If you want to go further down this rabbit hole, read on....
Here's where it gets more interesting. I added support for storing data, not just directives, with Margo, so you can arbitrarily serialize any data you want, either in JSON, YAML, plain text, or the default format like this:
# :: input: "population.csv", "infections.csv" ::
# :: input: "population.csv", "infections.csv" ::
So with margin notes like this, you can extend notebooks in other ways. With the above example, you can define the input and output files that result from running a notebook as a script, and then coordinate its execution with a tool like GNU Make.
Another use is to encode a Notebook's requirements.txt right into a Margo note. The Binder tutorial I posted above shows how to do this. But it let's take this even further...
... Since we can serialize arbitrary data to cells using Margo, authoring applications can use add features that aren't supported in the notebook format. As a proof of concept, I built an editor that demonstrates hierarchical cell relationships in Margo-enhanced Jupyter notebooks
Here's what that editor UI looks like: https://margo-editor.netlify.app/ (remember, it's only written to demonstrate the cell hierarchy UI features and how they can be serialized into a valid nbformat document).
Here's a blog post with the links from this thread and some others: https://jakekara.com/margo/2021/05/05/margo-notebooks.html
I know I need to make a video or something to make this easier to dive into, but I don't know when I'll be able to do that, and I just wanted to get some of this out there.
I know I need to make a video or something to make this easier to dive into, but I don't know when I'll be able to do that, and I just wanted to get some of this out there.
Here's the repo for the Binder tutorial I posted earlier: https://github.com/margo-notebooks/modular-notebook-tutorial
