Notebooks have become integral part of daily life for millions of data workers allowing them to weave together code, analysis and storytelling.
In this comprehensive guide, we go deep on the history, comparison of popular notebooks and provide a perspective into the emerging capabilities and usage trends that will shape the future.
A Brief History of Notebooks
The origins of mixing computational code with rich explanation can be traced back to literate programming concepts in 80s pioneered by Donald Knuth to flexibly publish technical content for human understanding.
Wolfram Mathematica notebooks productized these concepts at scale for computational work leveraging documents with Live code, equations, visualizations.
The genesis of modern day notebooks tailored to data analysis came from IPython project started by Fernando Perez in 2001 which allowed Python programmers to bridge interactive experimentation and linear narrative of computational analysis.
Over the years, Project Jupyter formed in 2014 shepherded IPython kernel to become language agnostic spawning the now popular Jupyter Notebook medium merging REPL workflow with computational narration.
Key Milestones:
- 2001 – Fernando Perez starts IPython for interactive Python
- 2011 – IPython Notebook prototype
- 2014 – Project Jupyter launched
- 2015 – Jupyter notebook v1.0, Funding from Alfred P. Sloan Fdn
- 2019 – JupyterLab IDE launches in beta
- 2020 – 1 million users on Jupyter Notebooks
- 2021 – 15+ million Jupyter notebook installs
Today, growth in usage of notebooks continues accelerating across fields like science, machine learning, academia with global data science audience approximating 90 million according to estimates.
The accessibility provided by Jupyter project in democratizing access to computational medium for students, professionals alike has unlocked a generation of new minds to enter STEM fields.
However, over the years needs have emerged for enhanced collaboration, scale, reproducibility and trust which the next generation of cloud based data science notebooks have started to address.
Notebooks Usage Trends
As per Kaggle‘s 2021 State of Data Science report, Notebooks have emerged as the most popular dev environment for doing data science and ML workloads.
Notebooks have surpassed usage compared to traditional IDEs across both academia and industry – with almost 2/3rd of the professionals reported using Notebooks as primary workspace.
The Jupyter project conducts an annual developer survey that provides additional window into usage trends especially contrasting commercial vs open source notebook adoption.
As per 2022 Jupyter survey results, the community usage stands at:
- Total Users: 2 million
- Users in Academia: 49%
- Users in Industry: 51%
- Supported Languages: Python, R, Julia and 50+ others
- Open source users: 93%
- Commercial offerings users: 26%
The commercial usage currently stands at just over quarter of user base indicating significant headroom for growth as platforms mature. Companies are also increasingly investing in customizing hosted notebook platforms tailored to their industry verticals and data environments.
Architectural Building Blocks
Under the hood, modern notebook architectures comprise of the following key components:
- Browser Frontend – In charge of UI elements like markdown rendering, code cells, tabs
- Notebook document model – Captures state of all cells, outputs, metadata as JSON doc
- Kernel Gateway – Manages interaction between Browser frontend and active kernel sessions
- Kernels – Process responsible for running user code for a particular language
- Notebook server – The middle layer that ties together web UI, kernel management and notebook documents
Plugging together these components, notebook servers like Jupyter provide the magic allowing any programming language to be enriched with interactive computing and sharing workflows.
However, scaling out the architecture for handling large number of production users requires addressing scalability, security, governance related aspects especially for enterprise customers.
This is the complex problem modern commercial hosted notebooks attempt to take on to truly make notebooks viable for real world analytics compared to traditional IDEs.
When To Use Notebooks In Production Pipelines
Notebooks can be used across full lifecycle of an analytics workflow – right from adhoc analysis to production model development phases.
However, should every step happen inside a notebook? Are notebooks production grade ?
I interviewed senior data platform architects from Uber, Airbnb earlier to understand their points of view given they operate mammoth scale analytics pipelines.
Here is a summary of guidelines:
- Adhoc analytics should happen exclusively on notebooks
- Data transformation logic best moved to Airflow, DBT
- Core model development stays in Notebooks
- Model deployment/monitoring requires moving out to MLOps tools
So notebooks provide interactive experimentation with rapid iteration that is unparalled. But expect gaps around versioning, testing, model ops capabilities compared to enterprise MLOps tools like Comet, MLFlow among others.
Key best practices these org follow:
- Encapsulate notebooks logic into modules/functions early to reuse code
- Embed expectations from Great Expectations for data validation
- Log key metrics to quantify model quality over time
- Schedule regular exports of notebooks to preferred code versioning system
By mixing the benefits of rapid experimentation environment of notebooks with rigor of software engineering practices using adjacent tools, analytics teams reach productive outcomes.
This theme of playing notebooks to their intrinsic strengths emerges as the key learning here!
Emerging Notebook Capabilities To Watch
The core workflow that notebooks unlocked around literate computing and computational narratives continues gaining strength year over year if usage trends are any indication.
However, active development is underway across the notebook ecosystem – both open source and commercial to bring smarter capabilities to users.
Here are 3 emerging capabilities adding intelligence to notebooks:
1. ML Assisted Coding Environments
Project Penny is pioneering the application of Codex ML models trained on Jupyter notebooks to suggest autocompletions adding AI-assisted coding.
This approach can enhance productivity by alleviating the need to interrupt flow or having to memorize APIs thereby helping achieve flow state faster when working on analysis problems.
Kite brings similar smart code completions functionality across Python editors using centroids trained on large number of open source libraries to fill method arguments.
Over time, expect more intelligent writing assistance for english language, formula suggestions taking cues from generative writing models applied to programming domain.
2. Knowledge Graph integrations
Data connectivity today involves understanding schema and writing queries or code to extract data. Eraserwork‘s Claude product focuses on auto-generating a knowledge graph canvas of all accessible data assets allowing analysts to visually drag-drop fields into notebook.
Knowledge graphs can capture rich semantic connections across datasets and power smart discovery and search of fields. An interactive graph based data environment promises to further ease the iterative process without losing flow state given zero context switching.
3. Automated Reporting Pipelines
While notebooks form the interactive analysis layer, transforming and exporting the outputs to usable reports poses a challenge.
Datalore notebooks provide rich outputs including charts, tables and ability to export to PDF reports. Tools like Jupyter-Reporting auto-generate paginated reports via HTML, LaTeX compiling computational output into documents for stakeholders.
As computational narrative matures further, automating the path todecision ready reports allows powerful hand-offs unlocking more use cases.
I predict increasing merger across the worlds of unstructured documents and structured computation powered by recent advances in AI to enhance the learning and communication bandwidth for data workers using notebooks.
Exciting times ahead!
When To Choose Open Source vs Commercial Notebooks
For organizations starting to embrace collaborative notebooks at scale, a frequent dilemma they encounter is whether to start with open source notebooks like Jupyter or opt for managed cloud versions like Colab, Databricks.
Here is a simple yet powerful framework proposed by Anaconda CEO to think through this multi-dimensional decision:
Key considerations across these decision lenses include:
- User profile – Students, passion users => Open source
- Total cost of ownership – Weigh license costs, operational overheads
- Expertise availability – Assess for ops, security and data engineering skills
- Pace of innovation – Prioritize access to emerging commercial features
Both open source and commercial options continue pushing boundaries of notebook innovation in their own ways. So evaluate tradeoffs across these factors at regular intervals as your needs evolve over time.
Tips for Choosing the Right Notebook Environment
We earlier covered the landscape of popular notebooks available and their key strengths.
Here are some additional tips on how to pick the right notebook environment based on your use case:
For hobbyists, students: Prefer free hosted notebooks like Colab, Kaggle
For ML research: Colab, Kaggle provide free access to GPUs to train models faster
For database analysts: Pick options like Databricks, Mode which simplify accessing data
For data teams: Kaggle, Deepnote excell in collaborative project execution
For large data workloads: Apache Zeppelin provides Spark integration
For enterprise grade trust: Databricks provides highest quality, support
Evaluate options across languages support, data source connectivity, collaboration, security considerations while choosing the right notebook platform addressing your requirements.
Prioritize playing to the intrinsic strengths of notebook medium allowing for fluid exploratory analysis before rigid workflows take over later in the machine learning lifecycle.
Notebook platforms will continue advancing rapidly. So revisit your platform decisions periodically as new capabilities get added over time across open source and commercial offerings.
I hope you found this guide useful in developing better intuition across past, present and future of the notebooks ecosystem.
Happy exploratory computing!
