streamtools
At The New York Times R&D Lab, I co-developed Streamtools, an open-source tool for manipulating real-time data. It provided a visual, flow-based interface where users could connect a vocabulary of operations to build live data-processing systems. Working from my colleague Michael Dewar's initial vision, I designed the project's core visual language and processing engine.
The project emerged from a belief that streaming data would fundamentally change how we reason about information, shifting analysis from static batches to continuous processing. We built streamtools to explore this future, creating a responsive environment that encouraged abductive reasoning by letting patterns emerge from the data rather than testing predetermined hypotheses.
The interface allowed for immediate understanding and live manipulation of running systems. We used streamtools to power several lab projects, including Delta, processing tens of gigabytes of event data daily with minimal latency. It demonstrated that complex data infrastructure could be made both legible and interactive.