A look inside the gorgeous world of "Dark Data"

Today, NASA's Jet Propulsion Lab data scientists showed us some of what they've been doing with the massive amounts of data they've collected — and how you can do it, too.

Video: Visualization of the 1236 exoplanet candidates observed by Kepler, created by Jer Thorpe


In a Q&A, JPL data scientist Rob Witoff, explained how they turn the terabytes upon terabytes of data they work with into visualizations, saying "Our most impactful visuals have been to showcase the existence of information in data that hadn't previously been considered. We think of this as illuminating 'Dark Data'."

And you can get ahold of some of that same data from NASA to create visualizations of your own, like the one Witoff recommended above from Jer Thorpe:

We think of science data in 4 levels from 0 (raw, unprocessed instrument data) to level 4 (modelled, processed, validated and ready to use and publish new discoveries). Raw data is unique to an instrument and largely unusable by anyone without the instrument scientist's models, calibrations and domain knowledge. Level 4 data is, however, readily available. We publish this across NASA's Distributed Archives (here), our Planetary Data System (here) and more recently the NASA Earth Exchange (the NEX) that's pushing these files to the cloud for scalable availability to the world. The President's Open Information Mandateis also helping us focus on the broader availability and ease of access of our data. The last piece I'll mention is also my favorite — the use of publicly available storage within our data processing pipelines. The engineers that build MSL's downlink pipeline chose to make all imagery data public immediately, which the world has accessed millions of times here.

Share This Story

Get our newsletter



Couldn't we just cal "Dark Data" by some other, more clever name? Like, I dunno...