I/O and Data Formats

Welcome Back

Yesterday:

• Coder
• Intro to AWS
• Team Git Repo

Welcome Back

Today:

• I/O
• Containers
• Parallel Computing Intro
• Programmatic Cloud Access
• Team Projects

Session Overview

Input/Output

Effective and Efficient Input and Output will make or break a project.

Avoid if possible:

• Iteratively reading files into a large object with a sequential loop.
• Reading in a monolithic large file only to extract summary statistics on one variable.
• Reading in a large file, then saving it to a new in-memory object with each processing step.

Data Formats

The Foundation for Performant Data Science:

Performant Data Formats

• Push-down filtering
• Parallel Access
  • Subfiles make up complete dataset
• Lazy-loading

Components of Performance

• Size
• Speed
• Zero (or close to zero) Copy

Components of Performance

Performance Takeaways

Seconds matter…

The corollary: To solve big data problems, make them small data problems.

Session Overview

• Input/Output
• Data Formats
  • Parquet
  • Zarr/Icechunk
  • Visualization Formats

Data Formats

Cloud Native Formats

Parquet

• Apache Parquet
• Apache Iceberg
• Columnar Data Store
• Tabular Data
• The Better CSV
• Orders of Magnitude Improvements in Size and Speed
• Lazy loading in R (Vroom), Python (Polars, DuckDB, Dask), and JavaScript
• Stable, well supported, community
• Underlying foundation for many AWS Serverless Offerings

Zarr/Icechunk

• Zarr Project
• Icechunk Project
• Chunked, compressed, N-dimensional arrays (think Numpy)
• Python, Rust, JavaScript
• Push-down Filtering, Parallel, and Lazy Loading
• Dask
• Standard for Gridded Data
• Icechunk Webinar Next Tuesday

Data Formats

Cloud Native Formats

Data Conversion

• Do it yourself: Read it in as one format, export to another:
  • AWS Lambda is very good at this.
• Use conversion tools:
  • Pangeo Forge
  • Kerchunk
  • Virtualizarr

In Action:

• Parquet
• Zarr

Parquet and Zarr

1. Open up the intro-to-io/intro-io.ipynb notebook.
2. Play