The positive tests are coming mainly from Western states, including California, Washington and Arizona. Regardless of where you're located in the U.S., though, most people are recommended to get their annual flu shot in September or October, and the same goes for this flu season.
The annual flu shot is the best way to prevent hospitalization and death from seasonal influenza. Nearly everyone ages 6 months and older is recommended to get a flu vaccine, and older adults have several special options, including a high-dose shot, a vaccine containing an extra immune-stimulating ingredient called an adjuvant, and a brand-new mRNA vaccine.
Given that flu season is off to an earlier start this year, our latest poll asks whether you plan to get vaccinated or have gotten vaccinated already. Tell us your thoughts in the comments below.
With the growing number of SRv6 implementations that netlab supports, it made sense to create a set of topologies to help you get started exploring the SRv6 world. The sample topologies include:
In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.
However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:
We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.
Challenges
As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:
Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.
This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.
TimeSeries Abstraction
Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.
Time Slices and Immutable Windows
This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.
This slice information is critical and is maintained in a dedicated metadata table:
Time Slice Metadata Storage
Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.
TimeSeries Data and Control Plane
With that background, let’s look at the different solutions we built for data tiering.
Solution 1: Streaming with Apache Kafka and Flink
Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.
Streaming with Apache Kafka and Flink
Getting Data Into S3 Efficiently
To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.
Enter the Compaction job.
Compaction Job to Optimize Reads
The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.
The job further optimizes the Parquet files in the following ways:
Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.
Compacted and Sorted Parquet File Layout
Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!
On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.
Compaction Burndown Per Day
Streaming Cold Tier Read
Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:
TimeSeries Tiered Storage Read Path
The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.
Cold Tier Read Latency in Seconds
At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:
Cold Tier Read Requests Per Second
Why not stop here?
The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:
Additional Infrastructure and Operational Cost
We wanted to improve this further and asked ourselves:
Can we avoid streaming data into the object store and the associated data transfer costs?
Can we reduce the compute and operational overhead of new components?
That led to a more radical question: Can we read directly from existing Cassandra backups in S3?
Solution 2: Cassandra Cold Storage for Immutable Data
At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:
Read directly from these S3 backups instead of paying a separate data transfer cost to S3.
Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!
New TimeSeries Read Path
To make this viable in production, we had to answer a series of “How…?” questions.
How Can We Read the Backups Efficiently?
We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:
Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:
SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.
ROCS re-compacts all SSTables once, aligning them precisely with token ranges:
ROCS SSTable Boundaries
ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:
Tail Latency Improvement in Reads after ROCS
Once the immutable Time Slices have been optimized, it is time to trigger backups.
How Would We Leverage Backups?
Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:
Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.
How Would Cassandra Read From Backups?
Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.
-- create table for a given Time Slice CREATE TABLE data_20260616_00_04 ( ... ..., PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key) ) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC) ... AND compaction = ... AND compression = ... AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable ...
The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.
Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:
State Machine for Cassandra Backup Memtable Initialization
This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:
Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.
Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:
Reading from Cassandra Backup Memtables
To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.
We further optimize cluster and table settings to reduce the number of S3 calls:
Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:
Automating Time Slice Lifecycle Transitions
Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.
How Would Cassandra Handle The Additional Load?
Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.
To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:
Isolated Cold Tier Cassandra Topology
The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.
Real World Performance
Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:
Storage Reduction During Rollout: from 80% disk usage to 7%
At the same time, it serves thousands of reads per second from cold tier storage in any given region:
Cold Tier Read Requests Per Second
Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:
Cold Tier Read Latency: blue — p90 latency, brown — p99 latency; latency measured in seconds)
A couple of months ago Jim Highsmith explained how
practitioners writing for each other forms a distinct style of writing. He
now adds a section to the article with more detail on the craft of how we
write in this style. How the writer shows up in the prose, its four
principles, the role of story, how to sensibly use AI, three levels of
editing, and clarity of the audience.
Over its five-year history, P99 CONF has hosted quite a few speakers who’ve offered pointed takedowns of the namesake metric. At last year’s conference, Adrian Cockcroft didn’t explicitly state that P99s are BS…but he did allude to it. If you don’t know Cockcroft, he’s spent decades architecting, scaling, and optimizing resilient, high-performance systems at giants like Sun Microsystems, Netflix…
Not every presentation needs slides, but when they earn their place,
they work as a visual channel that complements the speaker rather than
replacing them. Sumeet Gayathri Moghe sets out the
principles that follow from that idea — control over the rate of knowledge
exposition, tight coupling between slides and speaker, minimalism by
default, visuals that say more than words can, and never sending your
slides out in advance.
Log in
