Data warehouse storage: the problems and what we build
Warehouse storage grows with history tables, layers, recovery windows and copies, not only with source rows. Agnotic Technologies builds the pipelines, models and retention jobs that keep it in line with your requirements.
Last reviewed by the Agnotic Technologies engineering team
What drives data warehouse storage growth?
Data warehouse storage grows with rows loaded per day, average row size and retention, then multiplies through history tables, storage layers, recovery windows and replicas, and shrinks only by the compression your data achieves on the platform. Agnotic Technologies builds warehouses, pipelines and retention jobs that keep each of those drivers visible and under control.
Why does warehouse storage outgrow the plan?
Stored volume rarely tracks source data one for one. These are the causes data teams run into most.
History tables keep every version
Type 2 slowly changing dimensions add a row each time a tracked attribute changes, and daily snapshots add a full copy per snapshot kept. Stored rows then outpace the business records behind them.
Layers copy the same data
Raw, staging and modeled layers can each keep the full history. Staging tables that are never truncated quietly become a second copy of the raw layer.
Recovery windows hold overwritten data
Time travel and fail-safe features keep changed and deleted data for a set number of days. Busy merge jobs and daily full reloads make this line grow.
Logical and physical figures get mixed up
Warehouses report uncompressed and compressed size, and some bill on one or the other. Comparing a logical figure with a physical one is off by the whole compression ratio.
Borrowed compression ratios
Compression depends on data types, sort order, repetition and the platform's encodings. A ratio taken from someone else's data makes a plan look precise while being wrong.
Copies outside the warehouse
Cross-region replicas, exports for disaster recovery and backup snapshots each hold another copy of the compressed data, and they are easy to leave out of a plan.
What do we build to keep storage under control?
Agnotic Technologies builds the warehouse and the jobs around it to your requirements: your sources, your history rules and the platform you choose.
Ingestion pipelines per source
Batch and change data capture pipelines that land each row once and record row counts and bytes for every load, so growth is measured rather than guessed.
Models with explicit history rules
Dimensional models where each table's history method, such as type 2, snapshot or overwrite, is chosen on purpose and documented with the reason.
Partitioning and clustering
Tables partitioned by date and clustered on common filters, so retention jobs drop whole partitions and queries scan less data.
Retention and lifecycle jobs
Scheduled jobs that age out data past its retention period, truncate staging tables and move cold history to cheaper storage tiers.
Compression test loads
Representative sample loads on your chosen platform that measure the real ratio per large table before the warehouse is sized.
Storage monitoring
Dashboards built on the platform's table storage views that show logical bytes, physical bytes and recovery storage per table over time, with alerts on unexpected growth.
Recovery and replica settings as code
Time travel windows, replicas and backup schedules defined in infrastructure code, so every change to them is reviewed alongside its storage effect.
What is the difference between logical and physical storage?
Logical storage is the uncompressed size of your rows. Physical storage is what sits on disk after columnar compression, plus anything the platform keeps for recovery.
Warehouses report both, and some bill on one or the other, so a storage plan should show both. Start with logical data kept, then physical data after your measured ratio, then the recovery and backup copies on top.
Keep the two apart when you compare figures with a vendor. A logical figure and a physical figure for the same data can differ by your whole compression ratio.
How do you measure your own compression ratio?
Ratios depend on data types, sort order, repetition and the platform's encodings, so measure them on your own data.
Load a representative sample, ideally a full day from each large source, into a test table on the platform you plan to use.
Read the uncompressed or logical bytes and the stored or physical bytes from the platform's table storage views.
Divide logical bytes by physical bytes. Repeat for each large table, because a table of free text compresses differently from a table of codes and amounts.
If the sources differ a lot, plan each source group with its own ratio and add the results.
Why do history and recovery multiply stored volume?
Stored volume can grow faster than the source data, because the platform keeps more than one version of the same rows.
History tables keep every version of a record. Recovery windows keep the versions you overwrote or deleted. Layers and replicas copy all of it again.
- A type 2 dimension that changes often can hold several rows per business key.
- A daily full reload into a table with a long recovery window keeps the replaced data for every day in that window.
- Staging tables that are never truncated quietly become a second full copy of the raw layer.
- A replica in a second region doubles physical storage, whatever the compression ratio.
How does stored volume change year by year?
Until the retention period is reached, stored volume rises every year. After that, the oldest year ages out as a new one arrives, so stored volume follows your growth rate instead.
Plan the initial build from the first year and capacity and lifecycle rules from the year retention is reached. A new source arriving later is best planned as its own addition from that year.
For an industry example, the energy data warehouse page shows how meter and grid data shape a warehouse design.
How does Agnotic build a data warehouse?
We start from your sources, history rules and retention policy, then measure compression with test loads on the platform you choose. The pipelines, models and lifecycle jobs follow from those decisions.
Our data engineering services team builds the pipelines, partitioning and retention jobs. Our cloud consulting services cover landing zones, storage tiers and cost controls around the warehouse.
Which platform rules change stored volume?
Facts from each warehouse vendor's own documentation, checked against the source on 21 Oct 2026.
BigQuery offers two storage billing models set at the dataset level: logical billing on the uncompressed size of the data, which is the default, and physical billing on the compressed size on disk.
Source: Google Cloud, BigQuery datasets introductionBigQuery's time travel window can be set from two to seven days, and a fail-safe period keeps deleted data for a further seven days after the time travel window for emergency recovery.
Source: Google Cloud, BigQuery data retention with time travel and fail-safeSnowflake's default Time Travel retention period is 1 day; with Enterprise Edition or higher, permanent databases, schemas and tables can be set to any value up to 90 days.
Source: Snowflake documentation, Understanding and using Time TravelSnowflake Fail-safe provides a non-configurable 7-day period, starting when the Time Travel retention period ends, during which historical data may be recoverable by Snowflake.
Source: Snowflake documentation, Understanding and viewing Fail-safeIn Amazon Redshift, a compression encoding specifies the type of compression applied to a column of data values as rows are added to a table, and ENCODE AUTO is the default for tables.
Source: Amazon Redshift Database Developer Guide, Compression encodings
Trusted by global innovators
Where should you go next?
- Data engineering servicesPipelines, warehouse models and data quality checks.
- Energy data warehouseWarehouse design for meter, grid and asset data.
- Cloud consulting servicesLanding zones, storage tiers and cost controls.
- Precision medicine data storageStorage planning for FASTQ, BAM, CRAM and VCF files.