Big Data
Big data describes datasets whose volume, velocity, variety or variability is too much for one database server to store and process in time, so they need a scalable architecture that spreads storage and computation across many machines. The term names the engineering problem the data creates, not a fixed size threshold.
Key takeaways
- Defined by characteristics, not by a size in terabytes
- The classic three Vs are volume, velocity and variety; variability is often added as a fourth
- Each characteristic forces a different architecture decision
- Many workloads called big data still fit one well-tuned database
- Distributed tools such as Hadoop and Spark move computation to where the data sits
What Big Data means
The idea behind big data is that the old answer, a larger server, stops working. When data arrives faster than one machine can write it, grows larger than one machine can hold, or comes in too many shapes for one schema, the system has to be split across many machines that work in parallel.
That is why NIST frames big data around scalable architectures rather than a number. A dataset that is big data for an analysis on one laptop may be routine for a cloud warehouse, and a modest dataset arriving as a fast stream can still need distributed processing.
Agnotic Technologies builds data platforms for clients who have reached that point, and helps them confirm it before they take on the cost. Storage choices are compared on our data warehouse vs data lake vs lakehouse page, and streaming patterns are covered on our real-time data warehouse services page.
What are the Vs of big data?
The Vs of big data are the characteristics that make data hard for one system to handle. The original three are volume, velocity and variety.
Volume is how much data must be kept. Velocity is how fast it arrives and how quickly it must be processed. Variety is how many formats and structures it comes in, from tables to documents, images and logs.
Variability is often added as a fourth V: the data's rate, format or meaning changes over time, such as a seasonal surge or a source that changes its schema without notice.
In big data analytics, the Vs matter because they decide which tools and patterns a pipeline needs before any query is written.
When is a workload not big data?
Many workloads described as big data are not. These signs say a single database or warehouse is still the right tool:
- The full dataset fits on one server's storage with room to grow, and indexed queries finish in acceptable time.
- Data arrives in batches, such as a nightly file, not as a continuous stream that must be processed within seconds.
- Sources share a few known schemas that can be modeled as tables.
- The slow part is a missing index, a poor query or an unpartitioned table, not the hardware.
What does big data look like in practice?
Healthcare claims history is a volume and variety problem. A payer keeps years of claims, eligibility records and clinical documents in different formats, and analysts need to query across all of them. That points to columnar storage partitioned by date and one shared data model across sources.
Machine sensor streams are a velocity problem. Vibration and temperature readings from many machines arrive continuously, and an alert is only useful within moments. That points to a stream processor that aggregates readings in time windows, with the raw history kept in low-cost storage for later analysis.
Which architecture decision does each big data characteristic force?
Ask the test question for each characteristic. A yes puts the decision in the right column on the table.
| Characteristic | The question that tests for it | The decision it forces |
|---|---|---|
| Volume | Will the data outgrow one server's storage, or will full scans take too long on one machine? | Distributed or cloud object storage, partitioning and columnar file formats |
| Velocity | Must new data be processed within seconds of arriving, without a pause? | Stream processing on a durable message log, not only nightly batch jobs |
| Variety | Does the data arrive as tables, documents, images and logs with no shared schema? | Schema-on-read storage, a common data model and a catalog of sources |
| Variability | Do volumes surge or formats change without warning? | Elastic compute, schema change detection and data quality checks at ingestion |
Facts and primary sources
NIST SP 1500-1r2, Volume 1 of the NIST Big Data Interoperability Framework, sets out a definition of big data and related terms, and describes the opportunity to address the volume, velocity and variety of data through new scalable architectures.
Source: NIST SP 1500-1r2ISO/IEC 20546:2019 gives a conceptual overview of the field of big data and provides the terminological foundation for big data related standards.
Source: ISO/IEC 20546:2019The Apache Hadoop HDFS design document describes HDFS as built for applications with large data sets, and lists as a design assumption that moving computation is cheaper than moving data.
Source: Apache Hadoop, HDFS ArchitectureThe Apache Spark documentation describes Spark as a unified analytics engine for large-scale data processing, with APIs in Java, Scala, Python and R.
Source: Apache Spark documentation
Explore this on Agnotic
Where Big Data shows up in how we build.