Teams are often told they need a big data platform before anyone has checked whether their data needs one. Agnotic Technologies sizes your workload, reviews what you already run, and gives you a platform decision and an implementation plan that starts with one use case.
Trusted by global innovators
Big data consulting
Big data consulting is deciding whether your data needs a distributed platform and, if it does, which one to build and in what order. Agnotic Technologies sizes your volume, arrival rate, formats and query patterns, reviews any cluster or lake you already run, then writes the platform decision and a phased roadmap.
Many teams that ask for a big data platform need a well-tuned database or a cloud warehouse instead. We say so when that is the answer, because a smaller platform costs less to run and is easier to staff.
When a distributed platform is the answer, the engine detail lives on its own pages: Apache Spark for batch processing, Apache Kafka for event streams and Apache Cassandra for high-write operational data. This page covers the decision and the plan.
Platform sizing table
Start with the smallest platform that meets each trait, and move up only when the signal in the last column appears. The answer is per workload, not per company.
| Workload trait | Smallest platform that usually meets it | Signal that the next tier is needed |
|---|---|---|
| Data volume | One relational database, such as PostgreSQL, with partitioned tables and the right indexes | Reports still slow the application after partitioning and indexing, or backups no longer finish in their window |
| Arrival rate | Scheduled batch loads into a cloud warehouse | Loads overrun their schedule, or the business needs data within minutes instead of the next morning |
| Variety of formats | A cloud warehouse with semi-structured column types for JSON | Images, documents, logs or sensor files must be kept raw and processed with code, not SQL alone |
| Query concurrency | A cloud warehouse with separate compute for dashboards | Many teams and tools query the same data and each needs its own compute over one shared copy |
| Freshness need | Hourly or daily batch | Decisions or product features depend on events seconds old, which calls for a streaming platform |
| Processing shape | SQL transformations inside the warehouse | Jobs need custom code, model features or joins too large for one machine, which points to a lakehouse with a distributed engine |
When a row lands on streaming, the design continues on our real-time data warehouse page. The storage choice between warehouse, lake and lakehouse is set out on our comparison page.
Platform assessment
For teams that already run Hadoop, Spark, a data lake or a warehouse that grew without a plan. Each check ends in a keep, fix or retire decision.
Example: Hundreds of tables and jobs, and nobody sure which still feed a report
Approach: We read query and job history to list the tables, jobs and dashboards that actually ran in recent months, and mark the rest for archive or removal.
Example: A bill that keeps growing while usage looks flat
Approach: We split spend by cluster, job, storage tier and team, and find idle clusters, oversized nodes and data kept long after anyone read it.
Example: Nightly jobs that fail and are rerun by hand
Approach: We list failures and reruns from the scheduler and trace the worst offenders to skew, small files, schema drift or missing tests.
Example: One engineer who understands how the cluster works
Approach: We map the skills each platform needs against the team you have, because a platform nobody can operate is a risk whatever its features.
Example: Broad read access to raw data that holds personal information
Approach: We review roles, service accounts, encryption and audit logs, and list the data sets that need tighter access before anything moves.
Decision record
We write one decision per platform layer, with the options considered, the choice and the reason, so the next engineer knows why the platform looks the way it does.
Governance
Adding these to a platform that is already full is slow, so we set them up with the first use case.
Big data implementation goes better one use case at a time. Each phase ends with something a team uses, not a platform waiting for users.
One report, product feature or model with a named owner, a clear source list and an agreed definition of done. It decides what the platform must support first.
Phase 1Storage, one ingestion path, one compute engine, the catalog and access roles, all defined as code and sized for the first use case only.
Phase 2The first use case goes live with its results reconciled against the current source of truth, and the owning team signs off before more is added.
Phase 3Each new use case reuses the platform and adds only what it needs. Old jobs, copies and clusters are switched off as their consumers move.
Phase 4End to end big data applications do not stop at a dashboard. When results feed a product, the serving layer needs its own design.
Results written to a fast store such as PostgreSQL or Redis, shaped for the queries the application runs, so the app never queries the lake directly.
REST or GraphQL endpoints over the serving tables, with caching, pagination and rate limits, versioned so the platform can change behind them.
Scores, segments and flags pushed into CRM, billing or support tools on a schedule, with each run logged and replayable.
Model features computed once on the platform and served with the same definitions used in training, so predictions match what was tested.
Each served data set carries its last update time, and the application shows a safe fallback when a load is late.
Row filters applied in the serving layer, so one customer's results never reach another customer's screen.
Infrastructure
Big data infrastructure services cover the platform layer under the use cases. We define all of it as code, so environments match and can be rebuilt.
Scope and cost
We do not publish prices or rate cards. Big data implementation cost depends on your sources and use cases, so these are the factors we size on the scoping call.
Databases, files, event streams and third-party APIs each need their own ingestion path and tests.
Reviewing an existing cluster or lake adds discovery work, and moving off it adds a migration.
How much past data must be loaded and reconciled before the first use case can go live.
Daily batch is the simplest. Minutes or seconds add streaming infrastructure and on-call.
Catalog, lineage, masking and audit needs, which grow when data includes personal or regulated information.
APIs and serving stores for product features add application work on top of the platform.
Founders who need to know whether their database will hold up or a warehouse should come next. Big data consulting for startups often ends with a smaller platform than expected.
Teams running Hadoop or a self-managed Spark cluster that costs more each year and has few people who can operate it.
Lakes full of raw files with no catalog, where nobody can say which data sets are current.
Recommendations, usage analytics or customer reporting that must be served inside the application.
Several warehouses and lakes after a merger or reorganization that need one target platform and an order for moving.
References
Definitions and guidance from NIST and the platform owners that shape how we size a big data platform.
NIST SP 1500-1r2, Volume 1 of the NIST Big Data Interoperability Framework (2019), describes the opportunity in big data as addressing the volume, velocity and variety of data through new scalable architectures.
Source: NIST SP 1500-1r2: Big Data Interoperability Framework, Volume 1, DefinitionsNIST SP 1500-6r2 (2019) sets out the NIST Big Data Reference Architecture, with roles for the System Orchestrator, Data Provider, Big Data Application Provider, Big Data Framework Provider and Data Consumer, plus a Management fabric and a Security and Privacy fabric.
Source: NIST SP 1500-6r2: Big Data Interoperability Framework, Volume 6, Reference ArchitectureThe Azure Architecture Center describes a big data architecture as one that handles data too large or complex for traditional database systems, and notes that the threshold varies by organization, from hundreds of gigabytes to hundreds of terabytes.
Source: Azure Architecture Center: big data architecturesThe Apache Spark project describes Spark as a multi-language engine for data engineering, data science and machine learning on single-node machines or clusters.
Source: Apache Spark project site
