Capture
- Log-based CDC from PostgreSQL, MySQL, SQL Server or Oracle
- Debezium or a managed CDC service
- Initial snapshot, then ongoing changes
Real-Time Data Warehouse
Real-time data warehouse and streaming services
Nightly batch loads suit finance reports and are too slow for stock positions or fraud signals. Agnotic Technologies builds real-time data pipelines that capture changes from your operational databases, stream them into warehouse tables continuously, and keep each use case on the latency tier it really needs.
Latency tier per use caseLog-based change data captureExactly-once where it mattersOperational analytics and reverse ETL
Trusted by global innovators
Real-time data
What is a real-time data warehouse?
A real-time data warehouse is an analytics store that receives changes from operational systems continuously instead of in nightly batches, so dashboards and applications query data that is seconds or minutes old. Agnotic Technologies builds the change data capture, streaming, processing and table layers behind it, and decides with you what actually needs real time.
Real time costs more to run than batch, because the pipeline never stops. We keep batch wherever a daily refresh is enough and stream only where a decision depends on fresh data.
Our data engineering service covers batch pipelines and warehouse design, and plant floor sensor data has its own industrial IoT platform page. This page is about business databases and business events.
Decision table
Which use cases need real time, and which pattern fits?
We fill this in for each use case before choosing any technology.
| Use case | Latency tier | Pattern | Why |
|---|---|---|---|
| Month-end finance reporting | Batch, daily | Scheduled extract and load, then transforms | Figures are reconciled and signed off; fresher data adds cost, not value |
| Inventory position across stores or warehouses | Micro-batch, every few minutes | CDC into a lakehouse table with frequent merges | Stock decisions need recent counts, not every single event |
| Fraud or abuse signal | Streaming, seconds or less | Stream processing on events, with rules or a model scoring each one | The decision must happen before the transaction completes |
| Operations dashboard for a shift or a fleet | Streaming into a real-time OLAP store | CDC and events into tables built for fast aggregate queries | People act on what they see during the shift |
| Machine learning features | Mixed: streaming for fresh features, batch for history | One feature definition computed in the stream and backfilled in batch | Training and serving must see the same feature logic |
| Customer insights pushed to CRM or support tools | Micro-batch reverse ETL | Warehouse models synced to operational tools on change | Sales and support act on warehouse insights inside their own tools |
Ask what decision changes if the data is an hour old. If none does, batch is the cheaper answer.
Reference pipeline
How does a database change reach a warehouse table?
The five layers we build, from the source database's log to the tool where someone acts on the data.
Stream
- Apache Kafka or a managed equivalent
- Schema registry with compatibility rules
- Topics per table or per business event
Process
- Apache Flink or Spark Structured Streaming
- Event time windows and watermarks
- Deduplication and late data rules
- Checkpoints for recovery
Store
- Apache Iceberg or Delta Lake tables with upserts
- A real-time OLAP store for dashboards
- Compaction and snapshot expiry jobs
Serve and act
- BI and operational dashboards
- Reverse ETL to CRM, support and marketing tools
- Alerts and application APIs
Low latency serving
How do you serve fresh data to an application in milliseconds?
Low latency applications get fresh data through a serving pattern chosen per query: a precomputed key-value lookup in a cache, a real-time OLAP store for aggregates, a search index for text and filters, or a direct warehouse query where a slower answer is fine. Agnotic Technologies picks the pattern from the query and the freshness it needs.
| Serving pattern | Freshness | Query it suits | Failure point |
|---|---|---|---|
| Precomputed key-value lookup in a cache | As fresh as the stream job that writes it | One record by key: a balance, a stock level or a model feature | Cache and source drift apart when an update event is lost, so keys need expiry and a rebuild path |
| Real-time OLAP store | Shortly after each event is ingested | Aggregates over recent events, with filters and group-by | Large joins at query time slow it down, so data is joined before it is loaded |
| Search index | After each index refresh | Free text, facets and filters across many fields | A mapping change needs a full reindex, so index aliases allow a swap without downtime |
| Direct warehouse query | As fresh as the last merge into the table | Rare or ad hoc queries and internal tools | Concurrency limits queue requests, so it never sits behind a public endpoint without a cache |
Before choosing tools we set a latency budget per hop: ingest, process, store, query and the network back to the user each get a share, and load tests measure each share on its own. Caching and load testing are covered on our Redis development and performance testing pages.
Failure points
Where do streaming pipelines go wrong?
Problems we design against from the first pipeline.
Challenge
A source team renames a column and the pipeline breaks overnight.
Agnotic approach
Schemas are registered with compatibility rules, so a breaking change is rejected at the producer instead of found downstream.
Challenge
Totals double after a restart.
Agnotic approach
Processing is transactional or idempotent end to end, and sinks upsert on a key, so replays never double count.
Challenge
Late events land in the wrong window or vanish.
Agnotic approach
Windows use event time with watermarks and an allowed lateness, and late events update results instead of being dropped.
Challenge
Thousands of small files make queries slow.
Agnotic approach
Compaction runs on a schedule, and merge frequency follows the use case's latency tier.
Challenge
Deletes in the source never reach the warehouse.
Agnotic approach
CDC captures deletes, and the table format applies them as row-level deletes.
Challenge
The CDC connector falls behind and the source log fills its disk.
Agnotic approach
Replication lag and retained log size are monitored, with alerts well before the source is at risk.
What do we build for real-time and operational analytics?
The components of a real-time data warehouse, built and handed over with runbooks.
CDC connectors
Log-based capture from your operational databases, with snapshots, permissions and log retention set up with your DBAs.
Streaming backbone
Kafka or a managed equivalent, with topics, retention, access control and a schema registry.
Stream processing jobs
Joins, windows, deduplication and enrichment in Flink or Spark, with checkpoints and replay.
Real-time tables
Upserting lakehouse tables and real-time OLAP stores, with compaction and retention jobs.
Data contracts
Schemas, owners and compatibility rules agreed with source teams and checked in CI.
Reverse ETL
Warehouse models pushed back to CRM, support and marketing tools, so operational analytics reaches the people who act on it.
Running cost
How do we keep always-on pipelines affordable?
Streaming is billed every hour of the day, so cost control is part of the design.
- Batch stays batch wherever a daily or hourly refresh meets the need
- Micro-batch merges instead of per-event writes where minutes are fine
- Topic and snapshot retention set by what is actually replayed
- Compute sized to the stream's real volume, with autoscaling on managed services
- Each pipeline tagged to the use case that pays for it, so cost is visible
- Unused topics, jobs and tables found and retired at a regular review
Who it is for
Which teams need real-time data?
Teams whose decisions depend on what happened minutes ago, not yesterday.
Retail and ecommerce operations
Inventory, order status and fulfilment views that follow the business during the day.
Fintech and payments
Fraud, risk and limit signals evaluated while a transaction is still open.
Logistics and field operations
Fleet, job and exception dashboards for dispatchers and shift leads.
SaaS companies
In-product analytics that show customers their own fresh data.
Sales, support and success teams
Warehouse insights synced into the tools they already use through reverse ETL.
Technologies
Which technologies do we build streaming pipelines on?
Open source projects and their managed equivalents on your cloud.
Apache KafkaDebeziumApache FlinkApache IcebergDelta LakeSpark Structured Streaming
Facts and sources
What do the Kafka, Debezium, Flink and Iceberg docs say?
Project documentation behind the patterns on this page, each linked to its primary source.
Debezium describes log-based change data capture as capturing all data changes with very low delay, while avoiding the extra CPU load that frequent polling puts on the source database.
Source: Debezium documentation: featuresApache Kafka guarantees at-least-once delivery by default; exactly-once delivery is available within Kafka through the transactional producer and read-committed consumers, while exactly-once delivery to other systems generally needs their cooperation.
Source: Apache Kafka documentation: designApache Flink documentation explains that some events arrive after the watermark has passed their timestamp, so streaming programs must expect late elements, and holding watermarks back too long delays window results.
Source: Apache Flink documentation: timely stream processingVersion 2 of the Apache Iceberg table spec adds row-level updates and deletes, using delete files so individual rows can be changed without rewriting the immutable data files.
Source: Apache Iceberg table spec
Scope and cost
What drives the cost of a real-time data warehouse?
Build cost and running cost depend on the same factors, so we estimate both together on the scoping call.
Source systems
How many databases and event sources feed the pipeline, and which engines they run.
Change volume
How many changes per day each source produces, which sizes the stream and its storage.
Latency tiers
How many use cases need streaming rather than micro-batch or batch.
Transformations
Joins, enrichment and business rules applied in the stream.
Destinations
Warehouses, OLAP stores and operational tools the data must reach.
Operations
Whether your team or ours runs the pipelines after launch.
Frequently Asked Questions
We do not publish prices. Cost depends on the number of sources, change volume, how many use cases need streaming, the transformations and the destinations. Running cost depends on the same factors, so we estimate both on the scoping call.
Operational analytics puts fresh data in front of people who act during the day, such as shift leads or support agents, often inside their own tools through reverse ETL. A classic warehouse serves planning and reporting on data that is a day old.
Usually not. Most cloud warehouses and lakehouse table formats accept streaming inserts or frequent merges. We add a streaming path beside your batch loads and move only the use cases that need it.
Log-based CDC reads the database's transaction log instead of querying tables, so the load on the source is low. It still needs log retention, permissions and monitoring, which we set up with your database team.
Within Kafka, exactly-once is available through transactions. Across other systems it depends on the sink, so we make sinks upsert on keys, which keeps replays safe even when delivery is at least once.
Often not yet. Startups usually get more from a clean batch warehouse first. We add a real-time path when one product decision clearly depends on fresh data, and build it so more use cases can follow.