Topics
- One topic per business event type, such as order placed or payment captured
- Names that show the owning team and the event
- A written contract for each event's fields
Apache Kafka
Apache Kafka consulting and development services
When Kafka sits between your services, a wrong partition key or acks setting shows up as events out of order, lost writes or consumers hours behind. Agnotic Technologies designs topics, durability and consumer groups for Kafka as an application backbone, and moves clusters to KRaft.
Topic and key designDurability settingsConsumer lagKRaft migration
Trusted by global innovators
Kafka consulting
What does Apache Kafka consulting cover?
Apache Kafka consulting is designing and running Kafka as the event backbone between your applications, not only as a feed into analytics. Agnotic Technologies designs topics, partitions and keys for ordering, sets replication and acknowledgments for durability, builds consumer groups that keep up, secures the cluster and moves it from ZooKeeper to KRaft.
We work on new event-driven systems and on production clusters that lose events, fall behind or need a version upgrade.
Change data capture into a warehouse, schema registry rules and stream processing jobs are covered on our real-time data warehouse page. Where to split services is covered on our microservices consulting page.
Event design
How should topics, partitions and keys be designed?
Kafka keeps order only within a partition, so the key decides both ordering and parallelism.
Keys
- The key is the entity whose events must stay in order, such as an order ID
- Events with the same key land in the same partition
- No key only where order does not matter
Partitions
- Enough for the consumers you plan to run in parallel
- Partitions can be added but never removed
- Adding partitions changes which partition a key maps to
Consumers
- One consumer group per downstream use
- Idempotent handlers, since redelivery happens after a rebalance
- Dead-letter topics for events that cannot be processed
Settings decision table
Which Kafka settings matter most, and what goes wrong?
The settings we review first on any cluster. The starting rule is where we begin before measuring your own workload.
| Setting | What it controls | Safe starting rule | Failure when it is wrong |
|---|---|---|---|
| Partition count | How many consumers in a group can read a topic in parallel | Plan for peak parallel consumers with room to grow, since the count can rise but never fall | Too few and consumers cannot scale out; too many and brokers carry more files and recover more slowly |
| Replication factor | How many brokers hold a copy of each partition | 3 for production topics, the typical setup in the Kafka documentation | 1 or 2: one broker failure can lose data or stop writes |
| acks (producer) | When the producer treats a write as done | all, which is the producer default | acks=1 or 0: a leader failure right after the write can lose the record |
| min.insync.replicas | How many in-sync replicas must have a write when acks is all | 2 with a replication factor of 3 | 1 gives acks=all no extra safety; 3 with a factor of 3 stops all writes when one broker is down |
| retention.ms and retention.bytes | How long, or how much, data a topic keeps | Longer than the longest consumer outage you must recover from, plus any replay needs | Too short and a lagging consumer loses events; too long and disks fill |
| cleanup.policy | Whether old records are deleted by age or compacted to the latest value per key | delete for event streams, compact for topics that hold current state per key | delete on a state topic loses latest values; compact on an event stream drops history consumers need |
| max.poll.interval.ms (consumer) | How long a consumer may go between polls before the group treats it as failed | Longer than the slowest batch takes, with batch size tuned to match | Too short and slow batches trigger rebalances and repeated processing |
These are starting rules, not answers. We confirm each one with load tests on your message sizes and consumer code.
Consumer groups
Why do Kafka consumers fall behind?
Consumer lag is the gap between the newest offset and what a group has processed. We alert on it per group and per partition.
Challenge
One partition lags while the rest keep up, because a few keys carry most of the traffic.
Agnotic approach
Rethink the key so hot entities spread out, or route hot keys to their own topic, and alert on lag per partition, not only per group.
Challenge
The group rebalances again and again, and processing stops each time.
Agnotic approach
Keep processing inside max.poll.interval.ms, use cooperative rebalancing or the newer consumer group protocol, and give consumers static membership where restarts are routine.
Challenge
Events are processed twice after a restart.
Agnotic approach
Commit offsets after processing, not before, and make handlers idempotent with an event ID check.
Challenge
Lag grows steadily at peak, because events arrive faster than consumers can handle them.
Agnotic approach
Add consumers up to the partition count, speed up the handler, or add partitions with a plan for the key remapping.
Challenge
One bad event fails over and over and blocks its partition.
Agnotic approach
Send events that still fail after retries to a dead-letter topic with the error, and alert the owning team.
KRaft migration
How do we move a cluster from ZooKeeper to KRaft?
Kafka 4.0 and later run only in KRaft mode, so ZooKeeper clusters must migrate on a 3.x release before they can upgrade.
1.
Reach a migration-ready version
Brokers move to a 3.x release that supports the migration, such as the 3.9 line, with metadata and client versions checked first.
Prepare2.
Add the KRaft controller quorum
A set of KRaft controllers starts in migration mode and copies the cluster metadata from ZooKeeper.
Controllers3.
Move brokers to KRaft
Brokers restart one at a time against the new controllers, while ZooKeeper stays available as the way back.
Brokers4.
Finalize, then upgrade
Once every broker runs in KRaft mode and checks pass, the migration is finalized, ZooKeeper is retired and the cluster can move to Kafka 4.x.
FinalizeHosting
Self-managed or managed Kafka?
We describe each option on its own terms and choose with you, based on operations, features and the cloud you already use.
Self-managed Apache Kafka
- Virtual machines, or Kubernetes with an operator such as Strimzi
- Full control of versions, settings and tiered storage
- Your team or ours owns upgrades and on-call
Amazon MSK
- AWS runs the control plane: creating, updating and deleting clusters
- Standard Kafka APIs for producing and consuming
- Fits teams already on AWS
Confluent Cloud
- Kafka run as a cloud service by Confluent
- Connectors and schema tools from the same provider
- Fits teams that want the cluster operated for them
Azure Event Hubs Kafka endpoint
- The Kafka protocol on an Azure service, with no Kafka cluster to run
- Existing clients often connect through configuration changes only
- We test which Kafka features your clients rely on first
How do we secure and operate a Kafka cluster?
Every client authenticates, every connection is encrypted and every topic has an owner.
TLS on every listener
Encryption between clients and brokers and between brokers, with certificates rotated on a schedule.
SASL authentication
SCRAM or OAUTHBEARER for applications, or mutual TLS where certificates are already managed, so every client has its own identity.
ACLs per topic and group
Each service writes only the topics it owns and reads only the ones it consumes, with no shared superuser accounts in application code.
Client quotas
Produce and fetch quotas per client, so one noisy service cannot slow the cluster for everyone.
Retention and tiered storage
Retention set per topic from replay needs, with older log segments moved to remote storage such as Amazon S3 where tiered storage is turned on.
Monitoring and alerts
Under-replicated partitions, consumer lag, request latency and disk use, with alerts routed to the owning team.
Sizing
How do you size a Kafka cluster?
Instead of a Kafka cost calculator, we size clusters from measured inputs and then load test. Big data Kafka cost follows these factors, priced at your provider's own rates.
Write throughput
Peak messages per second and average message size, per topic.
Read fan-out
How many consumer groups read each topic, since every group reads the full stream.
Replication factor
Each copy multiplies disk use and network traffic between brokers.
Retention
How long data stays on brokers, and whether tiered storage moves older segments to cheaper storage.
Partitions
Total partitions across the cluster, which drives broker memory, open files and recovery time.
Hosting model
Self-managed brokers, a managed service or a Kafka endpoint each charge for these inputs in a different way.
Which applications use Kafka as their backbone?
Order and payment flows
Services that publish each step as an event, keyed by order so every step is handled in sequence.
Services that should not call each other directly
Events instead of chained API calls, so one slow service does not stall the rest.
Device and telemetry ingestion
High-volume readings collected once and read separately by alerting, storage and analytics.
Audit and activity logs
An append-only record of who did what, kept as long as retention requires.
Startups planning for growth
Big data Kafka for startups pays off once several services need the same events; before that, a managed queue may be enough.
References
Kafka references we design against
Apache Kafka design docsApache Kafka upgrade notesApache Kafka security docsAmazon MSK Developer GuideAzure Event Hubs for Apache Kafka
Facts and sources
Facts behind our Kafka designs
Guarantees and rules from the Apache Kafka documentation and the managed service providers that shape our settings.
Kafka writes events with the same key to the same partition, and guarantees that a consumer reads each partition's events in exactly the order they were written.
Source: Apache Kafka documentation: introductionThe Kafka documentation gives a replication factor of 3, min.insync.replicas of 2 and producers using acks=all as a typical durable setup; when fewer in-sync replicas are available than min.insync.replicas, acks=all writes fail with an exception.
Source: Apache Kafka topic configuration: min.insync.replicasApache Kafka 4.0 supports only KRaft mode, with ZooKeeper mode removed, and the upgrade notes say clusters in ZooKeeper mode must be migrated to KRaft before they can upgrade to Kafka 4.x.
Source: Apache Kafka documentation: upgrade notesAmazon MSK provides the control-plane operations for Apache Kafka clusters, such as creating, updating and deleting them, while applications use standard Kafka data-plane operations to produce and consume data.
Source: Amazon MSK Developer Guide: what is Amazon MSKAzure Event Hubs provides an Apache Kafka endpoint, and Microsoft notes that Kafka applications can often connect by changing only the connection string in their configuration, with no code changes.
Source: Azure Event Hubs for Apache Kafka overview
Frequently Asked Questions
It depends on how many clusters you run, the version gap and how much change you can make at once. We plan it in stages with a way back until the final step, and agree the plan after reviewing your clusters.
Enough for the number of consumers you expect to run in parallel at peak, with room to grow. Partitions can be added later but not removed, and adding them moves keys to new partitions, so plan the count early.
Within one partition, yes. Events with the same key go to the same partition, and consumers read each partition in order. Order across partitions is not guaranteed, so the key must be the entity whose order matters.
Yes. We review the current settings, consumer groups and client versions, then change them in steps with monitoring, so producers and consumers keep running during the work.
We do not quote prices here. Running cost follows throughput, fan-out, replication, retention and the hosting model, at the provider's published rates. We estimate it from your measured traffic during scoping.
Usually not at first. A managed service lets a small team use Kafka without running brokers. Self-managed Kafka makes sense when the team needs settings or features a managed option does not offer.
AI tools may help draft producer and consumer code or runbooks, but settings, ACLs and migration steps are reviewed by engineers and proven with load and failure tests before production.