Spark jobs that once ran in minutes now take hours, fail on skewed keys or cost more than the reports they feed. Agnotic Technologies writes, tunes and tests Apache Spark jobs for batch processing, and tells you plainly when a smaller engine would do the job.
Trusted by global innovators
Spark development
Apache Spark development is writing, tuning and testing Spark jobs that process data too large or complex for a single database query. Agnotic Technologies builds batch pipelines in PySpark, Scala and Spark SQL, fixes slow and failing jobs from the Spark UI, adds automated tests and plans upgrades to Spark 4.
We start by checking that Spark is the right engine. Some workloads run faster and cheaper as warehouse SQL or on one machine, and we would rather say so before writing the first job.
Streaming jobs, watermarks and exactly-once sinks are covered on our real-time data warehouse page. Platform choice and the wider roadmap sit on our big data consulting page.
Engine choice
Big data processing does not always mean Spark. We match the engine to the data size, the logic and where the data already lives.
Example: Joins and aggregates over tables already loaded in Snowflake, BigQuery, Amazon Redshift or Azure Synapse Analytics
Approach: Keep the logic in SQL inside the warehouse. Moving it to Spark adds a second engine to run and pay for, with nothing gained.
Example: Files or tables that fit on one large machine, processed with DuckDB, Polars or pandas
Approach: Run on one machine with no cluster to manage. Spark also runs on a single node, so code can move to a cluster later if data grows.
Example: Large joins, custom Python or Scala logic, model features or many files across object storage
Approach: Spark sized to the job, with partitioning and file layout planned so the job scales by adding executors.
Example: Events that must be processed within seconds of arriving
Approach: Spark Structured Streaming or Apache Flink, designed on our real-time data warehouse page rather than here.
Diagnosis table
We read the Spark UI before changing any setting. Each symptom points to a different cause, and the wrong fix often makes the job slower.
| Symptom in the Spark UI | Usual cause | Fix we apply |
|---|---|---|
| One or a few tasks in a stage run far longer than the rest | Data skew: a handful of join or group keys hold most of the rows | Confirm adaptive query execution skew join handling is on, salt the hot keys, or broadcast the smaller side of the join |
| Large shuffle spill to memory and disk | Too few shuffle partitions for the data, or wide rows carried through the shuffle | Start from a higher partition count and let adaptive query execution coalesce it, and select only the columns the next step needs |
| Thousands of very short tasks | Many small input files, or far too many shuffle partitions | Compact the table's files, set target file sizes on write, and let adaptive query execution merge small partitions |
| Executors lost with out of memory errors | Large results collected to the driver, oversized partitions, or Python functions holding rows in memory | Replace collect calls with writes, repartition before the heavy step, and swap row-by-row Python functions for built-in or vectorized pandas functions |
| Long scheduler delay before tasks start | Executors waiting for resources, or a driver busy planning a huge number of files | Right-size executors and dynamic allocation limits, and cut file counts with compaction and partition pruning |
| Sort-merge join where one side is small | Missing table statistics, so Spark cannot see that a broadcast join would fit | Collect statistics, or add a broadcast hint once the small side is confirmed to fit in executor memory |
Adaptive query execution has been on by default since Spark 3.2.0, so on older clusters the first fix is often the upgrade itself.
Runtime
The same Spark code can run in several places. We choose by who will operate it, what you already run and how jobs are scheduled.
Testing
Untested Spark jobs fail in production on data the developer never saw. We test logic, schemas and data at different stages.
Transformations written as functions and tested with a few hand-built rows, using PySpark's assertDataFrameEqual and assertSchemaEqual, or ScalaTest for Scala jobs.
Each job checks its input schema before it runs, so an upstream column change stops the job with a clear error instead of writing wrong results.
Row counts, null rates, uniqueness and value ranges checked on every run, with the run marked failed when a rule breaks.
Before release the job runs on a realistic sample in a test environment, so skew and memory problems appear before production.
Unit and schema tests run on every pull request, and jobs move between environments through the pipeline, never by hand.
Spark 4.0 changed defaults that can change results, not only speed, so we treat the upgrade as a tested migration.
Every job is listed with its Spark, Scala, Java and Python versions and libraries. Spark 4.0 drops Scala 2.12 and JDK 8 and 11, so older builds are recompiled first.
InventorySpark 4.0 turns on ANSI SQL mode by default, which raises errors on invalid casts and overflows that older defaults let through. We run each job with ANSI mode on while still on Spark 3 and fix what fails.
TestOld and new runtimes process the same inputs, and row counts and key totals are compared before any job switches.
CompareJobs move in groups, starting with those with the fewest dependents, and a rollback path stays in place until each group is stable.
SwitchDaily and hourly jobs that clean, join and aggregate data from object storage and databases into tables analysts and applications use.
Compaction, file sizing and clean-up of old snapshots on Apache Iceberg or Delta Lake tables, so reads stay fast as tables grow.
Jobs that rebuild history after a logic change, written to run in slices and resume after a failure.
Feature tables computed at scale for training and scoring, from the same code for both.
Checks inside the pipeline that stop bad data before it reaches a report.
Schedules and dependencies in Apache Airflow or the provider's own scheduler, with retries and alerts set per job.
Compute spend
Pipelines that grew with the data until they overran their schedule.
Jobs moving from a self-managed cluster to a managed service, Kubernetes or a lakehouse platform.
Big data Spark for startups usually starts with a few jobs that replace scripts which no longer finish on one machine.
Clusters on Spark 2 or early Spark 3 releases that need a planned path to a maintained version.
Notebook code turned into tested, scheduled jobs.
References
Behaviors and policies from the Apache Spark project documentation that shape how we tune, test and upgrade jobs.
Adaptive query execution in Spark SQL uses runtime statistics to choose the query plan, has been on by default since Apache Spark 3.2.0, and can split skewed tasks in sort-merge joins into roughly evenly sized tasks.
Source: Spark SQL performance tuning guideThe Apache Spark 4.0.0 release notes make Scala 2.13 and JDK 17 the defaults, drop Scala 2.12 and JDK 8 and 11, and turn on ANSI SQL mode by default.
Source: Apache Spark 4.0.0 release notesOn Kubernetes, spark-submit creates the Spark driver in a pod, the driver starts executors in their own pods, and the executor pods are cleaned up when the application completes.
Source: Apache Spark documentation: running Spark on KubernetesPySpark includes testing utilities such as assertDataFrameEqual and assertSchemaEqual, which the PySpark documentation says work with any test framework or CI test pipeline.
Source: PySpark documentation: testing PySparkThe Apache Spark versioning policy plans a release every three months, with six months of maintenance for major and feature releases and 18 months for the LTS release, the last feature release in each major line.
Source: Apache Spark versioning policy
