Challenge
Jobs read HDFS paths written into code and configuration.
Agnotic approach
We scan code and Oozie definitions for paths and replace them with catalog table names before the move.
Hadoop Migration
Hadoop migration services
Your Hadoop cluster still runs the jobs, but the hardware, the support contract and the people who know it are all running out. Agnotic Technologies moves the data, tables, jobs and access rules to a cloud lakehouse, and proves the results match before the cluster is switched off.
HDFS to object storageHive to Iceberg or DeltaJobs and schedulesReconciled results
Trusted by global innovators
Hadoop migration
What is a Hadoop migration?
A Hadoop migration moves the data, tables, jobs, schedules and security rules of an on-premises Hadoop cluster to cloud object storage and managed engines, then retires the cluster once results match. Agnotic Technologies plans and runs the move for the client, from HDFS and Hive to Spark jobs, Oozie workflows and Ranger policies.
We are often asked for Hadoop implementation. New clusters are rare now; the work buyers bring us is moving a running cluster to an open table format such as Apache Iceberg or Delta Lake on AWS, Microsoft Azure or Google Cloud.
Copying files is the easy part. The hard part is the jobs, schedules and access rules that grew around the cluster, and proving the new platform gives the same answers before anyone switches off the old one.
Mapping table
Where does each Hadoop component go?
The mapping we fill in during discovery. The target depends on your cloud and the engines your team can maintain.
| Hadoop component | Target option | Main migration risk |
|---|---|---|
| HDFS | Cloud object storage: Amazon S3, Azure Data Lake Storage or Google Cloud Storage | Large numbers of small files, which query engines handle poorly; compact them during the move |
| Hive tables and metastore | Apache Iceberg or Delta Lake tables in a managed catalog | Formats other than Avro, Parquet or ORC, and bucketed tables, which in-place migration does not carry over |
| MapReduce jobs | Rewritten as Spark or SQL jobs | Logic hidden in custom Java mappers and reducers with no tests |
| Spark jobs | Managed Spark: Amazon EMR, Azure Databricks or Google Cloud Dataproc | Spark version jumps, and HDFS paths written into job code |
| Oozie workflows | An orchestrator such as Apache Airflow | Coordinators with data-availability triggers and decision nodes that need redesign, not translation |
| HBase | A managed wide-column store, or HBase on a managed service | Row key design and read latency that applications depend on |
| Kerberos and Ranger | Cloud IAM, catalog permissions and table-level access controls | Column masks and row filters with no one-to-one equivalent |
We name target services as options for your cloud, not as recommendations of one vendor over another.
Migration order
In what order does a Hadoop cluster move?
Six steps, run one business domain at a time.
Inventory the cluster
Tables, file counts and sizes, jobs, schedules, users and Ranger policies, read from the metastore, YARN history and Oozie.
Choose the target per workload
Table format, catalog and engine for each workload, recorded in the mapping table and agreed with your team.
Move one domain end to end
Data, tables, jobs, schedule and access for one business domain, so the pattern is proven before it repeats.
Run in parallel and reconcile
Old and new pipelines run on the same inputs, and row counts, checksums and key aggregates are compared table by table.
Switch consumers
Reports, notebooks and downstream systems point to the new tables one group at a time, with a way back.
Decommission
Final copies, archived logs and a sign-off per domain before the cluster nodes are turned off.
Spark vs Hadoop MapReduce
Should MapReduce jobs become Spark code or SQL?
Spark vs Hadoop MapReduce matters most during a rewrite: Spark can keep intermediate data in memory across stages, while MapReduce writes map output to disk before reducers read it, and chained jobs pass data through HDFS. Agnotic Technologies turns simple jobs into SQL and complex ones into Spark DataFrame code, then reconciles every rewritten job's output.
| Job type | Rewrite as | Why | How outputs are reconciled |
|---|---|---|---|
| Simple aggregation or filter | SQL on the new engine | Readable by analysts and quick to review | Row counts and totals per partition against the MapReduce output |
| Multi-step joins chained across several jobs | One Spark SQL or DataFrame job | One job replaces the chain and the intermediate files written between jobs | The final table compared by key, with spot checks of the old intermediate outputs |
| Custom Java mapper and reducer logic | Spark DataFrame code, with the rules ported and unit tested | Business rules hidden in Java need tests before they move | Fixed input files run through old and new code and compared row by row |
| Iterative or machine learning jobs | Spark, caching the working dataset between passes | Each MapReduce pass rereads its input; Spark reuses the cached data | Scores or model outputs compared within a tolerance agreed with the job owner |
| Rarely run job nobody owns | Keep it wrapped as it is, or retire it | A rewrite costs more than the job is worth until an owner confirms it is needed | Job history checked for real use before it is retired |
The Hadoop MapReduce tutorial describes map output being sorted, spilled to disk and merged into one file before reduce tasks fetch it. No rewritten job goes live until its output matches the MapReduce run on the same input.
Why are teams leaving Hadoop now?
Three pressures usually arrive together: support for the distribution is ending or getting more expensive, the hardware is due for refresh, and the people who know the cluster are moving on.
The ecosystem has moved too. Apache Oozie, the scheduler behind many clusters, was retired in 2025 and moved to the Apache Attic, so it no longer gets releases.
Open table formats now give cloud storage the table features teams used Hive for, and let several engines read the same tables.
Failure points
Where do Hadoop migrations go wrong?
Problems we plan for on every cluster move.
Challenge
Small files make the new tables slow and costly to query.
Agnotic approach
Compaction runs as part of the move, and write jobs change so they stop creating small files.
Challenge
Results differ slightly and nobody can say why.
Agnotic approach
Reconciliation compares counts, checksums and aggregates per table, and every difference is explained before sign-off.
Challenge
Ranger column masks disappear on the new platform.
Agnotic approach
Each Ranger policy is mapped to a target control and tested by querying as each role.
Challenge
Everything is migrated first and tested last.
Agnotic approach
One domain goes end to end first, access and schedules included, so problems show up early and small.
Challenge
A Hive table is migrated in place and the original is lost.
Agnotic approach
We test on a snapshot of each table first, which leaves the source untouched, and migrate in place only after checks pass.
Security
How do Kerberos and Ranger policies move?
Access rules move with the data, and are tested before any user switches.
- Every Ranger policy is exported and mapped to a cloud IAM role, catalog grant or table permission
- Service identities replace Kerberos keytabs, with secrets held in the cloud provider's secret store
- Column masks and row filters are rebuilt as views or catalog policies, then tested per role
- Audit logs from the new platform go where your security team reviews logs today
- Storage encryption keys are managed by you, in your own cloud account
- Access is tested by querying as each role before any user moves
Who it is for
Teams that bring us a Hadoop migration
Teams with a cluster that still matters but cannot stay where it is.
Enterprises with an ageing cluster
A distribution near end of support, on hardware due for refresh.
Data teams short of Hadoop skills
A cluster run by a shrinking team, moving to services more engineers know.
Organizations consolidating platforms
Hadoop workloads joining an existing cloud warehouse or lakehouse.
Teams with Oozie-heavy pipelines
Many scheduled workflows that need a supported orchestrator.
Regulated data owners
Clusters holding sensitive data, where access rules must move intact and be proven.
Hadoop implementation for startups
Young companies asked to run Hadoop, who usually need a managed lakehouse instead; we scope that honestly.
Scope and cost
What drives the cost of a Hadoop migration?
We do not publish prices. These are the drivers we size during discovery.
Data volume and file counts
How much data moves, and how many small files need compaction.
Jobs and languages
How many Spark, MapReduce, Hive and Pig jobs exist, and how much custom code they hold.
Schedules
How many Oozie workflows and coordinators need redesign.
Security rules
Ranger policies, masks and row filters to map and test.
Target platform
Cloud, table format and engines, and whether the landing zone exists.
Parallel run
How long old and new run side by side before each domain switches.
Formats and engines
Open formats and tools we migrate to
Open standards keep your data readable by more than one engine.
Apache IcebergDelta LakeApache Parquet and ORCApache SparkApache AirflowCloud IAM
Facts and sources
Facts behind a Hadoop migration
Project documentation that shapes the migration plan, each linked to its primary source.
The HDFS NameNode keeps an image of the entire file system namespace and block map in memory, and a typical HDFS block size is 128 MB, which is why very many small files strain a cluster.
Source: Apache Hadoop documentation: HDFS ArchitectureApache Iceberg's snapshot procedure creates a temporary Iceberg copy of a Hive table that reads the original data files without changing the source table, so a migration can be tested first.
Source: Apache Iceberg documentation: Spark procedures, table migrationIceberg's in-place migrate procedure supports Avro, Parquet and ORC data files and fails on bucketed tables, because bucketing is not preserved.
Source: Apache Iceberg documentation: Spark procedures, migrateApache Oozie, the workflow scheduler for Hadoop jobs, was retired in February 2025 and its move to the Apache Attic was completed in April 2025.
Source: The Apache Attic: Apache OozieApache Spark can persist a dataset in memory across operations: each node keeps the partitions it computes and reuses them in later actions, which the Spark documentation calls a key tool for iterative algorithms.
Source: Apache Spark, RDD Programming Guide
Frequently Asked Questions
Rarely, because a new on-premises cluster is seldom the best answer today. When a client asks for a Hadoop implementation, we look at the workload and usually propose cloud object storage with an open table format and managed engines instead.
We do not publish prices. Cost depends on data volume and file counts, the number and type of jobs, Oozie workflows, security rules, the target platform and the length of the parallel run. You get a written proposal after discovery.
Often, yes. Apache Iceberg can register existing Parquet, ORC or Avro files as an Iceberg table, and Delta Lake can convert Parquet tables in place. Bucketed tables and other formats need a rewrite.
Both are open table formats. The choice usually follows the engines and catalog you plan to use and what your team already knows. We set out the trade-offs for your workloads during discovery rather than starting from a preference.
Old and new pipelines run on the same inputs, and we compare row counts, checksums and key aggregates table by table. Every difference is explained and signed off before consumers switch.
Yes, to whichever cloud you run or choose. The mapping table is the same; the target services differ.
AI tools may help translate old job code or Oozie definitions, but each converted job is reviewed and must pass reconciliation against the original output before it replaces anything.