Migration becomes difficult when every workload gets its own orchestration pattern, deployment process, and transformation logic. Capacity also matters: one Event Hubs throughput unit supports up to 1 MB per second or 1,000 ingress events per second, according to Microsoft’s Event Hubs limits. This guide covers the architecture, migration decisions, delivery patterns, and cutover controls we use to keep complexity contained. Last verified: August 23, 2026.
We build Azure Data Pipelines around one control plane that coordinates ingestion, transformation, validation, release, and recovery across nightly batch and continuous streams. We keep stable SSIS workloads running through Azure-SSIS Integration Runtime first, then modernize selectively with reusable, metadata-driven components, source control, automated tests, and staged cutovers.
How Should We Choose the Architecture for Azure Data Pipelines?
We start by choosing one primary control plane for scheduling, dependencies, retry policy, deployment, and operational visibility. That does not require one compute engine for every job. It means batch processes, package execution, and streaming services follow one set of engineering and release rules. Azure documentation treats a pipeline as a logical grouping of activities and notes a soft limit of 120 activities per pipeline, a useful reminder to compose workflows rather than create one giant control flow.
For a mixed estate, we separate four concerns: ingestion, transformation, orchestration, and serving. The control plane coordinates them. A stream processor remains continuously active for real-time events, while the orchestration layer deploys it, monitors its health, and coordinates downstream validation or backfill.
| Tool Category | Orchestration | Streaming | Low-Code Transformation | CI/CD | Governance | Operational Burden | Best Fit |
|---|---|---|---|---|---|---|---|
| Azure Data Factory | 5 | 2 | 4 | 5 | 3 | 4 | Hybrid Azure estates and SSIS transition |
| Fabric Data Factory | 4 | 3 | 5 | 4 | 4 | 5 | Fabric-centered low-code integration |
| Managed Spark Engineering Platform | 4 | 5 | 2 | 5 | 5 | 3 | Code-first batch, CDC, and streaming |
| Stream Analytics | 1 | 5 | 3 | 4 | 3 | 5 | Managed event-stream transformations |
| Event Services | 1 | 5 | 1 | 3 | 3 | 4 | Durable event ingestion and buffering |
These are fit scores, not performance benchmarks. A score of five means the category is a strong native fit, while a lower score signals that another service should own that responsibility. We use the matrix to prevent teams from confusing an event broker with an orchestrator or a low-code transformation tool with a full streaming runtime. For a wider platform design, see our unified analytics platform architecture.
How Should We Assess and Classify Legacy ETL?
Before moving a package, we inventory jobs, owners, schedules, sources, targets, dependencies, recovery steps, data-quality controls, and downstream consumers. We also record custom drivers, network paths, secrets, peak concurrency, and the business impact of delayed data. This inventory turns migration from a technology project into a series of defensible operating decisions.
The integration runtime choice belongs in that assessment. A self-hosted runtime can reach on-premises sources and supports custom components, while Azure-SSIS Integration Runtime is purpose-built to execute SSIS packages in managed Azure compute. We validate networking and package behavior before promising a lift-and-shift path.
| Decision | Choose It When | Implementation | Gate Before Progression |
|---|---|---|---|
| Retain | The job is stable and close to retirement | Keep it where it is temporarily | Named retirement date and accountable owner |
| Wrap | Existing package logic works and needs cloud scheduling | Run packages on Azure-SSIS Integration Runtime | Functional, recovery, and network tests pass |
| Convert | Logic repeats across projects or sources | Rebuild as metadata-driven templates | Reconciliation and reuse tests pass |
| Rebuild | Dependencies or design prevent reliable operation | Create a new governed workflow | Design, test suite, and backfill plan approved |
When Should We Retain a Pipeline?
We retain only when a workload has a defined sunset date, clear support ownership, and no practical benefit from moving it first. Retention is a sequencing choice, not permission to ignore risk.
When Should We Wrap an SSIS Package?
We wrap packages when the business needs cloud scheduling, monitoring, and connectivity without immediately rewriting proven logic. Azure-SSIS Integration Runtime can scale by node size and node count, but it is dedicated compute, so we schedule it around real execution windows instead of leaving it active unnecessarily.
When Should We Convert Repeated Logic?
We convert transformations that appear in multiple projects, especially repeated ingestion, audit, retry, schema, and watermark patterns. Metadata should hold source, target, load mode, and transformation selections, while the tested template holds the implementation. Our legacy ETL modernization resources can help teams identify the components worth standardizing.
When Should We Rebuild a Pipeline?
We rebuild when hard-coded credentials, unsupported dependencies, brittle sequencing, poor restartability, or missing quality controls make a compatibility layer expensive to operate. Teams should approve the target design, test suite, data backfill plan, and ownership model before replacing a workload.
How Do Azure Data Pipelines Coordinate Batch and Streaming?
A practical architecture has two workload lanes and one engineering model. The batch lane extracts scheduled data, lands it in durable storage, transforms it, validates it, and publishes it to warehouse or lakehouse consumers. The streaming lane ingests events, processes them continuously, writes durable outputs, and exposes health signals such as lag and throughput.
The same control plane should own deployment, secrets, alert routing, release approval, lineage, and reconciliation across both lanes. We do not schedule a streaming processor every few minutes as if it were a nightly copy job. Instead, we run it continuously, monitor capacity and backlog, and use batch processing for backfills, late data repair, and full reconciliation.
The operating model converges at storage, quality checks, governance, and release discipline. We keep raw event data long enough for replay, make batch loads idempotent, and apply consistent checks before downstream data becomes trusted. That is how we avoid maintaining separate teams and separate release processes for “real time” and “overnight” work.
How Do We Reuse and Ship Pipeline Components?
Duplicated work usually starts when every project hard-codes its own connector settings, environment values, transformation logic, and release process. We fix that by treating pipelines as versioned engineering assets, not portal-only configurations. The reusable unit is a template plus metadata, test coverage, documentation, and an owner.
How Should Repositories and Branches Work?
We keep development work in source control, isolate changes in feature branches, and use pull requests before merging into the shared branch. Test and production environments receive approved releases through CI/CD, rather than direct edits.
What Should We Parameterize?
We parameterize environment endpoints, compute choices, schedules, secrets references, private connectivity, and source-specific settings. We do not parameterize business logic so heavily that a template becomes impossible to understand. The goal is a stable pipeline skeleton with controlled metadata variation.
What Must Automated Tests Check?
We test artifact validity, connection behavior, schema expectations, row counts, checksums, idempotency, alert paths, and recovery from a deliberate failure. Before enabling triggers, we run post-deployment checks against the target environment and record the release that made the change.
How Should We Release and Roll Back?
We use an immutable release artifact, environment-specific configuration, approval gates, trigger management, and a documented rollback to the last known-good version. That gives operators a safe response when a schema change, access policy, or transformation rule behaves differently in production. Our Azure pipeline modernization resources expand on this delivery discipline.
How Should We Phase Migration and Cutover?
We migrate in bounded domains, not as a single program-wide switch. Coexistence is useful when it has a purpose: compare outputs, measure timing, prove recovery, and expose undocumented dependencies. We set explicit gates for each stage so teams know whether to proceed, roll back, or keep the old path longer.
Use this eight-step checklist:
-
Inventory jobs, dependencies, schedules, sources, targets, and recovery requirements.
-
Establish identity, secrets, connectivity, monitoring, lineage, and cost tags.
-
Classify every workload as retain, wrap, convert, or rebuild.
-
Build shared templates and the Azure DevOps delivery foundation.
-
Migrate a low-risk pilot and run it beside the existing workload.
-
Reconcile row counts, checksums, business totals, timing, and failure recovery.
-
Cut over by domain with a named owner, rollback criteria, and support coverage.
-
Decommission only after an observation period and documented dependency sign-off.
We treat reconciliation as evidence, not a ceremonial sign-off. Lineage helps establish what moved through supported paths, but it does not replace source-to-target quality checks or business validation. Cost also belongs at each gate: compatibility compute may be the right short-term decision, but it should have an owner, budget visibility, and an exit path. Teams coordinating the target storage layer can also review our centralized analytics and lakehouse resources.
Build Azure Data Pipelines with Vision Board
At Vision Board, we help Azure data engineering teams turn this operating model into work their own engineers can run. We use the migration inventory to decide what deserves a short-term compatibility path, what should become a reusable component, and what should be retired. Then we work through architecture reviews, pipeline conventions, source-control practices, testing, and delivery gates with the people who will support production.
That focus matters when workloads span on-premises dependencies, nightly warehouse loads, and continuous event data. Our goal is not to force a one-time rewrite. It is to give teams a practical path to reliable, governed Azure data engineering, with the skills to improve pipelines after the migration.
We also help establish shared language between engineers, analysts, and delivery leaders, so every release has a clear owner, measurable validation, and a workable rollback decision. Start your team’s next learning or modernization plan with Vision Board.
FAQs on Azure Data Pipelines
These questions address the implementation choices that most often slow migration teams. We use the answers to keep architecture, modernization, and operational requirements aligned.
Can We Move SSIS Packages Without Rewriting Them?
Yes. We can run compatible packages on the managed SSIS runtime, validate output and recovery, then modernize only workloads whose cost, reliability, or reuse needs justify change.
Can One Architecture Handle Batch and Streaming Workloads?
One control plane governs releases, dependencies, validation, and alerts, while continuous stream processing handles events and the batch lane handles scheduled backfills, checks, and loads safely.
How Do Metadata-Driven Pipelines Reduce Duplicate Work?
Metadata keeps source, target, load mode, and transformation choices outside pipeline code. We reuse tested templates, so changed rules are released once instead of rebuilt for projects.
