Data and Pipeline Transfer from On-Premise to GCP
For a long time, there has been a recurring demand for a detailed understanding of networking and network architecture for data transfer from on-premises environments to Google Cloud Platform (GCP). In this series, I will explain the concepts from a foundational level, covering real-world enterprise-grade networking design, connectivity patterns, and implementation depth to provide a practical and in-depth technical understanding.
NishantFollow
GCP Data Architect
45 Minutes min•May 24, 2026
Views - 40
How the Data Transfer happen - “Start”
Requirement
We have pipelines in Abnitio and Terradata on prem for Data Warehouse. There is a strategic move to migrate the pipelines and data in Google Cloud Platform.
- Total Volume of data = 500TB
- Total Batch Pipelines to migrate = #400
- Total Near Real Time Pipelines = #65
- Total Real Time Pipelines to migrate = #135
- Total Machine Learning OR AI Models = #150
Threshold = Every nano seconds
Throughput = ~100GB/Sec
Retention = 7 Days raw logs (default)
Refresh Frequency = Every Seconds
Analytics Type = Aggregations, Counts, Metrics etc.
Latency = Less than 5 secs
Every month Costing (on-premise) = $65000
Type of Pipelines = ETL and ELT Both
Phase-1 : Discovery
Phase-2 : Cost Assessment
Phase-3 : Impact Assessment
Phase-4 : High Level Design
Phase-5 : Low Level Design
(A) Network Design
(B) Pipeline Design
(C) Operation Mannual
Phase-6 : Learnings Or Challenges
Comments (0)
No comments yet.