Picture this: your data embarking on an exciting journey, leaving behind its familiar on-premises home and venturing into the cloud. It’s like moving from a cozy house to a futuristic floating city. In this blog, we’re diving into the fascinating world of data migration – how data packs its virtual bags and takes the leap to the cloud. We’ll unravel the what, why, and how of this tech adventure, making data migration a breeze even if you’re not a tech wizard. Ready to hop on this cloud-bound ride? Let’s get started!
What is Data Migration ?
Data migration is like moving your digital stuff from one place to another, just like when you shift your belongings when you move to a new home. It’s about taking your important information, like files and records, and moving them from one computer system to another, often to the more modern and convenient ‘cloud’ storage.
Project 1 for a Data Engineer at Datartic
Congratulations to you getting hired as a Data engineer at Datartic. This will be your first project. We will be following Kanban style project management in this Data series. If you don’t know what that is you can check out my post : Mastering Projects with JIRA: Your Guide to Effective Project Management .
First Task : Data Migration from On-Prem SQL Server to Snowflake
TASK : You are in a small sized team given a project to migrate data from On-Prem SQL Server to Snowflake (Snowflake: Empowering Modern Tech Stacks with Cloud Data Warehousing). There is just 1 Database with ‘X’ no of Tables which are needed to be Migrated
There is a Team Lead , Senior Data Engineer and 2 other Engineers including you. You are to learn here so just hang tight and learn how this project is architectured and completed.
As mentioned before the Team will be following Kanban style for this project. We will be architecting the approach first to brainstorm ideas and approaches . So doing that lets say we come up with this based on the requirements :
1) Tech Stack : Azure , Snowflake , SQL Server and JIRA
2) Architecture : I am using Draw.io to currently draw this diagram.
Crafting architectures might sound like a time-consuming puzzle, but fear not – we’ve got your back. Now, let’s dive into the day with excitement and a clear understanding of what lies ahead. Time to unravel the mystery and move forward in style!
Every company has their own board with their own Columns defined in JIRA or any other project management software. Lets create a board for this project in Kanban style workflow !
After setting up everything… a basic board looks something like this
Think of these tickets as our problem-solving puzzle pieces. Each ticket is like breaking down a big challenge into manageable parts. Now, let’s roll up our sleeves and craft these tickets, capturing our data migration challenge piece by piece. As our team currently consists of just one person (yes, a one-person army for now!), we’ll take on the role of all the team members. Ready to dive in, assign tasks, and embark on this solo adventure of tackling the data migration puzzle? Let’s make it happen!
I have broken down these into 5 different tasks :
- Configure and setting up our access. This includes setting up Snowflake account, Azure Data Factory account and SQL Server.
- Move Data from SQL to ADLS2. As shown in our architecture diagram we need to figure out a way to move data into our Data Lake.
- Move Data from ADLS2 to SF . Now that we have our data in ADLS2, figure out a way to move data into Snowflake.
- Validations.
- Documentation
Whew, that was quite a journey! With our tasks neatly outlined, it’s time to wrap up Day 2. We’ve crafted a solid plan to guide us forward. Rest assured, the adventure continues. So, until we reconvene on Day 3, keep the enthusiasm alive and get ready for more data-driven excitement! See you soon!
Very Intuitive and Resourceful. Thanks for Sharing !