November 15, 2023

Migrating BI Platforms Doesn’t Have to Be So Painful

‍

Making the move from one business intelligence (BI) platform to another can be an intimidating task, especially when it sounds like a complex, tech-heavy procedure. But here's the thing: It doesn’t have to be. Migrating between BI platforms is less about the intricate details of the technology and more about organizing and liaising with the right individuals.

We’ll break down this migration into straightforward steps, making it less daunting than it appears at first glance. Let’s walk through how to turn a tech problem into a people problem—and a very solvable one at that.

Step 1: Pre-Migration Audit and Resource Assessment

The reality is that sometimes people problems are more difficult to solve than actual technology problems. That means you need to start by identifying an internal champion at your company who can help with monitoring adoption during and after the migration. Then follow these steps:

Defining Key Performance Indicators (KPIs) for Success

Define Key Performance Indicators (KPIs) that will help evaluate the success of the migration process. Having well-defined KPIs not only ensures that the migration is on the right track but also helps in identifying areas that require attention. In our experience, data accuracy, user satisfaction, and–most importantly–adoption rate, are all excellent for tracking and evaluating the success of the migration.

Estimating Migration Costs and Budgeting

Before you get started, you need to figure out what the cost of your human capital will be. If you are a large enterprise, this will mean having quarterly goals and budget tied to the move. If you are a smaller, more agile organization, this will likely lengthen the time to delivery on some of your other efforts–as your team will need to focus on the migration. Migration cost estimation is fundamental to ensure that adequate resources are allocated—you’ve got to know what you are getting into. Your estimation should include costs related to technology, labor, training, and any unexpected costs that might arise during the process—buffer at least 10%, but ideally 15%. This buffer will act as contingency if there any unforeseen challenges. This cost estimation will serve as a financial blueprint, guiding the allocation and management of resources throughout the migration process. To estimate the total cost, you need to understand both existing data pipelines, transformations, and reporting/dashboards.

Mapping Existing Data Pipelines for Migration

First things first: Make sure you understand the flow and structure of data within your existing BI platform. That means finding any extracts or static data sources that live in the application of the current BI provider.

Next, create a plan for transitioning this content either to the cloud data warehouse, or to keep in static. If you’re simply migrating data warehouses then that's easy: you can plan a straightforward migration to the new data warehouse. But if you have data that lives outside the warehouse, you need to plan where those resources should live.

For example, some platforms allow you to connect to a data warehouse and a spreadsheet at the same time. So in your new environment, is it ideal for that spreadsheet to continue as a static asset, or to be migrated to the warehouse? You’ll have to decide based on how frequently that data changes, and how it could (or couldn’t) flow into a bigger system.

Next, handling CSV files in migration can be straightforward yet critical. Some BI platforms allow direct ingestion of CSV files, a simple approach that may face challenges in data governance and scalability as data volumes grow. A more structured alternative is migrating CSV data into a data warehouse before connecting to the BI platform, which aids in better data organization and leveraging the BI tool's full analytic power.

Regardless of the method chosen, maintaining data governance practices is crucial to ensure data accuracy, consistency, and security. Automating the migration of CSV data with the right tools can save time aiding in a smooth transition. Performance considerations are also vital; migrating CSV data to a more efficient data storage solution like a data warehouse can significantly improve performance and scalability.

If you’re paying for a separate data prep tool, do some analysis to see if you can remove the need for that platform altogether. Use data lineage on your data sources to gain a comprehensive visual representation of the relationships between different data elements. If you can consolidate your transformations into either your warehousing platform or your BI tool, you will save big on platform costs and streamline the skillsets that your team needs to be successful. Check out Sigma’s integrations here.

Cataloging Reporting Assets for Migration

Identifying potential technical challenges at the outset is crucial for a smooth migration. Challenges might include data compatibility issues, system integration hurdles, or even discrepancies in data security protocols between the old and new platforms. Having a comprehensive understanding of these challenges and a well-thought-out plan to address them will significantly reduce the risk of unforeseen issues disrupting the migration process.

First, list out your reporting assets by the recency and volume of access. Plan to migrate any assets that have been used within a certain time threshold, say, the last three months. Contact those content owners and key stakeholders to confirm the need to migrate those assets in their current state.

For anything older or less popular than your threshold, create an offboarding plan with the owner of the content.

Next, work with your Champion and main report creators to classify the assets you plan to migrate as High/Medium/Low based on technical complexity and the effort it will take to fully migrate it. For instance, Low effort might mean less than an hour, whereas High is more than a day. The typical distribution might look like 5%/30%/65% (High/Med/Low).

Planning for Security, Access, and Organization

The last part of “taking stock” is the most important. Switching to a new BI platform makes security essential. Assess your security approach by making sure your new platform checks all of the biggest boxes of security, and figure out which approaches are best based on your organization’s IT infrastructure. Sigma’s security options include oAuth, Single Sign On (SSO,) and AWS Private Link. You also should also identify the workspace organization and team structure you’d like to use and make a plan for implementing it.

"Migration is a people exercise, and the technology is simply the tool to make it happen."

Sigma also offers features in the product to streamline data visibility and security. Consider how to leverage user attributes for dynamic role switching (if using RBAC), RLS, etc. Read more about the best practices for setting up security in Sigma to make sure you’re taking full advantage.

Step 2: Executing a Smooth BI Platform Migration

The proof of concept,
The migration,
Hypercare

Validating Migration with a Proof of Concept

Testing is a critical component of the migration process. Before full implementation, a proof-of-value or test migration should be conducted to identify any issues that might arise. This proof should be as thorough as possible, covering all aspects of the migration process. It will provide a clear understanding of any adjustments that need to be made before the actual migration, ensuring a smoother transition to the new BI platform.

The first asset we’ll often migrate is the one with the greatest impact on the business—which often has the most frequent and widespread use. Migrating this successfully will demonstrate that the most important use cases can be moved with ease, and will gain the trust of the main business stakeholders. This gets users onto the platform and interacting the way that you want, which helps build momentum for the rest of the process. (All of this is of course dependent on good change management, which is a key foundation to any large transition. More on change management below).

Prioritizing Content for an Efficient Migration

For the rest of the approved content, there are two approaches to the migration. You can either gather a team to help move it, or you can have teams move their own assets.

If you choose to have users migrate their own content, you need to have a strong communication plan. The communication plan should be broken down into tiers based on the importance of the content. Teams that have products with high usage should have a more robust communication process. The goal is to set the expectations for when content needs to move, upskill them on the new technologies, and hold them accountable for the actual content migration with regular check-ins.

The other option is to have a centralized team make the migration. When prioritizing the content for this migration, you need to focus on the content that is the most important, which is the content frequently used, recently used, or accessed on a regular basis by key stakeholders.

In terms of timeline, you will always be constrained by the budget you’ve set. This is why it's so important to migrate the most important content first. This way, the important analytics and reporting is ready when the cutover is made. And the less important analytics and reporting (which might not be be serviced during the migration) is the analytics and reporting that is archived–not the important stuff that was somehow saved for last.

In the final stage, set up a hypercare team that supports the long-tail of needs after the initial switch has been made.

Separately, be sure you have an onboarding plan that will help your content owners and business users feel successful on this new technology. Provide multiple paths for onboarding including low-touch support through videos, and high-touch support like office hours and 1-1s.

Step 3: Managing Stale Data During BI Migration

While it’s always tempting to call a big project “done,” don’t say it too soon.

We recommend implementing a Data and Content Lifecycle Management plan (DCLM). DCLM involves defining policies on how data, dashboards, and reports should be handled, archived, and deleted, ensuring that these assets are managed systematically throughout their lifecycle. Applying DCLM principles during the migration will aid in deciding which data, dashboards, and reports should be migrated, archived, or purged, hence reducing the complexity and cost of migration.

Utilizing specific tools or technologies for archiving and retrieving stale data, outdated dashboards, and unused reports can significantly streamline the process. Tools like data archiving, content management, and data warehousing solutions can be employed to ensure that stale or outdated assets are stored securely and can be efficiently retrieved when needed. It's also beneficial to consider technologies that ensure the integrity and accuracy of the archived data and content, making the retrieval and re-activation process smoother.

Stale data can also unnecessarily inflate migration costs and complexity if not handled appropriately. For data deemed obsolete or less relevant, consider archiving it in a cost-effective storage solution rather than migrating it to the new platform. This approach keeps it accessible for potential future needs without cluttering the new BI environment.

However, always ensure that the storage medium is secure, compliant with industry regulations, and can efficiently re-activate the data if required. If a reactivation need arises, the data should be retrieved from the archival storage and subjected to validation and cleansing processes to ensure its accuracy and integrity. This might involve cross-referencing with current datasets, rectifying any inconsistencies, and formatting it to be compatible with the new BI platform's standards.

Step 4: Overcoming Common Pitfalls in BI Migration

Establishing feedback loops with stakeholders is crucial for continuous improvement during the migration process. Regular feedback sessions will provide insights into how the migration is perceived and where adjustments are necessary. It also creates a channel for addressing concerns and suggestions, making the process more collaborative and adaptive to the needs and expectations of all involved parties.

Monitoring and evaluating the migration against the set KPIs is fundamental for ensuring that the process is on track to achieve the desired outcomes. Utilizing monitoring tools and conducting regular evaluations will provide a clear picture of the migration's performance, enabling timely adjustments and ensuring that the goals are met efficiently.

Proper change management during migration is also pivotal to ensuring smooth transitions and user adoption. It involves clear communication of the migration's goals, timeline, and potential disruptions, along with providing adequate training and resources for users to adapt to the new environment. Neglecting this aspect can lead to confusion, resistance, and inefficiencies post-migration, turning the potential boon of a new system into a pitfall of lost productivity and disillusioned stakeholders. While the technical aspects of migration are undeniably crucial, the human element, if mishandled, can become the Achilles' heel of the entire process.

Finally, no matter when and how you do it, data migration will take longer than you think. Make sure to free up the capacity of your people to fully undertake a migration. Also make sure your execs have realistic timing and expectation. Finally, remember that no technology can automate this—on any platform. Migration is a people exercise, and the technology is simply the tool to make it happen.

FAQs

What are the Key Performance Indicators (KPIs) crucial for a successful BI migration?

Identifying and defining Key Performance Indicators (KPIs) like data accuracy, user satisfaction, and adoption rate are instrumental in evaluating the success of a BI migration process.

How can I estimate the costs associated with migrating to a new BI platform?

Assessing the costs related to technology, labor, training, and unforeseen circumstances is pivotal. Buffering at least 10-15% for unexpected costs and understanding existing data pipelines, transformations, and reporting structures are key steps in estimating migration costs.

What is the significance of mapping existing data pipelines during a BI migration?

Understanding the flow and structure of data within your existing BI platform is essential for a seamless migration. This includes identifying static data sources and planning their transition to cloud data warehouses or keeping them static based on your organization's needs.

How do I ensure security and organization while migrating to a new BI platform?

Assessing your security approach, making sure your new platform complies with industry security standards, and identifying the workspace organization and team structure are critical steps to ensure security and organization during the migration.

What steps are involved in executing a smooth BI platform migration?

Executing a smooth migration involves validating the migration with a proof of concept, prioritizing content for efficient migration, and ensuring a well-structured plan for managing stale data during the migration.

How can Data and Content Lifecycle Management (DCLM) aid in BI migration?

Implementing a DCLM plan helps in managing data, dashboards, and reports systematically throughout the migration, aiding in deciding which assets should be migrated, archived, or purged, hence reducing the complexity and cost of migration.

Get started in Sigma with these three use cases, and check out our Input Tables hands-on lab here to learn how to write back data to the warehouse.