Combining Atlan, Snowflake, and Sigma to Democratize Access to Well-governed Data
At a Glance
- North, a leading payments solution provider processing over $100 billion in annual transactions, aimed to improve data discovery and governance atop their Snowflake- and Sigma-powered modern data stack
- Adopting Atlan as their active metadata platform, North has since achieved end-to-end visibility of 225,000 data assets, securing sensitive data, and increasing adoption of self-service analytics
- In less than a year, their team is projecting $1.4mm in annual efficiency gains and a 200% growth in Sigma adoption while realizing a 700% increase in tagged Snowflake assets which has contributed to mitigating millions of dollars in risk
For over 30 years, North has established itself as a leading payments solution provider, now processing over $100 billion in transactions every year. With a diverse portfolio of over 20 brands and subsidiaries, they support a broad range of payment needs, supporting brick-and-mortar and online stores through payment gateways, mobile solutions, and more. Committed to serving industries that span restaurants to government entities, North aims to be the easiest end-to-end payments provider to do business with.
With a vast data estate comprising 225,000 assets, totaling 41 terabytes on Snowflake, the responsibility of managing and governing North’s data belongs to Daniel Dowdy, Vice President of Data Analytics & Governance, serving for 13 years at the company. Joining Atlan at Snowflake Summit 2024, Daniel shared his team’s journey, building and governing a modern data stack that supports both internal data consumers and their flagship products.
What do 20 subsidiaries and brands, and 100,000+ merchants, mean for our team? It means we have a lot of data. We use that data to support some of our most popular products.”
Daniel Dowdy, Vice President, Data Analytics & Governance
North’s flagship offerings, Payanywhere and Payments Hub, equip merchants with anything from hardware and software, to secure online portals, simplifying the complexities of payments for merchants.
Payments Hub, a comprehensive payments and operations platform, serves as a command center for everything from transaction tracking and reporting to inventory management and fraud protection. To deliver critical, transaction-level reporting to their customers, North relies on Snowflake.
“We utilize Snowflake for both of these on the backend to provide near real-time reporting and transaction-level reporting to our customers,” Daniel shared. “Without Snowflake, this just wouldn’t be possible.”
Before Snowflake, Data Silos
Before Snowflake, North struggled with unchecked and unclear data. Simple tasks like pulling raw data or generating reports could take days or even weeks, hindered by ETL bottlenecks and data silos scattered across multiple servers and databases, or Excel files circulated across the organization.
Once a file goes out, that data is already stale. Not to mention that once that file leaves you, users can then modify it and make their own source of truth. So, you’re struggling with multiple sources of the truth.”
Daniel Dowdy, Vice President, Data Analytics & Governance
The lack of a single source of truth complicated efforts to drive trust in North’s data, and without clear lineage and governance, controlling that data once it left their hands proved difficult.
After Snowflake + Sigma, Accurate, Real-time Data
In 2019, North adopted Snowflake as their enterprise data lake, significantly improving their data science, analytics, and reporting capabilities. By shifting from outdated batch processing to real-time CDC solutions, their team traded cumbersome ETL workflows and embraced the efficiency of modern ELT processes.
“We went from a ‘Data Desert,’ where retrieving information took days, to an oasis where data was available in real time, and we could query it in seconds,” said Daniel.
While Snowflake represented a significant leap forward in the availability and reliability of data, Daniel’s team were still responsible for fielding simple requests for data access. To reduce this workload, and to make data even more accessible to a growing number of consumers, their team introduced Sigma, a business intelligence tool that empowers users to independently access, analyze, and visualize data, offering crucial information in a self-service manner.
With its familiar, Excel-like interface, business users quickly embraced Sigma, resulting in a skyrocketing adoption rate, with annual user growth now exceeding 200%.
Beginning with 10 users in 2019, we now have over 350 users in Sigma, with an annual growth rate of more than 200% year-over-year. Now, Sigma is our self-service analytics platform, and it’s taken a lot of burden off our data team.”
Daniel Dowdy, Vice President, Data Analytics & Governance
North has since engineered a modern data stack, setting up robust pipelines that seamlessly ingest and replicate data from diverse sources like Oracle, MySQL, Salesforce, and S3. This wealth of data is then channeled into powerful tools such as Qlik, Snowpipe, and Kafka, all converging within Snowflake.
As the volume and variety of data grew, so did the demand for access, shifting the challenge from mere data accessibility to the more complex issues of data governance and discovery.
“What became the problem was how to find the data, and how to govern it well,” Daniel shared. “Atlan became the central point addressing all of these problems.”
Unlocking Snowflake and Sigma with Atlan
Connecting to over 225,000 data assets within Snowflake and Sigma took just hours with Atlan, driving unprecedented visibility on a large scale.
“After partnering with Atlan, it took probably a couple of hours before we were integrated with Snowflake and Sigma, connecting to over 225,000 assets. We’re so happy with the results,” Daniel shared.
This near-instant connection transformed data discovery at North, once a chaotic experience of jumping through multiple silos, suddenly making it simple to navigate and understand their data estate.
Atlan started getting us this immediate value through data discovery. The way I keep thinking about it was the early days of the internet. You had access to this vast amount of information, but it was really hard to find. Then, search engines made it easy to run a search and find relevant content. For us, Atlan became Google for Data.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Moving beyond simple data discovery, Daniel’s team introduced data glossaries, definitions, and comprehensive terms, enabling deeper dives into data, and utilizing lineage to understand upstream and downstream dependencies with ease.
With this valuable context unlocked, their next step was to deliver it directly to business users, who primarily consume data within the Sigma, without switching to other tools.
“That’s where Atlan’s Chrome Extension became a game-changer,” Daniel shared.
Atlan’s Chrome Extension transformed data consumption within Sigma. With context about their data assets—dashboard descriptions, data definitions, tags, lineage, notifications, and upstream quality signals—now available directly in Sigma, users no longer had to jump between tools to accomplish their tasks.
Available in a sidebar, Atlan’s Chrome Extension made this information instantly accessible, without training or change management.
“The initiative required no training,” Daniel explained. “Because it’s a sidebar and intuitively shows this information, we would just add it, then users were automatically up and running, improving their efficiency, and immediately seeing the benefit.”
Issue Resolution for Engineers and Consumers, Alike
With context about their data assets made available in Sigma, Daniel and his team began work to improve the process of troubleshooting and communicating issues related to these assets. Upon receiving a ticket, North’s engineering team once spent significant time searching through Sigma to identify the assets referenced in the request. Without a direct link between tickets and these assets, resolution would often be delayed, and work would be duplicated.
“It was really inefficient,” Daniel shared. “Sometimes even the same engineers would be working on the same project, and they didn’t even know it because multiple tickets were submitted by multiple people.”
By integrating the Atlan Chrome Extension with Jira and Slack, North’s data consumers could raise issues directly in their tool of choice, with a hyperlink to specific assets included automatically, allowing Data Engineers to understand when and where issues occurred.
For data consumers in Sigma, this integration offered a comprehensive history of all Jira tickets and Slack conversations relating to specific data assets, and for engineers, a means of preventing duplicate tickets or multiple engineers working on the same issue, and time savings by better understanding similar, past issues.
Eliminating duplicate work, or eliminating your engineers working on the same project without knowing about it? These efficiency gains are huge, and the savings add up really quickly.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Introducing Snowflake Dynamic Data Masking
As a payments solution provider, North handles vast amounts of sensitive financial data. While improvements in data discovery and enrichment are crucial, Daniel’s team realized the value of these enhancements would be limited to a small number of trusted users without robust data governance and documentation securing these assets.
To secure sensitive data and implement stringent access policies, North’s data team chose to utilize Dynamic Data Masking with Snowflake, offering comprehensive column-level masking for sensitive data, and policy-based control for users.
In order to utilize Dynamic Data Masking, Daniel and his team began the process of identifying specific data assets that these policies would apply to. Given the scale and complexity of their data landscape, they recognized that a granular, asset-by-asset approach would be not only time-consuming, but potentially error-prone. Searching for a way to identify and secure these assets programmatically, Daniel turned to Atlan.
“How do you find and tag thousands of assets? You don’t Google it, you Atlan it. It’s really simple,” Daniel explained.
Governing Thousands of Assets, at the Click of a Button
Rather than painstakingly locating these assets, then tagging them one-by-one, their team utilized Atlan Playbooks, rules-based bulk automations that could identify thousands of assets based on selected criteria, then automatically apply actions, such as adding chosen tags and policies.
By the time I added a filter, Atlan scanned our 225,000 assets and almost immediately found 45 matches. As I changed the filters to add new ones, it constantly updated to show me how many assets meet that criteria. Hours of manual research and testing, before Atlan, was converted into a couple seconds.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Quickly identifying thousands of assets to better-secure, Daniel’s team used Playbooks to build workflows that would automatically apply the right tag to the right asset, then propagate that tag upstream, at the click of a button. Then, with Atlan’s Two-way Tag Sync, these tags would serve as the foundation for Dynamic Data Masking at North, automatically applying masking policies based on how and from whom data was queried.
By combining Atlan Playbooks, Two-way Tag Sync, and Automated Lineage with Snowflake Dynamic Data Masking, North successfully governed thousands of data assets in record time with a programmatic approach, avoiding the painstaking work of identifying, tagging, and recording policies one-by-one across a 225,000-asset landscape.
Without Atlan tracing upstream and downstream, helping us understand potential exposure and implement policies, this process would have been complicated and very manual. But with Atlan, you can not only trace data lineage and organize the complexity, but you can see popularity and usage metrics that help prioritize where efforts are needed the most, or where the biggest impact could be made.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Through this programmatic approach, Daniel’s team affected a 700% increase in tagged assets, significantly improving their security and compliance, and paving the way for a significant increase in the number of their colleagues trusted to use data.
“We’ve used Atlan to boost the speed and accuracy of our asset tagging by more than 700% since implementation,” Daniel shared. “With the ability to discover data faster, apply tags, and reverse sync that data back into Snowflake, we added more than 2,000 governance assets in under a year. That’s just the tip of the iceberg.”
Millions in Value, in Months
In less than one year of being equipped with Atlan, Daniel and his team transformed the way North uses and governs its data, unlocking the potential for millions in value through efficiency gains, cost reductions, and enhanced risk mitigation.
By thoughtfully converging crucial context about Snowflake and Sigma into a user-friendly knowledge base in Atlan, North’s technical and business users are projecting savings of $1.4 million per year in effort through accelerated data discovery and reduced effort understanding lineage.
“That’s $1.4 million of duplicate work, manual and tedious research. That’s time we can now spend on changing things, versus trying to navigate data like it’s the internet in 1997,” Daniel shared.
And with built-in insights in Atlan, near–instantly identifying unnecessary spend in their Snowflake environment, Daniel and his team estimate an annual cost savings of more than $20,000 by deprecating more than 10,000 idle assets and inefficient queries.
“That savings was basically handed to us on a silver platter with no extra work,” Daniel remarked.
Beyond cost savings and efficiency gains, and existential to a company that processes sensitive financial data, lies significant risk mitigation. By increasing their tagged assets by 700%, quickly and in bulk, North’s governance and security improved almost overnight, to the tune of millions of dollars in reduced risk.
We’ve been able to reduce our risk and exposure by millions and millions of dollars.”
Daniel Dowdy, Vice President, Data Analytics & Governance
A 200% Increase in Self-service Analytics, with Room to Grow
While these improvements are impressive, whether in data discovery, with context about their data assets supporting a 200% year-over-year increase in self-service analytics adoption, or in Data Governance, driving millions in risk mitigation, it is the combination of these programs that’s set to transform the way North uses data.
By affecting a leap forward in the way that North manages access control and ensures security across a massive data estate, the way Daniel’s team defines a trusted user of data is evolving and expanding significantly.
“Data Governance gave us the confidence to safely and securely share more data in Sigma,” Daniel explained.Now assured that the most sensitive data is secure, appropriately tagged, and governed by the right access policy, North can confidently offer data and self-service analytics capabilities to more business users than ever before, enhancing the already significant value they’ve yielded from combining Atlan, Snowflake, and Sigma.
We were in a state of near-paralysis trying to manually determine who should see what. Utilizing a combination of improved governance policies, tags, and dynamic data masking, we were able to use data with confidence. Due to the enhanced user experience, combined with the increased data availability, we expect to continue or exceed our self-service environment growth rates.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Mere years after adopting Snowflake as the foundation of their Modern Data Stack, and less than a year after adopting Atlan, North’s data estate has transformed from, as Daniel described, a “Data Desert” into an “Oasis.”
“Since the data never leaves our umbrella, there’s no more flat files that are shared,” Daniel explained. “There’s no more multiple versions of the truth, and our data has a level of confidence, resulting in faster and more informed decisions.”
Despite the incredible accomplishments of North’s data team, building self-service analytics and Data Governance capabilities from scratch in record time, Daniel believes there’s still so much more to accomplish.
In the months and years to come, their data team is eager to use Atlan AI to simplify complex queries into natural language, and accelerate the enrichment of their data assets. And by supporting a growing number of Data Stewards to independently share and document knowledge, a more informed and empowered community of business users is emerging.
Reflecting on their journey with Atlan, Daniel emphasized that beyond the right technology, team, and strategy lies a crucial element: The right partnership.
One of the most important things about Atlan has been the people. They’re data people that have worked in the industry, understand what you’re going through, and understand what you’re talking about from the moment you start the conversation.”
Daniel Dowdy, Vice President, Data Analytics & Governance
Photo by Jonas Leupe on Unsplash
