No silver bullets: Building the analytics flywheel

Investing in analytics engineering was a game-changer for our ability to deliver value as a data organization, but it didn’t happen overnight. In this talk, we’ll walk through our journey towards analytics engineering and how you can convince your organization to invest in it as well.

Follow along in the slides here.

Browse this talk’s Slack archives #

The day-of-talk conversation is archived here in dbt Community Slack.

Not a member of the dbt Community yet? You can join here to view the Coalesce chat archives.

Full transcript #

Julia Schottenstein: [00:00:00] Welcome everyone to your next session at Coalesce ‘No silver bullets: Building the analytics flywheel. I’m Julia Schottenstein, and I’m part of the product team at dbt Labs. And I’ll be your host for this. And this talk we’ll hear from Aula Education, data lead Kelly Burdine.

I love this talk so much because all software investments and more people on the data team will likely help improve a company’s data capabilities.

There’s no one panacea. Kelly’s going to talk about leaning into analytics, engineering, workflows, and how that was a big unlock and accomplishing her team’s goals. And actually we’re also joined by Erica Pullum and Lewis Davies who are senior analytics engineers on the Aula team. And we’ll be hanging out in Slack to help answer questions.

So if you’d like to contribute to the conversation, please join the dbt Slack channel #coalesce-analytics-flywheel, and Kelly, and her team will follow up there. So without further ado, Kelly over to you.

Kelly Burdine: Thanks, [00:01:00] Julia, I’m super excited to be here. As Julia said, I’m Kelly Burdine and I’m the data lead at Aula, and Aula is an ed tech startup that builds a learning engagement platform for higher education. I’ve been at Aula for just over two years, but I’ve worked in data analytics for more than 10 years. And I’ve had the opportunity to build out a data function from the ground up twice. I’m at two different organizations, including.

And what I’ve learned from my experiences at creating value for your organization by leveraging data is really challenging. There isn’t an easy fix or a silver bullet to quote my presentation title, to creating that value. But when I joined Aula two years ago, I was determined to do things differently and the results were maybe even a bit more surprising than I had initially.

So today I’m going to walk through how I created what I’m calling the analytics flywheel that helps us build momentum. So we can better tackle these challenges and give you some tips for incorporating this at your own organizations. Like Julia mentioned throughout this presentation, I’ll actually be [00:02:00] incorporating the perspectives of my entire team and they’ll be manning the Slack channel today to answer any of your questions.

But first we’re going to actually dive into my backstory, just a little bit. So, like I said, I’ve had the opportunity to be the first analytics hire at two different organizations now. And in my last company, I was a bit more naive on what it took to build out a really scalable and reliable data function. In fact, I thought that I had every piece of the puzzle I needed to be successful with really just a few simple tools.

I had a centralized cloud database. Ask the soft some software engineers to dump our various data sources in there. Using custom ETL pipelines, we set up a modern cloud BI tool and gave access to all of our data to our entire data organization. I’ve been created a couple of training classes to teach employees, had a query and visualize it, visualize our data, thinking a bit of data literacy was the last piece to achieving data democratization and self-service analysts.

So here I was thinking I had everything I needed to be successful, and I was going to deliver all these powerful [00:03:00] insights across the business to lower turn and increase conversion, improve the user experience, save money, etc. And for a short time it worked, people had access to data they’d never seen before I was analyzing trends and delivering insights to company leaders.

It felt awesome. However, as our company and our team and our data volume group, a lot of problems started to arise. And I’m guessing that a lot of you can probably relate to some of these problems. We would often have cases where data would suddenly go missing or it’d be incorrect. It was difficult to clean and analyze the data quick enough to impact decision-making in our fast moving and growing organization.

Metrics and company-wide dashboards would suddenly change. And the problems were often first noticed by our stakeholders and not the data team. And we wouldn’t even know why the numbers changed. Stakeholders were using different reports with different numbers, for the same metrics, [00:04:00] leaving everyone really confused.

And as with most data, it’s inherently messy and riddled with nuances. So everyone on my team was spending 90 to a 100% of their time just cleaning, prepping, transforming data. This means almost 0% of the time, but doing actual analysis and delivering insights to decision. So the results of all these problems is that most people in the organization didn’t trust our data.

They didn’t want to self-service in any way. And everybody needed help from a data analyst, but analysts weren’t available because they were completely bought down by just cleaning data and fixing problems. So then the data requests line starts to form and eventually people really just get sick of waiting in line.

And so they aren’t going to use data for decision-making. So then I came to this like really harsh reality that our team was more of a cost to our organization than an asset. And we just weren’t producing enough value. And unfortunately I see the situation all the time. These problems that I mentioned were [00:05:00] also experienced my by my other teammates at Aula and many other people that I’ve talked to at other organizations.

And this happens in different industries and different size companies. And even in different countries, we’re all seeing the same thing. And this happened a few years ago, but this is really still true today. Like earlier this year, the Harvard business review published a report saying virtually every Fortune 1000 company was investing in.

But only a quarter of them were actually thinking that they were actually data-driven. When I had this realization a few years ago, I really wanted to believe that there was that silver bullet solution. I tried adding more tools. I tried hiring more people and I was constantly iterating on our sprint process, but everything I did just felt like this uphill battle and I wasn’t achieving any results.

And so when I started analyzing these problems at a deeper level, I realized there’s this common thread. We have a ton of processes that were really manual and repetitive, which didn’t get better with [00:06:00] more data, more people or more tools. It was actually just making the problem more widespread. And these are the same problems that software engineering, best practices resolve, and we’re analytics engineering actually can.

And, at the time I knew about analytics engineering. I was part of the dbt Slack community, and I had even hired somebody with the title of analytics. But we weren’t really doing it. I realized that analytics engineering, it wasn’t just a title. It’s not just a suite of tools. It’s really more of a philosophy or practice that needed to be rooted in everything we did.

So as I had this realization, I started developing main vision for doing things differently. And coincidentally Aula approached me around the same time and asked me if I would come start and build out their data. So I pitched them my vision and high level strategy and they heard me. And so here was this excellent opportunity to really test my hypothesis of building out and running everything with an analytics [00:07:00] engineering mindset.

Plus, I had the added benefit of being able to iterate really quickly at a small company.

[00:07:08] Creating an analytics flywheel #

Kelly Burdine: And what I discovered was actually created this five wheel effect. So for those not familiar with a flywheel, it’s a mechanical device that takes some effort to put in motion. And initially it was really slow. And that motion is hard to detect, but once it creates this energy, it gains momentum, which allows it to continually spend without needing to put much additional effort into it.

And, so I done all this effort upfront and Al and all these small steps. Just building out and setting that foundation of infrastructure and best practices and processes that they don’t mix engineering and everything after that became so much easier, it didn’t feel like an uphill battle anymore.

And now we have all this momentum where we can just deliver on outcomes and it’s been. So comparing to what I’ve seen before and what we have at all and how, we have accurate and reliable reporting. Our ingestion pipelines are robust and timely when source data, business logic changes.

We [00:08:00] can handle those quickly and painlessly. And most importantly, though, we have far more time to spend doing analysis and deliver insights. And most of our decision makers across the company are making real efforts to using. And it really feels like we’re just getting started and the momentum is going to allow us to move even faster to do more incredible things.

So hopefully by now, some of you are wondering like what I actually did at Aula to create this flywheel and what you can do to create one of your own organization. So there were really three high-level processes that I used at Allah that I think could be implemented at any organization. So the first one is getting organizational.

[00:08:43] Get organizational buy-in #

Kelly Burdine: Now I mentioned analytics engineer, I think is it’s more than just a title or a person. It’s more than a set of tools. It’s really an entire practice. And to really bring analytics engineering into the core of everything you do, it’s going to require change and for some teams major change. So it’d be really difficult to [00:09:00] do this all by yourself.

So you’re going to want somebody. And at first it might start off as just one person. Maybe it’s the whole team, maybe it’s a wider organization, but you’re going to need some buy-in. So the first thing that I did at Aula was identify an advocate that could back me up and really support my initiative.

[00:09:18] Identify an advocate #

Kelly Burdine: So an advocate is someone with one or more of the attributes listed on this slide. So for me, this was our VP of product. He had previously seen DW really powerful. He was disappointed that our product team wasn’t making sense. Based off of data and they were working more off of output that outcomes.

He wanted to be measuring objectives and he wanted to be data-driven. He also, the bonus was he was part of the executive team. So he was able to help me convince others along the way to really support this initiative. So he was a great advocate. The next thing that I did and that, everyone should do is really present the benefits to your organization and to your team, whoever you need to convince to make this happen.

[00:09:57] Articulate the benfits to the organization #

Kelly Burdine: Okay, so this might be your line manager. Again, maybe it’s [00:10:00] your team, maybe it’s somebody else. But a mistake I made early in my career was framing the problem instead of the benefits. I actually remember in my last organization, going to our VP of engineering and telling him how difficult all of our analyst jobs were because our vent logging pipeline was a hot mess.

I showed a multiple examples of problems that we had encountered and told them how much time you’re spending a trusting all these data. But what I didn’t focus on was how having better event logging would have actually benefited our organization because we would have had all this additional insight into how our product was getting used and the solution would actually make it easier for engineers as well.

So I learned from those lessons, luckily, and it all out when I made my pitches for resources or other types of support to build that analytics engineering foundation, I really focused on how it would benefit. I certainly had plenty of examples of problems in my back pocket that I could use if I needed to reiterate something, but that’s not what I focused on.

I really focused [00:11:00] on, being able to leverage data for better decision making and self-service analytics. Those are some of the benefits that I really tried to clarify. Now, at this point, you haven’t gained much momentum in your flywheel. In fact, it probably still feels like you’re pushing that boulder uphill.

But you’re actually starting to inch forward and you’re putting that wheel in motion, which is super critical and to continue putting it in motion. The next thing you want to do is get a quick win. So find something small that you can do to quickly show some value. A common mistake is wanting to tackle a problem in its entirety with some ideal solution.

[00:11:38] Start small and show value #

Kelly Burdine: This could take months to really reap the benefits. And you’re still trying to get those around you on. You need that momentum. So this is a similar approach to the concept of building a minimal viable product, where you want to build the smallest thing to be able to quickly test your assumptions or in this case shown the value.

So maybe this is something like adding automated test to your most [00:12:00] problematic model. Maybe this is moving one of your transformations in diversion control, maybe it’s switching one data product to a new tool. Just find something small that you’re confident will show value, so you can continue to get more organizational buy-in and continue to push that flywheel forward.

Now, depending on the size of your organization or your position within the team, the process of pitching for support or asking for resources might not be worth it in the beginning. As Grace Hopper a very famed computer scientists once said it’s easier to ask forgiveness than it is to get permission. So maybe that small thing you build to show value is something you just tackle on the side and present back to the team.

Showing something that you’ve already tried can be really powerful. You still want an advocate to help support you. And again, you don’t want to go overboard here, but I’ve had reports do this over the years and show me some pretty amazing things that maybe we would have otherwise not priorities.[00:13:00]

[00:13:01] Build a roadmap #

Kelly Burdine: So now at this point you have the attention of some, or all your teammates and hopefully like a senior leader or someone else in the organization that can really help, get support. You’ve proven out the value on a small dose, but what’s next? How are you going to get there? Just if you’re pitching a new product on the shark tank, right?

You have to have a good plan for succeeding. You can’t just say you have to have a plan for how to get. You don’t need to plan out the next five years, but what does say, like the next six months look like collaborate together as a data team to build the roadmap so that everyone feels a sense of ownership.

We actually did this recently at outlet to plan out our next phase of analytics maturity. We started by identifying our biggest problems, brainstorm solutions together. We voted on the best options. We overlaid those on an impact matrix. And then we use that to prioritize and define. And then we present that back.

We presented that back to our company to really, again, [00:14:00] continue to get organizational buy-in to push that flywheel forward. So at this point in your process, you don’t actually have a lot to show for all the work that. But you’ve actually put on a lot of effort and you’ve moved your flywheel quite a bit.

And now you’re at that top where things are starting to get a little bit easier. So you really want to be able to start accumulating visible results so that you can gain that energy. All right. So the next thing that I focused on an on, and what I recommend that you do as well is to start implementing the essential practices of analytics engineering one piece at a time. So each of these should have a visible result in impact.

[00:14:44] Implement essential practices #

Kelly Burdine: And what do I mean by essential practices? All of the core processes and best practices and tools that really comprise analytics engineer, I’ve listed some of the most common ones here, but this list is definitely not exhaustive.

dbt Labs gave a couple talks this morning around this [00:15:00] topic. So I definitely encourage you to check those out if you’re newer to analytics engineering But this is where you can really build out that analytics, engineering foundation and start to deliver visible results so that flywheel can start moving a little bit faster.

Now at Aula, I had the benefit of starting from scratch. So I was able to set a lot of this up from the beginning, but what was tricky for me was making sure that I had the buy-in from leadership to take that extra time upfront to get it right, but also continuing to make sure it was delivering value along the way.

I couldn’t simply ignore the data needs of the business. It took a lot of effort to get organizational buy-in. So if you lose that, buy-in because you’re not delivering any value along the way. You lose your momentum. And it’s crucial to deliver value along the way, because you have to have that momentum to keep the flywheel going.

So for me, I focused on implementing a lot of engineering or [00:16:00] analytics, engineering, best practices within a small subset of data. If I tried doing it for all of our data, it would have taken a very long time. To get to the point where business users could even start using the data. In fact, they’re probably would still be working on it would not be here at Coalesce talking to you today.

So the way I did this and figured out what data set to work on first is that I gathered input from the leadership team. I chose like the top business question or a few questions that we wanted to answer. And for all of these questions were around users and whether or not they were taking certain actions within our cloud. There’s a few key options they were really concerned about. And so using that, I was able to create just like a handful of high quality data models using all the best practices that I knew to help answer those questions.

Then I trade several end-users on how to use those models. So they could, self-serve a lot of their. And I could go back to focusing on implementing more essential practices and more data. And this process worked really well and allowed me to quickly [00:17:00] accumulate visible results and gain momentum in my flywheel.

Now this process might not work as well. If you aren’t starting from scratch. So what can those data teams do? So my teammates and slack are ready to share some of their best stories for implementing essential practices on existing teams and organizations. But let’s take a look at just a couple of quick examples.

So let’s say that you want to get more collaboration in your code, okay? You could start by just adding a review step in your process that every deliverable has to go. This is especially helpful for those words, with a very decentralized data structure or a process that feels very siloed. And so honestly at first, this is going to feel like a time suck to your analysts and your engineers.

So you’re going to have to hold people accountable, so they stick to it. But what starts to happen is that you start catching problems during that review process that you would have otherwise missed. And this prevents rework, which takes more time than just doing a review. It also [00:18:00] prevents stakeholders losing trust when they stumble upon a problem that you missed.

And this is really crucial. Let’s another example is testing. Let’s say you don’t have a lot of testing in your scripts and everything. And so you want to implement more testing, right? And get testing everywhere. It’s going to take some time, but you could just have started off by having a hard rule.

Any model was script that gets touched or modified in any way. You don’t get to test attitude by chipping away at it like that. You end up having a fairly robust test coverage in just a few. Let’s say you have a lot of very large scripts or transformations or something like that. And you really want to get more modular with your code.

So things are easier to modify and test and everything else. So for this, you could really collaborate with your team by identifying say one particular script or model that you can break apart. We actually did this at my last organization where we had this crucial model, that powered 800%. And of course it had, it was like hundreds of lines of SQL.

And so nobody wanted to touch [00:19:00] it. It was really intimidating. You couldn’t even find where problems were. It was really difficult to work with. And so we started by just working together and identifying what’s the easiest way we can chunk it up into a few pieces. And then from there you take each of those pieces and you can chunk it up more.

And so we worked together that way to create these this is much more manageable set of transformation scripts. And then that process by collaborating together taught people, how do we do this? What is the right process for this? So that we can continue doing that? Now each essential practice that you implement probably doesn’t have a massive impact on your team by itself, but little by little, they combine to create some very meaningful impacts.

So at this point you’ve probably accumulated a lot of visible results and the team, and maybe even the wider organization is starting to gain new, positive energy from these results. And so this is where you can really start to gain more moments. And to do that, you really need to get that flavor, the spending by hiring the right people as you [00:20:00] grow your team and back fill your positions.

[00:20:02] Hire the right mindset #

Kelly Burdine: Now, I don’t mean to imply that you can’t hire anybody until your organization is bought in and you’ve implemented all the practices, right? Remember, this is a wheel, this is a very cyclical process. So it really could come at any time. At my last organization, I mentioned that I definitely tried throwing people at problems.

And sometimes we’d hire people that had experienced with the exact problems that we were, we were experiencing and they weren’t always brought in, they weren’t always bought in to the analytics engineering. So when we brought them in, they couldn’t meaningfully contribute to the momentum and struggled along with the rest of us and the existing problems we had, they couldn’t really solve them.

And so that it discovered that prioritizing hiring somebody with the right mindset over someone that has extensive experience in maybe your specific tool set or our stack or something like that can actually lead to more momentum. Now, I don’t mean to say that you can just hire somebody. Anybody. Obviously there is a minimum level of competency or technical [00:21:00] aptitude that’s needed for a position, but I found it much easier to teach somebody Python or how to use a different data warehouse or something like that than it is to convince them to adopt an analytics, engineering mindset.

And hiring right now is really hard. There just aren’t enough people out there with these skillsets. Although hopefully at the end of this week, the pool gets a lot bigger, but expect to have to actively recruit individuals, to join your team and be willing to live for people from other industries.

Other tech stacks, one of my colleagues who I hired hadn’t used many modern data tools before, including. But she believed in that analytics, engineering best practices and had gone above and beyond at our previous organization to implement many of these practices using other means and adding her mindset to the team, gave us a ton of momentum and helped us to continue to implement and enforce as essential practices.

And I believe there’s some other talks this week talking specifically about hiring. So I’ve [00:22:00] definitely checked those out. Now, once you get more people in your team, With this mindset, you’ll gain momentum quickly. You now have more people building that buy-in, you’re implementing better processes faster.

You can continue this process like a cycle to keep furthering your analytics, maturity, and spreading to maybe other data teams. If you’re a large company or other departments and priests, and you have this analytics flywheel that just spins and no longer takes these monumental efforts to tackle new data challenges in your company that deliver a ton of value.

So in conclusion, to increase the value of your data team it’s important to really build a solid analytics engineering foundation and that will help you gain this momentum and create this flywheel. But to do that, you’d have to get organizational buy-in. You have to break apart those best practices and chip away at it piece by piece while continuing to deliver [00:23:00] value and showing results along the way, and hiring the people with the right mindset.

And if you do that, you’ll have an analytics flywheel, and everything else will be easier. Thank you.

Last modified on: