I need better equipment… I tell myself while trying to jog off the extra pounds I gained in 2020. The earphones I am wearing slowly loosen from my ears with each step and I can feel each contour of the blacktop with my worn-out sneakers. I manage to finagle the earphones just right to listen to a new podcast Building the Backend by Travis Lawrence. He is interviewing Chris Bergh from Data Kitchen and a principal author of DataOps. I had heard the term before, DataOps, and read the manifesto, read the DataOps Cookbook, and I had researched how DataOps is the fusion of DevOps, Agile, and Lean Manufacturing. The stories Chris and his colleagues share about clients calling about bad data in reports, ETL pipelines failing in production, and clients who thought new features could be delivered quicker were all experiences I could relate with. I had been a convert for a little over a year and wanted to hear more.
As I slogged up a hill trying to convince myself to keep going, Travis asked Chris the question:
“Where do you see data architectures hitting in the next two to five years?”
To paraphrase Chris sees two camps emerging:
The low-code side, tools that generate code but just in the background using fancy interfaces; furthermore, these are geared towards data analysts and folks without a software development background.
The high-code side, tools that help build data lakes, data warehouses and they are tailored for those with software development backgrounds.
And Chris sees that the challenge for organizations will be how they define the processes that adhere to DataOps principles in order for the two camps to co-exist and bring value to the organization.
I understood exactly what he meant. For well over a year I have been working in both camps. On the high-code side, I was working SQL, C#, and JavaScript working with teams to build solutions with unit tests, implementing continuous integration, and having tools like Application Insights and StackTraceJS to proactively address issues with clients.
On the other side though, there was Power BI. Please don’t get me wrong, I like the tool and its capabilities are profound and definitely a reason I have stable employment. But across version control, testing, continuous integration, to monitoring, applying DataOps principles to Power BI is a challenge… I need better equipment.
It has taken me the better part of a year, but I think I have put together some concepts and accelerators to bring DataOps to Power BI, and I would like to share them on this blog with the tech community. Along the way I will touch on a DataOps principle, the challenges adhering to that principle with Power BI, and my thoughts on overcoming those challenges.
My hope is that you find these concepts and accelerators helpful, but I would like your feedback. My thoughts come from my experiences, my work with clients and their environments. Please let me know what you think on LinkedIn or Twitter. Maybe there will be enough discussions that eventually we improve upon these concepts and accelerators, or they make their way into the Power BI product. If DataOps principles become more inherent to Power BI’s feature set and this blog series becomes obsolete, all the better.