The Challenge
A global workforce management provider faced significant challenges with outdated billing and finance data models and needed a comprehensive overhaul to address inefficiencies. They sought support refactoring their system into a sustainable, systematic, and scalable pipeline.
The Approach
In order to support our client goals, YLD helped refactor their billing and finance data models. We started by taking their existing models, tables, and underlying code, and rebuilding them in a clean, structured way. This included optimising and restructuring their data pipelines to improve efficiency, maintainability, and scalability.
The key areas where we acted included pipeline optimisation and data model refactoring.
Pipeline optimisation
The existing code was opaque, lacked testing, and did not utilise proper templating to organise references and static data, making classification and mapping challenging. With the lack of documentation and structure, we focused on applying industry best practices to ensure clarity and maintainability.
For both billing and finance data, we delivered refactored models of their legacy code and structured them following the same principles and ways of working. These models followed industry best practices by breaking the code into smaller, manageable parts to make it easier to maintain and scale. By using design patterns for reusability and flexibility, these models improved readability with consistent formatting and clear naming conventions.
The code was refactored for higher extensibility, quality, and security, with robust error handling and logging to enhance maintainability. Additionally, separation of concerns and comprehensive documentation were prioritised, with a focus on collaboration through code reviews and adherence to security best practices.
Data model refactoring
Our work focused on modernising a legacy codebase while maintaining seamless output consistency for end users. The goal was to ensure the output tables, rows, columns, and insights, remained identical to the original system.
As part of the refactoring process, we enhanced the codebase by improving its extensibility, functionality, readability, and ultimately, quality. Additionally, we developed tools and utilities within dbt to streamline testing and manage models effectively. These utilities enabled continuous verification, ensuring the refactored models matched the legacy models one-to-one throughout the development process.
For the data model refactoring, we used SQL to optimise and restructure the database for better performance and scalability, while Jinja templating helped automate the generation of dynamic SQL queries, making the schema more adaptable. These enhancements improved the development workflow and provided a robust foundation for maintaining and expanding the system.
Our focus then shifted to ensuring the seamless operation of the refactored system under clearly defined service-level agreements. This involved maintaining high performance and reliability while proactively addressing any issues through robust alerting and logging mechanisms. With the foundational work complete, our efforts emphasised day-to-day enhancements.
The Deliverables
We delivered robust testing and monitoring frameworks to detect and address data issues (e.g., incorrect rows or columns) before reaching customers. This was critical for ensuring billing accuracy, a key client priority.
With the increased visibility into the pipeline’s operations, we reduced complexity, streamlined processes, and enhanced overall data reliability. By continuously optimising the pipeline and models, we ensured the system remains scalable, efficient, and ready to adapt to future challenges.
Closing the Engagement
By streamlining the codebase for consistency, we empowered new developers to quickly onboard and make meaningful contributions, significantly reducing ramp-up time. This foundation supported the seamless and scalable addition of features such as new columns, calculations, and insights.
Ultimately, our collaboration has delivered a robust, future-ready data pipeline that drives innovation, ensures data accuracy, and fosters trust with end customers. The platform can now handle the increased data load and complexity associated with supporting a growing global workforce.
YLD’s contributions enhanced the client's operational efficiency and positioned them to adapt swiftly to future business needs and opportunities.
View More Work
