Book a Call
Modernising Data Reporting for a Library Service: Automating and Standardising Library Analytics with Microsoft Fabric
The Opportunity
A library service and its network of member libraries needed to move away from manual, email-based data ingestion and inconsistent library metrics. Limited use of Azure and Power BI resources, no CI/CD pipelines, and restricted DevOps access were holding back reporting, analytics, and the ability to scale across the network.
The Delivery
Agile Insights implemented a metadata-driven data platform on Microsoft Fabric, including automated ingestion via Data Pipelines, a centralised Lakehouse with Bronze, Silver and Gold layers, and standardised Power BI datasets and reports built on consistent, governed metric definitions.
The Outcome
The library service now onboards new data sources through configuration rather than code, reports on consistent metrics across all libraries, and benefits from improved operational visibility, easier troubleshooting, and a scalable foundation for future growth.
Book a free consultation
Download the full case study
About the Client
The client is an Australian library service that supports a network of libraries, providing collections, research resources and lending services to the public.
The Challenge

Data ingestion relied heavily on email attachments and manual uploads, limiting scalability and reliability, and some systems, such as Over Drive, required combining multiple files just to produce usable datasets. Library names and structures differed across systems, requiring manual mapping to keep things consistent, and metrics like “turnover” were calculated differently from library to library, leading to inconsistent reporting. On the infrastructure side, there was minimal use of Azure resources or Power BI capacity within their existing tenant, no established CI/CD pipelines, and free-tier DevOps licensing that limited its usefulness for Fabric development. Many smaller libraries relied on static PDF snapshots rather than direct Power BI access, and variability in how KPIs were calculated made meaningful comparisons difficult.

Our Solution

Agile Insights moved the library service from manual ingestion to automated pipelines using Fabric Data Factory, maximising resource utilisation through an end-to-end platform spanning Lakehouse storage, Data Pipelines, and integrated Power BI. Data Pipelines automated the ingestion of structured and semi-structured data from multiple sources, including email attachments, Over Drive, and SharePoint, and triggered transformation workflows automatically. A Lakehouse built on OneLake provided a centralised, scalable store for Bronze, Silver, and Gold datasets, while shared data models enforced consistency by embedding centralised metric definitions and calculations directly into Power BI datasets and dataflows, so every library reports from the same logic. The project was delivered as an 8-week MVP: the first two weeks covered requirements and solution design, ingestion was automated and the Lakehouse established by week four, and weeks five and six focused on data transformation and KPI standardisation in Power BI.

 

Data Ingestion
Before: Data ingestion was dominated by manual uploads and email attachments, with no real-time or near-real-time capability. Usable datasets could only be produced by manually combining multiple files, such as those from OverDrive. After: Ingestion is now automated through Fabric Data Pipelines, and new sources are onboarded through a metadata-driven approach that relies on configuration rather than code. Connection details, scheduling, and pipeline configuration are all centrally managed.
Reporting & Metrics
Before: Inconsistent calculation of metrics such as “turnover” across libraries and manual mapping required due to differing library names and structures After: Centralised metric definitions embedded in Power BI datasets and dataflows and consistent Bronze, Silver and Gold architecture ensuring reliable reporting
Infrastructure & Technology
Before: Azure resources and Power BI capacity were used only minimally, and there were no established CI/CD pipelines. Free-tier DevOps licensing also restricted Fabric development. After: An end-to-end Fabric platform now spans Lakehouse, Pipelines, and Power BI, supported by reusable, parameter-driven ingestion and transformation pipelines. Lightweight CI/CD integration is planned for Phase II.
Data Accessibility
Before: Smaller libraries relied on static PDF snapshots rather than having direct access to Power BI. After: Metadata-driven execution logging and monitoring have improved operational visibility, and reporting access is now consistent and governed across the entire library network.
Why It Worked

Agile Insights delivered the solution through a structured program focused on:

  • Implementing a metadata-driven ELT framework for scalable, configuration-based onboarding
  • Centralising connection details, execution parameters, scheduling and pipeline configuration
  • Building reusable, parameter-driven ingestion and transformation pipelines
  • Improving operational visibility through metadata-driven logging, monitoring and audit
  • Establishing a standardised ELT framework to support future platform expansion
  • Enabling a consistent Bronze, Silver and Gold architecture for data quality and governance

The Impact

The library service now operates on a standardised, automated, and scalable Microsoft Fabric platform that has replaced manual, inconsistent processes with governed, reliable reporting across its library network. The metadata-driven framework reduces the effort needed to onboard future data sources, strengthens operational visibility, and provides a strong foundation for the reporting platform to keep growing with minimal ongoing maintenance.

Download the full case study here.

Download

Get in touch