Data Minq

Real-Time Report Updating System

The Gig Agency required a solution to track and display daily sales and traffic statistics using CreatorIQ and LookerStudio (formerly Google Data Studio). While CreatorIQ only provided the latest aggregated data, the agency needed a system allowing day-by-day comparisons across different time frames, giving them more granular insights into their sales and traffic trends.

bg
Real-Time Report Updating System

Our task involved designing a system that could store, process, and retrieve large volumes of data from CreatorIQ, format it for use in LookerStudio, and provide a user-friendly interface for managing the data. This project was completed in six weeks, including client training on the control panel and the LookerStudio dashboard.

Data Analysis and Requirement Gathering

  • Objective: Understand how CreatorIQ stored its data and determine what was needed to facilitate real-time reporting in LookerStudio.
  • Challenges: CreatorIQ only provided aggregated data, meaning there was no inherent day-by-day breakdown. This required us to break down the aggregated data into daily segments for detailed analysis; LookerStudio required formatted, real-time data that allowed for comparisons across different periods (e.g., daily, weekly, monthly).
  • Solution: We thoroughly analyzed CreatorIQ’s data API and LookerStudio’s data structure requirements. This allowed us to define the data flows needed to provide the client with the required daily breakdowns.

Database Design and Implementation

  • Objective: Design a robust database to efficiently store daily sales and traffic statistics while accommodating future growth and flexibility.
  • Key Features: we developed a database to store daily statistics extracted from CreatorIQ, allowing for the storage of historical data in a structured format that can be quickly retrieved; the database was normalized to handle large volumes of data efficiently, with tables structured to separate daily traffic, sales data, time frames, and user segments; the design allowed for future scalability, making adding new metrics or expanding the data schema easy if additional tracking features were required later.

Data Processing and Script Development

  • Objective: Develop scripts to automatically pull, process, and format the data from CreatorIQ for use in LookerStudio.
  • Challenges: CreatorIQ’s API delivered aggregated data in formats incompatible with LookerStudio's real-time reporting needs; the scripts needed to transform the aggregated data into daily segments and prepare it for LookerStudio’s visualizations.
  • Solution: we wrote scripts that automated pulling data from CreatorIQ, parsing the aggregated statistics, and splitting them into daily reports; these scripts also handled data formatting issues, ensuring that the processed data matched LookerStudio’s requirements, allowing seamless integration.

Control Panel Development

  • Objective: Create a control panel that simplifies data management for The Gig Agency’s internal team, enabling them to oversee and manage daily statistics without technical expertise.
  • Key Features: the control panel provided a high-level view of the daily statistics being pulled and stored, giving the team a real-time view of the data pipeline's status; users could review, filter, and modify stored data as needed, with options to trigger manual data pulls or schedule automated updates; the system included alerts to notify the team of any issues with data syncing or API connection failures, ensuring the team could quickly address any problems.
  • User-Friendly Interface: The control panel was designed with non-technical users in mind. Its intuitive interface required minimal training for the agency’s staff.

Integration with LookerStudio

  • Objective: Ensure seamless integration between the CreatorIQ data and LookerStudio for real-time reporting and visualization.
  • Key Features: LookerStudio was configured to pull data directly from the newly created database, with the processed data feeding into various dashboards for daily, weekly, and monthly comparisons; users can filter the reports by different periods, comparing day-by-day trends, identifying patterns, and making data-driven decisions; the reports provided rich visualizations, including line graphs, bar charts, and tables that showcased key metrics like daily traffic, sales volumes, and other KPIs.

Client Training

  • Objective: Provide comprehensive training to The Gig Agency’s team on the control panel and LookerStudio dashboards, ensuring they could independently manage and interpret their data.
  • Training Approach: we held interactive training sessions with the agency’s staff, demonstrating how to use the control panel to manage data pulls, troubleshoot issues, and monitor daily statistics; additionally, we provided guidance on using LookerStudio, teaching the team how to create custom reports, apply filters, and analyze data across different time frames; post-training support addressed any follow-up questions and ensured a smooth transition.

Challenges and Solutions

  • Aggregated Data Processing: One of the main challenges was working with aggregated data from CreatorIQ, which required careful parsing and reformatting to break it down into daily segments. We solved this by developing robust scripts that could efficiently process and format the data into usable day-by-day reports.
  • Real-Time Performance: It was critical to ensure that the system could process and display real-time data without delays. We optimized both the database structure and the data pipeline to quickly handle large volumes of information, providing accurate, up-to-the-minute reporting for the client.
  • User-Friendly Design: The control panel had to be intuitive for non-technical users. We focused on simplicity in the design, ensuring that even those without technical knowledge could easily manage the system and monitor daily sales and traffic statistics.

Outcome

The project was completed within six weeks, delivering a comprehensive solution that met The Gig Agency’s detailed daily sales and traffic tracking needs. The system now provides:

  • Day-by-Day Reporting: The agency can view and compare daily statistics across different periods, providing deeper insights into sales and traffic trends.
  • Automated Data Processing: The developed scripts automate the data pull and processing, ensuring up-to-date statistics are available in real time.
  • Easy Data Management: The control panel gives the agency complete control over its data, allowing it to manage and monitor it without technical expertise.
  • Dynamic Reporting with LookerStudio: The integrated LookerStudio dashboards provide dynamic and customizable reporting, enabling the agency to quickly visualize and analyze critical metrics.


testimonials

What our clients says

I had the pleasure of working closely with Grzegorz on several backend software solutions for various clients.

Greg’s ability to design, develop, implement, and improve complex software was integral to these projects’ success. I knew that if Greg was tasked with something, it would always be completed properly and on time- regardless of complexity.

He consistently delivered high-quality code, could quickly troubleshoot and resolve issues, and kept teams well-organized and on-track.

In short, Grzegorz is a highly reliable and capable developer, and any team will be better with him on it!

Mathew Arrington
Mathew Arrington

Marketing Operations @ The Gig Agency