VTION: Serverless Analytics for Privacy-First Mobile Usage Insights

Executive Summary:

VTION’s metrics were tied to Power BI DAX, while mobile usage data kept growing and every run revisited the full history. Rapyder rebuilt the metric layer with its AWS data and AI services: partitioned S3 storage, Glue ETL with bookmarks, Delta Lake, Athena SQL, Lambda and API Gateway. The new pipeline processes only incoming data, supports the customer’s own dashboard and keeps the results aligned with the legacy metrics.

About Customer:

VTION provides data-driven insights and tools that help brands plan, activate and measure digital media and e-commerce strategies. In a privacy-first environment where traditional consumer behaviour analysis is losing effectiveness, it offers consumer insights, audience activation and panel surveys to brands across CPG, gaming, OTT, BFSI and mobile handsets. The business depends on aggregated, privacy-compliant behavioural analysis derived from mobile application usage data to support marketing planning and performance measurement in a regulated environment.

Problem Statement:

VTION’s analytics depended on Power BI DAX formulas, while growing mobile usage data made full-history reprocessing increasingly inefficient. The team needed consistent metrics across audience filters, normalised event data and weighted panel calculations. It also needed to serve insights through an API to its own dashboard while maintaining privacy-first requirements.

Industry:

Industries, Media & Entertainment

Offering:

Offerings, Data Analytics

Business Challenges:

1. Power BI dependency: Metric definitions lived in Power BI DAX formulas, tying the business to one tool and one execution model.
2. Growing data volume: Volumes of raw mobile usage data were growing faster than the existing approach could process them.
3. Historical reprocessing: Every run reprocessed historical data, wasting compute and lengthening cycle times.
4. Metric consistency: Metrics such as Universe, MAU, install rate and session duration had to be calculated consistently across every slicer, and drift between definitions was a real risk.
5. Raw event preparation: Raw timestamped event data needed normalisation and weighted panel logic before any metric could be trusted.
6. API delivery need: Insight had to be delivered to a dashboard the customer builds and operates itself, which meant an API layer rather than a BI extract.
7. Privacy requirements: Everything had to stay aligned with privacy-first principles given the regulatory environment.

Solution Implemented:

1. S3 data lake: Established Amazon S3 as the central data lake, storing customer CSV files partitioned by application, date and device attributes for efficient querying.
2. Schema detection: Configured AWS Glue Crawlers to detect schema changes automatically and keep the Glue Data Catalog consistent.
3. Incremental ETL: Built PySpark-based AWS Glue ETL jobs with job bookmarks enabled so only newly ingested partitions are processed on each run.
4. Timestamp normalisation: Normalised raw timestamped event data, converting Unix timestamps to standard date-time formats and deriving year, month and day fields to support partition pruning.
5. Session metrics: Computed session durations from application lifecycle events and applied weighted analytical logic to derive standardised metrics.
6. Metric calculations: Calculated Universe with minimum threshold validation, App Installed %, Monthly Active Users, MAU as a percentage of Universe and of installed base, average time spent, usage frequency, and category and app-level rankings.
7. Delta Lake output: Wrote transformed datasets back to Amazon S3 in Delta Lake format with time-based partitioning for ACID-compliant storage and faster queries.
8. Athena migration: Converted the existing Power BI DAX formulas into equivalent Amazon Athena SQL queries, preserving metric continuity while improving scalability.
9. Parameterised queries: Built AWS Lambda functions that construct and execute parameterised Athena queries based on slicer inputs including Month, State, NCCS, Gender, Age Group, Category and App.
10. REST API: Exposed secure REST endpoints through Amazon API Gateway returning structured JSON for dashboard consumption.
11. Environment deployment: Deployed the architecture consistently across Development and Production with centralised monitoring of ETL execution, query performance and API invocation.

Technology Stack:

• AWS Lambda
• Amazon S3
• Amazon Athena
• AWS Glue
• Amazon API Gateway

Business Benefits:

1. Scales without redesign – The serverless architecture absorbs growing mobile usage volumes, letting VTION onboard more apps, categories, and users without re-architecting.
2. Incremental processing – Glue job bookmarks and partitioned data avoid recomputing history, cutting processing time and operational cost.
3. Faster insight delivery – Pre-aggregated datasets and optimised Athena SQL give low-latency API responses for Universe, MAU, install rates and engagement trends.
4. Consistent metrics across environments – Automated ETL, centralized metadata, and monitored query execution keep insights available and identical across Development and Production.
5. Lower cloud spend – Pay-per-use services, partition pruning and columnar storage cut Athena scan costs and remove fixed infrastructure overhead.
6. Metric continuity with the old stack – DAX logic converted to Athena SQL was validated against existing dashboard outputs, so numbers did not shift when the platform did.

Key Takeaways:

• Metric continuity – Converting DAX to Athena SQL and checking the results against the old outputs made the migration safe to trust.
• Threshold validation – Minimum-threshold checks helped protect the reliability of aggregated, privacy-first metrics such as Universe.
• Dashboard ownership – A parameterized API lets VTION keep control of the dashboard instead of depending on a fixed BI extract.