High-Volume Streaming Analytics
Building a high speed, high volume streaming analytics solution for enhancing customer targeting with exactly-once semantics is complex, resource intensive and difficult to maintain and continuously improve. This customer needed to collect data from batch files as well as real-time weblogs, mobile activity and CRM data. Using the Cask Data Application Platform (CDAP), the company created a solution whereby data was organized into customer hierarchies and modeled to deliver targeted ad campaigns and leverage promotional opportunities based on users’ locations. Advanced analytics were then performed for tracking these campaigns in real-time. In the end, this solution:
• Performed real-time and batch processing
• Processed approximately 38 billion transactions per day in real-time
• Accurately categorized customer activity into specific buckets and hierarchies
• Generated unique counts in real-time to understand audience reach and track behavior trends
• Generated hourly, daily, monthly and yearly reports on multiple dimensions
• Provided unique stat count on an hourly basis, rather than weekly, for line of business leader reports
• Supported exactly-once process semantics for reliable processing
• Reprocessed data without side effects due to bug fixes and new features
Benefits of Cask Solution
- CDAP simplified building the application and getting it to market faster.
- CDAP’s standalone runtime mode and testing frameworks allowed developers to build and test the application quickly and efficiently.
- The team replaced a MapReduce based batch system with a real-time system, delivering insights by the minute rather than days.
- CDAP’s exactly-once and transactional semantics provided a high-degree of data consistency during failure scenarios, making it easy to debug and reason the state of data.
- No distributed components were required to run functional application tests.
- CDAP provides unique stat count on an hourly basis, rather than weekly, for line of business leader reports.
- The application generates multidimensional reports for customer 360 insights, providing information on behavior and trends and illuminating customer churn.