Emerging Techniques in Agile Analytics
Innovation in agile analytics tools and processes continues rapidly, with leading organizations already benefiting from cutting-edge developments while more nascent areas show high promise.
Conversational Analytics
Advances in natural language interfaces are making querying data and uncovering insights far more intuitive. Conversational analytics allows users to ask business questions using natural phrases rather than rigid report structures or dashboard clicks. Behind the friendly interface, machine learning quickly maps the intent to pull relevant data visualizations, predictions and alerts.
For agile analytics, these interfaces promote collaboration and exploration. No longer bottlenecked by manual dashboard creation, business teams can gain insights at the pace they have questions and ideas. Look for broad adoption of conversational analytics adding velocity to analytics cycles within two years.
Automated Machine Learning (AutoML)
Manual, coding-intensive data preparation and modeling steps often dominate analysis timelines. AutoML promises to automate these technical tasks through AI that reviews raw data, engineers features, develops and compares modeling options.
Leading offerings like Google Cloud AutoML Tables, Azure Machine Learning, Amazon SageMaker Autopilot and others now integrate automated ML capabilities like no-code model building and deployment. While still maturing, AutoML can eliminate grunt work during analytics sprints to maximize time spent on high-value interpretation and decision making.
Confidential Computing
As real-time data underpins agile analytics, security and privacy controls receive greater focus. Confidential computing technologies like homomorphic encryption allow complex computations on encrypted data. Rather than expose raw personal information during analysis, it remains protected even while aggregated into overall metrics.
Look for confidential computing to enable analytics velocity while also satisfying emerging data privacy regulations. While currently cost prohibitive for broad deployment, expected advances in encrypted hardware will increase adoption.
Edge Analytics
Performing analysis on decentralized edge devices rather than centralized cloud servers promises major latency reductions. By analyzing data on gateways and endpoints like connected factory equipment or vehicles, actions activate instantly without roundtrips to distant data centers.
Edge computing platforms enable distributed agile analytics out on the true front lines rather than the core. As the Internet of Things (IoT) expands exponentially more intelligent devices, edge analytics innovations integrate insights directly into business processes.
Augmented Analytics
While end users are increasingly technically savvy, few have hardcore data science expertise. Augmented analytics platforms blend automation, smart recommendations and powerful natural language search across model building, analysis and sharing tasks. Rather than fully automated black box machine learning, users retain control through AI assistance.
Augmented functionality already assists agile analytics adoption by shrinking technical skill gaps. As capabilities grow more robust expect augmented analytics to become the dominant paradigm, drastically expanding those able to work in an insight-driven, analytically creative manner during each sprint.
Overcoming Data Quality Challenges
Delivering rapid analytical insights depends on reliable, accurate source data. But assembling trustworthy datasets often bogs down analytics teams. Between evolving data sources, fragmented tooling, inadequate data validations and steep learning curves for new staff, maintaining data quality requires significant effort. These headaches divert energy away from core analysis during agile sprints.
Mastering a few key techniques paves the way for smooth agile analytics in the face of real-world data issues:
Standardize Reference Architectures
Document guidelines for how core datasets integrate from departmental systems onto centralized warehouses and lakes used for analysis. This clarifies required data feeds and transformations for analytics teams while directing IT groups on expected interfaces and loads. Refine standards as new sources emerge.
Metadata-Driven Automation
Catalog metadata detailing source schema and validation specs in a central catalog. Automation tools like Apache Airflow then orchestrate movement and validation based on the catalog rather than handwritten scripts. This simplifies modifying pipelines as needs evolve while enforcing consistency.
Holistic Data Testing
Assess data quality across dimensions like completeness, conformity, consistency, accuracy and duplication at batch ingestion plus during analysis. Automated testing frameworks scale systemic inspection rather than shallow spot checks. Integrate these frameworks withdataDict management platforms for continuous feeds detailing trustworthiness.
Build DQ Dashboards
Give business stakeholders self-service visibility into state of analytics data quality via dashboards detailing volumes, freshness and incident metrics. Empower data consumers to understand tradeoffs in latest vs high quality datasets for their particular use case.
Incorporate Monitoring
Leverage log analysis and application performance management (APM) tooling to trace dataflows end-to-end. Identify breakpoints slowing reliable delivery for investigation and address root issues like underpowered ETL tools or overloaded databases.
With pragmatic architectural decisions, metadata management and testing automation in place, analytics groups can take control of data quality rather than being controlled by constant fire drills.
Interviews on Agile Analytics Successes
Hear directly from those leading award-winning agile analytics programs at their organizations about key lessons learned:
Leveraging External Expertise to Kickstart Culture Shift
Margaret Wright, Director of Analytics at Vision Health, a $2B medical supplier, discusses their agile analytics journey:
"We had identified slow delivery of insights and data-driven decisions happening 6+ months late as critical issues limiting fast growth. But a decade of traditional BI approaches left teams ingrained in waterfall practices ill-suited for the pace of change needed.
To catalyze the shift to agile processes we brought in agile coaches from the domain expertise firm Agile Analytics Partners for an intensive design thinking and visioning workshop involving IT, analytics and our main business units. That outside perspective explaining practical steps to increase collaboration and iterations based on their experience working with other data-driven organizations gave the credibility needed to sell skeptics."
Takeaway: External expertise can validate the vision and approach needed to get agile analytics over the initial adoption hurdles
Embedding Analytics to Drive Decentralized Value
Roger MacAdams, VP Online Sales at MultiGoods Retailers with $15B in annual revenue discusses shifting their analytics from centralized enablement to decentralized value delivery:
"Historically analysts were seen as report generators or data gurus removed from core operations. Our pivot to agile analytics focused on driving analysis directly into the business through techniques like embedded BI apps. For example, working closely with category managers we jointly built mini analytics apps enabling what-if modeling for pricing optimizations, assortment planning and promo mix selection that integrate directly into their natural workflow."
He continues, "By aligning analysts directly with key roles rather than through centralized requests and generic reporting, visibility into data-driven decisions has increased dramatically while also increasing business ownership rather than analysts dictating supposed insights."
Takeaway: Embedding actionable analytics across the business boosts decentralized decision making authority and efficacy
Sustaining Agile Analytics Excellence
Kareem Hassan, Chief Data Officer for Regional Bank, $60B in assets, weighs in:
"We‘ve seen great traction going from traditional quarterly reporting cycles to agile delivery organized in 2 week sprints. Business intimacy increased substantially while time-to-value decreased by over 40%. But keeping momentum going at scale has challenges.
We combat analytics user experience fatigue where stakeholders lose interest in the latest incremental insight by mantra ‘go broader before deeper‘. In other words, increase the distinct business questions addressed through agile processes rather than dive exhaustively into slender domains. This sustains curiosity in insights.
We also incent participation by different business units in analytics sprints through chargeback models – if you‘re not at sessions, you don‘t gain priority for specialist time afterwards. Embedded performance consultants track ROI on agile initiatives to showcase hard value."
Takeaway: UX design thinking and economic levers help incentivize ongoing involvement.
Sample Agile Analytics Project Plan
Use this sample project template to organize a new agile analytics initiative:
Vision
Describe the current limitations to be addressed and aspired future state in alignment with overall corporate strategy.
Example: Slow reaction to customer churn spikes inhibits retention. Apply agile analytics for more adaptive predictive models and tailored incentives.
Scope
Outline key business processes and decisions to enhance.
Example: Improve subscriber retention across video streaming customer base by optimizing targeting and personalization of churn incentives.
Stakeholders
List the business units, IT/analytics groups, subject matter experts and data trustees involved plus outline their responsibilities.
Example: Product owners – Marketing Campaign Team
Technical Team – Data Engineers, Machine Learning Devs
End Users – Campaign Specialists
Processes to Make Agile
Identify workflows, standards and handoff points to evolve to work iteratively.
Examples:
- Transition static quarterly ETL batch pipeline to dynamic micro-batch loading
- Standardize features store definitions cleansed attributes
- Establish fortnightly model retraining cycle
Technology Enablers
Detail supporting platforms for analytical agility and scale.
Examples:
- Cloud data warehouse foundation – Snowflake
- Machine learning automation – Azure ML Studio
- Collaboration space – Tableau Visual Analytics + Slack
Analytics Backlog
Prioritize key questions, metrics and experiments – valuable items that analytics cycles tackle iteratively.
Example:
- Propensity model accuracy
- Feature set optimization
- Churn risk thresholds
- Personalization uplift
- Campaign targeting expansion
Measurement Framework
Define key results metrics and tracking process against vision.
Examples:
- Decreased subscriber churn rate
- Increased retention campaign conversion
- Reduced customer acquisition costs (CAC)
With the project charter completed, agile analytics cycles can commence!
