(And why some may not materialize)
Few areas of business today are changing faster than where and how analytics are being used. Turn your head for a second and—boom—you're falling to the back of the pack. From big data and visualization to predictive modeling and more, analytics represents a rapidly evolving world of technologies and tools that few have time to keep up with. This makes it challenging to think about which trends really matter and which will prove short-lived, which are hype and which will deliver tangible, timely business value.
But think about them we must, if only to understand where the market might be moving and how fast. Because for all the uncertainty in the field, business leaders still have to make decisions and choices about the future.
That's why we've taken a fresh approach to analytics trends this year, focusing on developments that appear to be important while also pointing out areas of disagreement, even within our own organization.
Whether a particular trend will affect you immediately is hard to say. Yet thinking about the trend—and what it might mean if it is true for your organization—may be time well spent.
Conventional wisdom says companies are facing a large supply gap of data analytics talent, with significant shortages at all levels. Professionals who can deliver data-backed insights that create business value—not just number crunchers—are especially hard to find.
Is the conventional wisdom correct, or are there other forces at play? Large companies in traditional industries may be much less concerned about hiring lots of data scientists than are online organizations and startups.
There are also classification issues. There's massive confusion about what a data scientist actually is. For some, a person who can manage spreadsheets and do basic reporting might qualify, at least in his or her own mind. For others, data scientists are expected to blend statistical sophistication, data management skills, and business acumen. It's quite possible that the job of the data scientist, as currently defined, requires more attributes than any individual should be expected to have. Some companies are resolving the gap by creating teams of diverse skill sets. They're mixing and matching professionals to deliver a balanced response to your business analytics questions.
When you're recruiting analytical people, be clear what your needs are. Just saying you need a data scientist is like advertising a slot for a smart person who's good with numbers.Download PDF of full report
Companies are struggling at almost every level of analytics recruiting today. The team approach makes sense, but it assumes there are enough people to hire in a wide range of analytics-related roles.
That's not the case. Even as universities create many more analysts, it will be years before they enter the labor force and become effective. Great data scientists also rely on their past experience, and getting these new graduates to that point will take some time.
Some of the current talent crunch is a function of hoarding, not real demand. In response to predictions about impending shortages of qualified analysts, companies scrambled to recruit talent beyond what they actually needed. This led to experienced people being asked to carry out activities like straightforward reporting that could have been done with lower-level talent. It also led to lower-level talent doing busywork—e.g., cleaning data—that is better done by machines.
In addition, startups are hiring like crazy. Millions of dollars are being invested in identifying the high-value use cases, building the magical model to predict the future, and creating the subscription service to ingest data and extract insights they can monetize or sell.
This will likely all shake out over the next several years, with market forces and employees themselves driving the outcomes. Third-party solutions will cover many of the most valuable use cases—and internal teams will be relegated to more mundane work, if not cast adrift.
Today's data visualization tools make complex information accessible and understandable for almost any business user—and easily affirm the adage that a picture is worth a thousand words. Or, in this case, numbers.
Users who seek insights in big data have new tools that let them understand, explore, share, and apply data efficiently and collaboratively, often without analytics professionals. But that's where the risk comes in. Eager users may choose polished graphics over data preparation and normalization and rigorous analysis. That means they may gloss over important insights and produce erroneous results.
Sometimes glitzy visuals just create noise where plain old numbers would make sense. And overusing them can be patronizing. Visualization has a place in almost any data presentation, as long as it offers at least one of these benefits:
Faster observations of trends and patterns. Sometimes, the essence of a story lies in larger patterns that occur at an aggregate level.
Better retention. For some people, visualization makes it easier to understand and remember data. And visual tools can present concepts as stories.
Embedded feeds and user engagement. Advanced tools and live feeds let users tweak data and generate custom views to explore together in real time.
Management wants greater involvement with analytics-based decision-making. That's a positive trend. Just don't get caught up in all the sizzle. Visual analytics don't improve every decision, and some visuals don't add clarity to data.Download PDF of full report
Visualization is real, and it's valuable. Without it, analytics adoption would likely be going nowhere fast. But like any technology, there's a right way and a wrong way to use it. If your decision-makers want to use data—and you should increase their understanding of the data before they act—there may be no better tool. Of course, this doesn't imply you always have to visualize. Computers don't care about visual analytics, so if your decision is to be automated without human oversight or intervention, don't waste time with visuals.
The overuse of visualization can get in the way of clarity. Sometimes that's the result of making a complex story too simple, and sometimes it's the opposite—making simple facts seem too complex. We've all experienced that sense of dread when turning the pages in a document in which each page presents information in a different way than the page before. Ugh. Who has the time and patience to read this kind of cumbersome material?
More important is the risk of putting so much energy into a presentation that the actual analysis gets short shrift. If the underlying data and assumptions aren't valid, the analysis won't measure up no matter how visually interesting it may prove to be. With analytics talent in such short supply—particularly analysts who can creatively develop visual displays—this risk is clear and present.
Machine learning—the ability of computers to learn from data—has been around since the 1970s. In analytics, it typically involves the semi-automated development of predictive and prescriptive models that improve over time. The software learns how to better fit the data and separate signal from noise.
The challenge in analytics? Data scientists understand machine learning, but businesspeople don't. Managers hesitate to make decisions without hypotheses or human explanations behind them. Many organizations simply ignore machine-learning findings.
Today, big data projects move too quickly for hypothesis-driven analytics. For example, digital advertisers need to create thousands of new ad models a week, and each ad placement decision takes milliseconds. No human team is that prolific.
But that doesn't mean analysts and data scientists should just turn things over to machines. The leading machine-learning environments still need people to specify the variables that can enter models, adjust model parameters for better fits, and interpret content for decision-makers.
Machine learning can also make analysts more productive. For example, one IT vendor creates 5,000 models a year to fine-tune sales and marketing—using only four analysts. Before machine learning, twice as many analysts could manage only 150 models a year.
That industrialized approach, or model factory, is gaining adoption. Deep automation is still in the pre-Henry Ford stage, but the concept is likely to take off just as fast as conventional manufacturing did.Download PDF of full report
This trend is spot on. It reflects a long-developing technology that has reached critical mass. What's more, we're seeing considerable demand from client companies for people who are familiar with the approach. They know that machine learning is not—and will likely never be—a completely automated way to develop models.
We're also beginning to detect interest in new business models in this domain, such as "machine learning as a service." Consulting organizations and cloud service providers are likely to be the early adopters.
This trend has its place, but its value and capabilities can be overblown. There is a sizable "garbage in, garbage out" potential here.
Just because someone knows how to operate the sausage maker doesn't mean tasty sausage will turn out. Companies still should rely on smart human analysts to hypothesize about relationships in the data and find models that support or overturn those hypotheses. It may be a slower process, but their companies may be much less likely to get into trouble.
The reality, of course, is that this isn't a black or white issue. There are shades of gray. Ongoing thought should go into deciding how humans can maintain control, because they are ultimately accountable for the outcomes.
Discovery is essential in science-based research, development, and product innovation. But it's no longer restricted to the lab. Increasingly, discovery focuses on data management and analytics. Leading organizations are adopting data discovery platforms—technology environments that make big data manipulation relatively easy and inexpensive.
Big data, once restricted to online and startup organizations, is now available to everyone. In a traditional analytics environment, data exploration was slow, and few companies relied on analytics for big decisions. Online organizations had discovery processes that relied on expensive and time-consuming work. Now, in an effective discovery environment, organizations can consider more types of data, use more variables and cases in models, synthesize them for new applications, and analyze fast-moving data at speed.
In the past, some organizations established analytical sandboxes, but discovery environments are different. First, discovery environments are now the domain of the masses, not just analytics experts. Analytics-aware users want to interact with data, look for patterns, or perform drill-downs that traditional approaches couldn't. Second, as these environments ingest more data, their architectures are becoming more complex. Hadoop/MapReduce datastores handle the volume and specialized processing appliances boost speed and performance. Today's data economy involves developing data-driven decisions and products at scale—and fast.
So far, few companies outside R&D have developed these stage-gate data-discovery processes. But that's changing quickly. As data grows in importance as a resource, determining how to exploit it becomes more vital as well.Download PDF of full report
Production discovery environments are a central concern among many of our most sophisticated clients. It doesn't make sense to address an important domain for analytics without first exploring the data to unlock the real insights, assessing the distribution and quality of the data, testing possible relationships, and developing some trial models. This isn't the private domain of the data scientist, but an inclusive environment for both the amateurs and the experts to explore and discover.
Traditionally, these environments were called "sandboxes"—places with limited rules and limited oversight. As discovery becomes a mainstream way of answering ad hoc questions, these environments evolve from being nice-to-have playgrounds to a mission-critical setting. Availability, version control, and release management—all the dominion of the IT groups—become the order of the day, and the restrictions go up.
Discovery platforms are often a good idea for online organizations with a lot of unstructured data, but they may not be needed in many traditional industries. We've been able to develop analytics without these tools.
For some, the discovery movement may be a new way to sell big data appliances. Clients aren't necessarily asking for them, and this trend isn't taking off within the great majority of companies. Even the sandbox idea raised hackles among sceptical managers, and this sounds like the sandbox on steroids. With CEOs and CFOs demanding to see return on investment, organizations need to find the clear line of sight to show them that necessary value.
Analytics prospered first in well-structured domains like pricing and supply chain optimization. Next came marketing organizations, where data and statistics found a place alongside creative content. Now one of the last bastions of pure creativity—the entertainment industry—has begun to explore the ways analytics can help human judgment determine which movies, television programs, plays, and books customers want.
Predicting consumer interest in entertainment has been tricky. Many Hollywood films lose money. New television programs are canceled. And among the hundreds of thousands of books published each year, few sell more than 100 copies. An industry with such a low batting average is ready for help.
Both Netflix and Amazon are using consumer behavior data and direct testing to shape their original TV content, like House of Cards on Netflix and Alpha House on Amazon. In the film domain, some companies are using analytics to decide which movies to produce. Relativity Media, which has had a good track record backing or making profitable films, reportedly employs an analytics algorithm for that purpose.4 And yet another entertainment giant has an internal analytics group that works with its books and theater units to predict which books and plays will generate significant profits.
Entertainment still takes human creativity that analytics can't completely replace. However, entertainment executives want to back their creative judgment with well-chosen numbers. The entertainment of the future will likely be creative, inspired, delightful, and analytically sound.Download PDF of full report
It seems inescapable that the entertainment industry may become much more analytical over time, primarily because almost every aspect of the industry can be monitored and predicted now. With the advent of streaming video and other content over the Internet—a trend that is changing virtually every entertainment business—we now have the ability to know what is appealing to many consumers. If we produce content that doesn't engage the audience, it's probably because we didn't do our research and analytics. The entertainment industry executive who doesn't understand and consume analytics may be a dinosaur in a very few years.
It is certainly true that there is more data now about the entertainment options that consumers enjoy. But analyzing that data more extensively is unlikely to lead to more creative and entertaining media content. In the film industry, the movies that are designed on the basis of audience reactions to early screenings are not noticeably more creative or successful than those created by an inspired director. Too much focus on entertainment analytics will likely lead to a lowest-common-denominator effect involving less creativity rather than more.
Let's not forget that many of the world's greatest artists died as paupers. It wasn't until decades or centuries after their deaths that the brilliance of their work became widely appreciated. While good might be an immediate best-seller, great might not.
A few years ago, there were no Chief Analytics Officers, no Chief Data Officers, no Chief Science Officers, and no heads of Big Data. Today, there are plenty. But do these C-level analytics and big data positions really help organizations? There isn't a lot of data…yet. But the signs are encouraging.
It's possible to fold data and analytics into existing C-level responsibilities— even the CEO's. On the other hand, a recent Deloitte Analytics survey found most analytics functions reported to a business unit or division head.5 And even if analytics lands on a senior executive's plate, will it win the battle for that person's attention?
The "fold it in" option has precedents, such as Caesars Entertainment. But many organizations—including Facebook, AIG, and the Obama 2012 reelection campaign—have created dedicated analytics roles. To achieve its potential, analytics needs advocacy and oversight.
Then there's the Chief Data Officer (CDO) role, which is increasingly common in large banks. The idea is to combine responsibility for managing data with its application through analytics. Yet many CDOs spend more time on management than on analytics—possibly because they lack strong analytics backgrounds.
An organization that creates a senior role for analytics, whatever the title, has likely done a productive thing. It's focusing on doing more with data, generating insights, and putting them to use. The perfect should not be the enemy of the good.Download PDF of full report
When e-commerce began to rise around the turn of this century, many organizations created senior e-commerce roles to advocate for and guide the new capabilities they were building. It's probably a safe bet that their positions sped up their companies' initiatives in this important domain, even if the roles were eventually absorbed back into IT or marketing. The same can be said of Chief Analytics and Big Data Officers today. Eventually these capabilities may well become pervasive, but we should have someone to lead the use of them today and for the foreseeable future. Organizational structures always should be flexible, and when we see an important new capability arriving on the scene, we should create a role to manage it.
Analytics and big data may be important enough to deserve their own role, but creating it won't necessarily help organizations succeed. First, there are plenty of other CXO positions that care about analytics—CIOs, CFOs, and CMOs, to name three. If they have passion for the topic, why can't they lead its application across the business? A Chief Analytics Officer has to report somewhere anyway. Secondly, we've seen these arguments for new C-level roles such as Chief Security Officers. A few companies established them, but little evidence exists that the new role was any more successful than a CIO or CTO overseeing the area.
Adding a new C-level title almost indicates that the topic is faddish. Whatever happened, for example, to Chief Knowledge Officers? And why can't the Chief Strategy Officer spearhead analytics, given that strategy should be at the core of any analytics roadmap? The last thing analytics needs is the same fleeting popularity.
Big data has spawned a wave of new products and service offerings. Just as trends drove the development of new technologies like MapReduce and Hadoop, they are likely to usher in other new approaches as analytics and data management take root in more industries and businesses.
For Google, almost every offering is a data product. Another data-centric business, LinkedIn, helps boost customer recruitment and retention with tools like People You May Know, Groups You May Like, Jobs You May Be Interested In, and InMaps. Software companies are embracing data products too: Intuit recently acquired an organization of data scientists to help develop products from information its tax and business software accumulates.
Even in the industrial and manufacturing industries, data products and services are gathering momentum. GE, for example, places sensors in gas turbines, jet engines, and medical imaging devices, then services those products based on sensor data analysis. One technology company analyzes data from more than 300 million transactions its devices handle each day. By monitoring patterns, these companies can predict and head off maintenance problems before they occur.
Will data products and services ever out-earn or outperform traditional products and services? It's too early to say, but they have significant potential. More companies are now able to gather data from their operations, analyze it, and make it all available to customers. This can turbocharge your business—or your competitor's.Download PDF of full report
It makes perfect sense to examine and pursue the possibility of data products from online data, sensors, and other new sources. This trend was mentioned more than 20 years ago, when Stan Davis and Bill Davidson wrote the book 2020 Vision. They argued that information businesses could capture data and resell it to customers. In the big data era, virtually every business can become an information business. To make this happen, companies need to form Data Products teams that combine analytics, technology, and customer expertise. Many companies are already monetizing their data assets, and many more organizations will likely explore this more over the next several years.
There may be some opportunities for data products and services outside of the online industry, but it's not a significant growth opportunity. Instead, companies should maintain their focus on analytics for internal decisions. In other words, stick to your knitting. There are still many unexploited opportunities for that type of work, and the payoff is much more likely—and faster.
Before undertaking development of data products, perhaps companies should do a better job of using analytics to market and sell their existing products to existing customers.
Big data threatens the future of the enterprise data warehouse (EDW). Many companies covet Hadoop clusters, whose per-unit costs to store and process data are a fraction of EDWs'. As a bonus, Hadoop platforms can also perform some processing and analytics tasks.
Still, EDWs probably won't vanish anytime soon. They're structurally versatile and offer a desirable way to store and process data for analytics. EDWs have the security, reliability, concurrent user support, and manageability large organizations need. And the rise of "in-database analytics" has made EDWs even more popular.
However, alternatives to EDWs are increasing. One major bank maintained a large EDW for production applications, but also supported multiple data marts on smaller appliances for less permanent or security-critical applications. And it stood up a Hadoop cluster for unstructured data applications. The same company may also add "graph" databases for social network analysis and columnar databases for high-speed analytics on numerical data.
Choice is a good thing, but it brings complexity, confusion, and cost. Organizations should define processes for deciding which data and applications go where, and what circumstances will trigger a platform move.
A simple, all-EDW world may come to feel like the good old days. But when organizations start paying less per terabyte for other options, their CIOs will warm to the change.Download PDF of full report
The main issue with this trend is that it doesn't go far enough. Enterprise data warehouses are a vanishing breed, although companies may not turn over their installed bases quickly. You see the trend quite strongly in the online industry, where companies such as eBay still have large EDWs, but their data storage is now much larger in Hadoop clusters than in the EDW. And as Hadoop-like products improve in function and maturity, we'll likely see even more organizations adopting them. It's not surprising that you see EDW vendors putting their brands on Hadoop clusters and related technologies. They'll need to do that, and do it well, in order to stay in business.
The big data folks may be very excited about Hadoop, but the EDW has served thousands of organizations very well, and it's not likely going away. (In fact, we've had several clients recently purchase large EDW technology.) It tends to be more expensive than Hadoop, but it has a much better fit with the data management and analytics needs of large, established organizations that can't bet their futures on unproven open-source technologies. At some point, we may see hybrid Hadoop/EDW environments, and we won't even pay much attention to what's under the hood of our data warehouse technologies. But in the short run, the EDW is still king.
US Leader, Deloitte Analytics
Deloitte Consulting LLP
Deloitte Tax LLP
Deloitte Consulting LLP
Independent Senior Advisor
Deloitte & Touche LLP
Deloitte Financial Advisory Services LLP
1. Gartner's 2012 CIO Agenda Report
2. Big Data Executive Survey, NewVantage Partners, November 2012
3. What is a data scientist? The Guardian, March 2, 2012
4. Ryan Kavanaugh Uses Math to Make Movies, Relativity Media, November 2009; http://www.relativitymedia.com/News.aspx?pid=9507896c-078a-427b-8509-9d2c0127159f
5. Deloitte Analytics Advantage Survey, 2013; http://www2.deloitte.com/global/en/pages/deloitte-analytics/articles/the-analytics-advantage.htm