The ocean of data keeps rising and concepts such as big data are spawned in that ocean. That leads to statements that have been made to the effect that data is the new center of gravity of IT. Is technology really enabling an information-centric world, and if so, how does it tie to the more familiar and mundane world of operational IT? The rise of a concept called data-driven intelligence serves as the model for a new perspective and we can contrast that with more familiar application driven intelligence models.
One of the earliest names for information technology was “data processing” which encompassed the need for both data and processing power. However, the glamour of IT for many years was in application development. Applications, i.e. a processing or computing centric focus, ruled the roost. Throughout its life cycle from birth (creation) to death (deletion) most data remained within the control of applications which tended to make digital what had been manual business processes. Now applications that analyzed data after it had been created have also long existed (such as business intelligence and seismic processing), but these were a small fraction of practical IT uses.
Over time, that has changed into a snowball rolling down a snow packed mountain. The rise of data warehousing where data that was originally created for one purpose (such as fulfilling a sales order) was repurposed (and so creating a longer useful life for the data) to meet other business needs is one illustration. The rise of the Web where data exists that can be repurposed for numerous uses other than its creators originally intended has helped to drive these new developments.
Differentiating Between Application-Driven and Data-Driven Software Intelligences
In his book “Reinventing Discovery” (well recommended by the way), the author Michael Nielsen discusses data-driven intelligence and contrasts it with artificial intelligence and human intelligence. He defines data-driven intelligence as the ability of computers to extract meaning from data, and differentiates it from artificial intelligence, which he says takes tasks that humans are good at and aims to mimic or better human performance (such as chess playing) and human intelligence (such as our ability to process visual information). According to Neilsen, data-driven intelligence is complementary to human intelligence by solving different kinds of problems (big data anyone?). By the way, in many instances the combination of intelligences is useful.
That is a very useful and valuable perspective but for our purposes, let’s examine what it means from an IT perspective (as our focus will be a business versus scientific perspective). The following table puts application-driven vs. data-driven intelligence in perspective. Note that this excludes many other important types of software intelligence, such as operating systems and middleware, but instead focuses on what business uses to derive value and benefits.
Table: Comparing Application-Driven to Data-Driven Software Intelligence
| Application-Driven Intelligence | Data-Driven Intelligence | |
| Primary Goal | Substitute application intelligence for human intelligence in managing a process | Extract meaning and knowledge from data |
| Description | Data is created and managed to fit the needs of the application; typically, the creation of data is part of a process using the application. | The application is created and managed to fit the needs of the data, which may be (and likely are) created independent of the application |
| Illustrations | · Supply Chain Management (SCM)· Customer Relationship Management (CRM)· Content Management· Online transaction processing systems in general | · Big data· Data warehousing· Web search engine· Sensor based analysis· IBM’s Watson or Apple’s Siri |
Source: the Mesabi Group, November 2012
Setting a Few Things Straight
Some things to note:
- Application-driven intelligence tends to create, read, update and delete data to fulfill an initial purpose, such as a workflow process to manage order processing shipping and he collection of payments. In contrast, data-driven intelligence often takes already human or machine-generated data and uses it for a secondary or additional purpose, such as performing e-discovery on e-mail files. However, sensory (such as meter reading) information or machine/computer-generated information (such as logs or other information for the software-defined data center or software-defined storage) are created first and then analyzed by a downstream process (which may be in real-time) as appropriate.
- There is nothing new under the sun. Data-driven intelligence (such as statistical analysis using techniques like regression analysis, linear programming, and simulation modeling) have been around for a long time though more recently new concepts (such as data warehousing, online analytical processing, and data mining) have emerged. The problem has been that terms such as advanced analytics, business intelligence, and big data have tended to be looked at as valuable by businesses, but isolated IT islands. The totality of these developments do not get their credit for exponentially expanding the role of IT’s data-centric focus.
- Yes, there are hybrids. Data-driven intelligence can be inserted in an operational system (such as retail sales transactions) to check a credit card to see if it is fraudulent or at points within a supply chain process.
- Data-driven intelligence is an additive view that broadens our understanding and does not replace application-driven intelligence. Let software intelligences continue to multiply and add to our understanding and the value that we derive from IT.
What Can a Data-Driven Intelligence World View Do For You?
There are some key benefits from thinking with a data-driven intelligence mindset:
- Clearly, being able to distinguish between an application-driven intelligence solution and a data-driven intelligence application is important because the development methodology is different. Although both can use agile development methods, there are a number of key differences. For example, Ken Collier in his book Agile Analytics discusses the difference between agile development for data warehousing and business intelligence and that of traditional applications. All in all, you have to know what is different (skill sets, methodology, resources, time frame) for building or using an application-driven intelligence solution vs. a data-driven intelligence application.
- You don’t have to worry about trying to fit a project into a particular definition. Big data is a hot topic, but what is it exactly? Doug Laney (now of Gartner) introduced the popular concept of volume, variety and velocity. This is a very powerful and useful idea, but it does not precisely define what is big and what is little. Moreover, size alone does not determine value. Using a data-driven intelligence approach causes you to think about its overall value and the software technology that needs to be applied. If the project seems to fill the bill of big data, then call it that. If it does not but still delivers value, go ahead. Be benefit-driven – not label-driven.
- Recognize that collectively data-driven intelligence is the engine that makes more data-centric IT possible. This collective perspective encompasses all the pieces and gives a better sense of the total value that results when viewing the world through a data-driven intelligence lens.
Mesabi Musings
This is a short introduction to a broad subject and will require further discussion both from a general perspective as well as using specific product illustrations. Now, application-driven intelligence tends to focus on operational business processes. Data-driven intelligence tends to finally fulfill the needs of management information systems (an old term that fell into disuse because early OLTP and other systems were really not management information systems (MIS)) that aid in decision-making processes (which can be operational, tactical or strategic decisions) including both knowledge or information-acquisition as well as direct decision making. All in all, leveraging a data-driven intelligence lens in our world view expands our perception of where the role of data-centricity is going in IT.