The Evolving Role of Data Scientist'18

“The world is one big data problem.” - Andrew McAfee, Center for Digital Business, MIT/Sloan 

“Without Big Data, you are blind and deaf in the middle of a freeway.” - Geoffrey Moore

Data Science – once a new and novel niche in the enterprise – has now emerged as a driving force in the enterprise: a key component (if not the key component) of BI, overall strategic planning, and knowledge gathering.  Let’s look at the timeline.

 It started out and had its infancy in the domain of academia, especially the social sciences. Academic researchers developed skills and algorithms for detecting economic, housing, and medical trends in specified populations. Their work was generally funded by grants. Their focus was on soft, non-commercial questions regarding the quality of life and related issues, these questions addressed using traditional structured database models. 

Following this, Data Science moved into corporate culture with the simple role of solving “point problems” – specifically-enunciated questions needing precise quantitative answers, these answers nearly always derived from one narrow, pre-defined source of structured data. 

With the dawn of Big Data capabilities, Data Science emerged as a broadly defined endeavor breaking new ground, exploring the rapidly exploding and constantly flowing fonts of unstructured digital information in order to generate new questions, define new pathways of BI, and become a force unto itself creatively driving and expanding the knowledge-base of the enterprise. And what of the future? Olly Downs, chief Data Scientist at the analytics firm Globys, says: “The way the Data Science role is migrating is that you don’t just need to know your science and need to know about data – you need to now understand about technologies and the technology evolution to help you shape [strategy].… There are some big changes happening or about to come about … in how computation can be done and what sorts of algorithms are now scalable. Problems that were exponentially hard to solve or algorithms that were exponentially hard to run but would really solve a problem correctly for you – a Data Science problem, or an optimisation problem – it’s going to become a reality that you can perform those computations. Now you need to understand how new types of computers work, in addition to understanding how new storage paradigms and data-representation paradigms work, and also still have at your core the understanding of statistics, machine learning, and data science.”

Downs also sees the practice of Data Science spreading well beyond primary questions of markets, products, and customer service, and expanding dramatically to address more and more questions related to in-house efficiencies and cost-cutting. At such firms as General Motors, Data Science is already as much about safety, production practices, quality control, and human resources issues as it is about marketing, advertising, and customer demographics. 

Another major area of new innovation will be the art and tools related to performing Data Science on information generated by the so-called “Internet of Things” and wearable technologies. Here we are talking about tools and skills to help us manage and wrangle data related to ubiquitous location awareness along with location context. As Downs says: “It’s more about the [location] sensing going everywhere with people and with physical things and [the proliferation of] that. It is about contextualizing that in time, which we’re kind of used to, but also in physical or virtual space.” 

Throughout all of this, the definition of a Data Scientist – that rare Unicorn of the business world – will continue to be a creative “mash-up” of skills: hacker, analyst, programmer, statistician, communicator, market researcher, and – at times – clairvoyant. In the final analysis, few fields are as well positioned for exponential growth as is Data Science. The flood waters of Big Data will never stop rising – and with them both the promise and the problems of their enormity, velocity, and variety.

In other words, the promise and role of Data Science shall progress in step with the advance of technologies for generating, harvesting, and manipulating unstructured data. And these show every sign of conforming to the famous “Moore’s Law,” originally promulgated decades ago by Gordon W. Moore, co-founder of Intel and Fairchild Semiconductor, which says, generally, that the speed and capabilities of data processing roughly double every two years. Thus far, Moore’s Law has held up. 

Ernest Dimnet has commented: “Too often we forget that genius, too, depends upon the data [knowledge] within its reach, that even Archimedes could not have devised Edison’s inventions.” Thus, as the capabilities and challenges of data technology advance, so will the evolving and increasingly vital role of the Data Scientist.