Breaking the silence

Hadoop World San Jose 2015 helped me break my silence- a hiatus from self doubt. After a great and hectic experience learning about the successes achieved in the face of challenges, roadblocks, doubts and confusion; I feel invigorated and primed with optimism.  The energy of the silicon valley, the free flow of ideas, the honest articulation of problems, the candid questioning of assumptions has definitely had an inspiring effect on most folks at Hadoop World.

From the high fives of business executives and wise nods of data scientists to awe-filled exhilaration of IT implementers and managers, the variety of problems solved continues to grow.  I am sure each attendee leaves with big data in his brain and will process the information over a period of months.

The Story

The blind men exploring the elephant stuck a resonant cord and succinctly summarizes my key takeaways at the conference.

I had heard the story as a child and had in fact told it to my son with some variations of course; to illustrate the concept of “viewpoints”. A few wise blind men lived in a village together and experienced life together. Having heard about a majestic animal called the elephant they discussed and debated on the form this beast. Not coming to a consensus on what an elephant is they decided to explore one at a nearby temple. Each went to the temple separately with a visually endowed shepherd, prayed to the local deity, touched the elephant and returned to their home. One evening they gathered to discuss their findings and were bewildered. An elephant is “a huge fan”; “a sturdy tree trunk”; “a stout rope”; “a wall”; "a sharp spear" and “a strong snake” were the real physical experiences of the majestic beast.

The Moral

Hadoop seems to induce a similar bafflement for Business executives, IT managers, IT implementers and Data Scientists. The success stories, use cases and recommendations still fall short of explaining the utility of the beast that threatens to help you rule at its best or trample you just as easily.

Under these circumstances, I believe there are opportunities and real benefits- though not clearly or easily available to any of the audiences. Big data is HARD and complicated and requires all the stakeholders work collaboratively, mentoring each other to find the best application to solve.

Making sense of it all requires everyone to agree that the fundamentals of science, management, mathematics or your functional area are not being disrupted by technology. Technology is merely enabling more possibilities through more options and it is up to you all as a team to make the most for your organization or cause. I believe you don’t need data to answer all the questions- simply asking is sufficient in some cases.

The Examples

“Why are taxis not available during rains in Singapore?”  

Millions of Singaporean dollars spent with a consulting company, technology, IT infrastructure etc. got an answer that could easily be the deduced by asking a “Taxi driver” and a “Taxi operator”.

AirBnB figured out their customers needed a “Discovery” feature by using a elaborate A/B testing experiment that was eloquently summarized in a 40-minute session detailing the data, the model, the testing and the insights. The Q/A went on to further probe and clarify assumptions, accuracy, consistency and a host of mathematical terms. A good old fashioned product manager asking the question to either a customer or a visitor that did not convert would have unearth the same insight at a fraction of the effort.

Or a classic example from Microsoft on "Connected cows".

The true value of big data comes from scientific curiosity, logical data collection and analysis, building models, defining metrics for a solution and measuring success in the future. There are more than one ways to solve problems and businesses look for the most cost effective actionable method. A concept the data scientists seem to neglect in the pursuit of rationalizing the value of data sciences. Data sciences can solve real problems- there is no question about it.  But a truly collaborative approach that helps everyone learn can lead organizations to scale even greater heights.

The Players

Much was said of the roles involved, influenced and impacted as a result of big data, I believe in the roles but differ in their value and hence the following guidance to my colleagues in the field.

• The Business Managers should help define the question and continue to mentor IT and the data scientist on exploring the best, cost effective and speediest course to get to the answers.
• The Technologist should own the requirements, be thorough and include both the data scientist and the business stakeholders in helping understand execution and operations.
• The Data Scientist should lead in the enabling the business question with a scientific approach spanning data, functional and technical methods to define the best solution.

Together you have to push the boundaries of the “Art of the possible” for your business and there is no silver bullet for it.  Yes you will need to expand the data collection, data storage, analytics and operationalization- in time you will have a data-driven enterprise but there is no one way for you, your organization or your industry to map your way. You will define your own path and can be successful if you build on the core foundations of scale, flexibility and simplicity.

Bottomline 

Transparency, curiosity and scientific inquiry have helped mankind overcome great challenges and will continue to do so in an big data venture. You might need help and sometimes an independent viewpoint can help.