How does big data work

Simplistically speaking, big data starts with data that is already collected, or is being generated or a combination of both. Which is collected and brought together in ways not possible earlier to be analyzed to extract some learnings.

this is an iterative process that is repeated several times to  get finer and deeper insights that actually can make a difference in a business.

Big data vendors approach this process from various angles. IBM, Oracle, Microsoft have a good handle on the data, analysis and processing model so they push solutions that apply there. Hadoop vendors are good at the collect and converge and are building newer tools for analysis so like to come at this pie from that view. Others start with data- specific to certain applications like Machine data, web, mobile, geo, financial, geological, remote sensing, weather... you name it and they know how to handle it. build a few collection, convergence and analysis modules and you have a workable big data solution that at least provides directional guidance if not actionable.

IDC, ESG, Forester, Booz Allen Hamilton put their own spin to this with big data workflows, converged infrastructure or third platform or the data lake or a bunch of other catchy terms. This is basic science and my friends the data scientists would say we have been doing this in our own ways first on note pads, then on spreadsheets then on databases then on data warehouses and more recently on massively parallel processing data warehouses or data appliances for a while.

The only difference is the scale and the tools to interpret the variety of data has expanded in scope.

In the next set of blogs i will explore four strategies and tactics that help work your way through the big data minefield where we straighten the iterative process into

• Acquisition
• Storage
• Analysis
• Application

Strategy 1.  Start with systems that acquire the data and move towards application. A path suggested by Oracle, Microsoft, SAP, Teradata etc.

Strategy 2. Stitch together a system that connects the four components by partnering with different component vendors. A path taken by various consortia like EMC, AWS and others

Strategy 3. Start with application  through  data exploration and contextualization applications like Splunk for machine data and Palantir for fraud detection

Strategy 4. Build the acquisition, storage and analysis engine with Hadoop and connect it to the applications through custom tools. A path recommended by Cloudera, Pivotal, hortonworks and MapR

What can you really do with big data?

If any analyst, guru, visionary was asked the question; we would get an answer simillar to

 "transforms a business from a data-aware organization to data-driven organization"

This is management speak for good science: an organization does things, that generates data, which gets analyzed so you do things better, different, more or less based on what the effects are.  This paradigm does not change; call it data-driven, analytics-driven or a whole lot of other names; you just have more tools to work with so can see a bigger rearview mirror better.

does that make big data less useful? no- just make sure you know where you are and what you see.

The Big data pie

Following the money patterns is interesting, after much thought and staring at spreadsheets; the breakdown is

1. Professional services (35%): not a wonder since Apache Hadoop is free; its the vendors support that is the biggest slice of the pie
2. Compute (17%) is after all commodity hardware;
3. Storage (14%) was a surprise. Its the data that is growing but even with the falling storage costs.. this comes in a distant third.
4. Apps and Analytics (13%): the splunks, tableaus of the world are very small. compared to the large and growing pie.

Did I miss the professional services boat? probably, but the real work is yet to be done.

I have heard the toy elephant story many times, but seriously the numbers tell that the real elephant is IBM- well they took the elephant sized pie.. the rest well seem like crumbs.

Big data is big money

We are creating the data, we are storing, slicing dicing and presenting it but where is the money?

Big data is big business.  $18 billion last year, expected to grow up to $28 billion this year and keep on growing upto $50 billion in 2017. and who takes the largest slice of the pie?

Professional services take the biggest bite of the big apple. All the hadoop vendors are slightly under the #5 on the list Teradata which is less than half of #1 IBM.

Making the noise is one business and money is another business..

Data needs to be processed

When you have a lot of stuff, you need to be able to know what you are looking for and then find it. That's where processing comes into picture; and its a huge business.

In the big data world, "tools" make a huge difference to get some use out of the large banks of information we are collecting, creating, duplicating and throwing in one huge heap. Tools fall into a few buckets but broadly

1. Organizing tools that help you to put some structure to madness- these force you to make a plan and stick to it.
2. Search tools to help you locate something specific in a huge pile- these work great if you know whats in the pile and how to look for it. 
3. Presentation tools that organize stuff, sort it and make it look like its all neatly stacked. 

We would love services like that for the physical world but the apps, systems that do just these basic items focus on specifics and don't cross boundaries. For example, there are business card managers, calendar managers, mail managers, news feed managers, photo managers/organizers, video managers/organizers. try to mix any of these up and you have a fruit salad. 

This is the big data opportunity and tons of companies are jumping on the bandwagon!

Data needs to be stored

We create bytes to revisit, relive, make sense of the events or experiences at a later date. Real, hopeful or anticipated, once we digitize something, its there to stay.

Stay

To keep bytes from losing, we employ ingenious strategies. We make copies, tag-em, bag-em, sort-em, order-em, file-em, or in many ways archive-em. Sending ourselves e-mails, burn DVD's, copy on USB drives, or store them safely on the cloud so that if we need them in the future, we will have them. 

By just trying not to lose we compound the problem of a large number of bytes we create.

Speed

We are a hugely impatient lot when it comes to browsing. When we say for instance, keep 1000 photos on instagram or icloud or facebook, we cannot wait a couple of seconds for the poor site to bring it up. Now to give us our pictures really fast, these sites do something ingenious, they break down the pictures into smaller blurry images and throw them at you so you have something to work with. 

You guessed it right- that added more bytes. now you had large crisp files to store, not lose them and keep them coming your way fast so you multiplied them up. 

Pry-proof

Some of us dont like others watching our pictures, especially while we are moving them over the internet- so we do something called encryption. This in effect adds a dark screen over your pictures while moving your bytes from one point to another.  Even though this is not a huge addition, you temporarily create a mini monster for a short duration of time, that is created, stored, transmitted in a pry-proof manner.

So now you see, we have built a capacity to create digital monsters, fearing losses, we pad them to weather time and prying eyes; to overcome our impatience, miniaturize them. Now our already big and growing data got bigger and grew some more.

It all starts with data.

Let us dig a bit deeper into the first step of big data - the creation of the bytes by most of us going about our ways. This is by no means comprehensive but you should get an idea of the deluge we are responsible for.

The Eyes

The biggest source of bytes is the ccd chip; a small device that captures images in a device/feature called a camera.

Why do we use this device so much? well someone spent a lot of money to find out in 2005 - which makes for interesting reading. These instruments are everywhere, very easy to use and we might regret not using them later so we use them. I have so many great memories not on film or digital form that i wish i had; but that is a different story.

The point is we just make lots of memory in digital form and as it gets easier to make more crisper memories, we go right ahead in burst mode. Burst mode is taking 100's of pictures with the hope that one of them will be perfect- emotion, angle, light and no nosy noise in the background. Ahem, we then conveniently forget to delete the nosy and now have 99 nosies along with one perfect.

Just these habits with an ever increasing megapixel capability and we have just managed to create a monster album.

The Multipliers

Now take the huge bits of pictures, videos and then, sync them with icloud, box, skydrive, google drive, instagram, facebook and a bunch of other places popping up all over the place- with sync and share capability and we just multiplied the data.

Imagine you are an apple fan, you have a ipad and a iphone (die-hard fanboys have mini's, retina's imacs etc). You take One picture of a soaring eagle on a crisp sunny day and the picture is replicated across all your icloud connected devices. You can conveniently show off your great luck in witnessing the glories of nature with your device closest to your reach at all times. 

The price of convenience? A whole bunch of hard disks that keep multiple copies of your pictures on icloud or any other sharing service you subscribe to. I have heard they keep anywhere from 2-6 copies.. folks that know for sure- ping me on this. 

So every picture taken * the number of devices connected * the icloud storage service = bytes now in this world.

The Act

With the ubiquitous internet, we act a lot- like visit websites, click on a bunch of interesting items and so on just because we are browsing or surfing. In the simplest form a webpage has some text, a logo, some pictures and hosted on a complex system consisting of network equipment, computers, databases and software that makes sure you get the page you asked for 90% of the time or more. One web page view can generate anywhere from 4-40 requests to fetch text, stylesheets, templates, logos, pictures, etc that touches at a minimum three electronic devices that make one line of note each saying you asked for something and you either got it or did not.

The Multipliers

In a simplest form, one request generates 4 requests x 3 devices that is 12 lines of notes also known as logs at a minimum on a simple but professional system. If you add on the analytics, caching, reporting and tracking each added feature starts  multiplying the 4 in the request with that number and you can see this adding up rather quickly. 

Considering that just Google handles over 100 billion searches per month each search carries at least 27 requests (on chrome per the netmon for my computer); that translates to over five trillion (5400 billion) lines of logs. Assuming no other shady tracking.  

The Shadows

Many internet sites install trackers on your computer that watch where you go, what you do and build profiles on where you live, how much you earn and where you spend money. This adds requests to every click, scroll of internet browsing you do.

Multipliers

I don't want to spook you as that is another separate discussion.

Bottom-line

Big data is created intentionally or systemically- call it machine data, human generated data or behavioral data, this is a digital footprint we leave just by being online. I wonder what is the carbon footprint of all this as the bytes are stored for all eternity?