Goldmine or Money pit?


By Patrick Condren, Ph.D., Chief Information Officer

To even ask the question “Is Big Data a Goldmine or a Money Pit?” is almost heresy in business and in particular in IT, these days. However, it is a fair question given the enthusiastic headlong ‘gold’ rush by so many companies to ‘get into big data before it’s too late’. But is ‘big data’ the right answer for all businesses, and what is ‘big data’? Is what you really need ‘business insight’, but almost by osmosis, now believe that this can only be delivered by ‘Big Data’? Is the ‘Big Data hype’ forcing companies to rush into a solution, before defining the problem that needs solving?


Big data started life as meaning BIG data, on a scale of Google or Twitter, the ability to use truly massive global platforms to give us ‘indications’ of trends on a global or national scale in almost (or actual) real time – essentially from unstructured data stores.


Big data informing soccer strategies!

The Hadoop people gave me a great demo (at the Gartner CIO Symposium in Barcelona) of what they do for their sponsored global soccer team franchise ‘Tottenham Hotspurs’. The concern from management was that the fans would be angry with the sale of Gareth Bale for a record fee. Hence the request for the ‘big data’ team to come back with real data, that could dynamically influence the communication strategy (allow instant spin doctoring). The ‘Big Data’ team analyzed all Twitter, Instagram, Facebook etc., updates and tweets along with press publications and converted this data in to ‘sentiment’ indicators. As it turned out the fans were not ‘angry’, but hopeful that the money would be used to strengthen the team to pursue soccer glory. This ‘insight’ allowed the tailoring of communications and actions in order to satisfy fans and make the most out of the financial benefits and quickly recover from a very significant (and potentially disruptive) business event. A ‘Big Data’ win!

Be realistic about big data

Big Data, is a term and function that is almost at the peak of its ‘hype cycle’ ©Gartner. The Gartner ‘hype cycle’ is a method of measuring/tracking the life-cycle of new and emerging technologies. It is impartial and very well researched by true experts with no vested interests. The cycle essentially starts with the launch of something new, it then gets to a ‘peak of inflated expectations’ (hype peak), then many people get disillusioned with it (big dip to the bottom of the graph – the trough of disillusionment). From there it finds its best positioning and value propositions and climbs back up in popularity with the maturity of realistic expectations attached (from ‘slope of enlightenment’ to ‘plateau of productivity’). Big data is almost at the top of its ‘hype cycle’, inferring that it will next undergo a ‘trough of disillusionment’. It follows that we should see ‘Big Data’ go from being ‘the next big thing’ to ‘tanking in the ‘expectations ratings’ and seeming to be a ‘disappointment’. In truth Big Data will be as useful then as it is now; it is our expectations that will change. This in itself is not a reason not to have a ‘Big Data’ activity, but it is a time to be cautious and mindful, to set the right expectations based on reality, to be focused of the desired outcome and the ROI. Certainly there is value for many companies in Big Data, but is it right for yours?

The speed of change

The nature of big data means a considerable shift in traditional methods of building reports and analytics. The speed of change and potentially the scale of the data can overwhelm an IT organization. The collaboration of business SME and IT becomes critical to understand context, and therefore derive ‘value’ from the massive volume of unstructured data. The shift is from ‘truth to trust’, and the traditional TCO and ROI models are not effective measures. The ‘value’ from the big data analysis tends to come sporadically and often serendipitously, by trial and error (almost), therefore traditional planning, progress reporting (the delivery phases) and milestones can be challenging – it is more R&D than traditional software engineering. These things (added to skills required and cost of deployment) are not necessarily negative things, but need to be accounted for in expectations around deriving value from ‘Big Data’.

Meaningful use of big data

The real question to you is, what do you want ‘Big Data’ for? In technology most people want to work on ‘Big Data’ for their CV (resume)! Often times that desire can build an infectious enthusiasm to do ‘Big Data’ without the usual scrutiny (partly because the business wants it for the same reason too). In business there is a fear that this is another ‘dot-Com’ that has to be adopted ASAP. In the real world the question has got to be two fold,

  • Are your data and insight requirements ‘big’ (in this sense), and
  • What are you looking to achieve?


My belief is that in all cases what is actually required is ‘Insight’, be it customer insight, market insight, financial insight etc. And if this is so, then there are many options and blends of options that need to be considered to get the right Insight for your company. I’d caution using the term ‘Big Data’ as it may seem to predetermine the solution path, when what you really need is a conscious consideration of your ‘Insight’ needs, and then build whatever solution (or solution blends) are required to achieve that business need.

Dig deep

To finish on the question that started this piece, “Big Data: Goldmine or Money pit?”, lets us expand on the analogy. Miners do not just grab a shovel and start digging up the ground that they are standing upon, this was something of a failed strategy in 1849 (Goldrush). Mining is a risky business, with a high failure rate, but it can also be very lucrative. Miners carefully gather as much information that would indicate the presence of whatever ‘nuggets’ they wanted to mine. Then the best miners secure the correct tools and safely mine, looking for the buried riches. Sometimes with success some times without, but with the right processes and preparation, they were much more likely to succeed. We all know about the big successes, but we know very little about the huge amount of failures. Big Data is very similar, preparation; planning, intra-business cooperation and understanding of the desired outcome are the critical factors that will create success. The mother lode may well await your mining but, what do you want from it, the choice of tool and where to start will be the most important decisions that you will make in this venture. Mining is an expensive business but can offer great reward. By all means, mine away but be cognizant of the best strategies to be successful, the cost, the hype, the opportunities to collaborate, and if need be when to call to call the streak ‘spent’.

Protecting the data

A note… mining requires a license to mine. If you are mining ‘data’ you will need to be sure that you have the rights to that data, for the purposes that you are using it. This is true under PI, PCI, SOX, Safe Harbor, Basel et al. More and more the lines are blurred and in the rush to land a ‘winning streak’. There is a inherent danger that our use of data may cross legal boundaries and even national borders. There are more and more cases of data security and privacy breaching, and growing fines and penalties daily, the impact on corporate reputation can be extremely damaging. It is not that difficult to navigate a safe path, but it will be a feature in this space if companies do not put in place the proper arrangements, securities and ensure the correct rights and uses are adhered to.

So, Big Data: Goldmine or Money pit?

That is largely up to you! You may well find your mother lode, assuming that you will be secure in your license, mindful of your approach, have realistic expectations, allow a real agile style IT/Business collaboration without the traditional structures and measures. As with any mining you cannot predict when you will hit a streak, or even that there is a streak, but if you do then rewards can be enormous. And remember….


‘Audaces fortuna iuvat (latin)- Fortune favors the bold.’