We live in a Big Data world where information is growing at an exponential rate. Each day almost 5 billion pieces of content are shared on Facebook and Twitter. IBM reports that: “There are expected to be 1 trillion new devices connected to the Internet in the near future, which will help drive 44X digital data growth by the year 2020, 80 percent of which will be unstructured content and will require great effort to analyze.” 

The exponential growth in Big Data poses significant scalability and performance challenges for traditional data analytics, social media monitoring and business intelligence solutions. These solutions which have been architected to index, store and process the full firehose of Big Data will need to expend significant capital resources (e.g., storage, servers, infrastructure) to simply maintain their core service offering. Meanwhile, the response time required to analyze and process Big Data will continue to increase creating performance and usability related problems.  

At the same time as the explosion in data is occurring, there is another trend that has huge implications for Big Data. It should come as no surprise that in our fast-paced, multi-tasking, micro-blogging obsessed world, the shelf life of information is shrinking rapidly. In fact, recent studies by Bitly confirm our own research that the half-life of social content (i.e., half the clicks or engagement social links will ever experience) is about 3 hours.

What this means is that the value of information is shifting to real-time. The more timely the information, the more relevant it is. Conversely, the value and relevance of data erodes quickly, in a matter of hours. As an example, a shared link from 30 days ago is far less valuable in terms of its potential impact on user engagement and decision-making than a similar link that has just been shared.

Herein lies the Paradox of Big Data. Just at the moment when data is exploding, the value and relevance of information is diminishing with every passing second.

As a result, a moment of change has arrived for Big Data. Today’s analytics solutions need to scale with the growth of the Internet, identify relevance in real-time, discern trends from incomplete or partial data and be able to predict the impact of information as it happens. 

This is precisely the approach and solution we developed for TrendSpottr. A real-time, scalable data analytics service that that analyzes events as they occur to identify and predict trends from real-time data streams.

As noted during GigaOm’s recent Structure Big Data Conference: “With data science, the moment of change has arrived, and companies that will succeed will be the ones that develop tools to enable that real-time data.”