I’m convinced that social media and big data are two of the really biggest things within Business Intelligence the next few years. There are some good reasons for that:
- Social media has reached a critical mass. Facebook and Twitter would be on the top 5 largest countries in the world list (based on population), if they were countries.
- A lot of research progress has been made on automated analysing of blog postings and other forms of human created texts.
- Data Mining is also becoming more and more widely available.
- To stay competitive, companies must understand more about their customers and the world around them. The findings in social media will predict what you are going to see in your order books and even later in your financial results.
- Organisations like United Nations have started to automatically monitor news and social media, with the goal of finding out about crisises in real time. See here: http://www.unglobalpulse.org/
- The growing amounts of data requires increased computing power. However, we cannot any longer rely on hardware improvements to rescue us. In fact, CPU clock speed has stopped increasing 5 years ago. It is the end of the era of “don’t worry about CPU power”. Much like we now have to think about sustainable energy sources and using energy efficiently, we will need to build “sustainable software”.
- Social media produces huge amounts of data.
- This means that we have to scale out (use distributed computing) instead of scaling up. We need architectures that can handle very large amounts of parallelism.
- Platforms for scaling out (distributed computing), like Hadoop, have been released as Open Source. On top of Hadoop has also been built new databases like Cassandra (with good scalability and fault tolerance), HBase (useful for very large tables) and Hive (a scalable data warehouse platform). These can run on cheap hardware and are freely available.
- The worlds of traditional databases and big data are meeting. Microsoft has announced that they are integrating Hadoop and SQL Server 2012. A first preview will be available during 2012.
- The cloud fits very well with social media monitoring and big data. The whole architecture behind the cloud is scaling out, which is what big data really is about. Also social media is very efficiently monitored from the cloud since the data is external anyway.
Is big data and social media monitoring only for the biggest companies and organisations? Certainly not. Even as a small organisation, you will want to look into the future instead of the history (like many current BI solutions that focus mainly on economics). With the cloud it will be affordable and available even to the small companies.
Still not convinced about the impact of big data and social media monitoring? Read this article:
http://www.nytimes.com/2012/02/12/sunday-review/big-datas-impact-in-the-world.html?_r=1
