What is Big Data, Anyway?

>What is Big Data, Anyway? Big data is nothing new. What’s new(er) are the tools are technologies that are contributing to the democratisation of big data (see Tools and Technologies of Big Data for some technology basics on the Talend website).

While most organisations have been amassing big data for years, it is worth noting that some of them have been “doing” big data for years. By “doing”, I mean processing, leveraging, analysing, mining – anything else than just storing it. It’s one thing for Wal-Mart to process in excess of one million transactions per hour and store this historical data, or for the US Census Bureau to collect demographics on 300 million Americans – and it’s another thing for these parties to actually process it, massage it, and extract actionable information.

As we have all experienced first-hand, or heard of, a number of real-life (or rumoured) big data use cases have been in existence for years. Among some obvious and famed “big” use cases:

Credit card fraud detection: anyone who travels a lot for business is often led to some unusual spending patterns. Like buying French train tickets online from a US IP address, minutes after ordering a Kindle book and just before paying for an intercontinental flight. So when my credit card company calls me to check, I can hardly complain (and I am actually glad they call).

Retail: the decades-old beer and diapers mining story (allegedly a rumoured one), recently supplanted by the teen pregnancy one (it will probably take time before we know if this one is real or not!). Less prone to urban legend, the long string of coupons that print out at the register reminding you that you need dip for these crisps, or conditioner to go with this shampoo, or mulch to help your flowers grow, are the result of big data analysis.

Yield management: with airfares that vary on an hourly basis and hotel room prices continuously adjusted, the travel & hospitality industry has been a master of shifting through massive historical data to get the best possible price out of a finite inventory of airplane seats or hotel rooms – without leaving any unsold.

What are the commonalities between these “big” use cases? Credit card companies, retail chains, airlines & hotel chains are for the most part large and wealthy organisations. They did not wait for the recent big data frenzy to “do” big data. They invested a lot of money into hardware and software, hired the best talents, and built their infrastructure and algorithms, by trial and error. And for the most part, they have reaped the rewards of this investment.

What’s new is the democratisation of big data. In the Talend blog, we’ll be reviewing some of the less “big” use cases of big data, and how big data can help smaller, less wealthy organisations. To find out more why not reserve some time with Talend at our stand at Big Data Analytics 2012 by emailing mcarter@talend.com.

Yves de Montcheuil – VP of Marketing – Talend