Cutting through the complexity

Big data analysis requires high-level talent – no wonder there’s a shortage of the right skills says Yves de Montcheuil, VP Marketing, Talend

New technologies come and go at an alarming pace. As a result, there is often a time lapse between demand – and having the right skills and experience to fulfil it.

Big Data can influence decisions

In some ways big data is no exception. Already the term is used in the media and mainstream business, surpassing even the mobility and social media as a discussion topic. TV programmes have been based on it and gradually even the general public are recognising how it could influence how we make decisions and view the world.

And yet, analysis of big data is complex. It needs the best and brightest brains to work on the mathematically challenging calculations and algorithms involved. Even among the small group of data scientists who have the capabilities required, only a minority have the necessary experience. Technologies to manage and analyse big data are still young and only a few have been exposed to the tools, techniques and frameworks for any useful amount of time.

Also, broadly speaking, big data requires skills which are often in short supply. It takes deep domain or industry knowledge and an understanding of business at all levels. But it also needs to be blended with advanced computer programming skills. Even at a senior level this mix is much sought after and, therefore, expensive.

Specialist knowledge within big data is hard to find

Consequently, there are few professionals with the right specialist knowledge on the job market – and those that do have the skills can command huge fees or salaries. So, businesses need to develop expertise internally, training their staff in these areas. But they then run the risk that these employees will be poached by competitors, once their training is completed.

So what’s the next step? In fact, new tools are emerging all the time to help businesses master the complexity of this big data environment without becoming highly-technical specialists.

One of the key processing frameworks for big data, Hadoop, is still relatively new and very few IT professionals know how to use it. NoSQL, the database technology used on top of this is not yet widely understood and even experts in relational databases won’t necessarily be able to use it.

So the best new solutions are those which hide the underlying complexity, including the coding in which it is written. This makes data analysis less difficult, turning mere data scientists and anyone with data integration skills into ‘super’ big data developers.

New tools are extending the power of big data

These new tools enable users to work in an intuitive visual development environment and to build data flows graphically. In this way, raw data is translated into a much more accessible format. They also help to bridge the gap between data scientists who are experts at statistical analysis, but have few programming skills and programmers who struggle to interpret the data scientists’ requirements.

The latest data integration tools take this process one stage further. These combine two key functions – the visual development environment which allows a standard development team to do most of the development work, thereby freeing up senior expert developers to be more innovative and create new intellectual property (IP) for the business.

In other words, the technology is what’s keeping everyone happy. It simplifies the process for those without in-depth programming expertise, while enabling highly-skilled programmers to make optimum use of their abilities. Top-end consultants are only used as needed, leaving slightly more affordable employees to carry out the rest of the work, so helping stretch the budgets.

In a survey carried out by Talend last year, two of the top three barriers to big data named were ‘budgetary restrictions’ and ‘skills shortages’. Once the obstacle of complexity is overcome the latter should become less of a problem, opening up the big data promise to an increasing number of businesses.

Only then will they discover that technology is only the foundation of big data analysis. The real prizes will go to those who use the results creatively and wisely.


Visit the Talend stand at Big Data Analytics 2014, Hotel Russell, London. The event, taking place on 13th November, will bring together the brightest minds from the world of data science and analytics in a highly engaging conference format. Contact us to find out how you can gain your place as a delegate.