Challenges Created by Data Pipeline Development Errors

Article posted on : link to source

Over the past few years, I have repeatedly written about the endless benefits of big data. However, I don’t write enough about the problems that transpire when big data projects are improperly managed.

A few years ago, a colleague of mine operated a company that was trying to develop a variety of AI applications for startups in the Bay Area. Due to the sophisticated nature of the problems that he was trying to solve, he relied extensively on big data. He and his team members were very knowledgeable data scientists and web developers, so they were certain that the solutions they developed would be highly useful to their future clients.

They were in for a rude awakening when they finally pitched to a growing company in San Francisco. They had a presentation with the company and showed a demonstration of the final application. Unfortunately, the company swiftly rejected the proposal. The harsh reply indicated that other potential clients would probably be equally disinterested.

My colleague’s team reached a disheartening conclusion. They developed a poorly thought out big data application that nobody would purchase. Five months of their lives and over $10,000 was wasted.

What went wrong? The biggest mistake they made was delegating all …

Read More on Datafloq