Striim CTO Steve Wilkes was quoted in a Datanami article that discusses the growing importance and need of the data engineer, and why it’s so critical to developing and managing an organization’s data architecture.
While one of the buzzwords of today is “data scientists,” the article does a great job at highlighting just how necessary and important data engineers are to the growth and success of a business. It’s been documented that data scientists are a high priority, but it shouldn’t overshadow the fact that data engineers are also a well sought-after role.
Data engineers are essentially “plumbers” to an organization’s entire infrastructure, and play critical roles in providing data to the data scientists for analytical and transactional purposes.
One interesting aspect from the article notes that according to Glassdoor, there are 98,218 openings for data engineers, opposed to only 24,695; that’s nearly a 4:1 ratio.
Because of the shortage in data engineers, as well as the fact that more and more companies are working with its data in real time, in a streaming fashion, instead of throwing it in a data lake to operationalize later, data scientists are being tasked to have a say on the preparation side of data as well as the analytics.
Steve is quoted saying, “What we’re seeing more is the data scientist is getting involved in that initial data collection and starting to have more of a say in the data preparation side of thing – including doing in-memory data preparation, getting things in the right form, doing feature extraction – before it lands rather than after it lands,” Wilkes says. “People have been burned by [throwing the data into a lake]. They get very frustrated because of how difficult it is to get value.”
Learn more about the role of data engineers and why they’re so critical to businesses by reading “Why 2018 Will Be the Year of the Data Engineer.”