HOW TO START A CAREER IN DATA ENGINEERING

5 things you should know to start your career in data engineering

You must be a strong developer

Everyone agrees that you need strong developer skills for a data engineering job.

"You'll have to write scripts and maybe some glue code. Everything is code now: infrastructure as code, pipeline as code, etc. Courses are OK but nothing beats real-world experience.”

Data Engineers also need a love of or at least an interest in data, in finding patterns in data, otherwise they may find the work boring. Also, Data Engineers have to like and have the ability to create systems that are difficult and complex. Big data projects are 10 times more complex than small data. So it's a love of data combined with a love of programming to create data pipelines.

In addition to being comfortable coding, "You have to have the operations mindset that uptime is critically important. You have to be careful how you build your infrastructure for reliability, so that any changes won't break any of the pieces. Devops experience is very valuable. And you need DBA skills."

You need to know about a lot of technologies

"A data engineer has three main duties:

  • To ensure that the data pipeline – the acquisition and processing of data – is working.
  • To serve the needs of internal customers – the data scientists and data analysts
  • To control the cost of moving and storing data

"The critical skills are SQL, Python, and R, and ETL methodologies and practices."

"Just understanding the foundation of a language will allow you to work at any company."

But there's more to being a data engineer than knowing SQL and programming, "A qualified data engineer's value is to know the right tool for the job.

"The best options are to get professional training, read books, and work on big data projects. You have to both internalize the knowledge and practice it. If you've learned passively but never practiced, you won't be able to code a project, and that will come out in an interview. Practice practice practice!"

Most companies standardize on a single vendor's suite of cloud computing services, so Ng recommends, "Go deep on one of the big three: Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure. Understand how their service offerings can be used as building blocks for highly scalable and available data pipelines."

Experience beats education

How do you pick up all those skills? Typically, on the job. Everyone we spoke with told us it wasn't necessary to have an advanced degree to get a job as a data engineer.

Education has its place but experience makes the best engineer. There are people in customer-facing areas like support and customer success, if they have an interest in programming, move into a data engineering role. Most support people are so in tune with what customers are asking for in terms of custom data integrations that it's an easy sidestep for them, in the sense that they understand the use case and the thought process behind it. Generally, it's people who have a hand and the experience in the day-to-day use of the data."

"If you have only a bachelor's degree and want to get on a data engineering team, We recommend you make a personal project that shows what you can do, not just what you can talk about."

Social and communication skills are important

"Aside from hard technical skills, a good data engineer should also have certain soft skills and qualities"

  • Attention to detail: Data quality is extremely important when building pipelines. All downstream work is only as good as the quality and integrity of the data you're moving through the pipeline. You have to really care about and appreciate the "garbage in, garbage out" principle.

  • Appreciation for clean design: There's never one way to design and build a pipeline for moving data from point A to point B. A good data engineer should appreciate the elegance of clean and simple designs that are not over-architected.

  • Good communication skills: A lot of times there's a discovery period when you start to design a pipeline because your data is sitting in different silos that may be located in different areas of your infrastructure. You'll have to talk to people to understand the playing field before you design anything. This discovery step isn't easy, but it's a requirement for making sure you're building the right thing. A good data engineer should find satisfaction in helping their customers solve painful problems.

  • Excitement about working on back-end systems: Data engineers don't build a lot of UIs and front-end apps. They work deep in the systems stack, and in many cases they won't be able to point to something shiny and say "I built that!" You have to be OK with that and take pride in being the hero behind the scenes.

  • A love of learning: This isn't really data engineering-specific, it's just how the software engineering world operates. You have to keep up with new libraries, frameworks, and tools out there in the community. Things change fast and you need to be able to quickly understand, evaluate, and learn new tools if necessary.

The job is changing

While all of the above is important, data engineering is an evolving discipline.

We're seeing a shift to data services, which means a change in the job of the data engineer to delivering data services. When it was expensive to store and process, data was siloed. Few people had access to it, and it was hard to make changes to it. With the cloud, it's now cheap and easy to store and process data, so everyone is putting data into cloud data warehouses and allowing anyone in the company to connect to it. Data engineers are still responsible for the performance of that infrastructure.

As data becomes even more ginormous than it is today, it becomes more about infrastructure and sustainable processes than it does about single processes. The job is growing toward more being able to maintain things.