I recently posted about the challenges in data engineering hiring after talking with Andreas Kretz, founder of Learn Data Engineering Academy, who mentioned that many data engineers are either looking for new jobs or hoping to move into higher positions. At the same time, tech companies say they struggle to find the right candidates, even with hundreds of applications. This started an engaging discussion on LinkedIn, and here’s what I learned from the valuable insights people shared.
International Hiring and Salary Mismatches
One common issue raised was the difficulty of hiring international candidates. George Firican said, “I wonder if there are a lot of international applications in general and then it's hard to resolve the work permits situation.” Remote work has opened more global hiring opportunities, but companies still face hurdles with international hires.
Salary mismatches were also a big topic. Kristen Kehrer shared, “At least in the data science world, it looks like salaries have decreased from previous years. So although there are openings, the number of desirable openings is much smaller.” Terezija Semenski added, “From what I keep hearing, a company is trying to hire a senior by offering a salary for a junior.” These mismatches can push away experienced candidates who feel undervalued.
Another challenge is the amount of interest in data roles. Andreas pointed out, “The hype is over, plus supply and demand… Easy apply is a huge problem here. It was supposed to make applying easier, but now everyone is just applying to thousands of jobs.” With so many people applying to jobs at the click of a button, hiring managers have to sort through too many applications.
Unrealistic Job Descriptions and Automated Filtering Problems
The unrealistic nature of job descriptions was also a common complaint. Colleen Tartow said, “There are cases where the employer doesn’t know what skills/experience they need for the data engineer role, so they dump everything in a job description.” József Vass agreed, “Unrealistic job descriptions turn off great candidates… companies should keep postings simple.” Long, demanding job descriptions can scare off qualified people.
Automation in hiring brought up other concerns. Muhammad Abbas Ali explained, “So much confusion on how AI and ATS play a role… there is a big issue, and most of the posts you see on LinkedIn are about poor platform reach and auto-rejection.” With applicant tracking systems and AI filters, companies may reject good candidates too quickly based on strict algorithms.
Broader Skills Needed and System Challenges
There is also a need for engineers with broader skills. PEDRO CARDOSO pointed out, “There are WAAYYYYYY too many ‘artisan-trade-minded’ data engineers who can do some or many individual things and not enough ‘general-contractor-minded’ DE's.” He said that companies often want engineers who can manage a project, not just focus on one technical skill.
Some saw these issues as part of a broken hiring system. Wendy Turner-Williams said, “The recruiting pipes are broken right now on top of unclear R&R, poor pay, etc.” From unclear job roles to lower pay, these factors create a system that doesn’t meet the needs of job seekers or employers.
Finally, Kevin Petrie added that demand for data engineers is still high, especially to support AI and ML projects. He shared, “I believe that overall data engineers remain very much in demand… in large part to support AI/ML initiatives.” Despite these hiring challenges, companies still need data engineers as AI technology grows.
The disconnect in data engineering hiring comes from many areas: international hiring issues, salary mismatches, complex job descriptions, automated filtering, and the need for broader skills. To solve this, companies need clearer job descriptions, better expectations, and smarter systems to find the right candidates.
I also highly recommend that people upskill to stay relevant in the job market. Andreas Kretz runs Learn Data Engineering Academy - check it out for yourself.