The AI talent crises is real — and unlike the dot-com era

For the past five years, I’ve been advising AI leaders at Fortune 1000 companies on how to recruit and retain AI talent. Because none of them have Google-sized compensation budgets, I have advised them to ignore the hype about talent shortages and focus on building the high-performing team necessary to achieve their business goals. The most successful companies have built effective teams by investing in their existing employees and recruiting from nontraditional regions and backgrounds. The least successful have copied Google’s job descriptions and sent recruiters scurrying around for Stanford and MIT PhDs.

In other words, I helped these companies avoid budget-busting salary costs through better recruiting strategies. But these tactics are no longer working. The AI talent shortage has evolved into a crisis, and every AI leader needs to begin resetting expectations with their leadership. 

In coming months, I will write more about strategies for developing AI talent. But I’d like to first dispel a myth: Today’s AI talent shortage is nothing like the dot-com talent shortage in the late 1990s.

What hiring managers tell me

Almost every large company has budgeted for additional AI investments in 2021. As the worst of the pandemic began to ease, AI leaders started recruiting talent to hit their 2022 business goals.

I hear consistent stories from many of them:

• We found an awesome data scientist but couldn’t afford her. She’s already making more than my boss.
• I finally got my HR department to agree to hire remote workers. But we still can’t afford them because everyone else is doing the same.
• We had a good data engineer, but he left when a bigger company offered a salary that was $90K more than we’re paying.
• I asked my data scientist to focus on solving my biggest business problem. He told me he prefers to do experiments, and he quit on the spot.

Under pressure to fill positions, many AI leaders are starting to settle on unqualified job candidates. But the data scientist whose only experience is online certificates and school isn’t going to create a system that can transform your business.

So, yes, the AI talent shortage is real, and it’s getting worse. How does it compare to the dot-com talent shortage?

The dot-com bubble and talent shortage

I’m old enough to have evolved from employee to investor to employer in the 1990s. Starting in 1995, infrastructure investments (telecom, energy, hardware) and online businesses exploded. The technology was new and complex. Large teams had to put in months of effort to get even basic infrastructure working. 

Large retailers tried to become e-commerce companies overnight. Media companies tried to move their content online. Financial services companies embraced low-cost transactions. Every chief technology officer panicked and threw money at the Y2K bug. Venture capitalists poured billions into dot-com startups that raced to build platforms and audiences through insane marketing spending. And so on. 

By 1998 the limiting factor for these initiatives was human capital. The hiring frenzy took off. Ted Leonsis remarked, “Our talent shortage is so severe that America Online will hire anyone who can spell computer.” 

What followed was two years of insanity. Engineers left traditional sectors like consulting and banking in droves. Salaries ballooned, and employees started demanding perks. It all ended in March 2000 when the NASDAQ started falling. Companies shut down, projects were canceled, and massive layoffs ensued. 

The appeal of comparisons with the dot-com era

The dot-com talent shortage was temporary. Although the NASDAQ bubble happened from 1995 to 2000, the hiring insanity reached a fever pitch only in the final two years.

Many AI leaders are hoping the same pattern of short-term hiring problems will play out for AI. It won’t.

The AI talent shortage is different

The current situation is fundamentally different from the dot-com hiring crisis. Today we have bigger market players and different market forces.

The price setters won’t run out of money

The top 100 tech companies are setting a high market price for AI talent because they know losing the AI war is an existential risk. It’s a safer bet to stockpile talent—at almost any price—than take the chance of losing competitive advantage.

Let’s just consider one company: Microsoft. 

Microsoft has $130B of cash on hand. The company can afford to hire 10,000 data scientists at $600K per year on the interest it earns from its cash war chest. This investment is a no-brainer if it mitigates the risk of falling behind Google.

Apple, Facebook, Spotify, Netflix, Google, Amazon, and Tesla are in the same position. So is that hot startup that just raised a $100M venture capital round. 

AI requires fewer engineers and less overhead 

During the dot-com bubble, a small army of engineers was required to do anything. Although it was better to have an A+ team, an average team could still make slow, steady progress.

AI is different. Building intelligent applications that solve real business problems requires creativity that can’t be broken down into repeatable processes. Training deep learning models and configuring MLOps (machine learning operations) cloud infrastructure takes highly skilled researchers and engineers. Fortunately, much of the complexity of these systems is packed into complex models and infrastructure that a small team can manage.

You’ll need far less overhead to recruit and retain 10 rock-star employees than 100 average ones, even if the salaries are significantly higher. The overhead of hiring mediocre recruits created a financial time bomb in the dot-com era that doesn’t exist today.

CEOs recognize the imminent AI threat 

During the dot-com era, the competitive threat of the internet was abstract. Analysts suggested that the internet would crush old media and that rising stars like Amazon would bury established companies like Toys-R-Us. Although these predictions were on target, the changes didn’t happen as quickly as predicted. 

AI is part of the digital transformation happening in every industry, and every CEO is acutely aware of the threat. Corporate boards watched companies like Netflix, Facebook, and Google destroy industries like media and entertainment with the Internet, and they are now holding management accountable for avoiding the same fate with AI. For example, financial services companies are terrified by the prospect of Amazon or Apple aggressively entering their industry. 

AI creates opportunities and threats for every company, and most have only five years to become AI-driven organizations.

Hope isn’t a strategy

Too many AI leaders fail to act because they think things might change for the better. They see outlier situations like Zillow’s workforce reduction and hope that salaries will stabilize across the market through more supply or lower demand.

Although I cannot predict the future, I see little evidence that more favorable hiring conditions are imminent. Instead of hoping for the market to shift, a more realistic strategy is to begin resetting expectations with your leadership about the cost of entry into the AI race.