LinkedIn Scraping: Legal Risks, Tools & Best Practices

LinkedIn has a massive 1.3 billion registered user base. For recruiters and human resources, it is a goldmine of professional data. Businesses can monitor the competition. Sales prospecting finds decision-makers.

But getting access to this social network's data is difficult. Due to its high commercial applicability, LinkedIn scraping is a widely discussed topic.

Also, numerous companies have already scraped LinkedIn data. Some even went to court for it, which we'll explain in this article.

Scraping LinkedIn profiles is notoriously hard. To give you a better idea, we will provide practical, legal, and tool-based tips.

It is also crucial to understand the importance of ethical LinkedIn data use. Personal data protection regulations safeguard it worldwide, and LinkedIn scrapers must abide by them.

What is LinkedIn scraping?

LinkedIn scraping is the process of extracting data from this website automatically using LinkedIn scrapers. You can extract the following types of data:

• Company page (name, size, industry, description)

• Job lists (description, submission date, applicant count, salary)

• Individual profile information (name, location, education, skills, and work experience)

• Feed (posts, articles, reactions, comments)

• Groups and communities (name, member count, posts, events)

The list is not exhaustive, but gives a good view of LinkedIn data.

It's worth noting that using an API (Application Programming Interface) is an alternative data extraction method. An API is a set of rules that lets two different applications communicate with each other. In many cases, it allows seamless data sharing between two different platforms.

LinkedIn offers a variety of APIs. However, LinkedIn APIs are much more restrictive. Its 'Consumer API' is basically a 'Sign in with LinkedIn' tool. 'Apply Connect API' allows posting and applying to jobs without leaving the LinkedIn website.

There are more, but all of them are highly specific. It's nothing like scraping LinkedIn profiles on a large scale.

Why scrape LinkedIn?

LinkedIn has everything in one place: businesses, CEOs, employees, groups, and recruitment agencies. Many of them also share market data in one form or another. For a marketing professional, LinkedIn's scraped data opens numerous possibilities.

Market researchers use LinkedIn data to keep track of workforce migration, monitor competitors, and track overall industry development. Diving deeper, it is possible to anticipate business growth, its strategies, and peek into the employee teams that enable it.

Recruiters benefit massively from LinkedIn scraping. Instead of going over thousands of profiles manually, they can automate it. Recruiters can reach out to employees with the required skills and experience, even if they are not looking for a job at that moment.

Let's jump to sales. Sales prospecting uses LinkedIn data to map out businesses, gather contact information, and identify decision-makers. Outreach professionals personalize messages with accurate icebreakers to kickstart the communication.

For a real-world example, consider companies like PhantomBuster. Their description on LinkedIn states:

"The easiest way to find and engage with warm leads."

As their most popular use cases, they offer to extract leads from LinkedIn posts and scrape LinkedIn search results into spreadsheets.

Collecting and providing access to scraped data from LinkedIn is a business model for companies like this. That brings us to the next chapter: the legality of accessing LinkedIn profile data.

Is LinkedIn scraping legal?

The short answer regarding the legality of LinkedIn scraping, and web scraping in general, is that it depends. It depends on how and what data you extract, what you use it for, and whether you target LinkedIn profiles while logged in or limit the scope to public information only.

LinkedIn’s Terms of Service

LinkedIn in its Terms of Service (ToS) states very clearly that no automatic data extraction is allowed. Unauthorized scraping of LinkedIn profiles will lead to account suspension or even legal action.

The social network also actively protects its profiles from LinkedIn scrapers. If it identifies automated behavior like sending concurrent requests or accessing multiple profiles simultaneously, it will restrict access.

On the other hand, LinkedIn's Terms of Service are just private business rules. In reality, the court has ruled multiple times that scraping publicly available web data is allowed. That was at the center of the hiQ Labs v. LinkedIn Corp. case, which became the cornerstone of web scraping legality.

hiQ Labs v. LinkedIn Corp. case

hiQ started scraping LinkedIn pages in 2012. The data analytics company used the scraped data to map out skill gaps and predict employee loss. LinkedIn responded, resulting in the following timeline:

• 2012: hiQ starts silently scraping LinkedIn profiles;

• 2017: LinkedIn issues a cease-and-desist order, invoking the Computer Fraud and Abuse Act (CFAA);

• 2017: hiQ responds with a lawsuit against LinkedIn;

• 2017: The court rules in favor of hiQ, allowing scraping of public web pages and LinkedIn appeals;

• 2019: The Court of Appeals upholds the decision, favoring web scraping;

• 2020: LinkedIn escalates to the US Supreme Court;

• 2021: The Supreme Court sends the case to be re-evaluated;

• 2022: The Court rules that scraping public data is still allowed. However, LinkedIn proves hiQ broke its Terms of Service and wins the case.

The back-and-forth rulings did not provide a final answer. Instead, the result is two rules regarding data extraction:

1. Scraping publicly available data is allowed and does not breach CFAA principles;
2. Creating LinkedIn accounts, accepting its ToS, and then breaking it to scrape LinkedIn profiles is not allowed and carries civil liability.

Because hiQ created LinkedIn accounts and used them in violation of LinkedIn's Terms of Service, it lost the case. However, the court set the ongoing data extraction ruling, which allows scraping of public web pages.

Recent legal developments

In a more recent development, a US court reaffirmed the legality of scraping public data.

In 2023, Meta sued proxy and data analytics company Bright Data because it was scraping Meta's public profiles. The judge ruled in favor of Bright Data because it did not log in to Meta's accounts, nor did it have to accept its ToS.

But the real right-here, right-now battle happens over data extraction for AI training. Generative AI demands enormous amounts of information, and web scraping is an efficient way to get it.

The laws that regulate data extraction for AI training are very recent. The European Data Protection Board document, which we use as an example, was adopted only two months before writing this article.

We recommend reading it all, but here are three crucial points that apply to scraping individual LinkedIn profiles:

1. Data collection must serve legitimate interests.
2. Collection of personal data is necessary to pursue these interests.
3. Data collection does not interfere with personal data protection laws.

This is only the green light to start scraping. Businesses must also ensure data minimization, purpose and storage limitation, the right to information, and other lawful principles. A lot is going on, as outlined in the current data privacy laws.

Data privacy laws

The document refers to the European General Data Protection Regulation (GDPR). But similar laws appear (or already exist) worldwide. For example, the California Consumer Privacy Act (CCPA) in the US, personal Information Protection and Electronic Documents Act (PIPEDA) in Canada, and many more.

We can't review them all in one article, but we will summarize several essential principles in the table below.

It's best to keep these regulations close when targeting LinkedIn profiles. But keep in mind that regulations also differ.

For example, scraping publicly available personal data is typically allowed in the US. Meanwhile, GDPR requires a lawful basis to gather such information, even if it is easily accessible on the internet.

Ethical best practices

Besides lawful regulations, there are also ethical agreements to keep the web scraping industry transparent.

Firstly, inspect and respect LinkedIn's robots.txt file. This file outlines which parts of a website are open to bots and which are not. These instructions are primarily for web crawlers, but it's ethical to apply them to web scrapers too.

Do not use bots to manage multiple fake LinkedIn profiles. As the hiQ vs. LinkedIn lawsuit demonstrates, it is a high-risk activity. LinkedIn can rightfully take legal action against it.

Make sure your LinkedIn scraper does not overload the website with too many concurrent requests. Remember that LinkedIn is very strict regarding automated data extraction. You should set realistic limits, such as scraping roughly 40 LinkedIn profiles per day, and avoid sensitive personal data.

Lastly, LinkedIn offers a Compliance API. It is a paid service that allows regulated industries (healthcare, legal, etc.) to monitor and archive their employees' LinkedIn communications. This approach is very different from LinkedIn scraping, but an option worth considering for applicable businesses (although with heavy access restrictions from LinkedIn).

Technical challenges & platform defenses

LinkedIn scraping is considerably harder than targeting your typical website. The platform also does not sell LinkedIn data directly. Instead, they provide access to services like LinkedIn Sales Navigator and LinkedIn Recruiter.

To remain exclusive, they implement numerous protection methods to prevent LinkedIn data scraping. Firstly, a lot of its data is placed behind an authentication wall that requires logging in. This is what you will often see when browsing LinkedIn while logged out.

As the hiQ lawsuit shows, it's a bad idea to extract data from LinkedIn after logging in. But even if you scrape public data, for example, from LinkedIn search, you will run into restrictions.

Firstly, LinkedIn monitors each IP address's activity. If it makes too many concurrent requests or has a low IP trust score, LinkedIn will restrict access. The same applies to datacenter proxies. Efficient LinkedIn profile scrapers must rely on high-quality mobile or residential proxies.

But that's also not enough. LinkedIn uses request fingerprinting. It inspects each request's browser and HTTP headers. Unmodified LinkedIn scraper headers will immediately get blocked.

Tools like Selenium and Puppeteer can leave a 'HeadlessChrome' identifier in the user agent string, which also results in a block. To put it shortly, each request must transmit all of the attributes that an authentic human visitor does.

Another LinkedIn data protection method is behavioral tracking. Simple LinkedIn scrapers behave differently from real people browsing. They do not display the same mouse movements, pagination, or scrolling. LinkedIn takes all of this into account to identify and block scrapers.

Lastly, LinkedIn has a complex JavaScript-rich website structure. It requires good front-end knowledge to target the required elements to build a custom LinkedIn scraper.

Moreover, the social network uses code obfuscation techniques to prevent identifying and targeting these data points. All of this combined makes LinkedIn a particularly challenging website to scrape.

How to scrape LinkedIn

There's a lot to consider when scraping LinkedIn data. Because of how complex it is, we will not be coding an example of a LinkedIn scraper from scratch. Instead, we will provide guidelines and review readily available tools developed to handle LinkedIn's scraped data.

Plan your data goals

Defining your targeted data early on is best. Remember that GDPR, CCPA, and other data protection laws clearly define what you can and cannot collect. It is especially important if you plan to target individual LinkedIn profiles.

Choose the right tools

There are numerous LinkedIn data scraping services available. Some are cloud-based, like Apify, and don't require any software installation. Others, like Evaboot, are specifically developed to extract data from the LinkedIn Sales Navigator tool.

Keep in mind that you don't have to rely only on LinkedIn scrapers. You can instead choose to purchase structured, pre-built LinkedIn data.

Set up proxies and user agents

Large-scale LinkedIn scraping is impossible without proxies. If you send too many data extraction requests from a single IP address, LinkedIn will simply block it. Proxies provide thousands of alternative IPs to prevent that.

Many LinkedIn scraping services offer their own proxies. But that doesn't mean they are the best in terms of IP address quality and proxy type availability.

For example, Apify lets you use your own proxies and user agents. If you already have experience extracting data using this tool, it gives you more control over your LinkedIn scraper.

Extract, clean, and store your data

Extract only the bare minimum required, as instructed by data protection regulations. Scraped LinkedIn data is typically provided in three formats: JSON, CSV, and XLSX (Excel). If you keep this data on your device, it must be encrypted and secured from data leaks.

Your chosen tools should also ensure that it's cleaned. That means removing duplicate entries, stripping special characters and whitespace, and separating text fields.

LinkedIn scraping tools & platforms

There are dozens of LinkedIn scraping tools to choose from. We're not promoting any specific service, as businesses will choose what's best for them. However, we will mention three of them that represent different methods of scraping LinkedIn pages.

Apify

Apify provides APIs for data extraction, which it calls actors. These are separate applications with different capabilities, purposes, and prices.

It's a versatile tool that allows targeting of profiles, organizations, jobs, and any other kind of LinkedIn data. Apify also has a free trial.

It took 20 seconds for Apify to extract data from two public LinkedIn profiles. It then suggested downloading it in JSON, CSV, XML, Excel, and a few other formats. There are helpful actor tutorials, but Apify also lets you customize them per your needs.

Pros

• Fast LinkedIn scraping

• A wide selection of actors (scrapers)

• Offers a free trial

Cons

• Large-scale scraping gets expensive

Skrapp.io

You don't always need something complex and highly customizable. We found a tool called Skrapp.io, which is a simple B2B email finder. It has a Chrome extension that sits on top of LinkedIn and lets you extract emails with a few clicks.

Its dashboard is also easy to use. To try it out, we searched for Craigslist employee email addresses on LinkedIn.

The tool found 64 results in general, displaying lead name, job role, location, and email. It takes less than 10 minutes to learn to use Skrapp.io, which is a time saver for B2B outreach.

Pros

• Very easy to use

• Provides Chrome extension

• Offers a free trial

Cons

• Functionality limited to email search

PhantomBuster

Last on our list is PhantomBuster. It's an automation-oriented, no-code, cloud-based platform with a free trial.

This tool is designed to automate tasks on LinkedIn, turning days of manual labor into hours. To use all of these features, PhantomBuster requires connecting a LinkedIn account.

Afterward, we chose to extract leads from a LinkedIn search URL with the simple keyword 'proxies'. In about two minutes, PhantomBuster gathered over 30 leads with job titles and locations related to proxies.

Pros

• LinkedIn task automation

• No-code cloud-based platform

• Offers a free trial

Cons

• Requires connecting a LinkedIn account

Conclusion: Best practices for LinkedIn scraping

There's a lot you can achieve with scraped data from LinkedIn. Companies can monitor their competitors, employee movement, as well as industry growth patterns and skill demands.

It is especially important to do it legally and ethically. Two times US courts have approved scraping public data. But businesses must also consider the safety and privacy of any personal information, even if it is easily accessible online.

Data protection laws - GDPR, CCPA, and others - protect LinkedIn data worldwide, so you must always follow their principles.

Noteworthy LinkedIn scraping tools use multiple techniques to successfully collect such information. They use rate limiting to avoid server overload, but also mimic a real person browsing to avoid detection and bans. After meeting all safety criteria, marketers can scrape detailed profile information to build prospect lists and more.

FAQ