• What's required to scrape LinkedIn yourself? Understanding private vs public profiles, scraping HTML, bypassing the authwall, and more.
• What does the Proxycurl API accept, and what do you get?
• How to query the Person Profile Endpoint in Postman, Python, and Javascript.
• A measure of reliability: We'll query the Person Profile Endpoint 100 times and look at the status codes.

How would you build a LinkedIn scraper in a vacuum?

Scraping one field from an example profile

def get_languages(raw_html: BeautifulSoup) -> List[str]:
    languages = []

    language_section = [_ for _ in raw_html.body.main.find_all('section') if _.get('data-section') == 'languages']

    if len(language_section) == 0:
        return languages

    for language in language_section[0].div.ul.find_all('li'):
        languages.append(language.div.h3.get_text().strip())
    return languages

• Do this for every single field we're interested in.
• Keep up with a moving target. Any time LinkedIn rearranges any of its HTML, we'll have to adjust our parser, sometimes significantly, and sometimes all at once.
• Support internationalization. For example, Czech-language profiles need a dedicated parser.

Authwall: What is it and how do we deal with it?

LinkedIn's "authwall" blocks you from seeing most profiles while you're logged out.

import requests
from bs4 import BeautifulSoup


def has_authwall(url: str) -> bool:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    for s in soup.find_all('script'):
        if '/authwall?trk=' in s.get_text():
            return True
    return False

Scraping LinkedIn yourself is hard, but scraping LinkedIn doesn't have to be What do I put into the Proxycurl API, and what do I get out?

What you have

What you get

• skills
• inferred_salary
• personal_email
• personal_contact_number
• twitter_profile_id
• facebook_profile_id
• github_profile_id
• extra

Example walkthroughs

Postman

1. In the Person Profile Endpoint docs, find the orange button that says Run in Postman and click it. Then click "Fork Collection" and log in however you like. If you're logging in for the first time, the screen should look something like this:

Postman upon initially forking the Proxycurl collection

1. Add your Bearer Token to the environment as a variable. Limit it to Proxycurl. You can do this from the Variables tab or from the pop-up that appears when you start typing into the "Token" field. You can name this variable anything you like, but Bearer Token is a good name.
2. Verify that the Authorization type is set to "Bearer Token" and that you have typed {{Bearer Token}} into the Token field and click Save in the upper right-hand corner. Remember to click Save!! Your page should look like this:

Adding a Bearer Token variable Authorization in Postman

1. Now, we should be able to run something! On the left-hand side, under "My workspace," expand your Proxycurl collection and then the People API, and double-click the Person Profile Endpoint. You can un-check some of the fields if you want or modify others - for example, you might want to change use_cache from if-present to if-recent to get the most up-to-date info, but maybe you don't need the user's personal contact number this time around.
2. Once you've modified the fields to your liking, click the blue "Send" button in the upper left-hand corner.

Troubleshooting Postman

Python

The Shell-Python toggle in the Proxycurl docs

import os, requests

api_endpoint = 'https://nubela.co/proxycurl/api/v2/linkedin'
api_key = os.environ['PROXYCURL_API_KEY']
header_dic = {'Authorization': 'Bearer ' + api_key}
params = {
    'url': 'https://www.linkedin.com/in/johnrmarty/',
    'fallback_to_cache': 'on-error',
    'use_cache': 'if-present',
    'skills': 'include',
    'inferred_salary': 'include',
    'personal_email': 'include',
    'personal_contact_number': 'include',
    'twitter_profile_id': 'include',
    'facebook_profile_id': 'include',
    'github_profile_id': 'include',
    'extra': 'include',
}
response = requests.get(api_endpoint, params=params, headers=header_dic)
print(response.json())

response = requests.get(api_endpoint, params=params, headers=header_dic)
result = response.json()
print(result)

The Python debugger in PyCharm is fantastic!

JavaScript

1. Generate an object of params corresponding to what we want to fetch in our query.
2. Transform these params into a query string using URLSearchParams.
3. Make the request using fetch, including the bearer token as a header.
4. Catch any errors and otherwise print a JSON to the console.

const apiKey = "your_api_key_here";
const params = {
  url: "https://www.linkedin.com/in/johnrmarty/",
  fallback_to_cache: "on-error",
  use_cache: "if-present",
  skills: "include",
  inferred_salary: "include",
  personal_email: "include",
  personal_contact_number: "include",
  twitter_profile_id: "include",
  facebook_profile_id: "include",
  github_profile_id: "include",
  extra: "include",
};

const queryString = new URLSearchParams(params).toString();
const endpoint = `https://nubela.co/proxycurl/api/v2/linkedin?${queryString}`;

fetch(endpoint, {
  headers: { Authorization: `Bearer ${apiKey}` },
})
  .then((response) => {
    if (!response.ok) {
      throw new Error(`Network response not ok. Status code: ${response.status}.`);
    }
    return response.json();
  })
  .then((data) => console.log(data))
  .catch((error) => console.error("Error: " + error));

How reliable is Proxycurl?

codes = {}

for _ in range(100):

    response = requests.get(api_endpoint, params=params, headers=header_dic)
    code = response.status_code
    if code not in codes:
        codes[code] = 1
    else:
        codes[code] += 1

print(codes)

{200: 100}

Summary

• Postman
• Python
• JavaScript