How to Scrape websites WITHOUT using Selenium,Instead Using Internal APIs with Python

Most of my other tutorials have focused on scraping websites using Selenium, Chromedriver and Python. While this is a great way to learn to scrape, it has its drawbacks. The main one being its very labor intensive. For each site you want to scrape you in essence need a chrome instance open on your machine. For a single website scraper this is fine, but if we want to move onto bigger projects this will undoubtedly provide resource issues.

The antidote to this is to use a websites internal API to scrape the data as opposed to running a chromedriver. In this tutorial I will show you how to find the API on a given website and the basics of making a call to that API. The only issue here is that not every site has this API publicly accessible. We focus on betting sites here so for example Bet365 does not allow this and their internal API is wrapped in JS which is more difficult to decipher, while PaddyPower have their API front and center.

For the example today I will be using PaddyPower with this specific URL

https://www.paddypower.com/inplay?tab=football

Otherwise navigate to the site you wish to scrape, and get to the page with the info you require. Then you can right click anywhere on this page and select inspect to open the…