Web Scraping Using Python What is Data Extraction? Data extraction is a process that involves retrieval of data from different website sources. Firms extract data in order to analyze it, migrate the data to a data repository (data warehouse) or use it in their businesses. Web scraping with python3 requests and BeautifulSoup. Installation pip install -r requirements.txt requirements.txt. Requests2.19.1 beautifulsoup44.6.3 requests module for requesting the url and fetching response and bs4 (beautifulsoup4) for making web scraping easier. Requesting and Souping. Web Scraping com Python e BeautifulSoup. GitHub Gist: instantly share code, notes, and snippets. Requests# Well known library for most of the Python developers as a fundamental tool to get raw.
| '' |
| Example of web scraping using Python and BeautifulSoup. |
| Sraping ESPN College Football data |
| http://www.espn.com/college-sports/football/recruiting/databaseresults/_/sportid/24/class/2006/sort/school/starsfilter/GT/ratingfilter/GT/statuscommit/Commitments/statusuncommit/Uncommited |
| The script will loop through a defined number of pages to extract footballer data. |
| '' |
| frombs4importBeautifulSoup |
| importrequests |
| importos |
| importos.path |
| importcsv |
| importtime |
| defwriterows(rows, filename): |
| withopen(filename, 'a', encoding='utf-8') astoWrite: |
| writer=csv.writer(toWrite) |
| writer.writerows(rows) |
| defgetlistings(listingurl): |
| '' |
| scrap footballer data from the page and write to CSV |
| '' |
| # prepare headers |
| headers= {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'} |
| # fetching the url, raising error if operation fails |
| try: |
| response=requests.get(listingurl, headers=headers) |
| exceptrequests.exceptions.RequestExceptionase: |
| print(e) |
| exit() |
| soup=BeautifulSoup(response.text, 'html.parser') |
| listings= [] |
| # loop through the table, get data from the columns |
| forrowsinsoup.find_all('tr'): |
| if ('oddrow'inrows['class']) or ('evenrow'inrows['class']): |
| name=rows.find('div', class_='name').a.get_text() |
| hometown=rows.find_all('td')[1].get_text() |
| school=hometown[hometown.find(',')+4:] |
| city=hometown[:hometown.find(',')+4] |
| position=rows.find_all('td')[2].get_text() |
| grade=rows.find_all('td')[4].get_text() |
| # append data to the list |
| listings.append([name, school, city, position, grade]) |
| returnlistings |
| if__name__'__main__': |
| '' |
| Set CSV file name. |
| Remove if file alreay exists to ensure a fresh start |
| '' |
| filename='footballers.csv' |
| ifos.path.exists(filename): |
| os.remove(filename) |
| '' |
| Url to fetch consists of 3 parts: |
| baseurl, page number, year, remaining url |
| '' |
| baseurl='http://www.espn.com/college-sports/football/recruiting/databaseresults/_/page/' |
| page=1 |
| parturl='/sportid/24/class/2006/sort/school/starsfilter/GT/ratingfilter/GT/statuscommit/Commitments/statusuncommit/Uncommited' |
| # scrap all pages |
| whilepage<259: |
| listingurl=baseurl+str(page) +parturl |
| listings=getlistings(listingurl) |
| # write to CSV |
| writerows(listings, filename) |
| # take a break |
| time.sleep(3) |
| page+=1 |
| ifpage>1: |
| print('Listings fetched successfully.') |
It is a well-known fact that Python is one of the most popular programming languages for data mining and Web Scraping. There are tons of libraries and niche scrapers around the community, but we’d like to share the 5 most popular of them.
Most of these libraries' advantages can be received by using our API and some of these libraries can be used in stack with it.
The Top 5 Python Web Scraping Libraries in 2020#
1. Requests#
Well known library for most of the Python developers as a fundamental tool to get raw HTML data from web resources.
To install the library just execute the following PyPI command in your command prompt or Terminal:
After this you can check installation using REPL:
- Official docs URL: https://requests.readthedocs.io/en/latest/
- GitHub repository: https://github.com/psf/requests
2. LXML#
When we’re talking about the speed and parsing of the HTML we should keep in mind this great library called LXML. This is a real champion in HTML and XML parsing while Web Scraping, so the software based on LXML can be used for scraping of frequently-changing pages like gambling sites that provide odds for live events.
To install the library just execute the following PyPI command in your command prompt or Terminal:
The LXML Toolkit is a really powerful instrument and the whole functionality can’t be described in just a few words, so the following links might be very useful:
- Official docs URL: https://lxml.de/index.html#documentation
- GitHub repository: https://github.com/lxml/lxml/
3. BeautifulSoup#
Probably 80% of all the Python Web Scraping tutorials on the Internet uses the BeautifulSoup4 library as a simple tool for dealing with retrieved HTML in the most human-preferable way. Selectors, attributes, DOM-tree, and much more. The perfect choice for porting code to or from Javascript's Cheerio or jQuery.
To install this library just execute the following PyPI command in your command prompt or Terminal:
As it was mentioned before, there are a bunch of tutorials around the Internet about BeautifulSoup4 usage, so do not hesitate to Google it!
- Official docs URL: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
- Launchpad repository: https://code.launchpad.net/~leonardr/beautifulsoup/bs4
4. Selenium#
Selenium is the most popular Web Driver that has a lot of wrappers suitable for most programming languages. Quality Assurance engineers, automation specialists, developers, data scientists - all of them at least once used this perfect tool. For the Web Scraping it’s like a Swiss Army knife - there are no additional libraries needed because any action can be performed with a browser like a real user: page opening, button click, form filling, Captcha resolving, and much more.
To install this library just execute the following PyPI command in your command prompt or Terminal:
The code below describes how easy Web Crawling can be started with using Selenium:
Web Scraping Multiple Pages Python Github
As this example only illustrates 1% of the Selenium power, we’d like to offer of following useful links:
- Official docs URL: https://selenium-python.readthedocs.io/
- GitHub repository: https://github.com/SeleniumHQ/selenium
5. Scrapy#
Scrapy is the greatest Web Scraping framework, and it was developed by a team with a lot of enterprise scraping experience. The software created on top of this library can be a crawler, scraper, and data extractor or even all this together.
To install this library just execute the following PyPI command in your command prompt or Terminal:
Web Scraping Python Code
We definitely suggest you start with a tutorial to know more about this piece of gold: https://docs.scrapy.org/en/latest/intro/tutorial.html
As usual, the useful links are below:
- Official docs URL: https://docs.scrapy.org/en/latest/index.html
- GitHub repository: https://github.com/scrapy/scrapy
What web scraping library to use?#
Web Scraping Python Projects Github
So, it’s all up to you and up to the task you’re trying to resolve, but always remember to read the Privacy Policy and Terms of the site you’re scraping 😉.