Scraping - Revision history

Michael Murtaugh: /* Scraping HTML in Python with html5lib + css selectors (2022) */

2022-05-25T13:36:48Z

Scraping HTML in Python with html5lib + css selectors (2022)

← Older revision		Revision as of 15:36, 25 May 2022
Line 17:		Line 17:
	html5lib is the modern python library for parsing "HTML in the wild" ... it's basically guaranteed to handle any HTML content and will correct broken tags, etc. to make sure whatever you load is "well-formed". Alternatively, you can use a library like BeautifulSoup.		html5lib is the modern python library for parsing "HTML in the wild" ... it's basically guaranteed to handle any HTML content and will correct broken tags, etc. to make sure whatever you load is "well-formed". Alternatively, you can use a library like BeautifulSoup.

	pip install html5lib elementpath ~~cssselector~~		pip install html5lib elementpath cssselect

	<source lang="python">		<source lang="python">

2022-05-25T13:36:36Z

Scraping HTML in Python with html5lib + css selectors (2022)

← Older revision		Revision as of 15:36, 25 May 2022
Line 16:		Line 16:
	== Scraping HTML in Python with html5lib + css selectors (2022) ==		== Scraping HTML in Python with html5lib + css selectors (2022) ==
	html5lib is the modern python library for parsing "HTML in the wild" ... it's basically guaranteed to handle any HTML content and will correct broken tags, etc. to make sure whatever you load is "well-formed". Alternatively, you can use a library like BeautifulSoup.		html5lib is the modern python library for parsing "HTML in the wild" ... it's basically guaranteed to handle any HTML content and will correct broken tags, etc. to make sure whatever you load is "well-formed". Alternatively, you can use a library like BeautifulSoup.

			pip install html5lib elementpath cssselector

	<source lang="python">		<source lang="python">

2022-05-25T13:36:06Z

Scraping HTML in Python with html5lib + css selectors (2022)

← Older revision		Revision as of 15:36, 25 May 2022
Line 15:		Line 15:

	== Scraping HTML in Python with html5lib + css selectors (2022) ==		== Scraping HTML in Python with html5lib + css selectors (2022) ==
			html5lib is the modern python library for parsing "HTML in the wild" ... it's basically guaranteed to handle any HTML content and will correct broken tags, etc. to make sure whatever you load is "well-formed". Alternatively, you can use a library like BeautifulSoup.

	<source lang="python">		<source lang="python">
	from xml.etree import ElementTree as ET		from xml.etree import ElementTree as ET

2022-05-25T13:34:53Z

Scraping HTML in Python with html5lib + css selectors (2022)

← Older revision		Revision as of 15:34, 25 May 2022
Line 28:		Line 28:
	with open("index.html") as fin:		with open("index.html") as fin:
	t = html5lib.parse(fin, namespaceHTMLElements=False)		t = html5lib.parse(fin, namespaceHTMLElements=False)

			print (textContent(t.find(".//p")))

	</source>		</source>

2022-05-25T13:34:23Z

Scraping HTML in Python with html5lib + css selectors (2022)

← Older revision		Revision as of 15:34, 25 May 2022
Line 15:		Line 15:

	== Scraping HTML in Python with html5lib + css selectors (2022) ==		== Scraping HTML in Python with html5lib + css selectors (2022) ==
			<source lang="python">
			from xml.etree import ElementTree as ET
			import html5lib

			def textContent (elt):
			if elt.text != None:
			ret = elt.text
			else:
			ret = u""
			return ret + u"".join([ET.tostring(x, method="text", encoding="utf8").decode("utf-8") for x in elt])

			with open("index.html") as fin:
			t = html5lib.parse(fin, namespaceHTMLElements=False)
			</source>

			<source lang="python">
			from cssselect import HTMLTranslator
			import elementpath

			def querySelector (t, selector):
			for elt in elementpath.select(t, HTMLTranslator().css_to_xpath(selector)):
			return elt

			def querySelectorAll (t, selector):
			for elt in elementpath.select(t, HTMLTranslator().css_to_xpath(selector)):
			yield elt

			for p in querySelectorAll("p.test"):
			print (textContent(p))

			</source>

2022-05-25T13:30:38Z

← Older revision		Revision as of 15:30, 25 May 2022
Line 13:		Line 13:

	See [[Extracting parts of an HTML document]] and other recipes in the [[:Category:Cookbook]]		See [[Extracting parts of an HTML document]] and other recipes in the [[:Category:Cookbook]]

			== Scraping HTML in Python with html5lib + css selectors (2022) ==

2012-10-08T15:28:05Z

← Older revision		Revision as of 17:28, 8 October 2012
Line 10:		Line 10:
	Resources:		Resources:
	* http://us.pycon.org/2009/tutorials/schedule/2AM8/		* http://us.pycon.org/2009/tutorials/schedule/2AM8/
			* http://scrapy.org/ Python framework for custom scrapers

	See [[Extracting parts of an HTML document]] and other recipes in the [[:Category:Cookbook]]		See [[Extracting parts of an HTML document]] and other recipes in the [[:Category:Cookbook]]

2012-05-18T06:26:43Z

← Older revision		Revision as of 08:26, 18 May 2012
Line 11:		Line 11:
	* http://us.pycon.org/2009/tutorials/schedule/2AM8/		* http://us.pycon.org/2009/tutorials/schedule/2AM8/

	See [[Extracting parts of an HTML ~~page~~]] and other recipes in the [[:Category:Cookbook]]		See [[Extracting parts of an HTML document]] and other recipes in the [[:Category:Cookbook]]

2012-05-18T06:26:26Z

← Older revision		Revision as of 08:26, 18 May 2012
Line 11:		Line 11:
	* http://us.pycon.org/2009/tutorials/schedule/2AM8/		* http://us.pycon.org/2009/tutorials/schedule/2AM8/

	See the [[:Category:Cookbook]]		See [[Extracting parts of an HTML page]] and other recipes in the [[:Category:Cookbook]]

2012-05-18T06:25:42Z

← Older revision		Revision as of 08:25, 18 May 2012
Line 10:		Line 10:
	Resources:		Resources:
	* http://us.pycon.org/2009/tutorials/schedule/2AM8/		* http://us.pycon.org/2009/tutorials/schedule/2AM8/

			See the [[:Category:Cookbook]]