Wget - Revision history

Michael Murtaugh: /* Doing the same thing in Python */

2009-11-17T15:42:49Z

Doing the same thing in Python

@@ Line 83: / Line 83: @@
 	thepage = f.read()
 	cal = ics.parseData(thepage) # or just "raw" data (string)
 </source>

2009-11-17T15:18:57Z

Grabbing images

← Older revision		Revision as of 17:18, 17 November 2009
Line 50:		Line 50:

	<source lang="bash">		<source lang="bash">
	wget -r -nd -np --follow-tags=img -A.jpg,.jpeg http://www.colourlovers.com		wget -e robots=off -r -nd -np --follow-tags=img -A.jpg,.jpeg http://www.colourlovers.com
	montage *.jpg ../public_html/montage.jpg		montage *.jpg ../public_html/montage.jpg
	</source>		</source>

2009-06-25T13:48:53Z

"Mirroring" a site

← Older revision		Revision as of 15:48, 25 June 2009
Line 41:		Line 41:
	external content, such as embedded images, links to style sheets,		external content, such as embedded images, links to style sheets,
	hyperlinks to non-HTML content, etc.		hyperlinks to non-HTML content, etc.

			'''-nH'''
			Turns off creating a top-level folder.

	=== Grabbing images ===		=== Grabbing images ===

2009-06-25T13:34:31Z

"Mirroring" a site

← Older revision		Revision as of 15:34, 25 June 2009
Line 16:		Line 16:
	What are the options used here (from wget manpage):		What are the options used here (from wget manpage):

	-E<br />		'''-E'''<br />
	--html-extension		'''--html-extension'''

	If a file of type application/xhtml+xml or text/html is downloaded		If a file of type application/xhtml+xml or text/html is downloaded
Line 29:		Line 29:
	article.cgi?25.html.		article.cgi?25.html.

	-r		'''-r'''

	Recursive: Meaning it downloads links as well.		Recursive: Meaning it downloads links as well.

	-k		'''-k'''
	--convert-links		'''--convert-links'''

	After the download is complete, convert the links in the document		After the download is complete, convert the links in the document

2009-06-25T13:33:51Z

"Mirroring" a site

← Older revision		Revision as of 15:33, 25 June 2009
Line 32:		Line 32:

	Recursive: Meaning it downloads links as well.		Recursive: Meaning it downloads links as well.

			-k
			--convert-links

			After the download is complete, convert the links in the document
			to make them suitable for local viewing. This affects not only the
			visible hyperlinks, but any part of the document that links to
			external content, such as embedded images, links to style sheets,
			hyperlinks to non-HTML content, etc.

	=== Grabbing images ===		=== Grabbing images ===

2009-06-25T13:32:50Z

"Mirroring" a site

← Older revision		Revision as of 15:32, 25 June 2009
Line 29:		Line 29:
	article.cgi?25.html.		article.cgi?25.html.

			-r

			Recursive: Meaning it downloads links as well.

	=== Grabbing images ===		=== Grabbing images ===

2009-06-25T13:10:03Z

← Older revision		Revision as of 15:10, 25 June 2009
Line 5:		Line 5:

	== Examples ==		== Examples ==

			=== "Mirroring" a site ===

			<source lang="bash">
			wget -Erk -nH 'http://www.yoursite.com/yourdirectory/yourfile.php"
			</source>

			Source: http://www.dreamincode.net/forums/showtopic8317.htm

			What are the options used here (from wget manpage):

			-E<br />
			--html-extension

			If a file of type application/xhtml+xml or text/html is downloaded
			and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, this
			option will cause the suffix .html to be appended to the local
			filename. This is useful, for instance, when you’re mirroring a
			remote site that uses .asp pages, but you want the mirrored pages
			to be viewable on your stock Apache server. Another good use for
			this is when you’re downloading CGI-generated materials. A URL
			like http://site.com/article.cgi?25 will be saved as
			article.cgi?25.html.


			=== Grabbing images ===

	This simple two-line script uses wget to collect all the jpeg's from a website, then uses [[ImageMagick]]'s montage tool to combine them in a single image.		This simple two-line script uses wget to collect all the jpeg's from a website, then uses [[ImageMagick]]'s montage tool to combine them in a single image.

2009-06-04T15:01:54Z

Doing the same thing with just Python

← Older revision		Revision as of 17:01, 4 June 2009
Line 13:		Line 13:
	</source>		</source>

	== Doing the same thing ~~with just~~ Python ==		== Doing the same thing in Python ==
	<source lang="python">		<source lang="python">
	import urllib2		import urllib2

2009-06-04T15:01:06Z

@@ Line 11: / Line 11: @@
 wget -r -nd -np --follow-tags=img -A.jpg,.jpeg http://www.colourlovers.com
 montage *.jpg ../public_html/montage.jpg
 </source>

2008-10-14T16:26:27Z

← Older revision		Revision as of 18:26, 14 October 2008
Line 3:		Line 3:
	* http://en.wikipedia.org/wiki/Wget		* http://en.wikipedia.org/wiki/Wget
	* http://www.gnu.org/software/wget/		* http://www.gnu.org/software/wget/

			== Examples ==

			This simple two-line script uses wget to collect all the jpeg's from a website, then uses [[ImageMagick]]'s montage tool to combine them in a single image.

			<source lang="bash">
			wget -r -nd -np --follow-tags=img -A.jpg,.jpeg http://www.colourlovers.com
			montage *.jpg ../public_html/montage.jpg
			</source>