BaseSeed

[Matt Zimmerman]

On Thu, Sep 09, 2004 at 09:28:19AM +1000, James Gregory wrote:

> On Wed, 2004-09-08 at 14:04 -0700, Matt Zimmerman wrote:
> > On Thu, Sep 09, 2004 at 04:54:34AM +0800, John wrote:
> > 
> > > How else do you do this?
> > > summer at Dolphin:~$ time lynx -dump  http://www.x.com/ | tail
> > >  30. http://www.ebay.com/
> > >  31. http://www.paypal.com/cgi-bin/webscr
> > >  32. http://www.paypal.com/cgi-bin/webscr?cmd=p/gen/fdic-outside
> > >  33. http://www.paypal.com/cgi-bin/webscr?cmd=p/gen/privacy-outside
> > >  34. http://www.bbbonline.org/cks.asp?id=20111061155818568
> > > 
> > > 
> > > I regularly want a list of URLs for some reason, often to get a list of 
> > > files to download with wget or (sometimes) with curl.
> > 
> > You don't need a browser at all if you only want to extract URLs.
> > 
> > wget -O- http://www.x.com/ | urlview
> 
> You can also go to mozilla and click 'page info'. There's a links tab
> there with all the links for the page. But if you want to download
> everything on a page, wget -r will work.

Oh, there's also lftp in base, which is even better if what you want is to
download:

mizar:[~] lftp http://www.dmoz.org/
cd ok, cwd=/
lftp www.dmoz.org:/> ls
-rw-r--r--  --  img/odphead.gif
-rw-r--r--  --  about.html
-rw-r--r--  --  add.html
-rw-r--r--  --  help/helpmain.html
-rw-r--r--  --  link.html
drwxr-xr-x  --  Arts
drwxr-xr-x  --  Arts/Movies
drwxr-xr-x  --  Arts/Television
drwxr-xr-x  --  Arts/Music
drwxr-xr-x  --  Business
drwxr-xr-x  --  Business/Employment
drwxr-xr-x  --  Business/Real_Estate
drwxr-xr-x  --  Business/Investing
drwxr-xr-x  --  Computers
drwxr-xr-x  --  Computers/Internet
drwxr-xr-x  --  Computers/Software
drwxr-xr-x  --  Computers/Hardware
drwxr-xr-x  --  Games
drwxr-xr-x  --  Games/Video_Games
drwxr-xr-x  --  Games/Roleplaying
drwxr-xr-x  --  Games/Gambling
drwxr-xr-x  --  Health
drwxr-xr-x  --  Health/Fitness
drwxr-xr-x  --  Health/Medicine
drwxr-xr-x  --  Health/Alternative
drwxr-xr-x  --  Home
drwxr-xr-x  --  Home/Family
drwxr-xr-x  --  Home/Consumer_Information
drwxr-xr-x  --  Home/Cooking
[...]

-- 
 - mdz