[Bug 1291373] Re: wget. memory leaks

Tue Jan 3 06:24:05 UTC 2017

fd_read_body() seems to be riddled with memory leaks.

Below is output of running a wget -c -r, fetching a total of a few
hundred files, all of which were already found on local disk, and thus
not actually fetched.

I am using 1.13.4-2ubuntu1.4 on Linux 2.6.32-042stab117.14 x86_64

==26452== 2,082 bytes in 347 blocks are definitely lost in loss record 810 of 839
==26452==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26452==    by 0x430348: xmalloc (xmalloc.c:43)
==26452==    by 0x428464: strdupdelim (utils.c:224)
==26452==    by 0x42BD94: parse_charset (iri.c:72)
==26452==    by 0x41ACBC: gethttp (http.c:2287)
==26452==    by 0x41B622: http_loop (http.c:2769)
==26452==    by 0x4249AD: retrieve_url (retr.c:736)
==26452==    by 0x422894: retrieve_tree (recur.c:283)
==26452==    by 0x40541A: main (main.c:1388)
==26452==
==26452== 38,400 bytes in 300 blocks are definitely lost in loss record 834 of 839
==26452==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26452==    by 0x430348: xmalloc (xmalloc.c:43)
==26452==    by 0x423DB6: fd_read_hunk (retr.c:451)
==26452==    by 0x424573: fd_read_body (retr.c:295)
==26452==    by 0x41A962: gethttp (http.c:2588)
==26452==    by 0x41B622: http_loop (http.c:2769)
==26452==    by 0x4249AD: retrieve_url (retr.c:736)
==26452==    by 0x422894: retrieve_tree (recur.c:283)
==26452==    by 0x40541A: main (main.c:1388)
==26452==
==26452== 147,712 bytes in 1,154 blocks are definitely lost in loss record 835 of 839
==26452==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26452==    by 0x430348: xmalloc (xmalloc.c:43)
==26452==    by 0x423DB6: fd_read_hunk (retr.c:451)
==26452==    by 0x4245A8: fd_read_body (retr.c:355)
==26452==    by 0x41A962: gethttp (http.c:2588)
==26452==    by 0x41B622: http_loop (http.c:2769)
==26452==    by 0x4249AD: retrieve_url (retr.c:736)
==26452==    by 0x422894: retrieve_tree (recur.c:283)
==26452==    by 0x40541A: main (main.c:1388)
==26452==
==26452== 186,112 bytes in 1,454 blocks are definitely lost in loss record 837 of 839
==26452==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26452==    by 0x430348: xmalloc (xmalloc.c:43)
==26452==    by 0x423DB6: fd_read_hunk (retr.c:451)
==26452==    by 0x42453E: fd_read_body (retr.c:283)
==26452==    by 0x41A962: gethttp (http.c:2588)
==26452==    by 0x41B622: http_loop (http.c:2769)
==26452==    by 0x4249AD: retrieve_url (retr.c:736)
==26452==    by 0x422894: retrieve_tree (recur.c:283)
==26452==    by 0x40541A: main (main.c:1388)
==26452==
==26452== 540,606 (144,384 direct, 396,222 indirect) bytes in 9,024 blocks are definitely lost in loss record 839 of 839
==26452==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26452==    by 0x50F21B3: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x519CCB6: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x519FB8C: ASN1_item_ex_d2i (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x519FE70: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x51A021B: ??? (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x519F86C: ASN1_item_ex_d2i (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x51A0353: ASN1_item_d2i (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x51BD606: X509V3_EXT_d2i (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==26452==    by 0x42D09A: ssl_check_certificate (openssl.c:590)
==26452==    by 0x41908B: gethttp (http.c:1917)
==26452==    by 0x41B622: http_loop (http.c:2769)
==26452==
==26452== LEAK SUMMARY:
==26452==    definitely lost: 518,690 bytes in 12,279 blocks
==26452==    indirectly lost: 396,222 bytes in 18,047 blocks
==26452==      possibly lost: 20 bytes in 1 blocks
==26452==    still reachable: 395,620 bytes in 6,051 blocks
==26452==         suppressed: 0 bytes in 0 blocks

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to wget in Ubuntu.
https://bugs.launchpad.net/bugs/1291373

Title:
  wget. memory leaks

Status in wget package in Ubuntu:
  New

Bug description:
  Wget eats RAM when downloading big website. To reproduce run these on slow internet connection:
  bash$ wget -r -k -p -l inf -U 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' 'http://bigwebsite.com/' 2> stderr.txt
  After one ~6 hours wget eats 1.5 GiB of RAM.

  ProblemType: Bug
  DistroRelease: Ubuntu 12.04
  Package: wget 1.13.4-2ubuntu1
  ProcVersionSignature: Ubuntu 3.8.0-36.52~precise1-generic 3.8.13.14
  Uname: Linux 3.8.0-36-generic x86_64
  ApportVersion: 2.0.1-0ubuntu17.6
  Architecture: amd64
  Date: Wed Mar 12 15:29:05 2014
  InstallationMedia: Ubuntu 12.04.3 LTS "Precise Pangolin" - Release amd64 (20130820.1)
  MarkForUpload: True
  SourcePackage: wget
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1291373/+subscriptions