wget2 2.1.0 excessive CPU usage: all cores at 100%

[mrdc]

Hello,

I'm experiencing a bug in wget2 2.1.0: during site mirroring it occupies all cores at 100%. It's reproducible.

Command line I use:
wget2 --max-threads 15 -r -l 10 --reject-regex='wp-login.php?redirect_to=' --mirror --convert-links --adjust-extension --page-requisites --no-parent https://thetechnicgear.com

Tech specs:
Apple M1 Pro
macOS 14.5
wget2 installed via:
brew install wget2

wget2 --version

GNU Wget2 2.1.0 - multithreaded metalink/file/website downloader

+digest +https +ssl/gnutls +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
+iconv +idn2 +zlib +lzma +brotlidec +zstd +bzip2 +lzip +http2 +gpgme

Hope it helps.

[rockdaboot]

Hm, here on Linux, it still shows 10% CPU usage, which is quite high.
Can you try with --progress=none and report back?

[mrdc]

Hm, here on Linux, it still shows 10% CPU usage, which is quite high. Can you try with --progress=none and report back?

Now it shows 20% with the command line I've posted in the issue.
Using --progress=none drops CPU utilization to 7-12%, with some rare spikes up to 18%.

[rockdaboot]

Using --progress=none drops CPU utilization to 7-12%, with some rare spikes up to 18%.

That sounds reasonable downloads via HTTPS and also depends on the utilized network bandwidth and your CPU, of course.

I think, the progress bar issue is identified and can be fixed by limiting the number of bar updates per second. Just needs to be implemented =)

[woodpenker]

Hi, I just happend to be here and gave it a try, the Linux's perf result show no diff compared to --progress=none. The core CPU usage is dominate by the yylex symbol.

Samples: 8K of event 'cpu/cycles/Pu', Event count (approx.): 2809705225
Overhead  Command  Shared Object          Symbol
  11.71%  wget2    libwget.so.4.0.0       [.] yylex
   3.37%  wget2    libz.so.1.3.1.zlib-ng  [.] inflate_fast_avx2
   3.04%  wget2    libc.so.6              [.] cfree@GLIBC_2.2.5
   3.01%  wget2    libc.so.6              [.] _int_malloc
   2.79%  wget2    libc.so.6              [.] __memmove_avx_unaligned_erms
   2.53%  wget2    libwget.so.4.0.0       [.] parseXML
   2.53%  wget2    p11-kit-trust.so       [.] lookup_or_create_bucket.lto_priv.0
   2.10%  wget2    p11-kit-trust.so       [.] p11_attrs_matchn
   1.83%  wget2    libtasn1.so.6.6.5      [.] asn1_find_node
   1.81%  wget2    libc.so.6              [.] _int_free_chunk
   1.75%  wget2    p11-kit-trust.so       [.] p11_dict_ulongptr_equal
   1.75%  wget2    libc.so.6              [.] __strchr_avx2
   1.52%  wget2    libc.so.6              [.] __strlen_avx2
   1.47%  wget2    libc.so.6              [.] __libc_malloc2
   1.46%  wget2    libc.so.6              [.] __libc_calloc2
   1.41%  wget2    libc.so.6              [.] malloc
   1.29%  wget2    libtasn1.so.6.6.5      [.] asn1_array2tree
   1.19%  wget2    p11-kit-trust.so       [.] p11_index_free
   1.18%  wget2    wget2                  [.] _search_queue_for_free_job
   1.18%  wget2    libgmp.so.10.5.0       [.] __gmpn_mul_basecase_coreisbr
   1.09%  wget2    libc.so.6              [.] __memcmp_avx2_movbe
   1.04%  wget2    libc.so.6              [.] __strcmp_avx2
   0.99%  wget2    libgnutls.so.30.42.0   [.] _aesni_ctr32_ghash_6x
   0.94%  wget2    libc.so.6              [.] re_compile_internal
   0.93%  wget2    libwget.so.4.0.0       [.] wget_css_parse_buffer
   0.92%  wget2    libc.so.6              [.] _int_free_create_chunk
   0.91%  wget2    libwget.so.4.0.0       [.] getToken
   0.89%  wget2    p11-kit-trust.so       [.] sys_C_FindObjects.lto_priv.0
   0.83%  wget2    [unknown]              [k] 0xffffffffa7a01988
   0.82%  wget2    libgmp.so.10.5.0       [.] __gmpn_sqr_basecase_coreisbr
   0.79%  wget2    libwget.so.4.0.0       [.] html_get_url
   0.79%  wget2    libwget.so.4.0.0       [.] wget_strcasecmp_ascii
   0.76%  wget2    libtasn1.so.6.6.5      [.] _asn1_expand_object_id
   0.70%  wget2    libc.so.6              [.] __libc_calloc
   0.67%  wget2    libc.so.6              [.] __memset_avx2_unaligned_erms
   0.65%  wget2    libhogweed.so.6.10     [.] _nettle_ecc_secp384r1_modp
   0.62%  wget2    libwget.so.4.0.0       [.] iri_unescape_inline
   0.62%  wget2    p11-kit-trust.so       [.] index_select.lto_priv.0
   0.57%  wget2    libtasn1.so.6.6.5      [.] _asn1_change_integer_value.isra.0
   0.56%  wget2    libz.so.1.3.1.zlib-ng  [.] chunkmemset_avx2

wget2 --version

GNU Wget2 2.2.1 - multithreaded metalink/file/website downloader

+digest +https +ssl/gnutls +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
+iconv +idn2 +zlib -lzma +brotlidec +zstd -bzip2 -lzip +http2 +gpgme

[rockdaboot]

11.71% wget2 libwget.so.4.0.0 [.] yylex

That's CSS parsing. The parser is created by flex, not hand-written.
Did you also use 15 threads?

[woodpenker]

11.71% wget2 libwget.so.4.0.0 [.] yylex

That's CSS parsing. The parser is created by flex, not hand-written. Did you also use 15 threads?

yes, just copy and run the cmd above.