albertedwardson:perf - Branch - scrapy/w3lib

Improve performance and readability of some `w3lib.html` functions

#256Merged

Comparing

albertedwardson:perf

(

df06647

) with

master

(

c44e4b6

)

+67%

Improvement: 2

Untouched: 29

New: 1

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments, which may affect the accuracy of the results. Learn more about regression causes.

Benchmarks

32 total

test_benchmark_html[remove_tags_with_content]

tests/benchmarks/test_benchmark_html.py

+67%

193.6 µs115.7 µs

test_benchmark_html[remove_tags]

tests/benchmarks/test_benchmark_html.py

+60%

477.1 µs297.3 µs

test_benchmark_html[has_entities]

tests/benchmarks/test_benchmark_html.py

N/A

N/A55.2 µs

test_benchmark_html[unquote_markup]

tests/benchmarks/test_benchmark_html.py

+7%

142.9 µs133.3 µs

test_benchmark_html[get_meta_refresh]

tests/benchmarks/test_benchmark_html.py

+4%

860.1 µs825.6 µs

test_benchmark_html[replace_entities]

tests/benchmarks/test_benchmark_html.py

+1%

428.2 µs425.4 µs

test_benchmark_encoding[read_bom]

tests/benchmarks/test_benchmark_encoding.py

38.2 µs38 µs

test_benchmark_url[any_to_uri]

tests/benchmarks/test_benchmark_url.py

125.4 µs125 µs

test_benchmark_encoding[html_body_declared_encoding]

tests/benchmarks/test_benchmark_encoding.py

290.4 µs289.6 µs

test_benchmark_encoding[to_unicode]

tests/benchmarks/test_benchmark_encoding.py

24.5 µs24.4 µs

test_benchmark_url[url_query_parameter]

tests/benchmarks/test_benchmark_url.py

91.2 µs91 µs

test_benchmark_encoding[http_content_type_encoding]

tests/benchmarks/test_benchmark_encoding.py

59.4 µs59.3 µs

test_benchmark_encoding[resolve_encoding]

tests/benchmarks/test_benchmark_encoding.py

75.5 µs75.4 µs

test_benchmark_html[replace_escape_chars]

tests/benchmarks/test_benchmark_html.py

77.7 µs77.6 µs

test_benchmark_url[file_uri_to_path]

tests/benchmarks/test_benchmark_url.py

81.3 µs81.3 µs

test_benchmark_url[url_query_cleaner]

tests/benchmarks/test_benchmark_url.py

275.6 µs275.5 µs

test_benchmark_url[parse_data_uri]

tests/benchmarks/test_benchmark_url.py

604.4 µs604.2 µs

test_benchmark_url[path_to_file_uri]

tests/benchmarks/test_benchmark_url.py

165.7 µs165.7 µs

test_benchmark_url[add_or_replace_parameter]

tests/benchmarks/test_benchmark_url.py

636.8 µs636.9 µs

test_benchmark_url[add_or_replace_parameters]

tests/benchmarks/test_benchmark_url.py

317.1 µs317.2 µs

test_benchmark_url[safe_download_url]

tests/benchmarks/test_benchmark_url.py

562 µs562.4 µs

test_benchmark_html[get_base_url]

tests/benchmarks/test_benchmark_html.py

503.2 µs503.7 µs

test_benchmark_http[basic_auth_header]

tests/benchmarks/test_benchmark_http.py

49.2 µs49.3 µs

test_benchmark_url[safe_url_string]

tests/benchmarks/test_benchmark_url.py

4.5 ms4.5 ms

test_benchmark_encoding[html_to_unicode]

tests/benchmarks/test_benchmark_encoding.py

338 µs338.6 µs

Commits

Click on a commit to change the comparison range

Base

master

c44e4b6

-11.99%

Improve performance and readability of some functions

8373b0c

8 days ago

by albertedwardson

+0.03%

revert changes in `get_meta_refresh`

a575137

8 days ago

by albertedwardson

+73.94%

don't use `set` in `replace_entities`

f6af482

8 days ago

by albertedwardson

-0.25%

revert `replace_entities`

c0e3df7

8 days ago

by albertedwardson

+0.3%

up coverage, add missing tests

7dd5f8f

7 days ago

by albertedwardson

+5.4%

try `replace_escape_chars` with cache, remove `from re import Match`

0005a13

7 days ago

by albertedwardson

-0.34%

fix typing

8bee692

7 days ago

by albertedwardson

+0.48%

fix typos, simplify imports

ff9ac4b

6 days ago

by albertedwardson

-0.13%

try to improve `get_meta_refresh`

df06647

6 days ago

by albertedwardson

Home Terms Privacy Docs