mattsu2020:sort-memory-functions - Branch - uutils/coreutils

feat(sort): auto-tune buffer sizing from available memory

#8959Merged

Comparing

mattsu2020:sort-memory-functions

(

72201b2

) with

main

(

4c68e1e

)

-3%

Regressions: 1

Untouched: 103

Archived: 75

Benchmarks

75 archived benchmarks were run. If they were deleted in another branch, consider rebasing to remove them from the report. .

Failed

sort_long_line[160000]

src/uu/sort/benches/sort_bench.rs

Regression

-3%

1.8 ms1.9 ms

Passed

sort_mixed_utf8_locale

src/uu/sort/benches/sort_locale_bench.rs

95 ms94.8 ms

sort_dictionary_order[500000]

src/uu/sort/benches/sort_bench.rs

1.1 s1.1 s

du_summarize_balanced_tree[(5, 4, 10)]

src/uu/du/benches/du_bench.rs

8.5 ms8.5 ms

du_all_wide_tree[(5000, 500)]

src/uu/du/benches/du_bench.rs

18.2 ms18.2 ms

cp_recursive_wide_tree[(6000, 800)]

src/uu/cp/benches/cp_bench.rs

188.1 ms187.9 ms

sort_numeric[500000]

src/uu/sort/benches/sort_bench.rs

1.2 s1.2 s

split_bytes

src/uu/split/benches/split_bench.rs

497.1 µs496.8 µs

wc_bytes_synthetic[500]

src/uu/wc/benches/wc_bench.rs

163 µs162.9 µs

sort_accented_data[500000]

src/uu/sort/benches/sort_bench.rs

828.8 ms828.5 ms

cp_recursive_deep_tree[(120, 4)]

src/uu/cp/benches/cp_bench.rs

13.3 ms13.3 ms

hashsum_md5

src/uu/hashsum/benches/hashsum_bench.rs

32 ms32 ms

sort_ascii_only[500000]

src/uu/sort/benches/sort_bench.rs

807.3 ms807.2 ms

mv_force_overwrite

src/uu/mv/benches/mv_bench.rs

119.1 ms119.1 ms

tsort_wide_dag[100000]

src/uu/tsort/benches/tsort_bench.rs

135.5 ms135.5 ms

rm_single_file

src/uu/rm/benches/rm_bench.rs

103.4 ms103.4 ms

ls_recursive_mixed_tree

src/uu/ls/benches/ls_bench.rs

1.4 ms1.4 ms

mv_single_file

src/uu/mv/benches/mv_bench.rs

118.9 ms118.9 ms

tsort_complex_dag[50000]

src/uu/tsort/benches/tsort_bench.rs

84 ms84 ms

hashsum_sha512

src/uu/hashsum/benches/hashsum_bench.rs

111.7 ms111.7 ms

wc_lines_large_line_count[500000]

src/uu/wc/benches/wc_bench.rs

2.9 ms2.9 ms

wc_lines_variable_length[(50, 500)]

src/uu/wc/benches/wc_bench.rs

3.4 ms3.4 ms

mv_directory

src/uu/mv/benches/mv_bench.rs

3.1 ms3.1 ms

wc_chars_large_line_count[100000]

src/uu/wc/benches/wc_bench.rs

990.8 µs990.8 µs

rm_force_files

src/uu/rm/benches/rm_bench.rs

2.1 ms2.1 ms

wc_lines_extreme_line_lengths[(100000, 200)]

src/uu/wc/benches/wc_bench.rs

1.6 ms1.6 ms

ls_recursive_balanced_tree[(6, 4, 15)]

src/uu/ls/benches/ls_bench.rs

55.6 ms55.6 ms

wc_words_large_line_count[100000]

src/uu/wc/benches/wc_bench.rs

43.8 ms43.8 ms

rm_multiple_files

src/uu/rm/benches/rm_bench.rs

2.2 ms2.2 ms

numfmt_to_si_precision[10000]

src/uu/numfmt/benches/numfmt_bench.rs

35.5 ms35.5 ms

wc_default_large_line_count[100000]

src/uu/wc/benches/wc_bench.rs

48.9 ms48.9 ms

cut_fields_tab

src/uu/cut/benches/cut_bench.rs

33.9 ms33.9 ms

cut_fields_custom_delim

src/uu/cut/benches/cut_bench.rs

40.5 ms40.5 ms

numfmt_from_si[10000]

src/uu/numfmt/benches/numfmt_bench.rs

33.1 ms33.1 ms

numfmt_large_numbers_si[10000]

src/uu/numfmt/benches/numfmt_bench.rs

34.6 ms34.6 ms

expand_many_lines[100000]

src/uu/expand/benches/expand_bench.rs

100.9 ms100.9 ms

nl_large_file[10]

src/uu/nl/benches/nl_bench.rs

100.8 ms100.8 ms

rm_recursive_tree

src/uu/rm/benches/rm_bench.rs

13.7 ms13.7 ms

numfmt_round_modes[("towards-zero", 10000)]

src/uu/numfmt/benches/numfmt_bench.rs

34.5 ms34.5 ms

numfmt_round_modes[("down", 10000)]

src/uu/numfmt/benches/numfmt_bench.rs

34.6 ms34.6 ms

numfmt_padding[(10000, 50)]

src/uu/numfmt/benches/numfmt_bench.rs

47 ms47 ms

wc_bytes_lines_synthetic[2000]

src/uu/wc/benches/wc_bench.rs

120.2 ms120.2 ms

numfmt_to_iec[10000]

src/uu/numfmt/benches/numfmt_bench.rs

34 ms34 ms

numfmt_round_modes[("up", 10000)]

src/uu/numfmt/benches/numfmt_bench.rs

32.8 ms32.8 ms

wc_words_synthetic[2000]

src/uu/wc/benches/wc_bench.rs

10.3 s10.3 s

fold_many_lines[100000]

src/uu/fold/benches/fold_bench.rs

125.3 ms125.3 ms

hashsum_sha256

src/uu/hashsum/benches/hashsum_bench.rs

173.6 ms173.6 ms

expand_custom_tabstops[50000]

src/uu/expand/benches/expand_bench.rs

25.9 ms25.9 ms

unexpand_many_lines[100000]

src/uu/unexpand/benches/unexpand_bench.rs

261.4 ms261.4 ms

unexpand_large_file[10]

src/uu/unexpand/benches/unexpand_bench.rs

548 ms548 ms

nl_many_lines[100000]

src/uu/nl/benches/nl_bench.rs

79.7 ms79.7 ms

hashsum_sha256_check

src/uu/hashsum/benches/hashsum_bench.rs

173.6 ms173.6 ms

hashsum_sha1

src/uu/hashsum/benches/hashsum_bench.rs

72.5 ms72.5 ms

numfmt_to_si[10000]

src/uu/numfmt/benches/numfmt_bench.rs

33.8 ms33.8 ms

fold_custom_width[50000]

src/uu/fold/benches/fold_bench.rs

46.6 ms46.6 ms

hashsum_md5_check

src/uu/hashsum/benches/hashsum_bench.rs

32 ms32 ms

ls_recursive_wide_tree[(10000, 1000)]

src/uu/ls/benches/ls_bench.rs

51.4 ms51.4 ms

seq_formatted

src/uu/seq/benches/seq_bench.rs

99.7 ms99.7 ms

uniq_case_insensitive[10000]

src/uu/uniq/benches/uniq_bench.rs

1.7 ms1.7 ms

cut_bytes

src/uu/cut/benches/cut_bench.rs

8.4 ms8.4 ms

cut_characters

src/uu/cut/benches/cut_bench.rs

7.5 ms7.5 ms

seq_custom_separator

src/uu/seq/benches/seq_bench.rs

29.2 ms29.2 ms

seq_integers

src/uu/seq/benches/seq_bench.rs

26.7 ms26.7 ms

seq_with_step

src/uu/seq/benches/seq_bench.rs

13.6 ms13.6 ms

uniq_with_count[10000]

src/uu/uniq/benches/uniq_bench.rs

1.8 ms1.8 ms

uniq_heavy_duplicates[10000]

src/uu/uniq/benches/uniq_bench.rs

1.9 ms1.9 ms

split_lines

src/uu/split/benches/split_bench.rs

5.9 ms5.9 ms

tsort_linear_chain[1000000]

src/uu/tsort/benches/tsort_bench.rs

1.5 s1.5 s

sort_key_field[500000]

src/uu/sort/benches/sort_bench.rs

697.9 ms698 ms

b64_encode_synthetic

src/uu/base64/benches/base64_bench.rs

143 µs143 µs

sort_unique_locale[500000]

src/uu/sort/benches/sort_bench.rs

1.2 s1.2 s

tsort_tree_dag[(10, 3)]

src/uu/tsort/benches/tsort_bench.rs

34.1 ms34.1 ms

sort_case_sensitive[500000]

src/uu/sort/benches/sort_bench.rs

337.6 ms337.6 ms

sort_case_insensitive[500000]

src/uu/sort/benches/sort_bench.rs

427.6 ms427.7 ms

b64_decode_ignore_garbage_synthetic

src/uu/base64/benches/base64_bench.rs

146 µs146 µs

split_numeric_suffix

src/uu/split/benches/split_bench.rs

6.1 ms6.1 ms

cp_large_file[16]

src/uu/cp/benches/cp_bench.rs

293.2 µs293.3 µs

ls_recursive_long_all_balanced_tree[(6, 4, 15)]

src/uu/ls/benches/ls_bench.rs

141.5 ms141.5 ms

ls_recursive_long_all_deep_tree[(100, 4)]

src/uu/ls/benches/ls_bench.rs

3.1 ms3.1 ms

b64_decode_synthetic

src/uu/base64/benches/base64_bench.rs

146.2 µs146.2 µs

split_number_chunks

src/uu/split/benches/split_bench.rs

271 µs271 µs

ls_recursive_long_all_mixed_tree

src/uu/ls/benches/ls_bench.rs

4 ms4 ms

ls_recursive_long_all_wide_tree[(15000, 1500)]

src/uu/ls/benches/ls_bench.rs

147.7 ms147.7 ms

ls_recursive_deep_tree[(200, 2)]

src/uu/ls/benches/ls_bench.rs

2.1 ms2.1 ms

sort_mixed_data[500000]

src/uu/sort/benches/sort_bench.rs

738.8 ms739 ms

cp_preserve_metadata[(5, 4, 10)]

src/uu/cp/benches/cp_bench.rs

74.2 ms74.2 ms

du_human_balanced_tree[(5, 4, 10)]

src/uu/du/benches/du_bench.rs

10.5 ms10.5 ms

cp_archive_balanced_tree[(5, 4, 10)]

src/uu/cp/benches/cp_bench.rs

76.9 ms76.9 ms

cp_recursive_balanced_tree[(5, 4, 10)]

src/uu/cp/benches/cp_bench.rs

71.6 ms71.6 ms

du_balanced_tree[(5, 4, 10)]

src/uu/du/benches/du_bench.rs

9.3 ms9.3 ms

sort_unique_mixed

src/uu/sort/benches/sort_locale_bench.rs

88.5 ms88.6 ms

mv_multiple_to_dir

src/uu/mv/benches/mv_bench.rs

13.2 ms13.2 ms

sort_reverse_mixed

src/uu/sort/benches/sort_locale_bench.rs

94.2 ms94.2 ms

sort_mixed_c_locale

src/uu/sort/benches/sort_locale_bench.rs

94.7 ms94.8 ms

du_deep_tree[(100, 3)]

src/uu/du/benches/du_bench.rs

1.1 ms1.1 ms

sort_reverse_locale[500000]

src/uu/sort/benches/sort_bench.rs

830 ms831 ms

du_max_depth_balanced_tree[(6, 4, 10)]

src/uu/du/benches/du_bench.rs

33.1 ms33.2 ms

sort_ascii_utf8_locale

src/uu/sort/benches/sort_locale_bench.rs

59.2 ms59.3 ms

du_all_balanced_tree[(4, 3, 10)]

src/uu/du/benches/du_bench.rs

1.6 ms1.6 ms

sort_ascii_c_locale

src/uu/sort/benches/sort_locale_bench.rs

29.4 ms29.5 ms

sort_numeric

src/uu/sort/benches/sort_locale_bench.rs

22.7 ms22.8 ms

sort_german_locale

src/uu/sort/benches/sort_locale_bench.rs

95.5 ms95.8 ms

sort_german_c_locale

src/uu/sort/benches/sort_locale_bench.rs

95.4 ms95.8 ms

du_wide_tree[(5000, 500)]

src/uu/du/benches/du_bench.rs

-1%

11.3 ms11.4 ms

Commits

Click on a commit to change the comparison range

Base

main

4c68e1e

-3.03%

Merge branch 'uutils:main' into sort-memory-functions

e197e64

28 days ago

by mattsu2020

docs: add 'sysconf' to jargon wordlist Added 'sysconf' to the CSpell jargon dictionary to include the system configuration function term, preventing false positives in spell-checking for technical code.

7160f59

28 days ago

by mattsu2020

+3.03%

refactor(sort): extract buffer hint logic to separate module Moved buffer size calculation functions (e.g., automatic_buffer_size, file_size_hint) to a new buffer_hint module for better code organization and modularity. Removed the Linux-specific nix dependency as the memory hint functionality is now handled internally without external crates.

e694ae6

28 days ago

by mattsu2020

-0.06%

refactor(sort): Explicitly cast to u128 in physical_memory_bytes_unix Updated the `physical_memory_bytes_unix` function to explicitly cast `pages` and `page_size` to `u128` before multiplication, ensuring safe arithmetic and preventing potential overflow. Also added "libc" dependency to `fuzz/Cargo.lock` to support the changes.

0ebf12b

28 days ago

by mattsu2020

-0.02%

style(buffer_hint): remove unnecessary blank line in physical_memory_bytes_unix Minor code style cleanup to improve readability and consistency in the buffer hint module.

280e127

28 days ago

by mattsu2020

+0.02%

refactor(sort): remove unnecessary return statement in physical_memory_bytes The explicit return is redundant in Rust, as the last expression in a block is implicitly returned, improving code style and adherence to idiomatic practices.

ddb36bc

28 days ago

by mattsu2020

-2.94%

fix: correct typo in buffer_hint.rs comment Corrected "overcommitting" to "overcommit" in the comment for accurate spelling.

bfa172e

28 days ago

by mattsu2020

+3%

Merge branch 'uutils:main' into sort-memory-functions

d273a69

27 days ago

by mattsu2020

-3.03%

docs: add license header to buffer_hint.rs Added standard copyright and license header to the buffer_hint.rs file in the sort utility to comply with project licensing requirements and ensure proper attribution.

92a4574

27 days ago

by mattsu2020

+3.06%

Update src/uu/sort/src/buffer_hint.rs Removal of unnecessary variables Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>

f8de88e

27 days ago

by mattsu2020

-3.07%

docs(sort): add comment explaining memory detection limitation Added a comment in the `physical_memory_bytes` function for non-Unix or Redox targets to clarify why `None` is returned, improving code readability and documenting the absence of a portable API for detecting total physical memory.

5725d06

27 days ago

by mattsu2020

+6.31%

refactor(sort): enhance physical memory detection for Unix systems Updated cfg conditions in `physical_memory_bytes` to explicitly handle Linux and Android targets. Refined `physical_memory_bytes_unix` with improved error handling for sysconf calls, ensuring safer memory page and size calculations while removing fallback libc implementation for non-Linux/Android Unix systems.

f941f1c

27 days ago

by mattsu2020

-0.87%

refactor(uu/sort): remove libc dependency and use named constants for batch size Removed the libc crate dependency from Cargo.toml and Cargo.lock to reduce unnecessary dependencies. Refactored default_merge_batch_size() in sort.rs to use named constants (LINUX_BATCH_DIVISOR, LINUX_BATCH_MIN, LINUX_BATCH_MAX) for better code readability and maintainability, while preserving the same dynamic batch size calculation logic based on file descriptor limits.

5586e8a

27 days ago

by mattsu2020

-2.39%

refactor(sort): reorder imports in buffer_hint.rs for consistency Reordered the imports in the `use nix::unistd` statement to place `SysconfVar` before `sysconf`, improving code readability and adhering to alphabetical import ordering conventions.

c7298c9

27 days ago

by mattsu2020

-3.03%

fix Cargo.lock linux enviroments

7fd534e

28 days ago

by mattsu2020

Merge branch 'uutils:main' into sort-memory-functions

08a9548

26 days ago

by mattsu2020

Merge branch 'uutils:main' into sort-memory-functions

19fd282

26 days ago

by mattsu2020

-0.05%

Merge branch 'uutils:main' into sort-memory-functions

72201b2

26 days ago

by mattsu2020

Home Terms Privacy Docs