179-optionally-embed-tokenizer_configjson-2 - Branch - mozilla-ai/encoderfile

feat: parse padding settings from tokenizer_config.json

#183Merged

Comparing

179-optionally-embed-tokenizer_configjson-2

(

a1e255a

) with

main

(

05b9c67

)

Untouched: 20

Skipped: 20

Benchmarks

40 total

bench_seq_cls_softmax[(16, 2)]

encoderfile-core/benches/benchmark_transforms.rs

+1%

138.3 µs137.2 µs

bench_tok_cls_softmax[(16, 16, 2)]

encoderfile-core/benches/benchmark_transforms.rs

+1%

183.1 µs182 µs

bench_seq_cls_softmax[(32, 8)]

encoderfile-core/benches/benchmark_transforms.rs

+1%

147.7 µs146.9 µs

embedding_postprocess[(64, 512, 1024)]

encoderfile-core/benches/postprocessing.rs

+1%

312.1 ms310.5 ms

bench_seq_cls_softmax[(128, 32)]

encoderfile-core/benches/benchmark_transforms.rs

292.7 µs291.9 µs

token_classification_postprocess[(64, 512)]

encoderfile-core/benches/postprocessing.rs

61.3 ms61.2 ms

bench_embedding_l2_normalization[(16, 16, 16)]

encoderfile-core/benches/benchmark_transforms.rs

337.8 µs337.3 µs

token_classification_postprocess[(8, 16)]

encoderfile-core/benches/postprocessing.rs

3.6 ms3.6 ms

token_classification_postprocess[(16, 128)]

encoderfile-core/benches/postprocessing.rs

8.2 ms8.2 ms

bench_tok_cls_softmax[(32, 128, 8)]

encoderfile-core/benches/benchmark_transforms.rs

1.8 ms1.8 ms

embedding_postprocess[(8, 16, 384)]

encoderfile-core/benches/postprocessing.rs

4.6 ms4.6 ms

embedding_postprocess[(16, 128, 768)]

encoderfile-core/benches/postprocessing.rs

25.2 ms25.2 ms

bench_l2_normalize[Axis(0)]

encoderfile-core/benches/l2_norm.rs

1.6 ms1.6 ms

bench_tok_cls_softmax[(128, 256, 32)]

encoderfile-core/benches/benchmark_transforms.rs

35.3 ms35.3 ms

bench_l2_normalize[Axis(1)]

encoderfile-core/benches/l2_norm.rs

3 ms3 ms

bench_embedding_l2_normalization[(32, 128, 384)]

encoderfile-core/benches/benchmark_transforms.rs

65.3 ms65.3 ms

bench_embedding_l2_normalization[(32, 256, 768)]

encoderfile-core/benches/benchmark_transforms.rs

267 ms267 ms

sequence_classification_postprocess[64]

encoderfile-core/benches/postprocessing.rs

48.2 µs48.2 µs

sequence_classification_postprocess[8]

encoderfile-core/benches/postprocessing.rs

-1%

10.9 µs11 µs

sequence_classification_postprocess[16]

encoderfile-core/benches/postprocessing.rs

-2%

32.4 µs33 µs

The benchmarks below were skipped, so their baseline results are used instead. If they were deleted in your codebase, archive them to remove them from the performance reports.
Learn more about archiving benchmarks

bench_seq_cls_softmax[(128, 32)]

encoderfile/benches/benchmark_transforms.rs

Skipped

269.5 µs*

bench_tok_cls_softmax[(128, 256, 32)]

encoderfile/benches/benchmark_transforms.rs

Skipped

27.9 ms*

bench_embedding_l2_normalization[(32, 256, 768)]

encoderfile/benches/benchmark_transforms.rs

Skipped

266.1 ms*

embedding_postprocess[(64, 512, 1024)]

encoderfile/benches/postprocessing.rs

Skipped

311.3 ms*

bench_embedding_l2_normalization[(32, 128, 384)]

encoderfile/benches/benchmark_transforms.rs

Skipped

65.1 ms*

Commits

Click on a commit to change the comparison range

Base

main

05b9c67

+0.24%

update

8569288

1 month ago

by besaleli

-0.34%

update unit tests

14d2305

1 month ago

by besaleli

+0.05%

add unit tests

fb476ee

1 month ago

by besaleli

+0.09%

add no config test path

a1e255a

1 month ago

by besaleli

Home Terms Privacy Docs