fix: also detect verbose option in has_custom_options()
Address review feedback: has_custom_options() was missing the verbose
check, so df.show(verbose=True) without explicit format would also
fall through to the __repr__ path and ignore the option.
fix(scheduler): fix autoscaler underscaling after Ray upgrade
Two independent bugs caused the autoscaler to scale to fewer workers
than demanded:
1. Ratio underestimation: needs_autoscaling() runs after schedule_tasks()
drains pending tasks, so the demand/capacity ratio only reflected
residual demand. Track last_scheduled_count so the ratio uses total
demand (pending + just-dispatched).
2. Monotonic watermark: max_resources_requested in worker_manager never
reset, permanently blocking re-scaling after the first peak. Reset
the watermark when workers join or die so the autoscaler can
re-evaluate demand against the new topology.
Also strip zero-valued GPU/memory keys from Ray resource bundles and
add tracing to both decision points.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>