fix perf: aggregate_downsample()  #13058
fix perf: aggregate_downsample()  #13058

mhvk/astropy,pllim/astropy,astropy/astropy,mhvk/astropy,pllim/astropy,astropy/astropy,pllim/astropy,astropy/astropy,mhvk/astropy,astropy/astropy,mhvk/astropy,astropy/astropy,pllim/astropy,mhvk/astropy,pllim/astropy