[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[ ] I have confirmed this bug exists on the main branch of pandas.
import pandas as pd
pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time').groupby([]).idxmin().dtypes
# output:
# value int64
# dtype: object
# compare to the result on a non-empty DataFrame
# pd.DataFrame(data=[['2023-01-26',1]],columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time').groupby([0]).idxmin().dtypes
# output:
# value datetime64[ns]
# dtype: object
# the following lines can also reproduce the problem:
pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time').groupby([]).idxmax().dtypes
# output:
# value int64
# dtype: object
pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time')['value'].groupby([]).idxmin().dtypes
# output:
# dtype('int64')
pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time')['value'].groupby([]).idxmax().dtypes
# output:
# dtype('int64')
On a non-empty Series/DataFrame, calling .groupby.idxmin/idxmax returns Series/DataFrame with the index dtype (datetime64[ns]). But the same operation on an empty Series/DataFrame returns Series/DataFrame with the dtype of original column (int64), which is inconsistent.
The GroupBy.idxmin/idxmax should produce Series/DataFrame with the index dtype, like the result shown below:
pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time').groupby([]).idxmin().dtypes
# output:
# value datetime64[ns]
# dtype: object
commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : 3.10.8.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19045 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : Chinese (Simplified)_China.936 pandas : 1.5.3 numpy : 1.23.4 pytz : 2021.3 dateutil : 2.8.2 setuptools : 61.2.0 pip : 21.2.4 Cython : None pytest : 7.1.1 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.0.3 IPython : 8.3.0 pandas_datareader: None bs4 : None bottleneck : 1.3.4 brotli : 1.0.9 fastparquet : None fsspec : None gcsfs : None matplotlib : None numba : 0.56.4 numexpr : 2.8.1 odfpy : None openpyxl : None pandas_gbq : None pyarrow : 8.0.0 pyreadstat : None pyxlsb : None s3fs : None scipy : 1.9.3 snappy : None sqlalchemy : None tables : 3.7.0 tabulate : None xarray : None xlrd : None xlwt : None zstandard : None tzdata : None
Thanks for the report! Agreed on the expectation of retaining the index type, and confirmed on main. Further investigations and pull requests to fix are welcome!
I think it's mainly because here that keeps the dtype of the returned value consistent with the user input and does not return the dtype of the desired Index.
When using idxmin() on an empty DataFrame, it will report an error directly. I wonder if it is possible to report an error directly for DataFrameGroupy instead of returning a null value. Or is there any better way to handle this and any suggestions?
df3 = pd.DataFrame(columns=['time','value']).astype({'time':'datetime64[ns]','value':'int64'}).set_index('time')
df3.idxmin( )
# ValueError: attempt to get argmin of an empty sequence
I wonder if it is possible to report an error directly for DataFrameGroupy instead of returning a null value.
I think the mental model for groupby should be:
for group in groups:
results.append(f(group))
combine(results)
With this, the expectation is that idxmin is never actually computed, and so this shouldn't raise.
Or is there any better way to handle this and any suggestions?
The two easiest solutions I see are to (a) not call _wrap_applied_output or (b) add how as an argument to this method and add in idxmin/max specific logic, but both of these seem to be not good. A much more intensive, but better solution, would be to develop a specific, performant, algorithm for groupby idxmin/max and sidestep calling _wrap_applied_output altogether.
A much more intensive, but better solution, would be to develop a specific, performant, algorithm for groupby idxmin/max and sidestep calling _wrap_applied_output altogether.
Totally agree, I want to add
if f.__name__ in ['idxmin', 'idxmax']:
return _func_instead_warp_appied_ouput(......)
here.
Does this change break any design rules or have any potential problems?