Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this issue exists on the latest version of pandas.

  • [X] I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

A major bottleneck of Categorical.from_codes is the validation step where 'codes' is checked if all of its values are in range see source code. The issue is that the check cannot be disabled. However, it is probably a common use-case for 'Categorical.from_codes' to be utilized as a raw constructor, i.e. to construct a categorical where it is known that the codes are already in a valid range. This check causes a significant slowdown for large datasets.

So it would be good to have a parameter which can be used to disable the validity checking (it is probably not a good idea to remove the check entirely). Ideally one would also make the following coecion step disableable (see)

Background: I have a 17 GB dataset with a lot of categorical columns, where loading (with memory mapping) takes ~27 sec with present pandas, but without the Categorical.from_codes validation only 2 sec (I edited pandas source code for that). So more than a factor of 10 faster. Validation is not needed in this case as correctness is ensured when saing the data.


import mmap
import numpy
import time
import pandas

with open('codes.bin', 'w+b') as f:
    map = mmap.mmap(f.fileno(), 1024*1024*1024, access=mmap.ACCESS_READ)
    codes = numpy.frombuffer(map, dtype=numpy.int8)
    codes.flags.writeable = False

start = time.time()
c = pandas.Categorical.from_codes(codes, categories=['a', 'b', 'c'] )
print( f"from_codes time {time.time()-start} sec" )

del c
del codes


Installed Versions


commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : de_DE.cp1252

pandas : 1.5.3 numpy : 1.24.1 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 66.1.1 pip : 22.1.2 Cython : 0.29.33 pytest : 6.2.5 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None

Prior Performance

No response

© 2022 - All rights reserved.