[X] I have checked that this issue has not already been reported.
[X] I have confirmed this issue exists on the latest version of pandas.
[X] I have confirmed this issue exists on the main branch of pandas.
A major bottleneck of Categorical.from_codes is the validation step where 'codes' is checked if all of its values are in range see source code.
The issue is that the check cannot be disabled. However, it is probably a common use-case for 'Categorical.from_codes' to be utilized as a raw constructor, i.e. to construct a categorical where it is known that the codes are already in a valid range. This check causes a significant slowdown for large datasets.
So it would be good to have a parameter which can be used to disable the validity checking (it is probably not a good idea to remove the check entirely). Ideally one would also make the following coecion step disableable (see)
Background: I have a 17 GB dataset with a lot of categorical columns, where loading (with memory mapping) takes ~27 sec with present pandas, but without the Categorical.from_codes validation only 2 sec (I edited pandas source code for that). So more than a factor of 10 faster. Validation is not needed in this case as correctness is ensured when saing the data.
Example:
import mmap
import numpy
import time
import pandas
with open('codes.bin', 'w+b') as f:
f.truncate(1024*1024*1024)
map = mmap.mmap(f.fileno(), 1024*1024*1024, access=mmap.ACCESS_READ)
codes = numpy.frombuffer(map, dtype=numpy.int8)
codes.flags.writeable = False
start = time.time()
c = pandas.Categorical.from_codes(codes, categories=['a', 'b', 'c'] )
print( f"from_codes time {time.time()-start} sec" )
del c
del codes
map.close()
commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : 3.9.7.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : de_DE.cp1252
pandas : 1.5.3 numpy : 1.24.1 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 66.1.1 pip : 22.1.2 Cython : 0.29.33 pytest : 6.2.5 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None
No response
