A Hybrid Apache Arrow/Numpy DataFrame with Vaex version 4.0
A Hybrid Apache Arrow/Numpy DataFrame with Vaex version 4.0 A Hybrid Apache Arrow/Numpy DataFrame with Vaex version 4.0 The Vaex DataFrame has always been very fast. Built from the ground up to be out of core (the size of your disk is the limit), it pushes the limits of what single machines can do in the context of big data analysis. Starting from version 2, we added better support for string data, giving an almost 1000x speedup compared to Pandas at the time. To support this seemingly trivial datatype, we had to choose a disk and memory format and did not want to reinvent the wheel. ...
Source: vaex.io