fossilesque@mander.xyzM to Science Memes@mander.xyzEnglish · 1 year agoPandasmander.xyzimagemessage-square23fedilinkarrow-up1234arrow-down17
arrow-up1227arrow-down1imagePandasmander.xyzfossilesque@mander.xyzM to Science Memes@mander.xyzEnglish · 1 year agomessage-square23fedilink
minus-squareKausta@lemm.eelinkfedilinkEnglisharrow-up39·1 year agoYou havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note.
minus-squareQuizzaciousOtter@lemm.eelinkfedilinkEnglisharrow-up9·1 year agoI really don’t think that’s a lot either. Nowadays we routinely process terabytes of data.
minus-squareKausta@lemm.eelinkfedilinkEnglisharrow-up9·1 year agoYeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.
minus-squarewhotookkarl@lemmy.worldlinkfedilinkEnglisharrow-up4·1 year agoIt’s good to see the occult is still alive and well
You havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note.
I really don’t think that’s a lot either. Nowadays we routinely process terabytes of data.
Yeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.
It’s good to see the occult is still alive and well