Hi Robert, welcome to the list.  Opening big files is definitely tricky.  I believe (not sure) folks on this list usually write scripts to filter and aggregate these files, or load them into big data tools.  But I found someone else asking a similar question with a very useful answer here: https://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files

On Thu, Jan 26, 2023 at 11:04 AM Robert Garrigos <robert@garrigos.cat> wrote:
Hi,

I just enrolled this list, thanks to Dan Andreescu, who let me know
about it, and I have a question on processing clickstream data.

I downloaded a file for last month clickstream data
(https://dumps.wikimedia.org/other/clickstream/2022-12/clickstream-eswiki-2022-12.tsv.gz)
and have problems to open it and processing it.

The only programme I could open it was OpenRefine. Other programmes
(Numbers and LibreOffice) just couldn't cope with it.

I can use OpenRefine to do some transformation and delete some rows I
don't need, but even then, with some 1.5milion rows, I can not open it
with numbers or libreoffice to do sum of the column 4.

Which tools do you use to work with such big files?

Thanks.
--
========================
Robert Garrigós i Castro
https://garrigos.cat
+34 620 91 87 01
_______________________________________________
Analytics mailing list -- analytics@lists.wikimedia.org
To unsubscribe send an email to analytics-leave@lists.wikimedia.org