[Xmldatadumps-admin-l] FR, IO, DE, EN, SV Wiki Failures

Thu May 14 22:24:24 UTC 2009

Tomasz Finc wrote:
> Tomasz Finc wrote:
>> This was a nasty mine to step on. While updating the template file to 
>> correctly update the links in the status file a variable wasn't properly 
>> declared. One exception and the whole system spun out of control and died.
>>
>> That's just silly. Missing to update a status file on one iteration 
>> should not kill the build out right. I've added a fail safe so that if 
>> this happens again, the other portions of the build that are working 
>> just fine will continue to.
>>
>> Thus two lessons:
>>
>> 1) Depend on test environment more
>> 2) Add catch blocks to the code so that small failures do not bring 
>> the 	   system down.
>>
>> --tomasz
>>
>> _______________________________________________
>> Xmldatadumps-admin-l mailing list
>> Xmldatadumps-admin-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-admin-l
> 
>   |-sh(15466)---php(15467)-+-sh(15468)---bzip2(15469)
> 
> Amusingly enough the linux kernel did something really helpful here. 
> When the python worker died, the kernel re-assigned the build manager to 
> init and kept going.
> 
> What this means is that some of these are still going unmanaged. DE & ES 
>   are both chugging along and I'll be watching them closely until they 
> finish their run. Sadly we had to kill EN due to high load on the db.
> 
> I'll update the status pages to reflect which ones are still going.
> 
> So far its
> 
> EN: pending db stability
> FR: dead, restarted, running
> DE: running, ETA 2009-05-14 07:33:40
> ES: running, ETA 2009-05-12 21:12:5
> IO: finished
> SV: dead, restarted, running
> 
> 
> There are parallel DE & ES jobs that got kicked off after the failures 
> that I'll kill if the left over procs continue to get work.
> 
> --tomasz
> 
> _______________________________________________
> Xmldatadumps-admin-l mailing list
> Xmldatadumps-admin-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-admin-l

eswiki 20090504 finished happily last night and is now complete
dewiki 20090506 finished pages-meta-history.xml.bz2 last night and I'm 
running 7z compression now. This will take a very long time to finish 
but anyone needing the full history can consume the bz2 file which has 
the same content.

Both of these are really great to have as we hadn't had a full history 
dump from eswiki & dewiki since January 2009.

That leaves us with only enwiki & frwiki missing from our normal 
rotation. Both of which are moving well through their data sets.

--tomasz