[Engineering] Issues on Cluster

Fri Jan 29 21:28:39 UTC 2016

Hi Folks,
Data ingestion problem have been solved without data loss, but a lot
re-computation is needed.
I hope it'll be done over the weekend but in it's not I'll send an email on
monday.
Have a good weekend !
Joseph

On Fri, Jan 29, 2016 at 12:03 PM, Joseph Allemandou <
jallemandou at wikimedia.org> wrote:

> Hi,
> The hadoop cluster is still having issues this morning (ingestion from
> kafka fails on one partition).
> Please refrain from using it, it needs all possible resources to catch
> back :)
> Thanks
> Joseph
>
> On Thu, Jan 28, 2016 at 2:55 PM, Joseph Allemandou <
> jallemandou at wikimedia.org> wrote:
>
>> Hi again,
>> The data ingestion problem is fixed, and the cluster is starting the race
>> against new data coming in.
>> For the moment we are about 20 hours behind on the two big partitions
>> (text and upload), so catching up will take some time.
>> We'll keep you updated as things go along.
>> Cheers
>> Joseph
>>
>> On Thu, Jan 28, 2016 at 11:57 AM, Joseph Allemandou <
>> jallemandou at wikimedia.org> wrote:
>>
>>> Hi Analytics fellows,
>>>
>>> We are experiencing issues with loading data into the hadoop cluster,
>>> therefore blocking the full job pipeline.
>>> When fixed, the cluster will be heavily loaded trying to catch up, so
>>> please, be nice with it and don't run heavy jobs in the next hours.
>>> We'll keep you posted about resolution.
>>> Many thanks, and sorry for the inconvenience.
>>> Joseph
>>>
>>>
>>
>>
>> --
>> *Joseph Allemandou*
>> Data Engineer @ Wikimedia Foundation
>> IRC: joal
>>
>
>
>
> --
> *Joseph Allemandou*
> Data Engineer @ Wikimedia Foundation
> IRC: joal
>

-- 
*Joseph Allemandou*
Data Engineer @ Wikimedia Foundation
IRC: joal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160129/8034734e/attachment-0001.html>