[Labs-announce] Tools NFS disruptive maintenance scheduled on 11/2 is being postponed to 11/14

Mon Nov 14 18:15:40 UTC 2016

Most of tools should be back up now. Expected behavior for the next many
hours (Expected ~12-14 hours based on last sync of tools data, but might be
much longer)

Quoting from previous announcement [1]:

* The tools NFS share (/data/project and /home) will be read-only for the
> duration of the maintenance, so no new data or logs will get written to it.
> * New jobs cannot be submitted for the whole maintenance window - this
> means submitting jobs through cron or tools-mail will not function,
> although tools-mail can continue to send emails.
> * Current jobs might keep running, but won't get rescheduled if they die.
> If they do not die and aren't writing to NFS they should be fine.

[1]: https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html

On Mon, Nov 14, 2016 at 9:31 AM, Madhumitha Viswanathan <
mviswanathan at wikimedia.org> wrote:

> All of tools is down currently - unexpected side effect of the ongoing
> maintenance - we are working on it, I will update once it's back up.
> Apologies for the inconvenience caused.
>
> On Mon, Nov 14, 2016 at 7:59 AM, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
>> Hi all,
>>
>> This is starting in a few minutes. You can follow along progress on
>> #wikimedia-labs or at T146154 <https://phabricator.wikimedia.org/T146154>
>> .
>>
>>
>> On Fri, Nov 11, 2016 at 10:16 AM, Madhumitha Viswanathan <
>> mviswanathan at wikimedia.org> wrote:
>>
>>> Hi all,
>>>
>>> This migration is scheduled to happened on Monday 11/14. We will start
>>> at 08:00 PST/16:00 UTC. More details here - https://phabricator.wikimedi
>>> a.org/T146154.
>>>
>>> On Tue, Nov 1, 2016 at 11:47 AM, Madhumitha Viswanathan <
>>> mviswanathan at wikimedia.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We had previously announced a disruptive maintenance for tools[1] on
>>>> 11/2(tomorrow), but we are going to have to postpone it tentatively to
>>>> 11/14/2016.
>>>>
>>>> Apologies for the late notice, but the server in our secondary
>>>> datacenter in Dallas that hosts data backups is having issues since
>>>> yesterday[2], and we are currently still debugging and fixing the problems
>>>> there. We've decided not to proceed with this migration until our backups
>>>> there are online.
>>>>
>>>> As always, do let us know if there are any questions/concerns here or
>>>> on #wikimedia-labs.
>>>>
>>>> [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-Octob
>>>> er/004698.html
>>>> [2]: https://phabricator.wikimedia.org/T149567
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
>

-- 
--Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-announce/attachments/20161114/06835e69/attachment.html>