[Labs-l] Rolling reboots today (finished)

Wed Jan 28 06:53:38 UTC 2015

After many concerted Ops hours, virt1009 appears to be back up and 
running along with its respective instances.  That means that all 
instances that were up yesterday should now be back up and running.

If you find that your instance did /not/ come back up, feel free to 
respond to this email, email me off-list, or contact me on IRC tomorrow.

Unfortunately, this email does not entirely conclude the 'reboot 
everything' phase, because the NFS servers are still in line for a 
restart.  That will most likely happen tomorrow and result in temporary 
freezes of shared filesystems -- watch this list for updates.

Thanks for your patience with today's action -- security issues like 
this are always painful to deal with, but not dealing with them would be 
even more painful.

-Andrew

On 1/27/15 3:29 PM, Andrew Bogott wrote:
> Update:
>
> Most, but not all, labs instances have now been restarted.  Those 
> instances essential for Tools operation are working -- some tools are 
> back up, but those that depend on web access are still failing.  Yuvi 
> is working hard on this issue.  Coren is unfortunately out sick, so 
> it's difficult to predict when he'll be available to assist.
>
> One of the hardware nodes, virt1009, is suffering a serious OS 
> problem, so VMs local to that host are still shut off.  We're working 
> frantically on resolving this issue and I'll update you as soon as 
> there is any news.
>
> -Andrew
>
>
> On 1/27/15 9:41 AM, Andrew Bogott wrote:
>> Due to a newly-discovered security vulnerability[1], I will be 
>> patching and rebooting ALL labs instances today.  I'll coordinate 
>> with Coren on tool labs to minimize the breakage; however, everyone 
>> should be on alert to check the status of their services; some things 
>> may need hand-tuning.
>>
>> I'll send another email when the reboots are completed -- don't 
>> hesitate to speak up on IRC if you have concerns.
>>
>> -Andrew
>>
>> [1] http://www.ubuntu.com/usn/usn-2485-1/
>