[Labs-admin] Ways to go forward with Kubernetes

Fri Dec 9 15:50:32 UTC 2016

Been thinking about this a bit and here is my take atm.  Numbers reference
Yuvi's original outline here.  Initial thought is do 1, then look at 4 as
the best thing, because we will end up recreating a lot of what 3 provides
in the service of doing 2.  3 makes the most sense once 4 and 2 are
resolved.

I have poked at and seen some in-depth demos of openshift and it explicitly
does not wall off direct kubectl use, and I don't think anything ever
will.  It may be that there are security contexts in which the really
PaaS-y services are boxed in and restricted in certain ways but I'm not
worried about an overlay mucking things up too much for now.  What I've
seen is more straightforward and the productization is very modular in a
typical k8s way.  I could be proven wrong but so far that's the trend I
understand.  That being said.  I want to push forward and have the
migration of webservices be #1.  I think that use case is distinct enough
to warrant buttoning it up before we move on to straight adhoc or general
jobs.  Do we have some idea of how many webservices spawn jobs on the
grid?  is it like 5 or 100?  At the moment it's hard to reason on that part
of this without knowing how much of an outlier.

I think another thing is mostly necessary to say too: no-ux change is not
possible.

It doesn't matter how neatly we tie the bow on this the illusion will fade
away the moment ancillary things are different. No way should we rewrite
qstat etc to make believe k8s isn't the backend, and I think that
jstart/jsub are just the beginning of this path.  So I'm in favor of really
opinioned change overs that come as full scale changes in expectation on
our part and we present the conceptual uses as if it were a public API.  We
don't really announce or make any effort to proselytize until we feel good
about it and we don't muddy the waters with hand wringing over users who
only ever wanted to see output in one certain format.  I'm both sensitive
to the idea that our users are especially burdened by changes and I feel
like it does us no good to placate them.  Some amount of these changes are
going to be the paternalistic "for your own good" type of things and I
think that's necessary and as long as it's coupled with responsible release
and versioning of the /experience/ we are going to be ok. We won't be in
this place until 4 is nailed down (either way) even if that's painful and
hard to figure out how we are going to do it.

There is a lot more to say I guess but that's where I'm at for the moment :)

On Tue, Dec 6, 2016 at 4:18 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:

> So I worked a bit with Krinkle (who knows how to use kubectl already)
> on getting one of his new bots setup on k8s.
>
> Conclusions:
> Plain kubectl is no-go for most regular users. To mount NFS (and the
> nslcd socket, for passwd / group dbs), we *need* to write a custom
> yaml file.
>
> Other than that, it was fairly smooth. But this does mean we need to
> wrap kubectl run at least in some form. We could make that super
> minimal, but it's modifications from upstream either way.
>
> Other stuff (kubectl get pods, kubectl delete, logs, etc) are fine.
>
> _______________________________________________
> Labs-admin mailing list
> Labs-admin at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-admin
>

-- 
Chase Pettet
Engineering Manager -- Labs
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20161209/3b8ae840/attachment.html>