[Labs-l] New instance creation outage ongoing

Thu Jul 19 01:36:52 UTC 2012

OpenStack Nova does some inefficient queries for searching for
instances, especially when pulling metadata information. When
instances are created, we inject what's called "userdata". That
userdata is used by a service in ubuntu called cloud-init. cloud-init
pulls this userdata, along with the instance's metadata and does the
initial bootstrapping of the system.

For us, cloud-init, using the userdata, installs puppet, points it at
our puppet server and does a full puppet run. This process is
currently failing, as cloud-init thinks the metadata server is timing
out. Apparently other deployments of openstack are having performance
issues with metadata that are due to the same inefficient query.

There's two things we'll be doing about this:

1. We're working on a fix with the other openstack devs
2. We'll purge deleted instances from the database. Nova keeps them
for auditing purposes, but they are unneeded. This will bring the
query speed down to a level that is below the cloud-init threshold.
We're also waiting for another organization to push their solution for
this upstream, rather than mucking around the database ourselves.

Sorry for the inconvenience,

- Ryan