From Day One of eBay’s cloud journey, the e-commerce company has focused on keeping its developers happy, according to Suneet Nandwani, eBay’s senior director of cloud infrastructure and platforms. That’s led to several challenges and innovations at the company, the latest of which is the development of TessMaster — a management framework to deploy Kubernetes on OpenStack.
With the emergence of Docker, it became clear that containers are “a technology which developers love,” Nandwani told ZDNet.
eBay saw cluster managers as a potential way to automate its operations at scale, Nandwani added. “We felt our managability and operability… would actually improve with cluster management.” On top of that, advanced scheduling and cluster management could lead to more efficient infrastructure use and, consequently, cost savings.
The company explored its options and landed on Kubernetes for its elegant developer interface. But as one of the largest OpenStack implementations in the world, running Kubernetes didn’t seem feasible for eBay.
“Most of the companies employing Kubernetes internally are much smaller scale,” Nandwani explained. “When you have one cluster with tens of nodes, it’s one thing. If you have seven or eight with hundreds of thousands of nodes, it’s completely different.”
As it started to broach this problem in early 2016, eBay was dissatisfied with existing solutions like Magnum.
“We didn’t see anything in the community that could help us, so we decided to write our own management platform for Kubernetes running on top of OpenStack,” Nandwani said.
TessMaster can deploy Kubernetes on OpenStack, it can scale it out and it flex it up and flex it down, Nandwani said. It’s a work in progress, and eBay continues to build management capabilities into it. While eBay hasn’t made it open source yet, it intends to “because there’s no reason for us not to,” Nandwani said.
eBay currently has seven large clusters deployed with the help of TessMaster and plans to significantly scale out its use of Kubernetes. “There are a lot of new applications being developed, and even platforms being developed, at eBay targeted squarely at running on Kubernetes,” Nandwani said.
eBay’s 2013 move to OpenStack was similarly driven by its interest in appealing to developers.
“When we first built our eBay specific cloud… it worked pretty well, but there were a couple downsides — one was that it wasn’t very open,” Nandwani said. “We had to train our own developers to work on our cloud. The second thing was how to attract talent. The community is evolving very quickly, and we were standing still. We made a strategic decision around 2013 to change our implementation to OpenStack.”
And like containers, OpenStack presented a challenge for a platform as large as eBay.
“Because most of the deployments on OpenStack were much smaller than ours, there wasn’t much focus on running a cloud platform at massive scale,” Nandwani explained. “How do you monitor this cloud, how do you do capacity management, how do you do remediation in case something goes wrong, alerting… there are a whole bunch of problems that come into being when you are operating a cloud at a large scale.”
eBay put significant engineering effort into solving those problems, some of which have been shared and some which remain internal solutions. “We have developed a lot of maturity, from an engineering point of view, managing OpenStack at scale,” Nandwani said.
