eBay's practice on software loadbalancer

Traffic on ebay.com has been served by Netscaler hardware loadbalancers where there are thousands of Layer 7 rules routing traffic for many years. In this article, we are going to get very specific about how we implement H/W LB functionalities in S/W LB from end to end based on Kubernetes and Envoy.

There are some major components involved in each layer:

Altus admin console: It’s the front end service which provides UI for admins and PDs to managed L7 rules on both S/W and H/W LBs.
Zebra: A PaaS layers service with job orchestration for L7 rules automation also provides the backend restful APIs to admin console
LBMS: It’s a adaptor to both S/W and H/W LBs exposing unified APIs to Zebra.
UFES: It’s the backend service based on Tess (eBay’s Kubernetes ecosystem) and Envoy to implement H/W LB functionalities.
CMS: It’s the configuration management service in eBay which is used to persist the information in devices and other configuration information.

Also there are two very important concepts in eBay’s cloud platform:

WISB(What it should be): WISB it used to capture the user intention in the cloud world, like how many VMs the user requested or what kind of L7 rule user wants to provision in LBs.
WIRI(What it really is): Since cloud platform is a very complicated system where there are many resources like compute/storage/loadbalancer/DNS, after user submit the requests it may be failed at some point, so WIRI is the truth that how user request got executed.

Both WIRI and WISB information are persisted in CMS for the software loadbalancer.

Data plane architecture

Control plane architecture

Control plane sequence diagram

Key functionalities
1.WISB building
First step is to understand what do we have in H/W LBs to narrow down the scope that we are going to support in S/W LBs. There has been a lot of manual work in H/W LBs and we got about 1700 L7 rules configured. There are four types of layer 7 policies:

Responder: Generates HTTP response upon certain HTTP request.
Rewrite: Modifies the HTTP request or response.
Content Switching: Routes request to downstream applications.
Filter: Accept, Deny or Drop certain requests.

Since all rules are in Netscaler syntax, we need to build WISB in a more human readable format, so we defined the L7 rule WISB spec:

{
“name”: “”,//rule name
“priority”: “”,//rule priority
“topologyId”: “”, //the topology id that the traffic is going to
“vip”: “”, //which vip this rule is going to bind, it could be core/intl/akamai
“domain”:”“,//which domain this policy is applying to, could be mweb/dweb/webtier
“type”:”“,//the type of this rule, could be SEO/MOBILE/WEBTIER
“rule”: {
“attribute1”: [{
“operator1”: “value1”
},
{
“operator2”: “value2”
}],
“attribute2”: [{
“operator3”: “value3”,
“operator4”: “value4”
}]
},
“action”: {
“response”: {
“schema”: “”,//http or https
“server”: “”,//server adress “like www.ebay.co.uk”
“path”: “”,//path like “/rpp/womens-fashion”
“code”: “”/possible values: forward | responder | filter | rewrite
},
“name”: “”,//action name
“bypaasSafetyCheck”: true,
“type”: “respondwith”,
“group”: “responder”
}
}

Also here is how we support bi-direction convert from WISB to Netscaler syntax:

2.L7 rule automation
In Hardware Loadbalancer times, if there is no automation to support L7 rules creation/deletion, admins can still login to device and execute LB command directly. But in S/W LB, since there is no such a device like Netscalar, we need PaaS layer tools to manage L7 rules for both H/W and S/W LBs thus we can create a L7 rule to all LBs at same time. Within these tools, there will be no difference to create L7 rule on H/W or S/W.

Backend service
As aforementioned, Zebra is the backend service to provide job orchestration for L7 rules automation also the restful interfaces to create/delete L7 rules. Below is the work flow for Zebra:

Below are the technologies that we introduced for this component:
Spring quartz: is a richly featured, open source job scheduling library, we leverage on it to build a distribution system also enable feature like disaster recovery for quartz jobs with multiple stages.
Kafka:is an open source distributed streaming platform and we use it as a message queue. By using messaging bus, de-coupled each components in Zebra and reduced huge internal traffic also improved the extensibility and performance.
Hazelcast： is an open source in-memory data grid and we used it for Zebra configuration management.

Frontend console
Since this is an end to end solution, we also developed an admin console to Admins/PDs to managed L7 rules for both S/W and H/W LBs. We defined a work flow in the console:

We implement features including but not limited to below:
1.Create/delete L7 rules on H/W and S/W LBs.
2.Support strategy rollout
3.Support L7 rule conflict detection
4.Post-rollout validation for new created L7 rules.

3.S/W LB bootstrapping
For H/W LB bootstrapping, admins can login to an existing LB device and copy the configure to a new on-boarding LB within less than 1 hour. In S/W we also need to on-board and bootstrap hence we need automation support to create all existing L7 rules based on WISB to a newly set up S/W LB cluster. So Zebra support to bootstrap S/W with below Pre-conditions:

1.Manually set up public VIPs
2.Global policies set up as static configuration
3.UFES cluster registered in LBMS/CMS

With bootstrapping API, we can bootstrap a S/W LB to create 1700 rules within less than 1 hour.

4.LB drift detection
After introducing S/W LB, we have both H/W and S/W LBs coexisted and there are quite a bunch of LBs to server eBay public traffic. Ideally all SEO LBs configuration should be the same, there still might be some urgent situation that admins login to some LBs and manually change LB configuration. Hence we need to capture those drifts quickly and notify admins to ensure all LBs should have consistent behavior. Below is the work flow of drift detection:

We’ve set up an hourly cron job which will get all all LB drifts based on WISB and WIRI comparison result, and if any drift detected, it will send email alert to admins and also we can see the drifts in admin console:

Software LB Versus Hardware LB
Basically we need to implement all the features that are being used in H/W LBs through Kubernetes and Enovy. It’s almost impossible to implement every feature in H/W to S/W LB but we’ll figure out equivalent way for all being used functionalities.
Below are some examples that how we mapped H/W LB features to S/W LB from operational perspective:

Feature/Hardware LB/Software LB
Show all VIPs on LB/show csv/kubectl get ingress –all-namespaces.

Here is an example that what a L7 looks like in WISB and WIRI in both H/W and S/W LB:

WISB:

{
tier: “WEB”,
rule: {
path: {
equal: “”rt”“,
ignorecase: “true”,
get: “1”
}
},
action: {
group: “forward”,
},
vip: “www.ebay.com-1-443”,
priority: 316450,
topologyId: “returnnjsui-topo:ENVjctdqnenevk1x82”,
name: “rt-to-returnnjsui-core-ssl”,
scope: “vip”,
ingressType: “SEO”,
domain: “dweb”,
id: “returnnjsui-topo:ENVjctdqnenevk1x82-www.ebay.com-1-443-rt-to-returnnjsui-core-ssl-ingress”,
status: “COMPLETE”,
type: “INGRESS”,
}

NS WIRI

add cs policy rt-to-returnnjsui-core-ssl -rule “HTTP.REQ.URL.PATH.GET(1).SET_TEXT_MODE(IGNORECASE).EQ(\”rt\”)”
bind cs vserver www.ebay.com-1-443 -policyName rt-to-returnnjsui-core-ssl -targetLBVserver returnnjsui-web-1-443 -priority 316450

S/W LB WIRI

match:
NSName: rt-to-returnnjsui-core-ssl
case_sensitive: false
prefix: /rt
name: rt-to-returnnjsui-core-ssl_1
route:
cluster: returnnjsui-web-1-443

And for each ingress there is a route config and this policy is in www-ebay-com-1-443-route-config which contains all routes for https://www.ebay.com.

Current status and what’s next?
This is an new initiative in eBay also a cross team collaboration, we encountered a lot of challenges regarding with feature implementation also operational requirements, with continuous effort, we are getting more and more closer to the goal of this new project: to deploy globally distributed, flexible and elastic front-end to delight our customers with fast and consistent eBay experience.
Currently there are 3 domains that we are working on:

www.ebay.com, this is for desktop domain, thousands of SEO L7 rules configured.
m.ebay.com, this for mobile domain, thousands of L7 rules configured.
apisd.ebay.com, this is for native app domain, there are only few policies.

And we’ve built 6 kubernets clusters so far and each one hosts these 3 domains under different name space.

For native app domain S/W LBs we already ramp up 50% production traffic and they are working well together with H/W LBs. For the other two domains we can successfully bootstrap all the rules and run several round of traffic verification and they are still under testing and they are going to serve production traffic in the near future.