Transcript slides
Enabling Flow-level Latency Measurements across Routers in Data Centers Parmjeet Singh, Myungjin Lee Sagar Kumar, Ramana Rao Kompella Latency-critical applications in data centers Guaranteeing low end-to-end latency is important Web search (e.g., Google’s instant search service) Retail advertising Recommendation systems High-frequency trading in financial data centers Operators want to troubleshoot latency anomalies End-host latencies can be monitored locally Detection, diagnosis and localization through a network: no native support of latency measurements in a router/switch Prior solutions Lossy Difference Aggregator (LDA) Kompella et al. [SIGCOMM ’09] Aggregate latency statistics Reference Latency Interpolation (RLI) Lee et al. [SIGCOMM ’10] Per-flow latency measurements More suitable due to more fine-grained measurements Deployment scenario of RLI Upgrading all switches/routers in a data center network Pros Cons Provide finest granularity of latency anomaly localization Significant deployment cost Possible downtime of entire production data centers In this work, we are considering partial deployment of RLI Our approach: RLI across Routers (RLIR) Overview of RLI architecture Router Ingress I Goal Egress E Latency statistics on a per-flow basis between interfaces Problem setting No storing timestamp for each packet at ingress and egress due to high storage and communication cost Regular packets do not carry timestamps Ingress I Reference L R Packet Injector 1E 2 Egress Delay Overview of RLI architecture Latency Estimator Premise of RLI: delay locality Approach L Linear interpolation line 1 Interpolated delay 2 R Time 1) The injector sends reference packets regularly 2) Reference packet carries ingress timestamp 3) Linear interpolation: compute per-packet latency estimates at the latency estimator 4) Per-flow estimates by aggregating per-packet estimates Full vs. Partial deployment RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6 Full deployment: 16 RLI sender-receiver pairs Partial deployment: 4 RLI senders + 2 RLI receivers 81.25 % deployment cost reduction Case 1: Presence of cross traffic RLI Sender (Reference Packet Injector) Switch 1 Link utilization estimation on Switch 1 Switch 2 Switch 3 RLI Receiver (Latency Estimator) Switch 5 Cross Traffic Bottleneck Link Switch 4 Switch 6 Issue: Inaccurate link utilization estimation at the sender leads to high reference packet injection rate Approach Not actively addressing the issue Evaluation shows no much impact on packet loss rate increase Details in the paper Case 2: RLI Sender side RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6 Issue: Traffic may take different routes at an intermediate switch Approach: Sender sends reference packets to all receivers Case 3: RLI Receiver side RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6 Issue: Hard to associate reference packets and regular packets that traversed the same path Approaches Packet marking: requires native support from routers Reverse ECMP computation: ‘reverse’ engineer intermediate routes using ECMP hash function IP prefix matching at limited situation Deployment example in fat-tree topology RLI Sender (Reference Packet Injector) IP prefix matching RLI Receiver (Latency Estimator) Reverse ECMP computation / IP prefix matching Evaluation Simulation setup Trace: regular traffic (22.4M pkts) + cross traffic (70M pkts) Simulator 10% / 1% RLI RLI injection rate Regular Sender Reference Receiver Traffic Traffic Divider Packet Trace Cross Traffic packets Switch1 Cross Traffic Injector Results Accuracy of per-flow latency estimates Switch2 Accuracy of per-flow latency estimates Bottleneck link utilization: 93% 67% 10% injection 1% injection 10% injection CDF 1% injection 1.2% Relative error 4.5% 18% 31% Summary Low latency applications in data centers Localization of latency anomaly is important RLI provides flow-level latency statistics, but full deployment (i.e., all routers/switches) cost is expensive Proposed a solution enabling partial deployment of RLI No too much loss in localization granularity (i.e., every other router) Thank you! Questions?