Rendered at 13:52:09 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jldugger 6 hours ago [-]
It's little surprising that the author was doing perf work and not already comparing distributions. As the rest of the article outlines, you learn a lot more with more data!
It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.
hyperpape 1 hours ago [-]
The median developer doing performance work has no background in the subject, they just have a complaint from someone that something is slow, and no real idea how to solve that problem.
By that standard, this guy is doing quite well. It's a good post.
markstock 1 hours ago [-]
The author is spot on. As an HPC performance engineer and computational artist, I could tell numerous stories about it, but I'll summarize three:
The common method for evaluating MPI performance on a supercomputer is with the OSU microbenchmarks, and with the -z option will print tail latencies. These are far more useful values to predict real-world performance on the big machines.
Second, while means DO say something useful, they are the top part of a profiling hierarchy. Deeper levels are sampling experiments (with minor effects on performance) to tracing (potentially larger effects on the code being measured). Seeing parallel timeline traces of a complex app has helped me and my team debug many performance problems. Look at HPCToolkit, Score-P, rocprof, and other tools.
Lastly, the reason I do computer artwork in the first place was because of a bug in an old fortran77 code which simulated turbulent mixing in 3d. I couldn't find our code mistake at first, but then I visualized the computational elements with a raytracer I had been playing with, and the result was not only shockingly beautiful, but pointed me to the line of code with the bug.
jeffwass 5 hours ago [-]
An interesting statistical example if you haven’t seen it before is the Anscombe Quartet :
Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.
touisteur 3 hours ago [-]
Cool that the Wikipedia page links to the "Mean Dinosaur" paper https://dl.acm.org/doi/10.1145/3025453.3025912 that I love to get out each time someone sends me mean, median, or stddev to measure processing latency. By all means use stats, but always eyeball the dataset to check assumptions extracted from statistics, I guess, especially in this world of matplotlib and notebooks and agents.
alphabeta3r56 4 hours ago [-]
I have mostly worked with computational physics with discontinuous polynomial approximations. Stuff like this would be easily detected in those methods as anatomy brcsuse of assumption pf smoothness.
Statistics on the other hand is more accepting of discontinuous data due to it's basis in measure spaces.
Hence In general, you should know what your data should look like before you aim to detect anamolies. But also, it might be a good practice with new automated research actors to always use both approximations (measure theory based and otherwise), to figure out what's going on.
ndr 3 hours ago [-]
Median also lies more than 99th/100th percentiles because in many workloads the same client is making multiple parallel requests and often waiting for the slowest.
See this piece from Jeff Dean [0]
Quote:
"""
Component-Level Variability Amplified By Scale
A common technique for reducing latency in large-scale online services is to parallelize sub-operations across many different machines, where each sub-operation is co-located with its portion of a large dataset. Parallelization happens by fanning out a request from a root to a large number of leaf servers and merging responses via a request-distribution tree. These sub-operations must all complete within a strict deadline for the service to feel responsive.
Variability in the latency distribution of individual components is magnified at the service level; for example, consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). Even for services with only one in 10,000 requests experiencing more than one-second latencies at the single-server level, a service with 2,000 such servers will see almost one in five user requests taking more than one second (marked “o” in the figure).
Instead of the CDF I like to use one minus the CDF, the fraction of requests not yet completed at any given time. Then you can make it a log log plot showing the entire tail with details invisible in the CDF because in the CDF the tail is essentially a horizontal line at one.
Ratelman 2 hours ago [-]
"The “after” curve is higher than the “before” curve to the right of 140ms which means more requests are finishing slower than before."
Think they made a typo - "before" curve is higher than the "after" curve to the right of 140ms, which means more requests take longer than before to finish.
Similar to the ridgeline plot, you can see the distribution better without having to interpret derivatives as you do with the CDF plots.
mr_toad 2 hours ago [-]
Headline is unnecessary clickbait, the mean was still signaling that something was wrong.
DarkNova6 6 hours ago [-]
With only having read the headline (as it is customary), I dare say: Aggregation without distribution always means nothing.
xxs 4 hours ago [-]
mean w/o std dev is just pointless, I tend to reject them altogether.
If I am to record something like queue times (as in latency), I'd have multiple samplings of both time enqueued and entries (or approx entries size), provided the queues have a reasonable size() method (e.g. not O(n) and not blocking). Even having max entries increased with lower overall latency, it'd speak of bursty behaviour.
In that regard - Recording percentiles en masse (production) is rather expensive when there are thousands of metrics. The avg/mean + max require few bytes (sub cacheline) and it can be implemented lock-free.
It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.
By that standard, this guy is doing quite well. It's a good post.
The common method for evaluating MPI performance on a supercomputer is with the OSU microbenchmarks, and with the -z option will print tail latencies. These are far more useful values to predict real-world performance on the big machines.
Second, while means DO say something useful, they are the top part of a profiling hierarchy. Deeper levels are sampling experiments (with minor effects on performance) to tracing (potentially larger effects on the code being measured). Seeing parallel timeline traces of a complex app has helped me and my team debug many performance problems. Look at HPCToolkit, Score-P, rocprof, and other tools.
Lastly, the reason I do computer artwork in the first place was because of a bug in an old fortran77 code which simulated turbulent mixing in 3d. I couldn't find our code mistake at first, but then I visualized the computational elements with a raytracer I had been playing with, and the result was not only shockingly beautiful, but pointed me to the line of code with the bug.
https://en.wikipedia.org/wiki/Anscombe%27s_quartet
Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.
Statistics on the other hand is more accepting of discontinuous data due to it's basis in measure spaces.
Hence In general, you should know what your data should look like before you aim to detect anamolies. But also, it might be a good practice with new automated research actors to always use both approximations (measure theory based and otherwise), to figure out what's going on.
See this piece from Jeff Dean [0]
Quote:
""" Component-Level Variability Amplified By Scale
A common technique for reducing latency in large-scale online services is to parallelize sub-operations across many different machines, where each sub-operation is co-located with its portion of a large dataset. Parallelization happens by fanning out a request from a root to a large number of leaf servers and merging responses via a request-distribution tree. These sub-operations must all complete within a strict deadline for the service to feel responsive.
Variability in the latency distribution of individual components is magnified at the service level; for example, consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). Even for services with only one in 10,000 requests experiencing more than one-second latencies at the single-server level, a service with 2,000 such servers will see almost one in five user requests taking more than one second (marked “o” in the figure).
"""
[0] https://cacm.acm.org/research/the-tail-at-scale/#body-4
Think they made a typo - "before" curve is higher than the "after" curve to the right of 140ms, which means more requests take longer than before to finish.
Or am I the one misunderstanding?
Similar to the ridgeline plot, you can see the distribution better without having to interpret derivatives as you do with the CDF plots.
If I am to record something like queue times (as in latency), I'd have multiple samplings of both time enqueued and entries (or approx entries size), provided the queues have a reasonable size() method (e.g. not O(n) and not blocking). Even having max entries increased with lower overall latency, it'd speak of bursty behaviour.
In that regard - Recording percentiles en masse (production) is rather expensive when there are thousands of metrics. The avg/mean + max require few bytes (sub cacheline) and it can be implemented lock-free.