Cumulative Normal Distribution Chart Axis Transformation lazarus / 55-TAChart ---------------------------------------------------------------------- wp · 2011-10-03 19:25 Back in the old days at university we plotted statistical data on "normal probability" paper in order to test for normal distribution. In this specially-designed grid paper the cumulative probability distribution is distorted to a straight line if the data follow a normal distribution. In the attached project I am posting a special axis transformation ("TCumulNormDistrAxisTransform") for TAChart to achieve the same result on a computer. If it is of any interest for the TAChart development, feel free to add it to the library. The transformation uses the error function implemented in the unit "spe" of the numlib package. When writing the demo project I came across some questions and issues: - The random data are generated in the procedure "GenerateData". For this purpose, random numbers are created with a uniform or normal distribution. The numbers are added as x values to a ListChartSource. Since the list is sorted the cumulative probability can be calculated from the list index by normalization to the list count. I do this after adding all random numbers by direct manipulation of the ChartDataItems. Unfortunately, minimum and maximum of the ChartSource do not get updated this way -- please see the comments in this procedure. Is there a simple way to enforce recalculation of the extent of a ChartSource? - The second issue seems to be related. When probabilities are displayed as percentage values and the ChartAxisTransformation is active the axis labels (taken from a special ListChartSource, see below) are cut off above 1 although they should be range up to slightly less than 100. When debugging the TChartAxis.Draw procedure I found out that the FValueMax of the AxisDrawHelper is still at the value corresponding to the maximum 1. So, how can I enforce updating of the axis range? - Finding labels for this distorted axis is quite challenging. In the end, I simply used a special ListChartSource which is populated in the demo by the procedure "PopulateAxLabelSource". When you shrink the size of the program window, labels are partially overlapping. I thought, switching the axis' OverlapPolicy to opHideNeighbor would prevent that; but this does not seem to be the case. - The color of the DataPointCrosshairTool cannot be changed. I thought this had been working some time ago in another project. - Finally, a strange error: the object inspector no longer displays the property ShowLines of TLineSeries. I checked the source - this property is there, and it is published. What's going wrong here? ---------------------------------------------------------------------- Ask · 2011-10-04 03:32 Thanks, that is an interesting case, and I will consider including it after a little improvement. QuoteThe transformation uses the error function implemented in the unit "spe" of the numlib package.Hm, maybe I'm mussing something, but where is the Sqrt(2) factor? http://en.wikipedia.org/wiki/Error_function#Related_functions Quote// Add random test data to RawSource as x values --> random values will  // get sorted in ascending direction automatically. Although it does not matter in the demo, it is generally better to first fill source with value, and only then set Sorted to true. See http://wiki.lazarus.freepascal.org/TAChart_documentation#List_source QuoteUnfortunately, minimum and maximum of the ChartSource do not get updated this way -- please see the comments in this procedure.You can get updates by using Source.SetYValue instead of direct assignment. This, however, is slow, so your pattern of BeginUpdate/direct assignment/EndUpdate is correct. Since r32666 EndUpdate should invalidate caches. Quotethe axis labels are cut off above 1 although they should be range up to slightly less than 100This is due to a bug in you axis transformation -- TCumulNormDistrAxisTransform.GraphToAxis should apply transformations in the reverse order compared to AxisToGraph. This means first calculating CumulNormDistr, then multiplying by 100. In general, Percent property is redundant since it can easily be simulated by adding linear transformation in front of TCumulNormDistrAxisTransform. Ability to combine transformations is the main reason for TAxisTransformations component. QuoteFinding labels for this distorted axis is quite challengingIndeed. I was able to get semi-decent results by setting UseCount=true, Count=30, MinLength=5, UseMaxLength=false. However, this is surely the case where custom mark source is the correct solution. QuoteI thought, switching the axis' OverlapPolicy to opHideNeighbor would prevent that; but this does not seem to be the case.This seems to be a bug -- OverlapPolicy does work for the default source. Will investigate later. Edit: This is because you source supplied non-monotonic marks. Setting AxLabelSource.Sorted := true fixes the issue. TAChart sort-of-works with non-monotonic axis sources, but there are problems because some algorithms suppose monotonicity for efficiency reasons. QuoteThe color of the DataPointCrosshairTool cannot be changedSee http://wiki.lazarus.freepascal.org/TAChart_documentation#Drawing_mode Quotethe object inspector no longer displays the property ShowLines of TLineSeriesShowLines is planned for deprecation in favor of LineType. Hiding from object editor was taken as the first step of that. See SkipObsoleteProperties in TASeries.pas Finally, perhaps properties for the standard deviation and expected value should be added for the general case? ---------------------------------------------------------------------- wp · 2011-10-04 17:32 Thanks for the debugging, you're great. Quotebut where is the Sqrt(2) factor? You are absolutely right. I just found that InvCumulNormDistr(CumulNormDistr(x)) was different from x. With the sqrt(2), the result is correct: Code: [Select]function CumulNormDistr(x:double) : double; // Cumulative normal distribution // x = -INF ... INF --> result = 0 ... 1 begin   if x > 0 then     result := (speerf(x/sqrt(2)) + 1) * 0.5   else if x < 0 then     result := (1 - speerf(-x/sqrt(2))) * 0.5   else     result := 0; end;        QuoteSource.SetYValue I remember now that you had mentioned that in another thread. QuoteThis means first calculating CumulNormDistr, then multiplying by 100. Correct. The percents are working after this correction. QuotePercent property is redundant since it can easily be simulated by adding linear transformation in front of TCumulNormDistrAxisTransform. I was not aware of that. But maybe it's better to keep the Percent property in the transform since many other users won't know that either, and percentage presentation of probabilities is quite common. QuoteAxLabelSource.Sorted := true Issue fixed with that modification. QuoteThe color of the DataPointCrosshairTool cannot be changed The color of the crosshair tool is correct when DrawinMode is set to dmNormal -- I keep learning... Quoteproperties for the standard deviation and expected value That came to my mind after posting. It would mean transforming the data, but not back-transforming the axis labels. Then the origin of the axis (y=0) would correspond to the mean value, and the tick marks would be given in units of standard deviations, i.e y= +/-1 would correspond to +/- 1 standard deviation. Could you add that? ---------------------------------------------------------------------- Ask · 2011-10-05 07:54 I have committed initial version of your code, with reformatting and some refactoring. See axistransf demo since r32694. I have postponed both special marks source and mean/variance properties -- I'd like to think some more about those.