In high-throughput screening (HTS), one of the major goals is to select compounds (including small molecules, siRNAs, shRNA, genes, et al.) with a desired size of inhibition or activation effects. A compound with a desired size of effects in an HTS screen is called a hit. The process of selecting hits is called hit selection.
Contents
Methods for hit selection in general
HTS experiments have the ability to screen tens of thousands (or even millions) of compounds rapidly. Hence, it is a challenge to glean chemical/biochemical significance from mounds of data in the process of hit selection. To address this challenge, appropriate analytic methods have been adopted for hit selection. There are two main strategies of selecting hits with large effects . One is to use certain metric(s) to rank and/or classify the compounds by their effects and then to select the largest number of potent compounds that is practical for validation assays . The other strategy is to test whether a compound has effects strong enough to reach a pre-set level. In this strategy, false-negative rates (FNRs) and/or false-positive rates (FPRs) must be controlled .
There are two major types of HTS experiments, one without replicates (usually in primary screens) and one with replicates (usually in confirmatory screens). The analytic methods for hit selection differ in those two types of HTS experiments. For example, the z-score method is suitable for screens without replicates whereas the t-statistic is suitable for screens with replicate. The calculation of SSMD for screens without replicates also differs from that for screens with replicates.
Screens without replicates
There are many metrics used for hit selection in primary screens without replicates. The easily interpretable ones are fold change, mean difference, percent inhibition, and percent activity. However, the drawback common to all of these metrics is that they do not capture data variability effectively. To address this issue, researchers then turned to the z-score method or SSMD, which can capture data variability in negative references .
The z-score method is based on the assumption that the measured values (usually fluorescent intensity in log scale) of all investigated compounds in a plate have a normal distribution. SSMD also works the best under the normality assumption. However, true hits with large effects should behave very different from the majority of the compounds and thus are outliers. Strong assay artifacts may also behave as outliers. Thus, outliers are not uncommon in HTS experiments. The regular versions of z-score and SSMD are sensitive to outliers and can be problematic. Consequently, robust methods such as the z*-score method, SSMD*, B-score method, and quantile-based method have been proposed and adopted for hit selection in primary screens without replicates .
In a primary screen without replicates, every compound is measured only once. Consequently, we cannot directly estimate the data variability for each compound. Instead, we indirectly estimate data variability by making a strong assumption that every compound has the same variability as a negative reference in a plate in the screen. The z-score, z*-score and B-score relies on this strong assumption; so are the SSMD and SSMD* for cases without replicates.
Screens with replicates
In a screen with replicates, we can directly estimate data variability for each compound, and thus we can use more powerful methods, such as SSMD for cases with replicates and t-statistic that does not rely on the strong assumption that the z-score and z*-score rely on. One issue with the use of t-statistic and associated p-values is that they are affected by both sample size and effect size . They come from testing for no mean difference, thus are not designed to measure the size of small molecule or siRNA effects. For hit selection, the major interest is the size of effect in a tested small molecule or siRNA. SSMD directly assesses the size of effects . SSMD has also been shown to be better than other commonly used effect sizes . The population value of SSMD is comparable across experiments and thus we can use the same cutoff for the population value of SSMD to measure the size of siRNA effects .
SSMD can overcome the drawback of average fold change not being able to capture data variability. On the other hand, because SSMD is the ratio of mean to standard deviation, we may get a large SSMD value when the standard deviation is very small, even if the mean is small. In some cases, a too small mean value may not have a biological impact. As such, the compounds with large SSMD values (or differentiations) but too small mean values may not be of interest. The concept of dual-flashlight plot has been proposed to address this issue. In a dual-flashlight plot, we plot the SSMD versus average log fold-change (or average percent inhibition/activation) on the y- and x-axes, respectively, for all compounds investigated in an experiment. With the dual-flashlight plot, we can see how the genes or compounds are distributed into each category in effect sizes, as shown in the figure. Meanwhile, we can also see the average fold-change for each compound .