Data Audit
-> Defining Cleansing Goals
-> Data Cleansing Plan
-> Backup Data
-> Data Cleansing Operations
-> Verification
-> Documentation & Reporting
-> Review
-> Implementation of Preventative Measures
plain text
S006 Amy Yeewood A 10
S006 Amy Yeewood A 10
plain text
John Smith
Jon Smith
J. Smith
plain text
parameter estimates
standard errors
confidence intervals
p-values
model validity
plain text
缺失完全像随机事故一样发生。
plain text
p(B | Y_obs, Y_miss) = p(B | η)
plain text
为什么缺失，可以用已经观察到的数据解释。
plain text
p(B | Y_obs, Y_miss) = p(B | Y_obs, η)
plain text
正是因为这个值本身很特殊，所以它才缺失。
plain text
p(B | Y_obs, Y_miss, η)
plain text
Y1 Y2 Y3 Y4
✓ ✓ ✓ NA
✓ ✓ ✓ NA
✓ ✓ ✓ NA
plain text
Time1 Time2 Time3 Time4
✓ ✓ ✓ ✓
✓ ✓ NA NA
✓ NA NA NA
分析 A 和 B 的关系时，只要 A、B 不缺失就能用；
分析 A 和 C 的关系时，再换一批 A、C 不缺失的样本。
plain text
JP_i = β0 + β1 IQ_i
plain text
JP_i = -2.065 + 0.123 × IQ_i
plain text
P_i = β0 + β1 IQ_i + z_i
plain text
z_i ~ Normal(0, residual variance)
plain text
variance
normality
statistical power
regression assumptions
ANOVA assumptions
plain text
年龄 = 150
收入 = 10,000,000
plain text
年龄 = 18
收入 = 500,000
工作经验 = 0
plain text
|x_k - x_0| > tζ
plain text
如果一个点距离“正常中心”太远，超过了“正常波动范围”的若干倍，就可能是异常值。
plain text
independent and identically distributed Gaussian random variables
plain text
|x_k - x̄| > 3σ
plain text
x0 = median
ζ = MAD scale estimator
plain text
S = 1.4826 × median(|x_k - median|)
plain text
|x_k - median| > 3S
plain text
8,7,9,9,6,5,8,9,8,8,9
8,7,9,9,6,5,8,9,8,8,9,100
plain text
IQR = Q3 - Q1
plain text
median = (Q3 + Q1) / 2
Q3 = median + IQR / 2
Q1 = median - IQR / 2
plain text
|x_k - median| > t × IQR
plain text
x_k > Q3 + t × IQR => upper outlier
x_k < Q1 - t × IQR => lower outlier
plain text
The residual ε_j provides useful information about the outlier score of data point j.
plain text
真实值 - 预测值 = residual
residual 越大，越可能异常
plain text
正常点通常聚集在密集区域；
异常点通常远离密集区域。
plain text
Data Cleansing = detect + correct/remove corrupt/inaccurate data
Data cleansing process:
Audit -> Goals -> Plan -> Backup -> Operations -> Verification -> Documentation -> Review -> Prevention
Main operations:
Duplicates -> remove duplicates
Inaccurate -> validate/correct
Inconsistent -> consistency checks
Incomplete -> fill missing values
Abnormal/irrelevant -> handle outliers
Missing data mechanisms:
MCAR = unrelated to observed variables and missing value itself
MAR = related to observed variables, not missing value itself
MNAR = related to missing value itself
Missing data patterns:
Univariate = one variable missing
Monotone = dropout pattern, never return
General = scattered missingness
Deletion:
List-wise = remove entire row if any missing value
Pairwise = remove only for each analysis
Both require MCAR
Imputation:
Mean = fill with mean, simple but biased / underestimates uncertainty
Regression = fill with predicted values
Stochastic regression = predicted values + random residual, restores variability
Evaluate missing-data method:
Minimise bias
Maximise available information
Yield good uncertainty estimates
Outlier:
Not necessarily error; may be valuable abnormal signal
Impacts:
Increase error variance
Reduce statistical power
Decrease normality
Bias estimates
Violate model assumptions
Outlier types:
Univariate = single variable
Multivariate = n-dimensional space
Univariate detection:
|x_k - x0| > tζ
3σ rule:
x0 = mean, ζ = standard deviation
Hampel:
x0 = median, ζ = MAD
Boxplot:
x0 = median, ζ = IQR
Upper outlier: x_k > Q3 + t×IQR
Lower outlier: x_k < Q1 - t×IQR
Multivariate detection:
Linear models = residuals from hyperplane
Proximity-based = points outside dense regions