Time Series Detrending Methodology

1. Executive Summary

This document presents a comprehensive technical specification for the time series detrending methodology implemented for processing meteorological data from India Meteorological Department (IMD) grid datasets. The methodology employs a hybrid two-stage approach that combines linear regression for trend detection with Locally Estimated Scatterplot Smoothing (LOESS) for non-linear trend estimation and removal.

The primary objective of this detrending procedure is to remove systematic temporal trends from weather time series data while preserving the underlying stochastic variability essential for parametric risk modelling, actuarial analysis, and agricultural insurance applications.

2. Introduction

2.1 Background and Motivation

Meteorological time series data, particularly precipitation and temperature records from IMD grids, often exhibit systematic temporal trends attributable to climate change, urbanization effects, or instrumentation drift. For actuarial and risk modelling applications, these trends can distort probability distributions and lead to biased parameter estimates.

The detrending methodology addresses this challenge by identifying and removing trend components while preserving the natural variability structure required for stochastic simulation and risk quantification.

2.2 Scope of Application

This methodology is designed for application to:

• Yearly index values for a given payout structure
• Index build actual time series

3. Methodological Framework

3.1 Two-Stage Hybrid Approach

The detrending algorithm implements a two-stage process: (1) linear regression for trend detection and directionality assessment, followed by (2) LOESS smoothing for flexible non-linear trend estimation when detrending is warranted.

3.2 Stage 1: Linear Trend Detection

The initial phase employs Ordinary Least Squares (OLS) linear regression to ascertain the presence and directionality of temporal trends in the input series. The model specification is:

Yₜ = β₀ + β₁ × t + εₜ

Where:

• Yt = observed value at time step t
• β0 = intercept coefficient
• β1 = slope coefficient (trend indicator)
• εt = residual error term
• t ∈ {0, 1, 2, ..., n-1}

3.2.1 Trend Direction Interpretation

The sign and magnitude of β1 (slope) determines trend directionality:

Condition	Interpretation	Detrending Action
β₁ > 0	Upward (positive) trend	Apply if mode = 'upward' or 'both'
β₁ < 0	Downward (negative) trend	Apply if mode = 'downward' or 'both'
β₁ ≈ 0	No significant linear trend	Skip unless mode = 'both'

3.3 Stage 2: LOESS Trend Estimation

3.3.1 LOESS Algorithm Overview

When detrending is warranted based on the Stage 1 assessment, the module employs Locally Estimated Scatterplot Smoothing (LOESS), also known as LOWESS (Locally Weighted Scatterplot Smoothing), to estimate the non-linear trend component.

Algorithm Parameters:

Parameter	Value	Description
Smoothing Fraction (frac)	0.5	50% of observations used in each local regression window
Polynomial Degree	1	Locally linear fit (default in statsmodels.lowess)
Weighting Kernel	Tricube	w(u) = (1 - |u|³)³ for |u| ≤ 1

3.3.2 Tricube Weight Kernel

The LOESS estimator computes fitted values at each time point through iteratively weighted local polynomial regression. The tricube kernel function assigns weights based on the scaled distance from the focal point:

`w(u) = (1 - |u|³)³  for |u| ≤ 1, and w(u) = 0  otherwise`

3.3.3 Advantages of LOESS for Meteorological Data

1. Non-parametric flexibility: Captures non-linear trends (e.g., accelerating climate signals, regime shifts) without imposing rigid functional forms

2. Local adaptivity: Adjusts to heterogeneous trend behavior across different temporal segments

3. Robustness: Less sensitive to outliers compared to global polynomial fitting

4. Multiplicative Detrending Transformation

4.1 Trend Factor Computation

The trend adjustment employs a multiplicative ratio-based normalization approach. The trend factor at each time step is computed as:

Fₜ = max(Ŷ) / Ŷₜ

Where:

• Ft = trend factor at time step t
• Ŷt = LOESS-predicted value at time t
• max(Ŷ) = maximum predicted value across all time steps

Boundary Condition: When Ŷt = 0, Ft defaults to 1.0 to avoid division-by-zero errors.

4.2 Detrended Value Computation

The detrended observations are computed as:

Y*ₜ = Yₜ × Fₜ

Where Y*ₜ represents the trend-adjusted (detrended) value.

4.3 Interpretation of the Multiplicative Approach

This methodology effectively normalizes all observations to the maximum trend level, thereby:

1. Preserving relative variability: The coefficient of variation structure is maintained

2. Adjusting for systematic growth/decline: Earlier (lower-trend) values are scaled upward; later (higher-trend) values remain relatively unchanged

3. Ensuring non-negativity: Multiplicative factors preserve the sign of original observations (critical for precipitation, yield indices, etc.)

5. Discrete Variable Handling

For discrete random variables (e.g., count data, integer-valued indices), the module applies rounding to maintain distributional coherence:

Yₜ* (discrete) = round(Yₜ × Fₜ)

The random variable type is either explicitly specified or automatically inferred using the certain utility function based on the data characteristics.

6. Output Specification

The detrending function returns a structured dictionary containing the following components:

Key	Type	Description
rv_type	Str	Inferred or specified random variable type ('continuous' or 'discrete')
detrended_values	list[float|int]	Transformed series with trend removed
detrend_lookup	pd.DataFrame	Lookup table containing timestep, original values, LOESS predictions, and trend factors
coefs	Dict	Linear regression coefficients (intercept, slope) from trend detection phase

6.1 Detrend Lookup DataFrame Structure

The detrend_lookup DataFrame contains detailed intermediate calculations for audit and diagnostic purposes:

Column	Description	Usage
timestep	Integer index (0 → n−1)	Time dimension for plotting & analysis
value	Original observed values	Preserves raw input data
pred	LOESS-predicted trend values	Visualizes trend component
trend_factor	Multiplicative adjustment factors	Audits detrending transformation

7. Assumptions and Limitations

7.1 Key Assumptions

Assumption	Implication
Monotonic or smooth trend	LOESS may produce artifacts with highly irregular or cyclical trends
Positive-valued series	Multiplicative adjustment assumes Y > 0; zero/negative values require special handling
Sufficient sample size	Minimum 3 observations required; stability improves with n ≥ 20
Stationarity post-detrending	Assumes trend removal yields approximately stationary residuals
Trend-stationary process	Method assumes deterministic trend rather than stochastic trend (unit root)

7.2 Methodological Limitations

1. Smoothing Parameter Sensitivity: The frac=0.5 parameter is fixed and may not be optimal for all time series lengths and trend patterns. Consider adaptive selection for production use.

2. Edge Effects: LOESS estimates at series endpoints have higher uncertainty due to asymmetric weighting. Consider using robust trend estimation for the first and last observations.

3. Non-monotonic Trends: The method may not perform well for series with multiple trend reversals or structural breaks. Consider segmented detrending for such cases.

4. Outlier Sensitivity: While LOESS is more robust than polynomial fitting, extreme outliers can still affect local estimates. Consider robust LOESS variants for heavily contaminated data.

8. References

1. Cleveland, W.S. (1979). "Robust Locally Weighted Regression and Smoothing Scatterplots." Journal of the American Statistical Association, 74(368), 829-836.

2. Cleveland, W.S. & Devlin, S.J. (1988). "Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting." Journal of the American Statistical Association, 83(403), 596-610.

3. Dickey, D.A. & Fuller, W.A. (1979). "Distribution of the Estimators for Autoregressive Time Series With a Unit Root." Journal of the American Statistical Association, 74(366), 427-431.

4. Kwiatkowski, D., Phillips, P.C.B., Schmidt, P., & Shin, Y. (1992). "Testing the null hypothesis of stationarity against the alternative of a unit root." Journal of Econometrics, 54(1-3), 159-178.

5. Mann, H.B. (1945). "Nonparametric Tests Against Trend." Econometrica, 13(3), 245-259.

6. Kendall, M.G. (1975). Rank Correlation Methods. 4th ed. London: Charles Griffin.

9. Appendix: Mathematical Summary

Complete Detrending Pipeline:

Step 1 - Fit Linear Model:

(β₀, β₁) = OLS(Y, t)

Step 2 - Evaluate Detrending Condition:

• If correction = 'upward': apply = (β1 > 0)
• If correction = 'downward': apply = (β1 < 0)
• If correction = 'both': apply = True

Step 3 - Estimate Trend (if applicable):

Ŷ = LOESS(Y, t; frac = 0.5)

Step 4 - Compute Trend Factors:

Fₜ = max(Ŷ) / Ŷₜ

Step 5 - Apply Transformation:

Yₜ* = Yₜ × Fₜ

---