Regression Rank, Collinearity and Unstable Coefficients

1. Why can a regression fit without unique coefficients?

When columns of the design matrix are linearly dependent, multiple coefficient vectors produce the same fitted observations. Least squares can still identify a fitted projection while failing to identify each coefficient separately. A software routine using a generalized inverse may return one solution instead of raising an error. Statsmodels explicitly warns about this rank-deficient case. Successful execution therefore does not establish unique economic exposures. Inspect the rank of the actual design matrix, including any intercept and transformations, before interpreting individual coefficients. A displayed coefficient table can conceal redundancy that no increase in decimal precision or training fit statistic can resolve.

2. Work an exactly redundant synthetic design

Let x-one equal one, two, three; let x-two equal twice x-one; and let the response equal three times x-one. With an intercept, the fitted equation has intercept zero and requires b-one plus twice b-two to equal three. Coefficient pairs three and zero, one and one, or zero and one-and-a-half all fit the observations exactly. The three-column design has rank two, because the second predictor duplicates the first after scaling. The table reports these alternative solutions. They are constructed algebraic fixtures, not fitted investment factors, observed asset exposures or independent evidence that one predictor has a particular causal effect.

3. Near collinearity creates sensitivity rather than exact redundancy

If the second predictor is almost, but not exactly, twice the first, a unique least-squares solution may exist while individual coefficients become highly sensitive to small input changes. Opposing large coefficients can cancel within the observed sample, producing stable-looking fitted values and unstable exposure interpretation. A condition number compares how strongly a linear system can amplify perturbations under a specified norm. NumPy documents several norm choices, so record the convention rather than quoting an unlabeled diagnostic. There is no context-free threshold that proves an economic interpretation reliable. Consider data precision, predictor construction and the intended use of the coefficients.

4. Scaling can help numerics without adding information

Changing a predictor from currency units to thousands of currency units changes its coefficient and can change the numerical condition number. Centering a large-offset predictor can also improve arithmetic. These transformations do not create independent variation where two columns encode the same information. Preserve the transformation and translate coefficients back to their intended units before interpreting them. Check both scale-driven conditioning and substantive redundancy. Adding an intercept twice, combining a complete set of category indicators with an intercept, or including a total alongside all of its components can create exact rank loss even when each individual column appears sensible.

5. Solver conventions are part of the result

NumPy's least-squares documentation describes rank and singular-value outputs, including a cutoff determining which singular values are treated as zero. Different cutoffs or solver conventions can select different numerical solutions near a boundary. Store the actual design-column order, intercept policy, tolerance, effective rank and solver version. A pseudoinverse solution is one convention for choosing among equivalent fits; it is not proof that the selected coefficient allocation is economically identified. If a model card promises a unique slope, its input admission rule must reject the relevant zero-variation case. A broader multivariate contract needs correspondingly broader identification checks rather than relying on that one-dimensional safeguard.

6. Regularization changes the estimation question

A penalty can select a stable or unique coefficient vector when ordinary least squares is weakly identified, but it adds a modeling preference and tuning choice. Dropping a redundant variable also changes the coefficient interpretation. Neither step should appear as an undocumented numerical repair under an unchanged method description. Preserve the unpenalized diagnostic, chosen transformation and reason for the alternative. Any empirical comparison of penalties requires its own temporally valid development and evaluation design. This guide does not fit or rank such models. Its narrower purpose is to show why a well-behaved calculation and an interpretable estimated exposure are different acceptance questions.

7. Questions for an inspectable regression receipt

How many independent design columns remain after adding the intercept? Are repeated or mechanically linked predictors present? What units and centering were applied? Can small admissible changes reverse individual coefficients while leaving fitted values nearly unchanged? Does the reported rank depend materially on the solver tolerance? Common failures include treating no exception as identification, interpreting a generalized-inverse allocation as unique and using scaling to claim new economic information. Keep diagnostics with the fitted object and distinguish coefficient interpretation from forecast testing. Even full rank, acceptable conditioning and perfect reproduction do not establish a stable causal relationship or predictive value on untouched observations.

Primary references

These sources support methods and conventions. The example values are original constructions, not observations extracted from these documents.

• statsmodels: Pitfalls, rank-deficient and ill-conditioned designs
• NumPy: linalg.lstsq, rank and singular-value cutoff
• NumPy: linalg.cond, norm-dependent condition number

Related investigations