8/8/2019 Loess 1988

1/16

Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting

Author(s): William S. Cleveland and Susan J. DevlinSource: Journal of the American Statistical Association, Vol. 83, No. 403 (Sep., 1988), pp. 596-610Published by: American Statistical AssociationStable URL: http://www.jstor.org/stable/2289282 .

Accessed: 08/01/2011 18:24

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless

you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you

may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at .http://www.jstor.org/action/showPublisher?publisherCode=astata. .

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed

page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access toJournal

of the American Statistical Association.

http://www.jstor.org

http://www.jstor.org/action/showPublisher?publisherCode=astatahttp://www.jstor.org/stable/2289282?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/action/showPublisher?publisherCode=astatahttp://www.jstor.org/action/showPublisher?publisherCode=astatahttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/2289282?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=astata

8/8/2019 Loess 1988

2/16

LocallyWeightedRegression:AnApproach toRegressionAnalysisbyLocal FifingWILLIAM. CLEVELAND nd SUSAN J. DEVLIN*

Locallyweightedegression,r oess, is a way festimatingregressionurfacehrough multivariatemoothingrocedure,fittingfunctionfthe ndependentariablesocally nd n movingashionnalogous o how movingverage scomputedfor time eries.With ocal fitting ecan estimate muchwider lassof regressionurfaceshanwith heusual lasses fparametricunctions,uch s polynomials.hegoalofthis rticles toshow, hroughpplications,ow oess an be usedforthree urposes: ata xploration,iagnosticheckingfparametricodels,ndprovidingnonparametricegressionurface.Along he way, he following ethodologys introduced:a) a multivariatemoothingrocedure hat s an extensionfunivariateocallyweightedegression;b) statisticalrocedureshat re analogous o those sed n the east-squaresittingfparametricunctions;c) several raphical ethodshat re useful oolsfor nderstandingoessestimatesndcheckingheassumptionsnwhich he stimationrocedures based; nd d) theM plot, nadaptationfMallows's pprocedure, hichprovides graphicalortrayalf thetrade-offetween ariance ndbias, and which an be usedtochoose he mount fsmoothing.

1. INTRODUCTIONLocallyweightedegression,r oess, s a procedureorfittingregressionurfaceo datathrough ultivariatesmoothing:hedependentariablessmoothedsa func-tion f he ndependentariablesn movingashionnal-ogousto how a moving verage s computed or timeseries.Thebasicframeworks this.Letyi i = 1,n) bemeasurementsfthedependentariable,nd etxi

= (xi1, . , xip), i = 1, . .. , n,be nmeasurementsfp independentariables. uppose hat he data aregen-erated yyi= g(xi) + gi.As inthemost ommonlysedframeworkor egression, e suppose hat heeiare n-dependent ormal ariables ithmean andvariance 2.In theusualframework,e would lsosuppose hat isa memberf parametriclass ffunctions,uch s poly-nomials, utherewewill uppose nly hat isa smoothfunctionf the ndependentariables.With ocalfittingwe canestimate wide lassof smooth unctions, uchwider,nfact, hanwhatwecould easonablyxpect romany pecificarametriclassoffunctions.Smoothingy ocalfittingsactuallynold dea that sdeeply uriedn themethodologyf time eries,wheredata measured t equally paced points n timeweresmoothedy ocalfittingfpolynomialsMacaulay 931).Watson1964),Stone 1977), ndCleveland1979) ntro-duced ocal-fittingethodsnto hemore eneral ase ofregressionnalysis. astie ndTibshirani1986) ook ocalfittingne tep urther;n ny ituation here dependentvariable epends n independentariables, e cancarryouta local ikelihoodrocedure. leveland1979) ntro-ducedthespecificocal-fittingethodologyhat s thesubject f this rticle,ocallyweightedegression,ndDevlin 1986) expanded hemethodologynd addressed

* William. Cleveland s in Statistics esearch, T&T Bell Labo-ratories, urray ill,NJ07974.SusanJ.Devlin s inMeasurementsResearch, ellCommunicationsesearch, iscataway, J 8854. hisarticleenefitedreatlyrom iscussionsith revor astie,who haredhissubstantialxperience ith hebackfittinglgorithm.heauthorsaregratefuloJohn hambers,revor astie, on ettenring,ndColinMallows or elpfuluggestionsbout hemethods.hey lsothank woeditorsnd hree efereeshose ommentsedto substantialmprove-ment fthe xposition.

mathematicalroperties;n this rticle e furtherxpandthemethodology.he originalmethodologylso ncludeda robust ersionn whichM estimations ncorporatedothat he assumptionfnormalityan be relaxed, utwedo not ddress obustnessere.The pplicationsn his rticlellustratehreemajor sesof he ocal-fittingethodology.he firsts simplyo pro-videan exploratoryraphicalool; graphingmooth ur-faces hat re fittedo thedatacan give s nsightnto hebehaviorf hedata ndhelp schoose arametricodels.The second s to providedditional egressioniagnosticsto check he dequacy fparametric odels ittedo thedata.The thirds touse the oess stimates the stimatedregressionurface, ithoutesortingo a parametriclassoffunctions. hile resentinghese hree seswe intro-duce newmethods nd review ndapply omeoldones.InSection we ntroducehemultivariatemoother:tis a straightforwardxtension f the univariateoesssmootheriscussed yCleveland1979).Section hasanapplicationovelocity easurementsfgalaxy GC7531.Locallyweightedegressionsused ofit velocityurfaceas a functionfposition n the elestialphere.nSection4 we discuss he statisticalropertiesf loess. Fortu-nately,nalogs f the tatisticalroceduressed npara-metricunctionitting-forxample, nalysisfvariance(ANOVA) and t intervals-involvetatistics hosedis-tributionsrewell pproximatedyfamiliaristributions.Section has an applicationo measurementsf ozoneconcentrationndthreemeteorologicalariables. ocallyweightedegressions used o provide regressionurfaceandto carry utprediction.nSection we ntroduceheMplot, singMallows's p dea Mallows 966, 973)withappropriate odificationsor henew ontext,ndgraph-ing nestimate f mean quared rror gainst egrees ffreedomf thefit.The principalse oftheM plot s tochoose theamount fsmoothing,hat s, theneighbor-hoodsizeofthemultivariatemoother.ection has anapplicationo data from n industrialxperiment ea-

t 1988Americantatistical ssociationJournalf heAmericantatistical ssociationSeptember 988,Vol. 3,No.403,Applications Case Studies596

8/8/2019 Loess 1988

3/16

Cleveland and Devlin: Locally Weighted Regression 597suring heabrasion oss ofrubber pecimens. locallyweightedegressionnalysisuggestshat here s no in-teractionetween hetwo ndependentariables, o theregressionurfaces estimatedy dditive ittingHastieandTibshirani986).Section has an applicationo mea-surementsfNO, inengine xhaust. hehistoryfthesedata ncludes n estimationftheregressionurface yalternatingonditionalxpectationsACE) (Breiman ndFriedman 985), procedurehat ransformshedepen-dent ariable ndfits n additive urface othedata. Ananalysis y locallyweighted egressionhowsthat theregressionurface f these ata s such hat o nontrivialtransformationf hedata ould eadtoadditivity.ection9 describesimulationshat nvestigatehedistributionalapproximationsf Section . Section 0discusses uali-ficationsothemethodologynddiscussesthermethods.We also ntroduceraphical ethodologyn additionotheM plot.Because t s easier o discuss hesemethodswith raphs t hand,however, e introducehismeth-odologynthe pplicationsections. ections and 7 setforthonditioninglots, ection presentsomponent-residual lots, nd Section discusses iagnosticlots orcheckinghe ssumptions ade bout i.The hortenedame oesshas ome emanticubstance.A loess pronouncedlo is") is a deposit ffine lay rsilt long iver alleys; n a verticalross-sectionf arth,a loesswould ppear s a narrow,urve-liketratumun-ning hroughhe ection.

2. MULTIVARIATEMOOTHINGLocallyweightedegressionrovidesn estimate (x)oftheregressionurface t anyvaluex in thep-dimen-sional paceof the ndependentariables. et q be aninteger, here c q c n. The estimatefg at x usestheq observations hosexivalues re closest o x. That s,wedefine neighborhoodnthe paceof he ndependentvariables. achpointn theneighborhoodsweightedc-cordingo tsdistance rom; points lose o x have argeweight,ndpoints ar rom have mallweight. linearor a quadratic unctionfthe ndependentariables sfittedothe ependentariablesing eightedeast quareswith heseweights; (x) is taken o be thevalueof thisfitted unctiontx. Ofcourse,wemust o this ompu-tation or achvalue fx forwhich e want (x),andthusloess sa computer-intensiveethod,ut lgorithmsxistfor oing he omputationsfficientlyCleveland, evlin,andGrosse1988).To carryut ocallyweightedegressione must avea distance unction in the paceof he ndependentari-ables.Forone ndependentariable e etpbe Euclideandistance. orthemultiple-regressionase t s sensible otakep tobe Euclidean istancenapplicationshere heindependentariables re measurementsfpositionnphysical pace; forexample, he ndependentariablesmightegeographicalocation nd he ependentariabletemperature.fthe ndependentariablesremeasuredon differentcales, hen t s typicallyensible odivideeach variable yan estimatef scale before pplyingstandard istance unction.or theapplicationsfSec-

tions and 7, we divide ach ndependentariable y tsstandard eviation nd thenuse Euclidean istance. Inapplications here ne or more f theunivariateampledistributionsf the ndependentariables as outliers,tis sensible o standardize ith resistant easure f calesuchas the nterquartileange.)For the applicationfSection we use Euclidean istance ithoutdjustinghescale.Locallyweightedegressionlso requiresweight unc-tion nd a specificationfneighborhoodize.Theweightfunctionsed n ll of ur xamplessthe ricube unction:W(u) = (1 - U3)3 for 0 < u < 1, and 0 otherwise.Wenow showhow the weight unctions used. Let d(x) bethe distance f the qth-nearestitox. Then theweightfor heobservationyi,xi) is

wi(x) = W(p(x, xi)/d(x)).Thusw,(x) s a functionf isa maximumor iclose ox, decreases s thexi increase n distance rom , andbecomes for he th-nearestitox. Instead fthinkinginterms fq, thenumber fpointsntheneighborhood,we thinkntermsff = qln,thefractionfpointsn theneighborhood. s f increases, (x) becomes moother.The M plot,which s discussednSection , is an aid tochoosing inapplications.If ocally inear ittings used,thefittingariables rejustthe ndependentariables. f ocally uadratic ittingisused, he ittingariables re he ndependentariables,their quares, nd their ross-products.ocally uadraticfittingends o performettern situations here heregressionurface as substantialurvature,uch s localmaxima nd minimae.g., see the pplicationnSec. 3).3. NGC 7531 VELOCITYDATA:AN APPLICATIONILLUSTRATINGHEBEHAVIOR F THEMULTIVARIATEMOOTHERNGC 7531 s aspiral alaxyn he outhern emispherewith very rightnner ing. uta 1987)mademeasure-ments fthe velocities f thisgalaxy t a collectionfpointsnthecelestial phere hat overed bout200 arcseconds n the north-southirectionnd about135 arcsecondsnthe ast-westirection.hemeasurementserederived rom inespectrogramsaken t CerroTololoInter-AmericanbservatorynJuly ndOctober1981.Each spectrogramas made long narrowlit, nd thevelocity easurementseremade tpoints long he litbyobservingheredshift.he locationsf these elocitymeasurementsre shownnFigure .Ascanbeseenfromthefigure,here re sevenuniquepositionsf thenineslits,ince wo ositions ere sed wice; he even niqueslit ines ntersectt a point n themiddle f theobser-vationegion.hemaximumelocity easurements1,785kilometerser econd nd theminimums1,409km/sec.The data re catteredecause fmeasurementoise nddo notform smooth elocityield.The velocityurface as estimatedy ocally uadraticfitting ith = .4. Figure is a contour lot.Thefittedsurface oes good ob of ollowinghe nderlyingatternin the data. For example, he surface ollows he peaks

8/8/2019 Loess 1988

4/16

598 Journal fthe American StatisticalAssociation, eptember 1988130 I

90 -~ ~ ~~~90~~~~~~~~~

050 * 010 0~ ~~~~O00 00000

t~~~~~~o j0 . __70 is *

* 0*0 0 0 0C.) ~~~~~~0 0 .

3070 0 0 0

East Eat- West oordntlcons

00 * o 00 0* 0 0*. g 00

Figure0 NGC 0.0~~~~~~~~ I0 OgJ 0000 00~0 00*70 0~~~0 0

- 70 -30 10 50 90--o- ast East West oordinateArc econds)

Figure . NGC7531Velocityata. Theplot hows the ocationsnthe celestial phere t which heNGC 7531 velocity easurementsweremade.andtroughsn thedata:Themaximumalueofthe sti-mates tthe ositions here hemeasurementseremadeis 1,757km/sec,nd theminimumalue s 1,440km/sec.When ocally inear ittings used,thefit s poorer ndcannot rack he ubstantialurvaturenless istaken obevery mall, bout 1, nwhichasethe stimatedurfaceis verynoisy.Thevelocity atternevealed ythecontourss inter-esting. here ppears o be anaxisof ymmetryfabout1080 the axis s shown y thedottedine nFig.2). Aswe move rom orth osouth long his xis, hevelocityincreasesy bout 20km/sec.uppose hat he nlymo-tions fthegalaxy relativeo the arth)were rotationabout n axis hroughts enter nd recessionue to theexpansionf heuniverse.hen he elocityurface ouldbe linear, he ontours ould e straightinesparallel otheprojectionfthe xisofrotationn theviewinglanefromhe arth,nd hevelocitylong his rojectionouldbeequaltothe ecessionelocity.igure doesnot ollowsuch pattern. he velocitys not inear long he1080axis:As we move rom he enter utwardlong he xis,therateofchange f thevelocityecreases ather hanstayingonstant.urthermore,hecontoursrecurved,bendingnewaybelow he1,580 m/secontourndtheotherway bovethis ontour. evertheless,he ontourssuggest hat hepredominant otion fthegalaxyasidefrom herecession)scircular. he motionuperimposedon this otation, hich esultsn thebending f the on-tours,s notyetknown Buta1987).

+ ~~~~148050- 1440z 15001580

5251520 16001540 12156

1 801700-Z -2 1580CO)

1740

-50 -25 0 25 50- East East - West CoordinateArc econds)

Figure 2. NGC 7531 Velocity ata. The velocity urface was esti-mated by ocallyquadratic fitting ith = .4. The figure hows surfacecontours. The dotted ine has a slope of 1080;the surface is roughlysymmetric bout this ine.

4. STATISTICALROPERTIESThe oessestimate,(x), is a linear ombinationftheYi,

ng(x) = E li(x)yi,i=1

where he i(x)depend n Xk for k = 1, . . ., n, W, p,andf,butnot ntheyi.Let9i = A(xi) e the ittedalues,let i - i be theresiduals,nd ety = (yl, * *Yn)', 9 (Yi, , 9n)', and e = (i1, . . ., e)' Sinceeach9i is a linear ombinationftheelements fy,wehavethat9 = Ly,where (locallyweightedegression)is an n x n matrix nd = (I - L)y, where is thenx n identity atrix. his s analogous o parametriceastsquares: or east quares, he ittedalues reGy,whereG (Gauss) s the rojectionperatornto he pace pannedbythefittingariables.f we apply othG andL tothevalues of one of thefittingariables,we getthesamevaluesback.Oneway o write his s GG = G andLG= G. Butunlike ,L isneitherymmetricordempotent(Devlin1986).There re three ey ngredientsor iscussinghe am-pling ariabilityf the oessestimate:a) thatg(x) is alinear ombinationf heyi; b) the ssumptionhat hasa normal istribution;nd (c) theassumptionhat (x)estimates with obias.For ocallyinear ittinghe s-sumption f no bias can onlybe exactly ruewhen islinear, nd for ocally uadratic ittingt can only e ex-actly ruewhen is quadratic. evertheless,hegoal ofpart f thediagnosticheckingdiscussedn Sec. 5) andtheM plot discussedn Sec. 6) is to find stimates ith

8/8/2019 Loess 1988

5/16

Cleveland and Devlin:LocallyWeightedRegression 599negligibleias. Note that ack of bias also underlies hedistributionalesults f parametricegression.The major onclusionfthis ections that everal ta-tistics efinednalogously ith hose sed n fittingara-metric unctionsy east quareshave distributionshatarewell pproximatedy hose sed nparametricegres-sion.This s good news, ecausefamiliarechniquesanthus e used n makingnferencesasedon loess. n theremainderf his ection e presenthedistributionalp-proximations,nd in Section we describe imulationsthat tudied hequality f the pproximations.4.1 Distributionsf Residuals, FiltedValues, andResidual Sum of Squares

Because of the inearitynd normality,and e havenormal istributionsith ovariance atrices2 LL' anda2(I - L)(I - L)', respectively.own

16 = el = residualum fsquares.i=lBecauseoftheunbiasedness,(e's) = a2 tr(I- L) (I -L), andwecanestimate2 by

a= 1'81tr(I L)(I - L)'.Thus, ince hevariance fg(x) is

na2(X) = 2 i=l

wecanestimatetbyn

(72(= 2 12(x).i=lWe can pproximatehe istributionf quadraticorminnormal ariables uch as ' by the distributionf aconstant ultipliedya x2variable; hedegrees f free-domand the constant re chosen o that he firstwomoments f theapproximatingistributionatch hoseof thedistributionf thequadratic ormKendallandStuart1977). Let 61 = tr(I - L)(I - L)' and let 2 =tr[(I L)(I - L)']2. Using hismethodf pproximation,thedistributionf la 2)/Q52C2) is approximated y adistributionith 2l52 df, nd thedistributionf g(x) -

g(x))/d(x) s approximatedy a tdistributionith(x)df.Wecan use this esult o getapproximateonfidenceintervalsor (x) basedong(x).4.2 Analysisof Variance

Suppose hatNy ndAyare two ectorsffittedaluesfor woregressionrocedures.We thinkfN as yieldinga fit or nullhypothesisndA as yielding fit or nalternativeypothesis.or example,N might e linearleast quares o thatN = G andA mighte loess o thatA = L, or A mighte loesswith small alueoff, say.3, and N might e less with larger alueoff, ay 9,so thatN = L9g ndA = L3. Lety'RNY = y'(I - N)(I- N)'y andy'RAY y'(I - A)(I - A)'y be the esidualsum f quares f he wo its.fwe want o testN againstA, the ikelihoodatio est eadsus to y'RNy)I(y'RAy)

> c. Thuswe will se n nalogy ith NOVAa test asedon (y'RNY - y'RAy)/y'RAY. In this est he reduction ueto A in theresidual um f squares s compared ith heresidual umof squares fA. [Devlin 1986) discussedsomewhatifferentpproach otestingor he pecial asewhereN = G.] Let v1 tr(RN - RA), V2 = tr(RN - RA)2,(1 = trRA, and 52 = trR32.The idea is to use the two-moment2 pproximationor henumeratorf he fore-mentionedtatisticnd he enominator,nd pproximatethe est tatisticyan F distribution.hat s,

A (y'RNY - y'RAY)IV1(y'RAy)I(1is the est tatisticnd tsdistributions approximatedyan F distributionith 1/v2nd 11l62 f.We refer ov1 sthe numeratorivisor f theF test nd to v1/v2as thenumeratoregrees ffreedom.imilarerminologyoldsfor 1 and (51/2.

5. OZONEANDMETEOROLOGICALATA:ANAPPLICATIONLLUSTRATINGHEUSE OFTHE TATISTICALROPERTIES,IAGNOSTICCHECKING, ND CONDITIONINGLOTSThe data n this pplicationre 111measurementsffour ariables-ozone an airpollutant),olarradiation,temperature,ndwind peed-on 111days etweenMay1 and September0, 1973, tsites n theNew YorkCitymetropolitanegion Bruntz,Cleveland,Kleiner, ndWarner 974). We analyzed hesedata to describe hedependencefozone on themeteorologicalariables othat zoneconcentrationsan be predictedrom orecastsofthemeteorology.igure isa scatterplotatrixfthedata.The first tep n theanalysis fthesedata wastosmooth zone sa functionf hemeteorologicalariablesby locallyinear ittingith = .4.0 5o 100 150 5 10 15 20

soi .';' 11 X 00

150 1X 11015 '. I:t',.......;i100 ;!00 300 Z K 10O~~~~~~~20

Figure. Ozonen Meerloia at.Te iueisasatepo

matriS Wind Spere a

o 1020306 8 0

mtIx of11Wesrmntfooe indpe,Seperaue,adsolar adiation.he oal s topredicthe zone oncentrationsromthemeteorologicalariables.

8/8/2019 Loess 1988

6/16

600 Journal ftheAmericanStatisticalAssociation, eptember 1988The oessialmethodologyiscussedn hisrticle idensthedomain f pplicabilityompared ith hemuch-prac-ticed arametric-functionitting;evertheless,hemeth-odologysstill asedoncertainriticalssumptions.neis that he rrors,i, are ndependentlyndnormallyis-tributedith onstantariance. nothersthat hefitted

functionollowshepattern fthedata, hats,providesa nearly nbiased stimate. uchassumptions ustbechecked.When ssumptionsreviolated ecanoftenakecorrectivections imilar o those used in parametricregression.here lready xists wealthfdiagnosticro-cedures or egression odelsBelsley, uh, ndWelsch1980;Chambers, leveland,Kleiner, nd Tukey1983;CookandWeisberg982;Danieland Wood1971).Muchof t s applicable o locallyweightedegression;or x-ample, ne canmake a normal robabilitylotof iE ocheck henormalityssumption,ake plot fJeijgainst9ito check heassumptionf a constant ariance, nd75 - 0

so- 050 02 5

0-25 0

0-50 - --3 -2 -1 0 1 2 3

Normal uantiles75 -0|j[ 0

0 ~~~~~~~~0'F 50 -:2 ~~~~~~00~~~~~~~~o 0 0or5 0 00 250 025 - o 0 ?0 ? o? 0o P0 00 00 0 0 0

010 w 9 000 ooI I I I . Io 25 50 75 100 125

Fined aluesFigure4. Ozone and MeteorologicalData. Ozone was regressed onthemeteorologicalvariablesusing ocally inearfittingnd f = .4. Thetop anel s a normalrobabilitylot f he esiduals. he ottomanelis a graph f he bsolute esidualsgainst he ittedalues; he moothcurve s a loess fit o the ataof he lot,with = 2/3.The lots hownonnormalitynd a dependenceof variance n the evelof thede-pendent ariable.

graph&j gainst he ndependentariables o checkforbias.Figures and5 arediagnosticlots or he ocallyinearfit othe zonedata.Thetoppanel fFigure is a normalprobabilitylot fthe 6.Thecurvatureuggestshat he6ihave a distributionhat s skewed o theright. hebottom anelofFigure is a plotof silversus j. Thesmoothurves a locallyinear it o thepoints f heplotwithf = 3. The plot suggests hatthevariance of 6ide-pends nthe evelofg. Figure shows lots f&igainstthe ndependentariables. hecurves n thegraphsrelocallyinear itswith = 2. Nodistortionppearsnthetoppanel,but small ffectppears nthemiddle aneland more erious neappears n he ottomanel,whichsuggestshat heestimatedurface s notfollowinghepatternnthedata. Ofcourse,t spossible hat hedis-tortions alsocausinghe nadequaciesnFigure .

050 - 0

a: ~0 00~008

00 %00:3 04 io 6o% o

O 100 00 300SolarRadiation(D 0~~~~

50 0

a: 00 0 m0 000 0

80 8o 100TemnperatureM- 0

50

50 0~~~0 0 00 ~ ~~O o O~~~~0~~~~~~

? cO(D~~~ 00~~~0

S 10 15 20WindpeedFigure . Ozone ndMeteorologicalata.The esiduals or he zonedata regraphedgainst hendependentariables;hemoothurvesare oess fits othedata oftheplots,with = 2/3.Theplots ndicatethat he stimated egressionurface oes not it hedata.

8/8/2019 Loess 1988

7/16

Cleveland and Devlin:LocoallyWeightedRegression 601Wecould educe hedistortionydecreasinghevalueoff,which s .4. But since is already airlymall, ndsince hefittedurface as substantialurvature,ede-cided ocombathe istortiony witchingo ocally uad-ratic ittingith = .8. Thedistortionisappeared,utthe nadequaciesfFigure remained.huswetook he

cuberoots f theozoneconcentrationsnd againcom-puted locally uadratic itwith = .8. Thisestimatepassed hediagnostichecks.Figures-8 arethree-variableonditioninglots or helocallyuadraticit. neachpanel fFigure ,g sgraphedagainstemperatureor ixed alues f olar adiationndwind peed,and confidencentervalscomputeds de-scribednSec. 4.1) are shown t five alues ftempera-ture. or example,nthepanels fthebottomow, olarradiations 50 langleys;n thepanels f the eftmostol-umn,wind peed s 5miles erhour. igures and8 graphg gainstolar adiationndwind peed, espectively,orfixed alues fthe ther ariables. heconditioninglotsshow learlyhenonlinearityf he egressionurfacendthe nteractionmong he ndependentariables.One majorreasonforfitting regressionurface oozone data sprediction,ither etrospectiverprospec-tive.Wewant o predicthe everityf ozonepollutionfromctual rpredictedalues f hemeteorologicalari-ables.For example, uringheperiod fmeasurement,May1-September0, 1973, hereweremany ayswithmissingzone measurementsecauseofmalfunctioningequipment.woof hese ays,August 0 nd11,followedthree ays frelativelyigh oncentrations,22,89,and110parts erbillionppb),all ofwhichwere bovetheWind ped

5 10 157-4 0 3 3 290

0~~~~~~~~~~~~~~7'0? 4 - X < t 170 x

0CO)7-4 ] < H950

60 70 80 90 60 70 80 90 60 70 80 90Temperature

Figure . OzoneandMeteorologicalata. Because of he roblemsindicatedythediagnosticlotsnFigures and5,ozone wastrans-formedy ube ootsnd ocallyuadraticittingith .8wasused.This igurehows conditioninglot. ach anel hows slice f heregressionurfaces a functionf emperatureor ixedalues f olarradiationndwindpeed; he erticalinesre95% onfidencenter-vals.

Wind peed5 10 15

7 -4 ~~~~~~~~~~~~~9

7 ~ ~ ~ ~ ~ ~ ~ 74 - HS w l7 -4 - A 62

0 100 200 300 0 100 200 300 0 100 200 300SolarRadiation

Figure 7. Ozone and MeteorologicalData. Each panel ofthis con-ditioning lotshows a slice oftheregression urfaceas a functionfsolar radiation or ixedvalues oftemperaturend windspeed.federaltandardf80ppb.Did thepollutionpisode on-tinue n these wodays, rwas t reduced?Wecan usethe oess urfaceo estimatehemissingzoneconcentra-tionsfrom hemeteorological easurements.he rightand eft ndpointsfapproximate5% confidencenter-vals, llontheppb cale, rethefollowing:ugust 0-68and97; August 1-34 and 57.Thus zonemightavebeen omewhatlevated n the10th, utwith igh rob-abilitytdropped nthe11th.

Temperature62 76 90

7 X

7 ~~~~~~~~~~04 8 12 16 4170 a:a:~~~~~~~~~~~~~~~~~~~~c

0CO7-

4 ~~~~~~~~~~~~~~4 8 12 16 4 8 12 16 4 8 12 16

Wind peedFigure. Ozone ndMeteorologicalata.Eachpanel f hison-ditioninglot hows slice f he egressionurfaces a functionfwindpeedfor ixed alues f emperaturend olar adiation.

8/8/2019 Loess 1988

8/16

602 Journal ftheAmerican StatisticalAssociation, eptember 19886. THE M PLOT

Mallows 1966) invented procedure alled Cp forchoosing subset f the ndependentariables asedonestimatesf hemean quared rror or ach ubset. ater,Mallows 1973)extendedhis o a moregeneral lassofestimatesnd ppliedt ochoosinghe arameternridgeregression.We can also extend t to locallyweightedregressiono help choosethevalue off.The expectedmean quared rrorummed ver he i nthe ample nddividedby a2 isMf= LE (gf(xi) - g(xi))2 a2,

where henotationor hefittedalues, f(xi), owhas asubscripto show hedependencenf.Suppose hat sis anestimate fU2 from smoothing here , thevalueoff, ssmall, suallyntherange rom2 to 4.The deais to choose a small so that he bias ofgM(xi)willbenegligible,hich esultsn a nearly nbiased stimate fa . Now, etBf = fe - tr(I - Lf)'(I - Lf)

andVf= trL'Lf.

A simple erivationhows hatwe can estimate fbyMf= Bf+ Vf.Bf sthe ontributionfbiasto the stimatedmean quared rror,ndVfsthe ontributionfvariance.If,for particular, fs a nearlynbiasedstimate,henusing standard5-methodrgumentKendall ndStuart1977) he xpectedalue fBf snearly ,so the xpectedvalue fMf snearly f. f sf ncreases ias s ntroduced,Bfhas a positivexpectedalue, o the xpected alue fMf xceedsVf.HereVf sthe quivalentumber fparametersf thefit, measure f theamount fsmoothingonebythelocal-fittingrocedure.Weuse thisnamebecause fwehaddoneordinaryinear east quares, hen he peratormatrixfwouldbe replaced y G and trG'G = trG,thenumber fparameterssed n the fit. n the forth-comingpplications,f ecreases sf increases,omoresmoothingesultsn a smallerquivalent umber fpa-rameters.TheM plot s a graph fMf gainstVf or selectionoff valuesbetween and1; this etsus see the rade-offbetween he ontributionsfvariancendbias othemeansquared rror s f changes.t is alsohelpful,orudgingvariation ntheplot, o show nformationbout hedis-tributionfMfwhen here s no bias. We can proceedexactlys in Section .2. Let RNbe the matrix or heresidualum f quareswhen he moothingarametersf, hat s, , = Y'RNY, and et RAbe thematrix hentheparameterss. Then

==v (Y'RNY -Y'RAY)/Vl ? (51- n+? 2tr Lf= v1F? os - n ? 2 trLf.

As before,we approximatehedistributionf F by an Fwith 'l/v2nd 51152f, ndtherebypproximatehedis-tributionfMf.It is importanto emphasize hat heM plot s not n-tended oproduce ard-and-fastules or he hoice ff.Rather, yshowinghe trade-offetween ariance ndbiasas f changes ndsome nformationbout amplingvariability,tassistsnour udgmentfanappropriate.Sometimes e want ominimizehemean quared rror;thismighte the ase whenwewant ouse g(x) for re-diction.nother pplicationse may ecide hatowvari-ance s importantnd thus hoose nf thatnflateshebiassomewhat; hismight e the case when hesamplesize s small r we aresearchingor simple escriptionofthedatastructurehat aptures he alient eatures.nstill ther pplications e might ecidethat ow bias iscritical;his s often he casewhen he oessestimatesusedfor raphicalxploration,ince ur yes antoleratesomenoisebut annot ecover missed ffect. outinelychoosing byminimizingf s a poorprocedureecauseit gnores ariancend bias,which re mportanto con-sider nmost pplications.Mallows1973)made he amepoint bout heuse ofCp.]Furthermore,t theminimum,Mf soften lat ompared ithts ampling ariability,oa range fvalues off withdifferentariance nd biaspropertiesives he amemean quared rror.The Mplot an beusedformore eneral urposeshancomparingoess moothingsith ifferentalues ff.Forexample, e can addM fromny arametricit rMfromotherocal-fittingroceduresuch s additive ittingdis-cussed nSec. 8). We do this ycomputingvalueofMin mannernalogous o the omputationfMf, ndwitha2 still stimated yas.7. ABRASION-LOSSDATA:AN APPLICATIONNWHICH THEM PLOT S USED TO CHOOSE fAND AN ADDITIVE URFACEFITSTHEDATAAn ndustrialxperimentasrunmeasuringhree ari-ablesfor achof30 rubberpecimensDavies1957).Eachspecimen as rubbedwith n abrasivematerial,nd theabrasionosswasmeasured;he xperimentas to relatethis oss to measurementsf the hardness nd tensilestrengthfthe pecimens.igure is a scatterplotatrixof he ata,which eanalyzedy ittinglinear egressionmodel.We ntendoevaluate hismodel.nan nitial assover hedata anoutlier as foundndremoved; e ana-lyze heremaining9observations.Since heoutlier idnot esultn xtremealuesn ny f he nivariateampledistributions,he ndependentariables ere tandardizedbased on sample tandard eviationsomputedromll30observations.)Figure 0 s an Mplotwith = .3.Thecircles howMfversusVffor rangingrom = 1 (the eftmostircle)tof = .3 (therightmostircle) nsteps f 05. The lineMf = Vfhasbeendrawn; otethatMsmust ie on thisline.Theverticaline egmentsnd theirickmarks or-

tray he amplingistributionfMf,under hehypothesisof no biasand using hedistributionalpproximatione-scribedn thepreviousection: hetopof ach ine sthe

8/8/2019 Loess 1988

9/16

Cleveland and Devlin:LocallyWeightedRegression 603120 160 200 240

0 0 0 000 0 000 0-;?0 o0 0 0 00 0 0 0 0Hardness 0 0 0 0 000 000 0 0 0 1 a 0000 0.00 0

0 0240- 000:o 000000 0 000 0 0 0200- Oo 0 00 0 0 00 Tensile tength 00160 r00 0 00 0 0 0 0 0.00 00 0 0 0 00 000 0 0120- 0 0

00 0 0 0 00 0I

loalrgrsso smohig beas les Abrarsio oss

0 00- 0 ~~~~~~0 0

00e at smo0hnta loa 0egrbsi asIon osse 1

#00 00 000 0 000 0 0 000 00 0 0 0 00 000 000~~~~

50 70 90 50 150 250 350Figure.Abrasion-Lossata.The igures a scatterplotatrixfdata romn ndustrialxperimentnwhichbrasionosswas tudiedas a functionfhardnessnd ensiletrength.

95%point,he pper ickmarksthe 0%point,he owertickmark s the10%point, nd thebottom fthe ine sthe 5% point.The G on theplot s thevalueofM forlineareast quares.Note that heequivalent umber fparametersor he east-squaresits ess han hat or nylocal-regressionmoothing,ecause east squaresdoesmoredatasmoothinghan ocalregression.nAFigure10there s no clearly efined ointwhere heMfbeginprecipitousise, ndMf s flat ompared ithts amplingvariability,rom = .3 tof = .5; we chosef to be .5,preferringn estimatehathadas lowa variance s pos-sible,nview fthe mall ample ize,withoutntroducingundue ias.Notethat heA!valuefor east quares howsthat he inear-modelit n theoriginalnalysiss inap-propriate.Figure 1plots hefitwith = .5 n thefollowingay:Consider he opcurventhebottomanel.The valueofhardness asbeen set to 60. Thecurve s a graph fthefittedurfacegainst ensile trengthor his ixed alueof hardness. ortheother urves nthepanel,hardnesshas beenset to other alues.The graphn the bottompanel ssimilar,ut he onditioningson tensile trength.Thisgraphicalool s a two-variableonditioninglot hatcanbe usedgenerallyo explorebess fitswith wo n-dependentariables. fcourse,t sanalogouso the hree-variable onditioninglotsofFigures to 8. Figure 1reveals everalmportantropertiesfthe stimatedur-face.-n eachpanel he oururves averoughlyhe ameshape, arying ostlyn evel, uggestinghat heres ittleinteractionetween ensile trengthndhardness. ur-functionf hardness nd a nonlinear unctionftensilestrength.Figure 1 suggestshatwe incorporateack of nterac-

35 3 6 9G

25-

* 15C,)

5-

-5-3 6 9 1

EquivalentNumber f ParametersFigure10. Abrasion-Loss Data. TheM plot s a graphicalmethodforchoosing the smoothing arameter,f, n locallyweightedregression.The filled ircles show M statistics, stimates of the mean squarederror, or rangingfrom3 (rightmostircle) to 1.0 (leftmostircle). TheG shows theM statistic or linear east-squares fit.The M statisticsare graphed against their xpected values underan assumptionofnobias. The slanted line on the plot s y = x, so the vertical istance ofan M statistic o the ine is the contributionf bias to the estimate ofthemean squared error.The ends of the vertical ines show 90% in-tervals, nd the tickmarksshow 80% intervals f the distributionsftheM statisticsunderan assumption of no bias. On the basis ofthisplot,fwas chosen to be .5.

tion nd inearityfhardnessnto he moothing. e cando this y followinghe additive-estimationpproach fHastie ndTibshirani1986).An additivestimateonsistsof sum f mooth unctionsf he ndependentariables,g1(xi1) + gA(xip).he k are he omponentunctions.The salient eature f the estimates that lthoughheregressionurface s nonlinear,here s no interactionamong he ndependentariables.Additive stimationan be carried ut by using hebackfittinglgorithmrom rojectionelectionBreimanand Friedman985;Friedmannd Stuetzle 981;HastieandTibshirani986).Backfittingsan terativerocedure.In each iteration componentunction,aythekth, supdated y moothing,minus he um f he ther om-ponent unctionss a functionfXik. In our mplemen-tation he moothings carried utby oess.Thefinal itis a linear peratorpplied oy. Forthis eason hedis-tributionalesults fSection apply obackfittingswell,butwith replaced ythebackfittingperator.Additiveittingasusedfor he brasion-lossata,withthe omponentunctionorhardnessstimatedy inearleast squaresand the component unctionor tensilestrengthstimatedy oesswith aryingalues ff.Figure12 showsMi or hese its. he values f used nthe oesssmoothingange rom = 1 (leftmostircle) ofi = .3(rightmostircle)n teps f 05, ust s for hemultivariate

8/8/2019 Loess 1988

10/16

604 Journal fthe American StatisticalAssociation, eptember 1988350-

6250

150~ ~ ~~~~~~~~5173197220

50 - *60 70 80Hardness

350 -

~2501 ~~~~~~~~~~~60150 -67

7380

140 160 180 200 220Tensile trengthFigure 11. Abrasion-Loss Data. Conditioning lots show a loess fitto the abrasion-loss data, withf = .5. The graphs suggest the de-pendence onhardness is linear nd that here s no interactionetweentensile trength nd hardness.

loess nFigure 0.Also, the stimatefo2 iS the ame sthatnFigure 0. Theplot hows hat n additivemooth-ing anprovide n acceptable it o thedata;we choseftobe .75,preferringlow-variancestimate ithout n-dulynflatinghemean quared rror,gainnview f hesmall ample ize.Additiveits anbegraphed y omponent-residuallots.As before,et gr(xir)be the estimatedomponentunc-tions, nd et&i e theresiduals. o studyhepropertiesofthefitwe canmake neplotfor achcomponentunc-tion: r(xir) is graphedgainst ir or = 1 to n bycon-nectinguccessiveoints y ine egments,ndgr(xir) +&isgraphedgainst iry ircles. heseplots llowus tosee theform f the stimatedurface nd to see whetherany signalhas leaked into theresiduals. he plottingmethod ollows hat sed npartial esidual lots Land-wehr 983;Larsen ndMcCleary 972),where hecom-ponent unctionsave a differentorm.Figure 3 shows omponent-residuallotsfor head-ditive it o the brasion-lossatawith = .75. Thetoppanel hows learly heform hat henonlinearityakes;

35 I

25-

15CI)

5-

5-53 6 9 12

EquivalentNumber f ParametersFigure 12. Abrasion-Loss Data. Figure 11 suggests that n additivenonparametric moothingwith o interaction ill it he data. Thisfigureis an M plot for dditive fitswithhardness estimated by linear eastsquares and abrasion loss estimated by loess, with rangingfrom3to 1 insteps of 05. On the basis of thisplot, f was chosen to be .75.

there s a hockey-stickependence. logical ext tep ntheanalysis f thesedata wouldbe to fit parametricmodel n whichhedependenceftensiletrengthscon-tinuous ndpiecewiseinear.8. NO, DATA: AN APPLICATIONN WHICHTHEM PLOT S USED TO CHOOSE fAND ANADDITIVE URFACEDOES NOT FITTHE DATAThe data nthis pplicationre from nexperimentnwhich single-cylindernginewas runwith thanol rindoleneBrinkman981).There re110measurementsofcompressionatioC), equivalenceatioE), andNO,inthe xhaust. hepurpose f he nalysis astoseehowNO,depends n E andC. Therewere 8runswiththanol;for heseruns, varied rom 535 to 1.232,C tookoneoffive alues angingrom .5 to18,and thevalues fEand C werenearly ncorrelated.herewere 2runswithindolene; or hese uns, took ustonevalue,7.5,andE ranged rom665to 1.224.Rodriguez1985) nalyzedhese atausing CE (Brei-man and Friedman 985) andMORALS (Young,De-Leeuw, ndTakane 976),withype ffuel s a categoricalvariable nd C andE as continuousariables. n ACEanalysisheresultingurfaces an additive it o a trans-formationfthedependentariable. hus nACE fit otheNO, concentrationsesultsna surface ith o nter-action.Ourgoalwas toexplore hedata to see if an additivefitwas reasonable. o allowforgeneral nteractions,e

treated and type ffuel s a single ategoricalariablewith ix evels, inceC wasequal to 7.5 for ll indoleneruns nd to fivevaluesranging rom .5 to 18 for he

8/8/2019 Loess 1988

11/16

Cleveland and Devlin:LocallyWeightedRegression 605300 - 0

0

a 250- o0 0 00a: ~~~~~0

X 200-1[ ~~~~~8\0

150 - 0 00000 0 0100 00 0

120 160 200 240Tensile trength

180 , ,

'2 0~~~

45 60 75 90

0~~~~0 0 0

0 o0-90 ~~~~~00~~~~~~~~~~~-90 ~~~~~~~~~~0

000-180 -7 J45 60 75 90Hardness

Figure 3. Abrasion-Lossata.Component-residuallots howtheadditive it o the brasion-loss ata, with = .75. The urve n eachplots the stimatedomponentunctionor ne ndependentariable,andthe lottingymbols how he omponentunctionalueplus theresidual or ach observation.ethanol uns.Thus there re two ndependentariables,E and this ategorical ariable. urthermore,heNO.concentrationsere ransformedy uberoots. hus heloessanalysis onsistsf sixseparatemoothingsfcuberootNO. as a functionfE, one for ach levelofthecategoricalariable. he smoothernthis asewas ocallyquadratic ittingecause, s we shall ee, thefunctionaldependence f cube rootNO. on theequivalence atiohas a localmaximumndsubstantialurvature.Figure 4 sanMplot or he ocallyuadraticmoother;thevalueof is 4,and nmovingromeft oright goesfrom to 4 insteps f 05.Onthebasis f his lot waschosen obe .85;Mf umps onsiderablyorarger aluesoff.Thetoppanel fFigure 5 hows he ix ocal-regressionestimates,k(x) fork = 1-6, for hesix evelsof thecategoricalariable. ach estimate as computedt 50equallypaced alues fE from6 to1.15; et he 0valuesbedenoted yx,* or]j i to 50. Eachestimatesgraphed

100- l l l

ioo5-049 50-25-

20 30 40 50EquivalentNumber f Parameters

Figure 14. NO, Data. The data are from n experiment tudying hedependence ofNOxexhaust emissions on equivalence ratio, ompres-sion ratio, nd typeoffuel. The figure s an Mplot for ocally quadraticfitting, ith rangingfrom 4 to 1 in steps of 05. The type of fuel ndthe evel of compression ratio,which took on one of fivevalues, werebothenteredas categorical variables. On thebasis ofthe plot,fwaschosen to be .85.in thetop panel by connectinguccessive aluesby inesegments. he bottompanel ofFigure15 is an interactionplot. Each curve s a graphof

hk(Xj) - - h(xj)against ,.Figure 5 shows omethingmportant:or the thanolruns, here s a substantialnteractionetween and E.As C increases O1'3 enerallyncreases,ut he ffectsreduced s E increasesndeventuallyecomes earlywhenE is at its maximum alue. ndolene dds to thisinteraction,ecause tsbehaviors a functionfE isdif-ferent rom hat fethanolwith equalto7.5. Thus nadditiveitscompletelynappropriateor hese ata. TheM plotfor headditive its, s one would xpect, howsvery arge iases.)Furthermore,igure 5shows hat heform f the nteractions such hat nontrivialransfor-mation fNOxcannot ossibly emove he nteraction,whichmeans hatACE cannotead to a satisfactoryodelfor hesedata.

9. LABORATORY ND FIELDSIMULATIONSMonte Carlosimulations ithnormal iwere runtoinvestigatehedistributionalpproximationsiscussednSection . We constructedwide ollectionfdesignon-figurationsi.e., sets of valuesof the ndependentari-ables) forup to five ndependentariables. hree tems

weretudiednthe imulations:a) distributionf w2/a2,(b) confidencentervalsor (x), and c) ANOVA forN= linear east quares ndA = locally inear itting.he

8/8/2019 Loess 1988

12/16

606 Journal f theAmerican StatisticalAssociation, eptember1988Indolene = 7.5 EthanolC=12.0EthanolC=18.0 EthanolC= 9.0 .................Ethanol C=15.0 --------- EthanolC= 7.5

1.8 '

1.4-

0.8

0.2 -

N. -

02 -~~~~~x 0.1 ,/ \

0.0 /.y'~~~~s.0.1 ,

-0.2 _________________0.6 0.8 1.0 1.2EquivalenceRatio

Figure 5.NO,Data.The oppanel hows he ix eparate mooth-ings fNOx s a functionf quivalenceatio. hebottomanel howsthe urveswith mean curve ubtracted.hegraphs howa stronginteractionmong he ndependentariableshat annot e removedbya nontrivialransformationf hedependent ariable. hus n ad-ditive its not ossiblefor hesedata.distributionalpproximationsfSection wereexceed-ingly lose to thetruedistributionsor a) and b). For(c) theywere lose, xceptwhen hedegrees f freedomof he itwere large ractionfn;however,his ituationisnotrelevantnpractice.Werefer othese imulationsas laboratoryimulations,ecause hey mployrtificiallyconstructedesign onfigurations.n Section .1, (c) isinvestigated;nSection .2, a) and b) are nvestigated;and nSection .3, (c) is investigatedor modificationofthe oessprocedure.For normal i, thetruedistributionsf the statisticsinvolvedn a)-(c) depend nthe alue ffand he esignconfiguration.dataanalystancheck hedistributionalapproximationorany particularpplicationhroughsimulationsing hedesign onfigurationfthedataandthevalueoff used n the moothing. e call these ieldsimulations.f thediagnosticheckingf theresidualsshows hat he ample istributionf theresidualsswellapproximatedy normal istribution,hen he ieldim-

ulation an use samples rom he normal.f significantnonnormalityppears n theresiduals,hen amplinganbe from he ample istributionf he esiduals. wofieldsimulationsrediscussednSection .4.9.1 Laboratory imulations:Analysisof Variance

Inthis ection e discussaboratoryimulationsor est-ingN = linear east quares gainst = locally inearfitting.igure 6 shows ome of theresults or ne col-lection f 0 simulations;ach imulationmployed6,000replications, hich avehigh ccuracy ven at the .01significanceevel.The 60 simulationsmployed 8 designconfigurationsnd 4 values ff;not ll values ff wereusedwith achconfiguration,incewe limited ur nves-tigationso practicalituations.Therewereninedesign onfigurationsor = 1. Foreachof hree alues fn,100, 0, nd25,there ere hreesetsof valuesofthe ndependentariable. ach setwasoftheform -1[(i - .5)/n]for = 1, . .. , n, wherewas either he uniform,ormal, r Cauchy istribution.Simulations ith = .3, .5, and .7 were runfor achconfiguration,esultingn27 simulations.There were sixdesign onfigurationsor = 2. Foreachoftwovalues fn,50 and100, herewere hreeetsof values fthe ndependentariables. ach setwas de-rivedn thefollowing anner: ne independentariablewas nitiallyetequalto one ofthe ets fvalues sedforp = 1; the secondvariablewas initiallyetequal to arandom ermutationf thesevalues;and then hetwovariableswere rotatednd scaledto have correlationand variance . Simulations ith = .3, .5, .7, and .9n=100 n =50 n= 25

0.5 LX

00.5?o?0.0. . . .1.5- - 5 15 25 351.0

0 0 0 0 ~~~~~~p=1

o * O

-0 0 0

1.5 5 15 25 351.0 .

C) 0.5 . 00.0

*5 15 25 35

Numeratoregreesof FreedomFigure 16. Simulations.The figure hows the results of laboratorysimulationsnvestigatingheANOVA test for lobal linearity. n each----I.5 vetia sa is 5 ni cn

th resinfcne ee,an h oizna caei0dgeso

freeom thenumerator. Dhegraees farearnedom , h nme

of ndependentariables,ndn, he umberf bservations.he igureshows hat he istributionalpproximationsorkxceedinglyell.

8/8/2019 Loess 1988

13/16

Cleveland and Devlin: Locally Weighted Regression 607wererunfor achconfiguration,esultingn 24 simula-tions.Therewere hree esignonfigurationsor = 3. Onlythevalue ofn = 100wasused,and theconfigurationswere eneratedna mannernalogous o that or he asewith = 2 and n = 100. Simulationswith = .5, 7, and.9wererun or achconfiguration,esultingnnine im-ulations.Figure 6shows nformationbout hetest t the5%levelofsignificance.he valuesplotted n theverticalscales re5% minus he ctual ignificance,ndthehor-izontalcales re hedegreesf reedomf henumerator,that s, v Iv2. hepanels rearranged yp and n. Mostimportant,igure 6 shows hat heapproximating%significanceevel scloseto the rueevels neach ofthe60 simulations.he largestbsolute eviations 1.59%.Infact, he ituationsevenbetterhan hat, ecause helargest eparturesccurfor he argest egrees f free-dom,and thesevaluesare somewhatarger han hosetypicallysed npractice. orthe ases withessthan 0df, he argestbsolute eviations .94%. Similar esultsholdfor hedeviationst the 10% and2.5% levelsofsignificance.or he ormer,he argestbsolute eviationis 2.18%; for he atter, he argests 1.05%. Figure 6also shows hat hedeviation fthe true evelfromhenominalevel ncreasess p increases,s ndecreases, ras thedegrees ffreedomncrease.The good performancef the approximationsorANOVA occurs venthoughhenumeratorf theteststatistics not ndependentf thedenominator.he ap-proximationorks artly ecause hedependences notstrongndpartly ecauseunless orf isvery mall henumeratorscontributinghemost othe ariabilityf hestatistic.9.2 Laboratory imulations: onfidence Intervalsfor U2and g(x)The 60 simulationsescribednSection .1 were lsoused to investigateonfidencentervals orU2. For the90%confidenceevel, hemaximumbsolute eviationfthe ctual evelfromhenominalevelwas 50%; for he95% level hemaximum as 48%. Clearly,he pprox-imatingistributionserformedxcellentlynthese ases.The 27 simulationsor = 1 thatweredescribednSection .1 were also used to investigateonfidencen-tervals or (x) at twovalues fx: themean fthexiandthe argestfthexi.Forthe 0% confidencenterval,helargest bsolutedeviationwas .44% forthe mean and.65% for heextreme. or the95% interval,he argestabsolute eviation as 45% for hemean nd 65%fortheextreme. gain,theapproximationserformedx-cellently.9.3 OtherLaboratory imulationsIndistributionalpproximationsorANOVA,thediv-isors or he ums f quares, 1 or henumeratornd~2for he denominator,re notgenerallyhe sameas thedegrees ffreedomor he pproximatingdistribution,v2lIv2or henumeratornd b1'/2 for hedenominator.

Nevertheless,ne might ope hat 1 s close o v2 nd hat51s closeto 2, and then ake hedegrees ffreedomobe v1 nd 51. The 60simulationsescribedn Section .1were also used to investigatehis ne-momentpproxi-mation. or he10%, 5%, and2.5% levels f ignificance,themaximumbsolute eviationsre 3.84%, 2.68%, and1.62%,respectively.hecorrespondingalues or he wo-moment pproximationgiven n Sec. 9.1) are 2.18%,1.59%, nd 1.05%. The degradationnthe pproximationfor he ne-momentase s ust arge nough hatwe havecontinued ith he omewhat ore omplicatedwo-mo-ment pproximation.9.4 Field Simulations

As we stated arlier, data analyst an check heper-formancefthe pproximatingistributionnany ppli-cation y a field imulation.fthe approximatingistri-bution erformedoorly,he imulationistributionouldbe usedto make nferences.utwe havenotyet ncoun-tered napplicationn whichheresiduals ave sampledistributionhat s well pproximatedythenormal ndthe pproximatingistributionerformedoorly.We willillustratehe use oftwofield imulationsor woof theapplicationsnthis rticle.For the stimationftheozone surfacenSection , itis sensible o ask whetherheobserved urvaturenthefitted urface s significant,ecausethe estimatefthestandardrror fthe residualss a = .43,which s notsmall ompared ithhe ample tandardeviationf hecube root zoneconcentrations,hich s .89.To addresswhetheratawith hismuch oise an support ther hana globalfit,we carried ut ANOVA (describedn Sec.4.2), testinghe ocallyweighted egressionit gainstquadraticeast-squaresit. heF statistics 2.10 and theapproximatingistributionsF,with 9.2 nd89.0df.Thesignificanceevel s .011, o the urvatureshighlyignif-icant.We also ran a field imulation ith ,200replica-tions: he simulatedignificanceevelwas 010,which squite loseto the pproximatingevel.The result f the brasion-losspplicationnSectionwas a nonlineardditive it. incethenumberfobser-vations29)issmall,wemighteasonablyskwhetherhedatareally upport nonlinearegressionurface. huswetested he dditivemodel gainst lineareast-squaresfit: hesignificanceevelwas 00256,makinghenonlin-earity ighlyignificant.Ofcourse, he estneeds o beviewedwith ome aution, ecause hemodel rose fterseveralpassesof thefittingrocess nd becausef wasselectedromheMplot.)A field imulation as lsorun:Thesimulatedignificanceevelwas 00211,whichsquiteclose to the pproximatingevel.

10. DISCUSSION10.1 Locally WeightedRegressionforApplications

The methodologyntroducedere can be an integralpart f the nalysisn many egressiontudies. n fact,trepresents new approach,ompared ithwhat smost

8/8/2019 Loess 1988

14/16

608 Journal of the American Statistical Association, September 1988often racticedoday.Thismethodologyanpotentiallypenetrate regressiontudymostdeeplywhen he de-pendent ariable s a nonlinearunctionf the ndepen-dent ariables. oday, he womost ommonpproachesto fittingonlinearurfacesnapplicationsresearchingfor ransformationsfthevariableshatinearizehe ur-face ndfittingolynomialsf he ndependentariables.Thesemethods, owever, o not ead to a nearly ichenough lassof surfaceso modeladequatelyhewidevarietyf urfacesncounterednpractice. ut venwhenthe inal esult f regressiontudys aparametricurface,themethodologyanhelp ubstantiatehevalidityfthefit.10.2 Current estrictionso theMethodology

One currentestrictionf he pplicabilityfourmeth-odologysthe ssumptionfnormalityndconstantari-ance ofthe rrors. evertheless,uture orkmight elaxthis estriction.method or stimating(x) when he iareassumed nly obe symmetriclready xists: obustlocallyweighted egressionCleveland1979).What sneededforthisrobust rocedure, owever,s distribu-tional esultsimilarothosenSections and6. Smooth-ingtechniques ithout istributionalesults ften eavethe nalyst ith oo ittlemethodologyomake nformedinferences.Another urrentestrictions to studiesnwhich herelevance feach ndependentariable nexplaininghedependent ariable asalready eenascertained. o re-move his estriction,ork s needed o determine owto ncorporatento oessmethodologyroceduresor e-lecting subset fthe ndependentariables.10.3 TheCurse ofDimensionality

As thenumberf ndependentariables,, increases,a fixed umberfpoints, ,rapidlyecomesparse. hisis referredo as thecurse fdimensionality.omehavemistakenlyupposed hat he cursemakesmultivariatesmoothing-thats,smoothingith > 1-a method oavoid.Whatmust e avoided sallowing toremain ixedasp increases,ecause or ixed the quivalentumberofparametersf he it ncreasessp increases. fcourse,we mustmaintain ontrol ftheequivalent umber fparameters;his s doneby ncreasing.As long s wemaintainontrol nd donot llow he quivalentumberofparameterso become largefractionfn, we canexpectmultivariatemoothingo behave eliably.n thisarticlewe have successfullyarriedout multivariatesmoothingordata setswith wo nd threendependentvariables. owlkes 1986)demonstratedhat moothingwithmore han hreendependentariablessreasonableincertain ircumstances,venformoderatelyizeddatasets.Ofcourse, sp andf increase or ixed therewillbe a decrease n theamount f curvaturehat an beestimated ithouterious ias. This s not defectnthemethod ut a statementhat he morecomplicatedregressionurface ecomes, he arger must e to getgood stimatesf t.Exactly he ame onsiderationsb-tainwhateverhemethodfestimation.

10.4 WeightFunctions nd thePoorPerformanceof theUniformThegeneral orm f thetricube eightunction,ar-ticularlyhesmooth ontactwith at 1, enhances heperformancef ocallyweightedegression.nyreason-able function ith moothontact analsobeexpectedo

perform ell.Nevertheless,heuniformeightunction,with hediscontinuityt 1,performsoorly.A problemwith heuniforms that tsdiscontinuityresultsn ocalroughnessng(x) thatsalmost lways oiseandnot ignal. his s awell-knownhenomenonndigitalfilteringnd pectrumnalysis,hat oxcarwindows aveFourierransformsithide obes hat all ff lowly s afunctionffrequencynd thuspass unacceptablyargeamounts fhigh-frequencyoise Bloomfield 976). Asecond roblem ith heuniformeightunctions thatit eads to esssatisfactoryistributionalpproximations,because for heuniform,he eigenvaluesfL-which,again, re related otheFourier ransformftheweightfunction-donot end hemselvess well o the pprox-imationss toa continuouseightunctionuch stricube(Devlin1986).Wementionheweight-functionssue, npart, ecauseasymptoticesults or onparametricegressionhow hattheoverall orm ftheweight unctionoesnothave nappreciableffect ith espectomean quaredrrore.g.,PriestleyndChao 1972). This,however,houldnotbeinterpretedomean hat heformf theweight unctiondoesnotmatternallrespects.10.5 OtherMethods

Anotherpproach o smoothingdependent ariableas a functionf two or more ndependentariables sprojection ursuit, n iterative rocedure Friedman andStuetzle981).Ateach tage f he terations,i ssmoothedas a functionf a linear ombinationf the ndependentvariables. he inear ombinations chosen ogive max-imum eductionn he esidualum f quares. he mootheris similar o univariateocallyweightedegression,utwithmodificationsodecrease he omputationime ndwith method or hoosinghe mount f moothing.hemultivariatemoothingntroduced ere s attractivee-cause of its simplicity:or a particular, g(x) has astraightforwardefinitionnd is simply linear ombi-nation f theyi, o thestatisticalropertiesreeasytofathom. hissimplicityeadsto much fthe additionalmethodologyn this rticle. he fullprojection-pursuitalgorithmesultsn considerablyore omplicatedunc-tion f theyi,because he inear ombinationsf the n-dependentariables re chosen ominimizeheresidualsumofsquares.Consequently,lmost othings knownabout tsdistributionalropertiesHuber1985). n ad-dition,ull rojectionursuitlsohas ts estrictedomainof applicability;ot all regressionurfaces an be wellapproximatedy moderate umber f moothunctionsof inearombinationsfthe ndependentariablesHuber1985).Locallyweightedegressionalls nto classofregres-sion rocedureshat ome allnonparametricegression.

8/8/2019 Loess 1988

15/16

Cleveland and Devlin:Locally WeightedRegression 609Stone 1977),Collomb1981),WegmanndWright1983),andTitterington1985)reviewedther rocedures. anystudies fnonparametricegressionocused n asymptoticpropertiesuch as consistency,ormality,nd ratesofconvergencee.g., Benedetti 977;Devlin 1986;Hardleand Gasser 1984; Stone 1977, 1982; Wahba 1979). Forexample, tone 1977),using legantrguments,howedthe asymptoticonsistencyf a wideclass of nonpara-metricstimates.One well-knownonparametricegressionroceduressmoothingplines Henderson 924; Reinsch 967; Sil-verman 985;Wahba1978;Whittaker923).Splines avean attractiveroperty:hey rethe olutiono an intui-tively ppealingmathematicalriterion. nother ttrac-tivepropertys that hey ave a Bayesian nterpretation(Wahba 1978;Whittaker923). Weerahandind Zidek(1985) provided Bayesian nterpretationorunivariatelocallyweighted egression ith particular eightunc-tion.]Butsplines lso have omeunattractiveroperties.First, hey ptimize global riterionnd renot enerallylocal. Although,s Silverman1985)pointed ut,whenn is large and the amountof smoothings neither argenor mall, plinemethods ehave, o a good approxima-tion, s smoothingy ocal fittingith weightunctionwith xponentialecay; hus plinesrenearlyocal n hiscase.] A second unattractiveropertys that becausesplines rise as the result f an optimization,t canbedifficulto determineowthey perate n thedata. Incontrast, he operational haracteristicsf local-fittingproceduresre easier o fathom ecause heyre defineddirectly.or xample, ecause f tsdefinition,ne knowsthat he ocallyweighted egressionstimate, (x), is de-terminedy 100f%of thedata at eachx, for nyn andfor ny onfigurationfthexi exceptwhen ies nthexileavemore han 00f%ofthedata t a particularoint).It sconsiderably ore ifficulto determinehe ffectivebandwidthfa spline stimatet x (Silverman 984). nmany ases this s onlypossible y numericallyorkingoutthe coefficientsf the inear ombinationfyithatmakeup the stimate.Themost erious roblem ithplinesscomputational.Althoughfast0(n) algorithmsxistforone independentvariableSilverman 985;Whittaker923),fittingthinplate" plines o two r morendependentariablessan0(n3) computation Wahba 1984). The expected compu-tation ime f a loessestimate t a single alueofx is0(n). Fora fixed alueoff (i.e., a fixed umber fdegreesoffreedomfthefit), henumber fpointstwhich neneeds ocompute to characterizet for ractical ppli-cations s fixed: y using lendingunctionsndk - dtrees, ocal-fittingomputationsnpracticean bekept o0(n) time Cleveland et al. 1988) and are thus feasibleeven n computingnvironmentshatdo not havefast,powerfulrocessors.ote hat histrategys not vailableinspline moothing,ecause ne cannot etgat a singlevalue fxwithoutarryingut he ull ptimization.husanothertrategyhathas been employedor pliness tosolve n altered ptimizationhat equiresesscomputa-tion nd thatyields solution lose to the originalnewhen is large Wahba 984). ut he omputingsstill

substantialndcomplex,ndmany uestionsemainSil-verman 985).Two popularmethodsor hoosinghe smoothinga-rametern spline-fittingre cross-validationStone1974)andgeneralizedross-validationCraven ndWahba 979).Unfortunately,sers f hesemethodsenerallyocus x-clusivelyn themean quared rror,which nSectionwe criticizeds too limiting.ne exception,owever,sthework yClark 1980).Ofcourse, necoulduse cross-validationr generalizedross-validationn place of theM statisticochoose he mount f moothingor ocallyweightedegression,r onecoulduse theM statisticorsplines. hat s,thesemethodsor hoosingheamountof moothingrenotdependentn themethod f mooth-ing.[Receivedeptember986.Revised ecember987.]

REFERENCESBelsley, . A., Kuh,E., and Welsch, . E. (1980),Regressioniag-nostics, ewYork:JohnWiley.Benedetti, . K. (1977), On theNonparametricstimationfRegres-sionFunctions,"ournal f theRoyal tatisticalociety, er.B, 39,248-253.Bloomfield,. 1976),FouriernalysisfTime eries: n ntroduction,NewYork:JohnWiley.Breiman, ., and Friedman,. H. (1985), Estimatingptimal rans-formationsor orrelationndRegression,"ournalf heAmericanStatisticalssociation,0,580-598.Brinkman,. D. (1981), Ethanol uel-A Single-Cylindernginetudyof Efficiencynd Exhaust missions,"AE Transactions,0, No.810345, 410-1424.Bruntz,. M., Cleveland,W. S., Kleiner, ., andWarner, .L. (1974),"The DependencefAmbientzone nSolarRadiation, ind, em-perature,nd Mixing eight,"n Symposiumn Atmosphericiffu-sion ndAirPollution,oston:American eteorologicalociety, p.

125-128.Buta,R. (1987), The StructurendDynamicsfRinged alaxies,II:Surface hotometrynd Kinematicsf theRingedNonbarredpiralNGC7531,"TheAstrophysicalournalupplementeries, 4,1-37.Chambers,.M., Cleveland,W.S., Kleiner, ., andTukey,. A. (1983),Graphical ethodsorDataAnalysis, onterey,A: Wadsworth.Clark,R. M. (1980),"Calibration,ross-Validation,nd Carbon-14,II," Journalf theRoyal tatisticalociety,er. A, 143,177-194.Cleveland,W. S. (1979),"RobustLocallyWeighted egressionndSmoothingcatterplots,"ournalftheAmericantatisticalssoci-ation, 4,829-836.Cleveland,W.S., Devlin, . J., ndGrosse, . (1988), RegressionyLocalFitting: ethods, roperties,ndComputationallgorithms,"JournalfEconometrics,7,87-114.Collomb, . (1981), Estimation on-Parametriquee la Regression:RevueBibliographique,"nternationaltatisticaleview, 9,75-93.Cook,R. D., andWeisberg,. (1982),nfluencendResidualsnRegres-sion,NewYork:Chapman Hall.Craven, .,andWahba,G. (1979), Smoothingoisy ataWithplineFunctions: stimatinghe CorrectDegree of Smoothingy theMethod fGeneralizedross-Validation,"umerischeathematik,31, 377-403.Daniel,C., andWood, . (1971),FittingquationsoData,NewYork:JohnWiley.Davies,0. L. (ed.) (1957), tatisticalethodsn Researchnd Produc-tion3rd d.), NewYork:Hafner ress.Devlin, . J. 1986), Locally-Weightedultiple egression:tatisticalPropertiesnd tsUsetoTest or inearity,"echnical emorandum,Bell Communicationsesearch, iscataway,J.Fowlkes,. B. (1986), SomeDiagnosticsor inary ogistic egressionVia Smoothing"with iscussion),n Proceedingsfthe tatisticalComputingection, mericantatisticalssociation,p.54-64.Friedman,.H., and Stuetzle,W. 1981), Projectionursuit egres-sion,"JournalftheAmericantatisticalssociation,6,817-823.Hardle,W.,and Gasser, . (1984), RobustNon-ParametricunctionFitting," ournalftheRoyal tatisticalociety,er. B, 46,42-51.Hastie, . J., ndTibshirani,.J. 1986), Generalizeddditive odels,"Statisticalcience, ,297-318.

8/8/2019 Loess 1988

16/16

610 Journal f the American StatisticalAssociation, eptember 1988Henderson, . (1924), A New Method f Graduation," ransactionsoftheActuarialociety fAmerica, 5, 29-40.Huber, . 1985), Projectionursuit"with iscussion),heAnnals fStatistics,3,435-525.Kendall,M., and tuart, . S. (1977),TheAdvanced heoryf tatistics(Vol. 1, 4th d.), NewYork:Macmillan.Landwehr, . M. (1983), Using artial esidual lots o DetectNon-linearity,"echnical emorandum,T&T Bell Laboratories, urray

Hill,NJ.Larsen,W.A., andMcCleary,. J. 1972), TheUse ofPartial esidualPlots nRegression nalysis," echnometrics,4,781-790.Macaulay,. R. (1981),The moothingf Time eries, ewYork:Na-tional ureau fEconomic esearch.Mallows, . L. (1966), Choosing SubsetRegression,"npublishedpaperpresentedt the annualmeetingfthe AmericantatisticalAssociation,os Angeles.(1973), SomeCommentsnCp,"Technometrics,5,661-675.Priestley, . B., and Chao,M. T. (1972), Non-parametricunctionFitting," ournal f theRoyal Statisticalociety, er. B, 34, 385-392.Reinsch, . (1967), Smoothingy pline unctions,"umerischeath-ematik, 0, 177-183.Rodriguez, . N. (1985), A Comparisonf theACE and MORALSAlgorithmsn nApplicationoEngine xhaustmissions odeling,"inComputercience nd Statistics:roceedingsf the ixteenthym-posium nthe nterface,d. L. Billard, ewYork:North-Holland,pp. 159-167.Silverman,. W.(1984), "Spline moothing:heEquivalent ariableKernelMethod," heAnnals fStatistics,2,898-916.(1985), "Some Aspects f theSpline moothing pproachoNon-parametricegressionurve itting"with iscussion),ournaloftheRoyal tatisticalociety,er.B, 47,1-52.

Stone, . J. 1977), Consistentonparametricegression,"heAnnalsofStatistics,, 595-620.(1982), "OptimalRatesof Convergence or NonparametricRegression,"heAnnals fStatistics,0, 1040-1053.Stone,M. (1974), Cross-Validatoryhoice nd Assessmentf Statis-ticalPredictions"with iscussion), ournal f theRoyalStatisticalSociety,er.B, 36, 111-147.Titterington,. M. (1985), "Common tructurefSmoothingech-niques n Statistics,"nternationaltatisticaleview, 3,141-170.Wahba, . (1978), Improperriors,pline moothing,nd he roblemofGuarding gainstModel Errors n Regression,"ournal f theRoyal tatisticalociety,er. B, 40, 364-372.(1979), Convergence atesof ThinPlate'SmoothingplinesWhen heData AreNoisy,"nSmoothingechniquesorCurve s-timation,ds.T. Gasser ndM. Rosenblatt,erlin: pringer-Verlag,pp. 233-245.(1984), Cross-ValidatedplineMethods or heEstimationfMultivariateunctionsromData on Functionals,"n Statistics:nAppraisal, ds. H. A. Davidand H. T. David, Ames: owa StateUniversityress, p. 205-235.Watson, . S. (1964), Smooth egression nalysis,"ankhya,er.A,26, 359-372.Weerahandi,., and Zidek,J. V. (1985), Smoothingocally moothProcesses yBayesianNonparametricethods," echnical eport26, UniversityfBritish olumbia, ept.ofStatistics.Wegman,. J., ndWright,. W. 1983), SplinesnStatistics,"ournaloftheAmericantatisticalssociation,8, 351-365.Whittaker,. T. (1923), On a NewMethod f Graduation,"n Pro-ceedingsf he dinburgh athematicalocietyVol.41), pp.63-75.Young, .W., DeLeeuw,J., ndTakane,Y. (1976), Regression ithQualitativendQuantitativeariables: n Alternatingeast-SquaresMethodWith ptimal calingeatures," sychometrika,1,505-529.