ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf ·...

35

Transcript of ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf ·...

Page 1: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

����������� ��������������������������������� ���"!#���$���%���& �%��� '(�)�*�������,+-�.����' �/�(�102������������� ����3��.����' �/�(�

!4���(5���' �����6���

7�8:9<;>=@?6ACB�DFEG=IHJ8�;*KFLNMOBP8�QOQ)R�DFS�A TVUOQWL�MYX[Z]\OQ)E^DY_CA�QO`[A�;6LNMYabA�;<QY8];<Kdc/=@?e9>Z[;>;<=@fg(hji�k�lNm�nporq:s*m*tutuvxwyl�z�m<or{�v|s<m*t�}Vm*tum*k�ors�~J�Wk��F�y���������Wk��yt��$}Vm<tumek�ors�~F�Y�e����z]s�~]�[�x�

� orvx{�m��y}Pme� ��k�w[�O�[�����N�N���]�<�j���N�N���O��z]�� �bw[v|{Nm<ortuv��rm��[m/��k�m*wF�y�$tu [�|k�w¡k��[m/�[mb�Ik�npk�v�¢��P�*£N�N�N¤���k�m*w¥�m*�[m<¢ ��¦[o�k�w[s*m

§�¨p©*ª*«6¬¡­�ªN® h¯w° ¡k6�u�rm<orw°orm*s*q�lNwyv��rvxqNwY��t±��k��rvxt±�rvxsek���i�q��]m*�xvxwyl[��q�o(orm<l�orm*tutuvxqNwY��r~[m�k�i�q��yw:��q�²p�[k���kCvxt��r~[mVi�q�t±��s<orv��rvxsek��O²³k�s<�rq�oPk6´Om*s<�rvxwyl/�r~ymb Gm>ou²@q�ori�k�wys*m�µh¶²/�r~[mdk�i�qN�[w:��q�²C�yk���k·k�w¡�°s*q�i� y�[��k6�rvxqNw¡k��Vorm*tuq��[ors*m<t�k�ormdwym*k�o�vxw]¸¡wyv��rmN�i�k�w�¹Jk��xlNq�orvx�r~[i�tbº�vx�x�$ ]orq6{Nk��y��¹¥s*qNw�{�m>orlNmC�rq��r~ym/qN ]�rvxi�k��ptuqN�x�]�rv|q�wFµO»¼~[m*w�r~[v|t�vxtwyq����r~[m�sek�tumN��q�wymP~¡k�t$�rq%vxw:�uorq��[�ys<m�orm<lN�y�|k6orv|�<m<ort�k�wy��k��¶ [orvxq�orvy�]w[qeº��x�me�]lNm��rq�tu�[ y y�xm*i�m<w��W�r~ym�ke{�k�vx�|k��y�xmP�yk6��k�vxw½q�o��]m<o��rq��Gq:qNt±�$�r~ym� Gm>ou²@q�ori�k�wys*m�µh¯w]{�k�orv|k�wys*m/¾³q�o���w[q6º�w��[m< Gm*w¡�]m*wys*me¿pº�v��r~�orm*tu Gm*s<���rq%�uo�k�w[t±²@q�ori�k��rvxqNw�q�²Y�r~ymvxwy [�[�pvxt$kP²IormeÀ:�ym*w:�pq:s<s*�[ouorm*w[s*m�q�²ytu�ys�~½k6�¯ ]orvxq�orv]�]w[qeº��|m*�[lNm�µ�h¯wC�r~[v|tps�~¡k� [�rm<oe�ºm�v|w:�uorq��]�ys*mb�r~[m�s*q�wys*m* ]��q�²p��k�wylNm<w���{Nm*s<�rq�ort*�]º�~yvxs�~(s*q�i� ¡k�s<�r��¹ orm< [orm*tum*w:��r~[m�m*tutum*wys<m�q�²p�r~ym<tum��uo�k�wyt±²Áq�ori�k6�rv|q�w�vxw�{Nk6orvIk�wys*m<t*�Gk�wy� �¯º�q½s*�|k�tutum*tbq�²$k��xlNq��orv��r~yi�t*�W±�^k�wyl�m*w:���]v|t±��k�wys*m<Ã%k�w¡��Âj�^k�w[lNm*w:�� [orq� ¡k�lNk��rvxqNw[Ã[�:º�~[v|s�~�i�k��Nm��ytumq�²p�r~ym*tum�vxw�{�k�orv|k�w[s*m*t��rq�vxi� [orq6{�m� Gm>ou²@q�ori�k�wys*m�µ

Ä ÅyÆPÇGÈOÉ�ÊCË/ÌWÇOÍrÉPÆ

7�8:9�9<A�;<Q·Î�A�?�Z]Ï[QO=|9<=IZ[Q)=ÁÐCZ]QYA Z�Ñ�9<\OA�HJ8]=IQ)9*8]Ð>`GÐ%Z�Ñ�ÒY=IZ[ÓIZ[Ï]=@?�8�Ó$=IQGÑÔZ[;>HJ8�9>=ÁZ]Q·ÕO;<ZG?6ANÐ>Ð�Ö=IQOÏ�Ð�×GÐ�9>A�HJÐ�M[8]QYKJ8�HJ8:ØrZ[;V?*\Y8]ÓIÓÁA�QOÏ[A�Z�Ñ�?6Z[H�ÕOUG9>A�;VÐ<?6=ÁA�QY?�A]DyÙ�\OA/ÕO;<Z]ÒYÓIA�HÚZ]Ñ^ÕF8:9�9<A�;<Q;>AN?6Z]Ï[QO=I9>=ÁZ]Q¥=@Ðb9>Z�?6Ó@8]Ð<Ð�=IÑÔ×JZ]ÒOØrA�?e9*Ð�=IQy9<Z(?�8:9>A�Ï]Z[;>=ÁA�Ð�M[Ï[=IÛ[A�QJ9<\Y8:9�Z[ÒGØrA�?69<Ð�=ÁQÜ8(ÕY8�;>9>=@?6UOÖÓÁ8];C?�8�9>A�Ï]Z];<×dHJ8�×ÜÛ�8];>×ÞÝ�=@KGA�ÓÁ×]MWÝ�\O=ÁÓIA�Z]ÒGØrAN?e9<Ð/=IQßKG=|à^A�;<A�Qy9½?�8:9<A�Ï[Z];<=IANÐ�H�8�×ÞÒWA Û]A�;>×Ð�=ÁH�=IÓ@8�;ND:R�9rסÕO=@?�8]ÓyA�áO8�H�ÕOÓÁAV=@Ð\Y8�QYKOÝ�;>=I9�9<A�Q�KG=IÏ[=|9�;<A�?6Z[Ï]QO=I9>=ÁZ]Q$D:TV\Y8];<8[?e9<A�;*Ð�M�9rסÕO=Á?�8�ÓÁÓI×;>A�ÕO;>ANÐ�A�Qy9>A�K(8]Ð�âOáGA�K¡ÖuÐ�=Áã�A�=ÁHJ8�Ï[A�Ð%ä¯Ð>8�×Üå�æ½Ò¡×dåNæ½ÕO=|áGA�ÓÁÐ*çeM�H�UYÐ�9PÒFAC?6Ó@8]Ð<Ð�=IâYA�K(=ÁQy9>Z�Z]QYAZ�Ñpå�è%?�8:9<A�Ï]Z[;>=ÁA�Ð^UFÐ�=ÁQOÏ/8 é�êIë:ì<ìeí îbé<ë�ï¯í¶ð�ñ%ò*óGñWé�ï¯í¶ð�ñYD�abUO=IÓ@KG=ÁQOÏ/Ð>UY?*\�8%?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q�ÑÔUOQY?eÖ9>=ÁZ]Q =@Ð�8/HJ8�ØrZ];�9>A�?*\YQOZ]ÓÁZ]Ï[=Á?�8�Ó[?*\F8�ÓÁÓIA�QOÏ]A[MN8[Ð�Û:8];>=@8�ÒO=ÁÓÁ=|9<=IANÐ�8]H(Z[QOÏ/Z[ÒGØrA�?69<ÐZ�ÑY9>\OA�Ð<8�H�A?6Ó@8]Ð<нH UYÐ�9�ÒFAdA�ÓÁ=IH�=ÁQY8:9<A�KpM�Ý�\Y=IÓÁAdKG=IàWA�;>A�QY?6ANнÒFA�9rÝVA�A�Q°Z]ÒOØrA�?e9*Ð�Z�Ñ�KG=IàWA�;>A�Qy9 ?6Ó@8]Ð<Ð�ANÐH UYÐ�9ÒWAb=ÁKGA�Qy9>=IâYA�KpD�E¡=ÁQY?�AV?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q½ÑÔUOQY?e9<=IZ[QYÐ�ÑÔZ];H�Z[Ð�9;<A�8]Ó]ÕY8�9�9>A�;>Q ;>AN?6Z]Ï[QO=I9>=ÁZ]Q9<8]Ð>`GÐ 8];>AJ9<Z¡Zß?6Z[H�ÕOÓI=@?�8�9>ANKô9>ZõÒWAÜÐ>סQy9>\OANÐ�=Áã�ANK÷ö�Ò¡×°\Y8�QYKYøOM�8]UG9>Z[HJ8:9>=@?�9<A�?*\OQO=@ùyUOA�ÐH UYÐ�9(ÒWA)KGA�Û¡=@Ð�ANK¼9<Zô?�Z]QYÐ�9>;<UY?e9(9>\YA�?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q¼ÑÔUOQF?e9>=ÁZ]Q.ÑÔ;<Z]Hú8õÐ>A69JZ]Ñ/ÓÁ8]ÒFA�ÓIANKA6áO8�H�ÕOÓÁA�Ð¥äÔ9>\OAd9>;*8�=ÁQO=ÁQOÏ·Ð�A�9*ç6DÙ�\OANÐ�A�9<A�?*\OQY=ÁùyUOANÐ ?�8]Q¼ÒFAÜKG=IÛ¡=@KGA�K°=ÁQy9>Z)9rÝbZ·?�8]H�ÕYÐ�M8]?�?�Z];*KG=ÁQOÏ�9>Z(9<\OACQ¡UOH�ÒFA�;�Z�Ñ$ÕY8];<8]H�A69>A�;<ÐP9>\YA�×¥8�;<A/;<A�ùyUO=Á;<=IQOÏFû]9<\OA�ö�H�A�H�Z[;>×�ÒY8[Ð�ANKOø8�ÓÁÏ]Z];<=I9>\OHJÐ�M�Ý�\O=@?*\�UYÐ>Ad8�ä¯?6Z]H�ÕY8[?e9eç½;>A�ÕO;<A�Ð>A�Qy9<8�9>=ÁZ]Q°Z�Ñ�9<\OA¥9<;<8]=IQO=ÁQOÏ·Ð>A69NM8]QYK°9>\YAö�ÓÁA�8�;<QO=ÁQOÏ[ø/9>AN?*\OQO=@ùyUOA�Ð�M�Ý�\O=@?*\J;>ANùyUO=I;<A�8]K�ØrUFÐr9<H(A�Qy9<Ð�Z]Ñ^8�?�Z]H�ÕY8];<8�9>=ÁÛ]A�ÓÁ×�Ð>H�8]ÓIÓYQyUYH�ÖÒFA�;�Z�Ñ�ÕF8�;*8�H�A69<A�;*ÐCä¯KGUO;<=IQOÏ(9<;<8]=IQO=ÁQOÏ¡ç�9<Z�?�Z]H�ÕOUG9<A�9>\OA�?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[QJÑÔUOQY?69>=ÁZ]Q�DFÙ�\O=ÁÐKG=ÁÐ�9>=ÁQY?69>=ÁZ]Q�?�8�Q�ÒWAÜÐ>Z]H�A�Ý�\F8:9�8];>ÒO=I9>;*8�;<×·ÒWA�?�8�UYÐ>AÜÐ>Z]H�Ad?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q¼8�ÓÁÏ]Z];<=I9>\OHJÐ�M

Page 2: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

ÑÔZ];�=IQYÐ�9<8]QY?6A�S�A�8];>QY=IQOÏdÛ]A�?69>Z[;CùyUY8�Qy9<=IãN8:9>=ÁZ]Q)Z[;CS[c���MF8];>A�\¡×¡ÒO;<=ÁKYÐ�D^Ù�\OA�KO=ÁÐ�9>=ÁQY?e9<=IZ[QÐ�A�;>Û[A�Ð�Z]UO;�ÕOUO;<ÕWZ[Ð>A]MQOZ]QOA�9>\OA�ÓIANÐ>Ð�M�ÒFAN?�8]UYÐ�AÞH�A�H�Z];<×¼ÒY8[Ð�ANK�8�ÓÁÏ]Z[;>=I9>\YH�Ð�Z�ѳ9<A�Q ;>A�ÓI×Z]Q�8)H�A69<;>=@?¥Ý�\Y=Á?*\�?�8�Q¼ÒWAdH�ZGKG=|âFA�K°9<Z·=IQF?6Z];<ÕWZ];*8:9>A�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q°=ÁQyÛ:8];>=@8�QY?�A�Ð�MÝ�\O=IÓÁA�ÓIAN8�;<QO=IQYÏ ÒY8[Ð�ANK¥8�ÓÁÏ]Z[;>=I9>\OHJÐV?6Z]QFÐ�=@Ðr9bZ�Ñ�Ð>A�ÓÁA�?69>=ÁQOÏ�8(?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q�ÑÔUOQY?e9<=IZ[Q�M¡9>\YAKGA�;<=IÛ:8�9>=ÁÛ]A�нZ]Ñ�Ý�\O=Á?*\¼?�8�Q°ÒFA¥?�Z]QYÐ�9>;*8�=ÁQOANKõ9<Z);>A��YA�?69�9>\OAÜÐ>8]H(A(9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Qô=IQGÖÛ�8];>=@8�QF?6A�Ð�DpÙ�\OA�9rÝbZÞH�A�9>\OZGKOнÑÔZ[;�=ÁQY?6Z[;>ÕWZ];*8:9<=IQOÏÞ=ÁQyÛ:8];>=@8�QY?�A�н8�;<AJKG=IàWA�;>A�Q[9�A�QOZ]UYÏ]\9>Z½ØrUYÐ�9>=IÑÔ×¥9rÝbZ�=IQYKOA�ÕWA�QYKGA�Qy9/Ð�AN?e9>=ÁZ]QFÐ�D

����� ���� �������������������� ��� �!�"����Ù$Zd?�Z]H�ÕOUG9<A�9<\OA�?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[QÞÑÔUOQY?e9<=IZ[Q�MWH�8]Q¡×ÜÕO;*8]?69>=@?�8]Ó�ÕY8:9>9>A�;>Q·;<A�?�Z]Ï[QO=|9<=IZ[Q)Ð�×GÐ�Ö9>A�HJÐ�M[8]QYK�Ð�A�Û]A�;*8�ÓGÒY=IZ[ÓIZ[Ï]=@?�8�ÓGH�ZGKGA�ÓÁÐ�M[Ð>=IH�ÕOÓÁ×�Ðr9<Z];<A�8]ÓIÓY9>\OA/A6áO8�H�ÕOÓÁA�Ð�M:9>Z[Ï]A�9>\OA�;�Ý�=|9<\9>\OA�=I;½Ó@8�ÒWA�Ó@Ð�M^=ÁQß8ÜH(A�H�Z];<×]D$#P8]?*\õ=IQY?�Z]H�=ÁQOÏÜÕF8:9�9<A�;<Qß?�8�Q)9<\OA�QôÒFA�?6Z[H�ÕY8�;<A�K�9<ZÞ8]ÓIÓ9>\OA·Ð�9>Z[;>ANK�ÕO;<Z�9<Z�9rסÕWA�Ð�M�8]QYK�9>\OA)Ó@8�ÒWA�Ó/8[Ð>Ð>ZG?6=@8:9<A�K�Ý�=I9>\ 9<\OA)ÕO;<Z�9>Z]9rסÕFAÞ9>\F8:9¥ÒWA�Ð�9H�8�9<?*\OANÐ�9<\OA/=ÁQOÕOUO9�?�8�Q¥ÒWA/Z[UG9>ÕOUO9�Ö±A�KpD¡Ù�\OAC8]ÒFZ:Û[A%H�A69<\OZGK¥=@Ð�9<\OACÐ>=IH�ÕOÓÁA�Ð�9VA�áO8�H�ÕOÓÁAZ�Ñ9>\OA&%&'(%Jð*),+*-/.*ë:ì('10½H�ZGKGA�ÓÁÐ�D�2)A�H(Z[;>×yÖ±ÒY8]Ð>A�KdH�Z¡KOA�Ó@Ð�;>ANùyUO=I;<AC9>\O;<A�A½9<\O=IQYÏ[Ð�ûY830:í@ì,-ï±ë�ñWé1'4%5'*ë:ì*ó6)7'�9<ZÜ?6Z[H�ÕY8�;<A�=ÁQOÕOUG9*Ð/9<ZdÕO;<Z�9<Z�9rסÕWA�Ð�MW8]Q.ð:óGï98WóGïpò*óGñWé�ïjí¶ð:ñ¥9<ZdÕO;<ZGKGUY?�A8�QôZ]UG9<ÕOUG9�Ò¡×õ?6Z[H ÒO=ÁQO=ÁQOÏd9<\OA�ÓÁ8]ÒFA�ÓÁнZ]ÑP9<\OA�ÕO;>Z]9>Z]9r×yÕWA�Ð�Mp8�QYKô8õì*ïuð*)<ë�:;'�ì�é7<�'(%&'�9<ZÒOUO=ÁÓÁKÜ9>\YA�Ð�A�9%Z�Ñ�ÕY;>Z]9>Z�9rסÕWA�Ð�DR%ÓÁÓ%9>\Y;>A�Aõ8]Ð>ÕFAN?e9*ÐJ\Y8�Û]A)ÒWA�A�Q 8�ÒOUYQYKO8�Qy9<ÓI×�9>;<A�8:9<A�K =ÁQ 9>\OAõÓI=I9>A�;<8�9>UO;<A]D>=CUO9>ÕOUG9

ÑÔUOQY?e9<=IZ[QYÐ�;*8�QOÏ[A/ÑÔ;<Z]H Ð>=ÁH(ÕYÓI×¥Û[Z�9<=IQOÏ�9>\YA�Ó@8�ÒWA�Ó@Ð�8]Ð<Ð>Z¡?�=Á8�9>ANK¥Ý�=I9>\Ü9>\OA@?Ü?6ÓÁZ[Ð>A�Ð�9bÕY;>Z]Ö9>Z�9rסÕWA�Ð�ä�A�ÖCB%AN8�;<A�Ð�9DB/A�=ÁÏ]\¡ÒFZ[;<Ð*çeMF9>Z�?6Z[H�ÕOUG9>=ÁQOÏ�8ÜÐ<?6Z[;>A ÑÔZ];½A�8[?*\õ?6Ó@8]Ð<Ð�8]н8dÓÁ=ÁQOA�8];?6Z]H�ÒO=ÁQY8:9<=IZ[QJZ�Ñp9>\YACKG=@Ðr9*8�QY?�A�Ð�9>Z�8]ÓIÓF9>\YA/ÕO;<Z�9<Z�9rסÕFANÐ�M[UYÐ>=IQOÏ�âOáGA�KFE GGå�HFZ];VÓIAN8�;<QOA�K3E I�H?6Z¡A(J¥?6=ÁA�Qy9*Ð�D¡Ey9>Z[;<8]Ï]A�Ð>?*\YA�H�A�ÐPÛ�8];>× ÑÔ;>Z[HÚÐ�9>Z[;>=ÁQOϽ9>\YA/A�Qy9<=I;<A�9>;*8�=ÁQO=ÁQOÏ Ð>A69�M]9>Z�ÕY=Á?*`¡=ÁQOÏ8�ÕOÕO;<Z]ÕY;>=@8:9<A/Ð�UYÒYÐ�A�9<ÐbZ�Ñ�=|9�ä¶Ð>A�A5E K*H±M¡?*\F8�ÕG9<A�;�æOMyÑÔZ];�8 Ð>UO;<Û]A�×Gç�9>Z(ÓIAN8�;<QO=ÁQOÏ�8]ÓIÏ[Z];<=|9<\OHJÐÐ�UY?*\�8]Ð�S$A�8];>QO=ÁQOÏ�c�A�?69>Z];L�CUF8�Qy9>=Áã�8�9>=ÁZ]Q�ä¶S[c��½çLEIå�M�HY8�QYK�Ï];*8]KO=IA�Q[9�KGANÐ>?�A�Qy9�D�_C=@Ðr9*8�QY?�AH(AN8]Ð>UO;<A�Ð�?�8�QJÒWA/8]ÐVÐ�=ÁH�ÕOÓIAC8]Ð�9<\OAN#PUY?6ÓÁ=ÁKOA�8�Q¥KO=ÁÐ�9<8]QY?6A[M[8]Ð<Ð>UOH�=IQOϽ9<\OA%ÕY8�9�9<A�;<QYÐP8]QYKÕO;>Z]9>Z]9r×yÕWA�Ð�8];>AÞ;>A�ÕO;<A�Ð>A�Qy9>ANK�8]Ð(Û[A�?e9<Z];*Ð�M�Z[;(H�Z[;>A�?6Z[H(ÕYÓIA�á�8]Ð�=ÁQ�9<\OA�Ï]A�QYA�;*8�ÓÁ=Iã�A�Kù[UF8]KG;*8:9<=Á?�H�A69<;>=@?5E|åNè*H�Z];�=ÁQÞA�Ó@8]Ð�9>=@?½H�8�9<?*\O=ÁQOÏ�H�A69<\OZGKOÐOEIåPI�H±D

Pattern tobe classified Prototype A Prototype B

QSRUT ®*V¡® �Ps*s*q�o��[vxwyl��rq��r~ymP���ys*�xv|�[m*k�w �[vxt±��k�w[s*m��r~ym� ¡k6�u�rm<orw½�rq��GmVs<�Ik�tutuv�¸¡me��vxti�q�ormPtuvxi�vx�|k�o�rq� ]orq��rq��¯¹] Gm�Vµ��)�Gm<�u�rm>op�[vxt±��k�wys*m$i�mek�tu�]orm$ºqN�[�I��¸¡wy�b�r~¡k6�^ [orq��rq��¯¹[ Gm$�)vxt^s*�xqNtum>op�Gm*sek��ytumv�� �]vx´Om>ort½i�k�vxwy��¹·�:¹·k¥orq���k��rvxq�w°k�wy�ôkJ�r~yvxs��]w[m*tut��uo�k�wyt±²@q�ori�k��rvxqNwY���¯º�qd�uo�k�wyt±²Áq�ori�k6�rv|q�wytº�~[vxs>~�tu~[qN�[�I� �xmeke{�m��r~ym�s*�|k�tutuvx¸ysek��rvxq�wJv|w�{�k�orv|k�w:�eµ

RÚÐ>=IH�ÕOÓÁA�ÒOUG9�=ÁQOA(J¥?6=ÁA�Qy9�ÕY8�9�9>A�;>Qô;<A�?6Z[Ï]QO=I9>=ÁZ]QõH(A�9>\OZGKß=ÁÐC9<Z�?�Z]H�ÒO=IQYA(8ÞÐ�=ÁH�ÕOÓÁAKG=ÁÐ�9<8]QY?6ACH�A�8]Ð>UO;<A]MyÐ�UY?*\Ü8]Ð>#�UY?�ÓI=@KGA�8]Q¥KG=@Ðr9*8�QY?�A/ÒWA69rÝbA�A�QdÛ]A�?69>Z[;<Ð�M]Ý�=I9>\Þ8�Û]A�;>×(ÓÁ8];>Ï[A

Page 3: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ð�A�9�Z]ÑFÕO;<Z�9<Z�9rסÕWA�Ð�D�Ù�\Y=ÁÐ�H�A69<\OZGK(=ÁÐ�=ÁQOA(J¥?�=IA�Q[9�ÒWA�?�8�UYÐ>A�8�ÓÁH(ZyÐr9�8�ÓÁÓGÕWZ[Ð<Ð�=ÁÒOÓÁAb=ÁQYÐ�9<8]QY?6ANÐZ�ÑV8Ü?�8:9<A�Ï[Z];<×ÜH�UYÐr9�ÒFA(ÕO;<A�Ð>A�Qy9C=ÁQ·9<\OA(ÕO;>Z]9>Z]9r×yÕWA�Ð>A69ND��uQ·9>\YA(?�8]Ð>A�Z�ÑP\F8�QYKGÝ�;<=I9�9>A�QKG=IÏ[=|9�;<A�?�Z]Ï[QO=|9<=IZ[Q�M^9>\Y=ÁÐ H(AN8�QYÐ�9>\Y8�9 KG=IÏ[=|9*Ð�Z]ÑbAN8]?*\°?6Ó@8]Ð<Ð�=IQ¼8]ÓIÓ�ÕWZ[Ð<Ð>=IÒOÓÁAJÕWZ[Ð>=|9<=IZ[QYÐ�MÐ�=Áã�ANÐ�Mb8�QOÏ[ÓIANÐ�M�Ý�;<=I9>=ÁQOÏ�Ð�9r×yÓÁA�Ð�MPÓÁ=ÁQOA)9>\Y=Á?*`¡QOANÐ>Ð>A�Ð�MVÐ>`]A�Ý%Ð�M�A�9<?]DIDÁDVH�UYÐ�9ÜÒWAõÐr9<Z];<A�KpD��uQ;>AN8�ÓPÐ>=|9<UY8:9<=IZ[QYÐ�Mp9>\O=@Ð 8]ÕOÕO;<Z[8[?*\ßÓÁA�8[KOн9>Z)=ÁH�ÕO;*8]?e9<=Á?�8�ÓÁÓI×ßÓÁ8];>Ï[A�ÕO;>Z]9>Z]9r×yÕWA¥Ð>A69<Ð�Z];�9<ZH(ANKG=ÁZ¡?�;>AV;>AN?6Z[Ï]QO=I9>=ÁZ]Q 8[?�?�UO;<8[?6×½8]Ð�=IÓÁÓÁUYÐr9<;<8�9>ANK�=IQ��=IÏ[UO;>ACå]D:R%Q�UOQOÓ@8�ÒWA�ÓÁA�K =IHJ8�Ï[APZ]ÑY89>\O=@?*`^M�Ð>Ó@8�Qy9>ANK·ö��[ø�H UFÐr9�ÒWAb?�ÓÁ8[Ð>Ð>=|âFA�K�Ò¡×�âYQYKO=IQOÏ�9<\OAb?�ÓIZyÐ�ANÐr9ÕO;<Z�9>Z]9rסÕFAb=IHJ8]Ï]APZ[UG9�Z�Ñ9rÝVZ�=ÁHJ8�Ï]ANÐ�;>A�ÕO;<A�Ð>A�Qy9>=ÁQOϽ;>ANÐ�ÕWA�?69>=ÁÛ]A�ÓI× 8�9<\O=ÁQ�M]UOÕY;>=ÁÏ]\y9�ö��[ø½8�QYK�8�9>\Y=Á?*`^M]Ð>ÓÁ8]Qy9>A�K�ö��[øYDR/?�?�Z];*KG=ÁQOÏ�9>Z)9<\OA #PUY?6ÓÁ=ÁKOA�8�Q�KO=ÁÐ�9<8]QY?6A·ä¯Ð�UOH Z]Ñb9<\OAdÐ<ùyUY8�;<A�Ð�Z]Ñ�9>\OA¥ÕY=|áGA�ÓP9>Z·ÕY=|áGA�ÓKG=|à^A�;<A�QF?6A�Ð*çeMO9>\OA)ö��[ø�=ÁÐ�?�ÓIZyÐ�A�;�DOÙ�\OA½;<A�Ð>UOÓI9%=ÁÐ�8]QÜ=ÁQY?6Z[;>;<A�?69%?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Q�DÙ�\OAJ?6Ó@8]Ð<Ð�=@?�8]Ó�Ýb8�×)Z]Ñ�KGAN8�ÓÁ=IQYÏÞÝ�=I9>\ô9<\O=ÁÐ�ÕO;>Z[ÒOÓÁA�H =@Ð�9>Z)UFÐ�AJ8)Ð>Z�Öu?�8�ÓÁÓÁA�Kdò('*ë�ï¯ó6) '

'�[ï )>ë]é�ï±ð )%Ý�\OZyÐ�A�ÕOUO;<ÕFZyÐ�A(=ÁÐC9<ZÞ?�Z]H�ÕOUG9<A(8d;>A�ÕO;>ANÐ�A�Qy9<8:9<=IZ[Q·Z�ÑP9>\OA(ÕY8:9>9>A�;>QYÐC9<\Y8:9½=ÁÐH(=ÁQO=ÁHJ8�ÓÁÓI×�8:à^A�?69>A�K�Ò¡×d9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�Z�Ñ�9>\OA ÕY8:9>9>A�;<QYÐ�9<\Y8:9½KGZJQOZ]9/H�Z¡KO=|ÑÔ×Ü9>\OA�=I;?�8:9<A�Ï[Z];<×]D�OZ];%?*\F8�;*8]?e9<A�;b;>AN?6Z[Ï]QO=I9>=ÁZ]Q�M[9>\YA�;>A�ÕO;>ANÐ�A�Qy9<8:9<=IZ[QdÐ>\OZ]UOÓ@KÜÒFA�=ÁQ¡Û�8];>=@8�Qy9bÝ�=|9<\;>ANÐ�ÕWA�?69b9<ZJÕFZyÐ�=I9>=ÁZ]Q$MYÐ�=Áã�A�?*\Y8]QOÏ]ANÐ�MGÐ>ÓI=ÁÏ]\y9%;<Z�9*8:9>=ÁZ]QFÐ�MGKG=@Ðr9<Z];>9>=ÁZ]QYÐ�MGZ];%?*\F8�QOÏ[A�Ðb=IQ�ÓÁ=IQYA9>\O=@?*`¡QOA�Ð<Ð�DbÙ�\OAõKGANÐ�=ÁÏ]Q 8]QYK�=ÁH�ÕOÓIA�H�A�Qy9<8�9>=ÁZ]Q�Z�Ñ�ÑÔAN8:9<UO;>AõA6á¡9>;*8]?69>Z];*Ð�=@ÐJ9>\YAõHJ8:ØrZ[;ÒFZ]9�9>ÓÁA�QYA�?*`)Z�ÑVÒOUO=ÁÓÁKG=ÁQOÏ�8¥ÕY8�9�9<A�;<Qõ;<A�?6Z[Ï]QO=I9>=ÁZ]QßÐ�×GÐ�9>A�H�D��OZ[;�A�áO8�H�ÕOÓÁA]MF9>\OA�ÕO;<Z]ÒOÓÁA�H=IÓÁÓIUFÐr9<;<8�9>A�KÜ=ÁQ��=ÁÏ]UO;<A�å�?�8�Q�ÒWA�Ð�Z[ÓIÛ[A�KdÒy×ÜKGA�Ð>ÓÁ8]Qy9>=ÁQOÏJ8�QYKÜ9>\Y=IQOQY=IQOÏ�9<\OA�=ÁH�8]Ï]ANÐ�DR%Q.8]Ó|9<A�;<QY8:9<=IÛ[A¥9<Z·9>\Y=ÁÐ(=@Ð 9<ZßUFÐ�A�8�Q�=ÁQ¡Û:8�;<=Á8]Q[930:í@ìeïuë:ñWé1' %&'*ë�ìeó6) '(?6Z[QYÐ�9>;<UY?e9<A�K

=IQõÐ>UY?*\·8JÝ�8�×¥9<\Y8:9C9>\OA(KG=@Ðr9*8�QY?�A�ÒFA�9rÝVA�A�Q·8JÕY;>Z]9>Z�9rסÕWA 8�QFK)8JÕY8:9>9>A�;>Q)Ý�=ÁÓIÓQOZ]9CÒWA8:à^A�?e9<A�K Òy×½=Á;>;<A�ÓÁA�Û:8]Q[9$9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐpZ�ÑO9<\OAbÕY8:9>9>A�;<Q�Z[;$Z�ÑY9>\OAVÕO;>Z]9>Z]9r×yÕWA]D ��=I9>\�8]Q=IQ¡Û:8�;<=Á8]Qy9CKG=@Ðr9*8�QY?�A�H�A�8[Ð�UY;>A[MYA�8[?*\·ÕO;<Z�9<Z�9rסÕFA�?�8�QõH�8�9<?*\ßH�8]Q¡×ÜÕWZ[Ð<Ð�=ÁÒOÓÁA�=ÁQYÐ�9<8]QY?6ANÐZ�Ñ�ÕY8:9>9>A�;<Q�M¡9<\OA�;<A�Ò¡×¥Ï[;>AN8:9>ÓÁ×J;<A�KGUY?�=IQYÏ�9<\OA�Q¡UOH ÒWA�;/Z�Ñ�ÕY;>Z]9>Z�9rסÕWA�Ð�;<A�ùyUO=Á;>ANKpDÙ�\OA/QY8�9>UO;*8�ÓWÝ�8�×�Z]Ñ$KGZ]=ÁQOÏ�9<\O=@ÐV=@ÐP9>Z(UYÐ�AÜö>KGA�ÑÔZ];<HJ8�ÒOÓÁA�ø�ÕY;>Z]9>Z�9rסÕWA�Ð�D[_CUO;<=IQOÏ�9>\YA

H�8�9<?*\O=ÁQOÏ�ÕY;>ZG?6ANÐ>Ð�M�AN8]?*\JÕO;<Z�9<Z�9rסÕFA�=@ÐbKGA6ÑÔZ[;>H�A�KJÐ>Z 8[Ð�9>Z�ÒFANÐr9VâO9P9<\OA%=ÁQY?6Z[H(=ÁQOÏ�ÕF8:9�Ö9>A�;<Q�DÙ�\OAdùyUY8�ÓÁ=I9r×õZ�Ñ�9<\OA¥âO9NM$ÕWZ[Ð<Ð�=ÁÒOÓÁ×ô?�Z]H�ÒO=IQYA�KôÝ�=|9<\�8�H�A�8[Ð�UO;<AJZ�Ñ�9<\OAd8�H�Z[UOQy9Z�Ñ�KOA6ÑÔZ];<HJ8:9<=IZ[Q�M�=ÁÐ(9<\OA�Q UYÐ>A�K 8]Ð�9>\OA·KO=ÁÐ�9<8]QY?6AÞH�A�8]Ð>UO;<A EIåPI*HjD���=I9>\ 9>\YAÞA�áO8�H�ÕOÓÁAZ�Ñ��=ÁÏ]UO;<Aßå[M9>\OA ö��[ø·ÕO;<Z�9<Z�9rסÕWAdÝbZ]UOÓ@K�ÒWA�;>Z]9<8:9<A�K.8�QYK�9>\O=@?*`]A�QOA�K.Ð�Z°8]Ð 9>Z°ÒWA�Ð�9H�8�9<?*\¼9<\OAd=ÁQY?�Z]H�=IQYÏ ö��[øOD�Ù�\Y=ÁÐ(8�ÕOÕY;>Zy8]?*\ô\Y8]Ð 9rÝVZßÐ�\OZ[;�9*?6Z[H(=ÁQOÏyÐ�D��=Á;<Ð�9�M�8·Ð>A69(Z�Ñ8�ÓÁÓIZ:ÝbA�KßKGA6ÑÔZ[;>HJ8:9<=IZ[QYнH UFÐr9�ÒWAJ`yQYZ:Ý�Qô8ÞÕO;<=IZ[;>=jD��OZ[;�9<UOQY8�9>A�ÓÁ×]M^9<\O=@Ð�=ÁÐ�9<\OA¥?�8[Ð�A(ÑÔZ[;H�8]Q¡×¼9<8[Ð�`GÐ�M�=IQY?�ÓIUFKG=IQYÏ�?*\Y8�;*8]?69>A�;(;<A�?6Z[Ï]QO=I9>=ÁZ]Q$D�EGA�?6Z[QYKpM�9<\OA·Ð>A�8�;*?*\�ÑÔZ[;�9>\OA)ÒWA�Ð�9�ÖH�8�9<?*\O=ÁQOÏôKGA6ÑÔZ[;>HJ8:9<=IZ[Q�=@Ð�Z]ѳ9>A�Q�A�QYZ];<H(Z[UYÐ>ÓI×°A6áGÕWA�QYÐ>=ÁÛ]A]M�8�QFK���Z[; UOQO;<A�ÓÁ=@8�ÒOÓÁA]DPTVZ]QOÖÐ�=@KGA�;/9>\OA�?�8]Ð>A�Z]Ñ�ÕF8:9�9<A�;<QYÐ�9<\Y8:9½?�8]QÞÒWA ;<A�ÕY;>ANÐ�A�Q[9<A�KÞÒ¡×ÜÛ[A�?69>Z];*Ð�D��OZ[;/A6áO8]H(ÕYÓIA[MO9>\YAÕO=|áGA�ÓGÛ�8]ÓIUYA�Ð�Z�Ñ^8(å�æCÒy×då�æCÕO=|áGA�ÓY?*\Y8�;*8]?69>A�;�=ÁHJ8�Ï]A�?�8]Q�ÒWA�Û¡=IA�ÝVANK 8[Ð9>\OA%?�Z]H�ÕFZ[QOA�Qy9*ÐZ�Ñ%8 G I]æ:ÖuKG=ÁH(A�QYÐ>=IZ[QY8�Ó�Û[A�?e9<Z];ND =CQOA¥ÕY8�9�9<A�;<Q�M$Z[;�Z]QOAdÕO;>Z]9>Z]9r×yÕWA]M$=ÁÐ�8)ÕFZ[=IQy9 =ÁQ¼9<\O=ÁÐG I�æ�Ö±KO=IH�A�QFÐ�=ÁZ]QY8]ÓbÐ>ÕY8[?6A]D�RCÐ>Ð>UOH�=IQYÏß9<\Y8:9�9<\OA·Ð�A�9JZ�Ñ�8]ÓIÓÁZ:Ý�8�ÒOÓÁA¥9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐ�=ÁÐ?6Z]Qy9<=IQ¡UOZ[UYÐ�M]9>\OA½Ð>A69�Z�Ñ$8�ÓÁÓF9>\OA�ÕY8:9>9>A�;>QYÐP9>\F8:9b?�8�Q¥ÒWA�Z]ÒG9*8�=ÁQOA�KJÒ¡×(9>;*8�QFÐrÑÔZ[;>H�=ÁQOÏ�Z]QYAÕO;>Z]9>Z]9r×yÕWA UFÐ�=ÁQOÏdZ[QOA Z[;C8Ü?6Z]H�ÒO=ÁQY8:9<=IZ[Q)Z�ÑV8]ÓIÓÁZ:Ýb8]ÒOÓÁA½9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ%=@ÐC8ÜÐ�UO;>Ѷ8]?�A=IQÞ9>\YA@G I�æ�Ö±_÷ÕY=|áGA�Ó�Ð�ÕY8[?6A[D�2)Z];<A½ÕO;>AN?6=@Ð�A�ÓI×[MGÝ�\OA�Q·8(ÕY8�9�9<A�;<Q��Ú=@Ðb9>;*8�QYÐ�ÑÔZ];<H�A�KôäÔA]D ÏYD;>Z]9<8:9<A�KYçP8]?�?6Z];*KG=ÁQOϽ9>Z(8�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Q��yä�������çVÝ�\O=@?*\dKGA�ÕWA�QFKOÐbZ]Q¥Z[QOA%ÕY8];<8]H(A�9>A�;� äÔA[D ÏFDG9>\OA�8]QOÏ]ÓÁA�Z�Ñ9>\OA�;<Z�9*8:9>=ÁZ]QWçeM¡9>\YA�Ð�A�9�Z]Ñ�8]ÓIÓ^9<\OA½9>;*8�QFÐrÑÔZ[;>H�A�KÜÕY8�9�9>A�;>QFÐ

��� �"!$#&%('*) ÑÔZ];�Ý�\O=@?*\ #+� �yä����,�ç.- ärå�ç=ÁÐb8�Z[QOA6ÖuKG=ÁH(A�QYÐ>=IZ[QY8�ÓW?6UO;<Û]A%=ÁQJ9<\OACÛ[A�?69>Z];bÐ�ÕF8]?6A/Z�Ñ�9<\OA/=ÁQOÕOUG9*Ð�D�uQJ9<\OA�;>A�H�8]=IQFKGA�;VZ�Ñ9>\O=@Ð�?*\Y8�ÕG9<A�;NM]ÝbA/Ý�=ÁÓIÓ^8�ÓÁÝ�8�סÐV8]Ð<Ð�UYH(A%9>\Y8�9bÝbA%\Y8�Û[A/?*\OZ[Ð>A�Q/�/ÒWA½KG=|à^A�;<A�Qy9<=Á8]ÒOÓIA/Ý�=|9<\;>ANÐ�ÕWA�?69b9<ZJÒFZ]9>\0�48]QYK���8�QFKÜÐ>UY?*\Þ9>\Y8�91�[ä2���>èyç � ��D

Page 4: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

��\OA�Qô9>\YAdÐ�A�9�Z�Ñb9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ�=ÁÐ�ÕF8�;*8�H�A69<A�;<=Iã�A�KßÒy×��.ÕF8�;*8�H�A69<A�;*Ð ����ä¶;>Z]Ö9<8:9<=IZ[Q�MW9>;*8�QYÐ>Ó@8:9>=ÁZ]Q$MFÐ<?�8�ÓÁ=ÁQOÏYMWA69*?�D�çeM^9>\YA�=ÁQy9>;<=IQYÐ>=@? KG=ÁH�A�QYÐ>=IZ[QõZ]Ñ�9>\OA(HJ8�QO=IÑÔZ]Ó@K � � =ÁÐ8:9JH�Z[Ð�9���D �YZ];�A6áO8�H�ÕOÓÁA]M�=IÑ/9<\OA)8�ÓÁÓIZ:Ý�8�ÒOÓÁA¥9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QYÐ�Z�Ñ�?*\Y8�;*8]?69>A�;(=ÁH�8]Ï]ANÐ8�;<A¥\OZ];<=Áã�Z]Qy9*8�ÓP8]QYK¼Û]A�;�9<=Á?�8�ÓbÐ�\O=Iѳ9<Ð�M;>Z]9<8:9<=IZ[QYÐ�M�8�QFK�Ð>?�8�ÓÁ=IQYÏYM�9<\OAÜÐ>UO;>Ѷ8]?�A¥Ý�=IÓÁÓbÒFAÞ8��ÖuKG=IH�A�QYÐ�=ÁZ]QF8�Ó^HJ8�QO=IÑÔZ]Ó@KpD�uQ)Ï]A�QYA�;*8�ÓjM¡9>\OA�H�8]QO=IÑÔZ]Ó@K�Ý�=IÓÁÓ$QOZ]9/ÒWA ÓÁ=IQOAN8�;ND�#�Û[A�Q)8¥Ð>=IH�ÕOÓÁA�=ÁHJ8�Ï]A½9<;<8]QYÐ>ÓÁ8�9>=ÁZ]Q

?6Z];<;<A�Ð>ÕFZ[QYKOÐP9>Z�8�\Y=IÏ[\OÓI×JQYZ]QGÖ±ÓI=ÁQOAN8�;V9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Qd=IQÜ9>\OA½\O=ÁÏ]\OÖ±KG=ÁH�A�QYÐ>=ÁZ]QY8]ÓWÕY=|áGA�ÓÐ�ÕY8[?6A[D��OZ[;dA�áG8]H�ÕOÓIA[MP=IÑ�9>\OAß=IHJ8]Ï]A·Z]Ñ 8 ö K[ø°=ÁÐd9>;*8�QYÐ>Ó@8:9>ANK UOÕ¡Ýb8];<K�MPÐ�Z[H�A·ÕO=Iá¡A�ÓÁÐZ[Ð<?6=ÁÓIÓ@8:9<AÜÑÔ;<Z]H Ý�\O=|9<A)9>Z°ÒOÓ@8]?*`.8�QYK ÒY8[?*`�Ð�A�Û]A�;<8]ÓV9<=IH�ANÐ�D 2õ8:9<?*\Y=IQOϼ8¼KGA�ÑÔZ];<HJ8�ÒOÓÁAÕO;>Z]9>Z]9r×yÕWAC9>ZJ8�QÜ=IQY?�Z]H�=ÁQOÏ�ÕY8:9>9>A�;<QÞQOZ:Ý 8�H�Z[UOQy9<ÐV9>Z(âYQYKO=IQOÏ(9<\OA½ÕFZ[=IQy9�Z]QÜ9>\OA�Ð>UO;>ÖѶ8]?6Ad9>\Y8�9�=@Ð�8�9(8õH(=ÁQO=ÁH UOH KG=@Ð�9<8�QF?6A¥ÑÔ;<Z]H29<\OAdÕWZ]=ÁQy9(;<A�ÕO;<A�Ð>A�Qy9<=IQOÏ�9<\OAd=ÁQY?�Z]H�=IQYÏÕY8:9>9>A�;<Q�DaVAN?�8]UYÐ�A(9<\OAdHJ8�QY=|ÑÔZ[ÓÁKô=@Ð�QOZ[QGÖjÓÁ=ÁQOA�8];�Mp9<\OA¥HJ8:9*?*\O=ÁQOÏ·?�8]QôÒWA¥Û[A�;<×õA6áGÕWA�QGÖÐ�=ÁÛ]A(8�QYK)UYQO;>A�ÓI=@8�ÒYÓIA[D^E¡=ÁH�ÕOÓÁA�H�=ÁQO=IH�=Áã�8�9>=ÁZ]QõH�A�9>\OZGKOÐ�Ð�UF?*\õ8]ÐCÏ];*8]KG=ÁA�Qy9CKOA�Ð<?6A�Qy9JäÔZ[;?6Z]Q�ØrUOÏ[8�9>A Ï[;<8[KG=ÁA�Qy9*çC?�8]Q·ÒFA�UYÐ>A�K·9>ZÜâYQYK·9>\OA�H�=IQY=IH�UOH(Ö±KG=@Ð�9<8�QF?6A(ÕFZ[=IQy9�M^\OZ:ÝbA�Û[A�;NM9>\OANÐ�A�H(A�9>\OZGKOÐCZ]QYÓI×�?6Z[QyÛ[A�;<Ï]A�9>ZÞ8)êIðNé*ë:êYH�=ÁQO=ÁH UOH�D �uQõ8]KOKO=|9<=IZ[Q�MF;<UOQOQY=IQOÏÜÐ�UY?*\·8]Q=|9<A�;*8:9>=ÁÛ]A½ÕO;<ZG?6ANKGUO;<A/ÑÔZ[;�A�8]?*\ÞÕO;<Z�9<Z�9rסÕFA½=@Ð�ÕO;<Z]\O=ÁÒO=I9>=ÁÛ]A�ÓÁ×¥A�á¡ÕWA�QFÐ�=ÁÛ]A[D�±Ñ�9>\OAÞÐ�A�9 Z�Ñ�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYн\Y8]ÕOÕWA�QYÐ 9>Z·ÒWAdÓÁ=ÁQOA�8]; =ÁQ�ÕO=Iá¡A�ÓbÐ>ÕY8[?6A]M$9>\OA�Q¼9>\YA

H�8]QO=IÑÔZ]Ó@K�=ÁÐV8�ÓI=ÁQOAN8�;PÐ>UOÒYÐ>ÕY8]?�A�ä¶8�ÕYÓÁ8]QOANç6D[Ù�\OA%HJ8�9<?*\O=ÁQOÏ�ÕO;<ZG?6ANKGUO;<A�=ÁÐ�9>\OA�QJ;>ANKGUY?6ANK9>Z�âYQYKG=ÁQOÏ�9>\YA%Ð�\OZ[;�9<A�Ð�9�KO=ÁÐ�9<8]QY?6A�ÒWA69rÝbA�A�Q�8CÕFZ[=IQy9%äÔÛ[A�?e9<Z];eç8�QYK(8/ÕYÓÁ8]QOA]û�8]Q�AN8]Ð>×[Öj9>Z]ÖÐ�Z[ÓIÛ[A/ùyUY8]KG;*8:9<=Á?%H(=ÁQO=ÁH�=IãN8:9>=ÁZ]QÜÕO;<Z]ÒOÓÁA�H�D¡Ù�\O=ÁÐbÐ>ÕFAN?6=@8�Ó^?�8]Ð>A/\F8]Ð�ÒWA�A�QÜÐ�9>UYKO=IANK¥=ÁQ¥9>\YAÐr9*8:9>=@Ð�9>=@?�8�Ó�ÓI=I9>A�;<8�9>UO;<A�8�QFKÜ=@Ð/Ð>Z]H�A�9>=ÁH(ANÐ%;<A6ÑÔA�;>;<A�KÜ9>Z¥8[Ð/7P;>ZG?6;<UYÐ�9>ANÐ�8�QY8]ÓI×GÐ>=ÁÐ E G � HjD��±9\Y8]Ð�ÒWA�A�Q�8]ÕOÕOÓÁ=IANKÜ9>Z¥Ð>=IÏ[QY8:9<UO;>A�Û]A�;>=IâF?�8:9>=ÁZ]Q EIåPG�H$8]QYKÜZ]QGÖ±ÓI=ÁQOA�?*\Y8];<8[?e9<A�;b;>AN?6Z]Ï[QO=I9>=ÁZ]QE G]æ*H±DÙ�\O=ÁÐ�?*\F8�ÕG9<A�;�?6Z[QYÐ�=@KGA�;<Ð�9<\OA�H(Z[;>A�Ï[A�QOA�;<8]Ó�?�8]Ð>A�Z]ÑbQYZ]QGÖ±ÓI=ÁQOAN8�;�9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ

Ð�UY?*\ß8]Ð%Ï[A�Z[H(A�9>;<=Á?�9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�Z�Ñ�Ï[;<8�×yÖ±ÓIA�Û]A�Óp=ÁHJ8�Ï[A�Ð�DWÎ�A�H�A�H ÒWA�;C9>\F8:9CA�Û]A�Q·8Ð�=ÁH�ÕOÓIAd=IHJ8�Ï[A�9>;*8�QYÐ>Ó@8:9>=ÁZ]Q¼?�Z];<;>ANÐ�ÕWZ]QYKYÐ�9>Zõ8)\Y=IÏ[\OÓI×ôQOZ[QGÖjÓÁ=ÁQOA�8];½9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q°=ÁQ9>\OAÞ\O=ÁÏ]\GÖuKG=ÁH(A�QYÐ>=IZ[QY8�ÓbÕO=Iá¡A�Ó%Ð�ÕF8]?6A[DÙ�\YAÜHJ8�=ÁQ�=@KGAN8õZ]Ñ%9>\OA�?*\Y8]ÕG9>A�;�=@Ð�9>Z°8�ÕOÕO;<Z�á¡Ö=IHJ8:9<AÞ9>\OA)Ð>UO;�Ѷ8[?6AÞZ�Ñ�ÕFZyÐ>Ð>=IÒYÓIAÜ9>;*8�QYÐ�ÑÔZ];<HJÐ(Z�Ñ�8ßÕY8�9�9>A�;>Q Ò¡×¼=|9*Ð(9<8]QOÏ]A�Q[9JÕOÓ@8�QYAÞ8�99>\OA�ÕF8:9�9<A�;<Q�MG9<\OA�;<A�Ò¡×¥;<A�KGUF?6=ÁQOÏ�9>\OA�HJ8�9<?*\O=ÁQOÏ�9>Z�âFQYKG=ÁQOÏ�9>\OA�Ð�\OZ[;�9<A�Ð�9%KG=ÁÐ�9<8]QY?6A�ÒWA6Ö9rÝVA�A�Q°9rÝbZ·ÕOÓ@8�QOANÐ�D�Ù�\O=@Ð KG=@Ðr9*8�QY?�Ad=@Ð ?�8�ÓÁÓIANK°9>\OA��Yë�ñ;:;'6ñFï���í@ìeïuë:ñWé1'�DÙ�\YAd;<A�Ð>UOÓ|9�Z�Ñ9>\OA�8]ÕOÕO;<Z�á¡=ÁHJ8:9<=IZ[Qd=@Ð�Ð�\OZ:Ý�QÞ=ÁQ��=ÁÏ]UO;<AOGGMG=ÁQÞ9>\OA�?�8]Ð>A�Z]Ñ�;<Z�9*8:9>=ÁZ]QÜÑÔZ];�\F8�QYKGÝ�;<=I9�9>A�QKG=IÏ[=|9*Ð�DVRb9J9>\OA�9<Z]Õ Z�ÑC9>\OA)âFÏ]UO;<A]M�=@Ð(9>\OA·9>\OA�Z];<A69>=@?�8]Ó�?6UO;<Û]A�=ÁQ ÕY=|áGA�ÓCÐ�ÕY8[?6A�Ý�\O=@?*\;>A�ÕO;>ANÐ�A�Qy9<Ð�A�ùyUY8:9<=IZ[Q ä�åNçeM�9<Z]Ï[A69>\YA�;�Ý�=|9<\.=I9<Ð(ÓÁ=IQYA�8�;J8]ÕOÕO;<Z�á¡=ÁHJ8:9<=IZ[Q�D7�Z[=IQy9*Ð Z]Ñ/9>\YA9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Qß?6UO;<Û]A(8�;<A(KGA�ÕO=@?e9>ANK·ÒWA�ÓÁZ:Ý ÑÔZ];½Û:8�;<=ÁZ]UYÐ�8�H�Z]UYQ[9*Ð�Z�ÑP;<Z�9*8:9>=ÁZ]Q äÔAN8]?*\8�QOÏ[ÓIAJ?6Z[;>;<A�Ð>ÕWZ]QYKOÐC9<Z·8�Û�8]ÓIUYAJZ�Ñ �ç6D�Ù�\OAJÒWZ�9>9>Z]H Z�Ñ �=IÏ[UO;>A G�KGA�ÕO=@?e9<Ð�9>\OA¥ÓÁ=ÁQOA�8];8�ÕOÕO;<Z�áG=IHJ8�9>=ÁZ]Q·Z]ÑV9<\OAJ?6UO;<Û]A �yä�������ç½Ï[=IÛ[A�QõÒ¡×)9<\OA¥Ù8�×yÓÁZ];�A6áGÕY8]QYÐ>=IZ[QßZ]Ñ��J8];>Z[UOQYK� � èYû

�yä2������ç � �yä2���>è[ç ��� �[ä2������ç� �

� �ä�� f ç�� �� �� ä�G[çÙ�\O=ÁÐ�ÓI=ÁQOA�8];�8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]Qß=ÁÐ�?�Z]H�ÕOÓÁA69>A�ÓI×ô?*\Y8�;*8]?69>A�;>=Áã�ANK·Ò¡×)9>\YAJÕFZ[=IQy9 � 8�QYKß9>\YA9<8�QYÏ]A�Qy9Û]A�?69>Z[;� ������� ��� ���� � D�Ù�8]QOÏ]A�Qy9�Û]AN?e9<Z];*Ð�M�8�Ó@Ð�ZC?�8�ÓÁÓÁA�K�9<\OA�S�=ÁAVKGA�;>=ÁÛ:8:9>=ÁÛ]ANÐ$Z]ÑO9>\YA9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q �[M¡Ý�=ÁÓIÓWÒFAC9>\YA�Ð>UOÒGØrAN?e9�Z�Ñ�Ð>A�?69>=ÁZ]Q��FDyRCÐb?�8�Q¥ÒWA½Ð�A�A�QJÑÔ;<Z]H �=IÏ[UO;<ADGGMÑÔZ];%Ð>H�8]ÓIÓ�8]QOÏ]ÓÁA�Ð�ä�� ����� å�çeMG9<\OA�8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]QÜ=ÁÐ�Û[A�;<×JÏ]Z¡ZGKpD��=ÁÏ]UY;>A! ·=IÓÁÓIUFÐr9<;<8�9>A�Ð 9>\YAÜKG=IàWA�;>A�QY?6AÜÒFA�9rÝVA�A�Q¼9<\OA #PUY?6ÓÁ=@KGA�8]Q�KG=ÁÐ�9<8]QY?6A[M$9<\OA¥ÑÔUYÓIÓ

=IQ¡Û:8�;<=Á8]Qy9/KG=ÁÐ�9<8]QY?6AÞäÔH�=ÁQO=IH�UOH KG=@Ðr9*8�QY?�A ÒWA69rÝbA�A�Q)HJ8]QO=|ÑÔZ[ÓÁKYÐ<ç/8�QYKÞ9<\OA�9*8�QOÏ[A�Qy9CKG=@ÐrÖ9<8�QF?6A]D��uQß9<\OA�âYÏ[UO;>A[MpÒFZ]9>\ô9>\YAJÕO;>Z]9>Z]9r×yÕWAJ8�QYKß9<\OAJÕY8:9>9>A�;>Q¼8�;<AJKGA6ÑÔZ[;>HJ8]ÒOÓIA)äÔ9rÝVZ]Ö

Page 5: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

),

+ .

α=1α=−2 α=−1 α=0 α=2

P

α α=0

α=1

α=−2α=−1

α=2

T

P+ .Tα

Pixel space

TP

∂∂ α=T

s(P, )

s(

s

QSRUT ®��Y® �WqN ��¡}Pm* [orm*tum<w���k6�rvxqNwJq�²p�r~[m�m<´Om*s<�Vq�²p�r~[m�orq���k��rvxq�wJv|wJ [v�¢[m<�Ftu yk�s*mNµ��(vI�[�[�xm��Gz]i�k��x�orq���k��rvxqNw[t�q�²Pk�w�q�orv|l�vxw¡k����[vxlNv��rvx�*me��vxi�k�l�m�q�²��r~ym��]vxlNv��Þ±¤�Ã]�^²@q�o½�]vx´Om>orm*w:�/k�wyl��xm�{�k��x�ym<t/q�²� µy��q��u�rqNi��]h¯i�k�lNm<t�qN�[��k�vxwyme� ��¹½i�q�{�vxwylCk��xq�wyl��r~[mV��k�w[lNm*w:���rq/�r~ymV�uo�k�wyt±²Áq�ori�k��rvxq�w�s*�[or{Nm²Áq�oP�r~ym�trk�i�m�q�orvxlNvxwyk��p�]v|l�v��rv|�<me��vxi�k�lNm��:¹ kN�[�[vxwyl�{Nk6orv|q��ytVk�i�q��yw:�rt�¾ � ¿�q�²^�r~[m���k�w[lNm*w:�{Nm*s<�rq�o��%µ

Ð�=@KGA�K�KO=ÁÐ�9<8]QY?6A�çeMbÒOUG9¥ÑÔZ[;ÜÐ>=IH�ÕOÓÁ=Á?�=|9r× Z];ÜA(J¥?6=ÁA�QF?6× ;>AN8]Ð>Z]QYÐ�M�=|9Þ=ÁÐÜ8�Ó@Ð�Z¼ÕWZ[Ð<Ð>=IÒOÓÁA)9<ZKGA6ÑÔZ[;>H Z[QOÓÁ×¥9>\YA�ÕY;>Z]9>Z�9rסÕWA½Z];�Z]QYÓI×J9<\OA�UOQO`¡QOZ:Ý�QÞÕY8�9�9>A�;>Q¼ä¶Z]QOA�Ö±Ð>=ÁKOA�KÞKG=ÁÐ�9<8]QY?6A�çeDR%ÓI9>\OZ[UOÏ]\ß=IQß9>\OA(ÑÔZ[ÓIÓÁZ:Ý�=IQYÏÜÝbA�Ý�=ÁÓÁÓ�?6Z]QF?6A�Qy9<;<8�9>A Z[QõUYÐ>=ÁQOÏÞ9<8�QYÏ]A�Qy9�KG=@Ðr9*8�QY?�A 9<Z

;>AN?6Z]Ï[QO=Áã�A/=ÁHJ8�Ï[A�Ð�M[9<\OA�H�A69<\OZGKÜ?�8�QÞÒFA�8�ÕYÕOÓI=ÁA�Kd9>Z�HJ8�Q¡×¥KG=Ià^A�;<A�Qy9�9r×yÕWA�Ð�Z]ÑÐ>=IÏ[QY8�Ó@Ð�û9>A�H�ÕWZ];*8�Ó�Ð>=IÏ[QY8�Ó@Ð�MOÐ�ÕWA�AN?*\�MOÐ�A�QYÐ>Z];�KO8:9*8ODÁDID

���� �L;������ ���� � ������������� ��* !�"����

Î%8:9<\OA�;�9<\Y8�Q(9<;>ס=ÁQOϽ9>Z�`[A�A�ÕJ8�;<A�ÕO;<A�Ð>A�Qy9<8�9>=ÁZ]Q(Z�ÑW9>\OA%9>;*8�=ÁQO=IQYϽÐ>A69NM�=I9P=ÁÐP8�Ó@Ð�Z½ÕWZ[Ð<Ð�=ÁÒOÓÁA9>Zd?�Z]H�ÕOUG9<A�9>\YA�?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[QÜÑÔUYQY?e9<=IZ[Q·Ò¡×ÜÓÁA�8];>QO=ÁQOÏ¥8¥Ð>A69CZ�Ñ�ÕY8];<8]H(A�9>A�;<Ð�ÑÔ;<Z]H 9>\YAâYQO=I9>A�9>;*8�=ÁQO=IQYÏÞÐ>A69NDWÙ�\O=@Ð�=ÁÐC9<\OA(8]ÕOÕO;<Z[8[?*\�9<8�`[A�Q·=ÁQõQYA�UO;*8�Ó�QOA�9rÝVZ[;>`GÐ�DFS�A�9��Jä # çCÒWA9>\OA�äÔUOQY`yQYZ:Ý�QFç�ÑÔUYQY?e9<=IZ[Q·9>ZÞÒWA�ÓÁA�8];>QOANKpM�� � ! ä # L ���Jä # L ç�ç.�������$�Nä #�� ���Jä #�� ç�ç.-(ÒWA�8âYQO=I9>A�Ð�A�9�Z]Ñ�9>;*8�=ÁQO=IQYÏÜKO8�9<8J9*8�`[A�Q)ÑÔ;<Z]H 8d?�Z]QYÐ�9<8]Qy9�Ð�9<8:9<=ÁÐ�9>=@?�8]ÓKO=ÁÐ�9>;<=IÒYUG9>=ÁZ]Q��¥MW8]QYKÓIA�9 �"!�ä # ç�ÒFAC8�Ð�A�9PZ�Ñ^ÑÔUOQF?e9>=ÁZ]QFÐ�M[=IQYKOA6áGA�K�Ò¡×�9<\OA/Û[A�?69>Z];�Z]ÑpÕY8�;*8�H�A69<A�;*Ð$#�D¡Ù�\OA�9<8[Ð�`

Page 6: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

E

Euclidean distancebetween P and E

PTangent Distance

Distance betweenS and S

S

SP E

P

E

QSRUT ®��Y® hj�x�x�yt±�uo�k6�rvxqNw(q�²W�r~[m����ys*�xv|�[mek�w��]v|t±��k�wys*m�k�w¡���r~ymP��k�w[lNm*w:���]v|t±��k�wys*mb�Gm<�jºm*m*w k�wy�� µ[��~ym�s<�[or{�m*t����½k�w¡�����orm* [orm<tum*w:�P�r~[m�tum<�rt�q�²^ GqNvxw:�rt�qN�[��k�vxwyme�(�:¹ k� y y��¹]vxwyl��r~[m�s�~yqNtum<w�uo�k�wyt±²Áq�ori�k��rvxq�wyt/¾@²@q�oVm<¢[k�i� [�xm��uo�k�wytu�|k6�rv|q�wytbk�w¡� orq���k��rvxq�wyt�¿��rq k�w¡� � µy��~ym%�xvxwym*tPlNq�v|w[l�r~]orqN�[lN~� k�w¡� � orm* [orm*tum<w���r~ymb��k�wylNm<w����rq/�r~ym*tumbs*�[or{�m<t*µy�Ptutu�yi�vxwyl%�r~¡k6��º�q�or�]vxw[l�tu ¡k�s*m~yk�t i�q�orm¥�]v|i�m<wytuvxqNw[t��r~yk�wô�r~[mJw��yi��Gm>o�q�²%s�~yq�tum*w°�uo�k�w[t±²@q�ori�k6�rvxqNwytd¾³qNwß�r~ym¥�[v|k�l�o�k�i��k�tutu�yi�m�� � ¿��r~ymC��k�wyl�m*w:��tu ¡k�s*m*t%�]q w[q���vxw:�rm<ortum*s>�%k�wy�¥�r~[m/��k�wylNm<w����[vxt±��k�w[s*m�vxt��[wyv|À:�ym*��¹�]m<¸¡w[me�Yµ

=Áн9<ZÞâYQYK¼8ÞÛ:8�ÓÁUOA�ÑÔZ];"# ÑÔ;<Z]H 9<\OA�âYQO=I9>A�9<;<8]=IQY=IQOÏ)KO8�9<8�Ð>A69 � Ð>UY?*\ô9>\Y8�9 �"! ÒWA�Ð�98�ÕOÕO;<Z�áG=IHJ8�9>A�Ð �÷Z]Q¥=I9<ÐV=IQYÕOUG9 # D�OZ];bA6áO8]H(ÕYÓIA[M �"!ôHJ8�×�ÒWA/9<\OA%ÑÔUYQY?e9<=IZ[QÜ?6Z[H�ÕOUG9>ANKÒy×)8ÜQOA�UY;<8]Ó�QOA�9½8]QYK�9<\OA�Û[A�?69>Z]; #4HJ8�×�ÒWA 9<\OA�QYA�UO;*8�ÓQOA69� Ð�ÝbA�=ÁÏ]\y9<Ð�MWZ];��"! H�8�×ÒFA 8¥ÕFZ[ÓIסQOZ[H(=@8�Ó�8�QYK # =I9<ÐC?6Z¡A(J¥?6=ÁA�Qy9*Ð�D���=|9<\OZ]UG9�8]KOKG=I9>=ÁZ]QF8�Ó$=ÁQGÑÔZ[;>HJ8:9<=IZ[Q�MGâYQFKG=IQYÏ8¥Û:8�ÓÁUOA�ÑÔZ[; #Ú=ÁÐ�8�Q)=ÁÓIÓIÖ±ÕFZyÐ�ANK)ÕO;<Z]ÒOÓÁA�H ÒFAN?�8]UYÐ�A 9>\OA 9>;*8�=ÁQO=ÁQOÏÜÐ>A69½KGZ¡A�Ð%QYZ�9CÕY;>Z:Û¡=@KGAA�QOZ[UOÏ]\�=ÁQGÑÔZ[;>HJ8:9<=IZ[Q¼9>ZôKG=@Ðr9<=IQOÏ[UO=@Ð�\�9>\OAÞÒFANÐr9JÐ>Z]ÓÁUG9>=ÁZ]Q�8]H�Z]QOÏõ8]ÓIÓb9<\OAÞ?�8�QYKO=ÁKO8�9>ANÐ#/Ð�DWÙ�\O=@Ð%ÕO;<Z]ÒOÓÁA�H =ÁÐ%=IÓÁÓIUFÐr9<;<8�9>A�K�=ÁQ �=IÏ[UO;<A �·äÔÓÁA6ѳ9*ç6DWÙ�\YA�KGA�Ð>=I;<A�KÜÑÔUOQF?e9>=ÁZ]Q � ä¯Ð�Z[ÓI=@KÓI=ÁQOANçb=@Ðb9>Z�ÒWA�8�ÕOÕY;>Z�áG=ÁH�8�9>ANKJÒ¡×d8�ÑÔUOQY?e9<=IZ[QYÐ� ! ä¯KGZ�9>9>ANKÜÓÁ=ÁQOANçVÑÔ;<Z]H ÑÔZ[UO;�A6áO8]H(ÕYÓIANÐ! ä # �����Jä # �jç>ç - � � L � f � � � � D�R/Ð$A6áGA�H�ÕOÓÁ=IâYA�K½=ÁQ�9>\YA�ÕO=@?e9>UY;>A[M69>\YA�âO9>9>A�K�ÑÔUOQY?e9<=IZ[Q � ! ÓÁ8];>Ï[A�ÓÁ×KG=ÁÐ<8�Ï[;>A�A�ÐpÝ�=I9>\ 9>\OAbKGA�Ð>=Á;>ANK½ÑÔUOQY?69>=ÁZ]Q � ÒWA69rÝbA�A�Q�9>\YAPA6áO8]H(ÕYÓIANÐ�MNÒOUG9�=|9=ÁÐ�QOZ]9�ÕWZ[Ð<Ð�=ÁÒOÓÁA9>Z�=IQOÑÔA�;�9>\O=@Ð�ÑÔ;<Z]H 9<\OA(9>;*8�=ÁQO=IQYÏÞÐ>A69�8�ÓÁZ]QOA[D 2·8�Q¡×)Û:8�ÓÁUOA�нZ]Ñ # ?�8]QõÏ]A�QOA�;*8:9<A�HJ8]Qy×KG=|à^A�;<A�Qy9bÑÔUOQY?69>=ÁZ]Q � ! MGÐ>Z]H�A/Z]Ñ$Ý�\O=@?*\¥HJ8�×(ÒWAC9>A�;<;<=IÒOÓÁA½8�ÕOÕO;<Z�áG=IHJ8�9>=ÁZ]QYÐ�Z]Ñ ��MyA�Û]A�Q9>\OZ[UOÏ]\ß9>\OA�×ß8];>A�=ÁQ°?�Z]H�ÕOÓÁA69<A�8]Ï];<A�A�H(A�Qy9½Ý�=I9>\ô9<\OA�9>;*8�=ÁQO=ÁQOÏ)Ð�A�9�D�abA�?�8�UYÐ>A�Z�Ñb9>\O=@Ð�M=|9�=@Ð�?6UYÐ�9>Z[HJ8�;<×)9>Z)8]KYK ö>;>A�Ï]UOÓ@8�;<=Áã�A�;*Ð<øOMOZ[;�8]KOKG=I9>=ÁZ]QF8�Ó�?6Z[QYÐ�9>;*8�=ÁQ[9*Ð�MW9<Z);>ANÐr9<;>=@?e9½9>\YAÐ�AN8�;*?*\�Z�Ñ�8�Q�8[?�?6A�ÕG9<8]ÒOÓÁA # D��OZ[;JA6áO8�H�ÕOÓÁA]MPÝVA�HJ8�×�;<A�ùyUO=Á;>AÞ9>\YA)ÑÔUOQY?69>=ÁZ]Q �"! 9<ZÒFA ö>Ð>H�Z¡Z�9>\FøOM�Ò¡×�8]KOKO=IQOÏß9>\YA·?6Z]QFÐr9<;<8]=IQy9(9>\Y8�9 � # �6f�Ð>\OZ]UYÓÁK.ÒFA�H�=ÁQO=IH�=Áã�ANKpD �±9J=ÁÐ=IH�ÕWZ];>9<8�Qy99>\Y8�9�9<\OA�;>A�Ï]UOÓ@8�;<=Áã�A�;$;>A��YA�?69<Ð�8%ÕO;<Z]ÕWA�;>9r×½Z�Ñ ��M:\YA�QY?�AV;<A�Ï[UOÓÁ8];>=Áã�A�;<Ð�;>ANù[UY=I;<A8]KOKG=I9>=ÁZ]QF8�Ó�8:Ö±ÕO;<=IZ[;>=^`¡QOZ:Ý�ÓÁA�KGÏ[A�8]ÒFZ[UG9�9>\YA½ÑÔUOQY?e9<=IZ[QÞ9>Z�ÒFA�H�ZGKGA�ÓIANKpD

Page 7: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

x2x1 x3 x4

F(x)

x x2x1 x3 x4

F(x)

x

wG (x)

wG (x)

QSRUT ®��W® �Ymek�orw[vxwyl/k�lNvx{�m*w½²Á�ywys>�rv|q�w¥¾³tuqN�xv|���xvxwyme¿�²Áorq�i#k��xvxi�v��rme�½tum<��q�²Fm<¢[k�i� y�xm*tP¾�� g �rq����e¿>µ��~[mb¸y�u�rm*��s<�[or{�m*t�k�orm�tu~[q6º�w��:¹��[q��u�rme� �xvxwym�µ¡�Ym<²Á� �[��~ymVqNw[�x¹ s*q�wyt±�uo�k�v|w:��vxt��r~¡k6���r~ymP¸y�u�rme�s<�[or{�mClNq:m*t%�r~[orq��ylN~¥�r~ym�m<¢[k�i� y�xm*t*µY}PvxlN~:� �F��~ym�¸[�u�rme�ds<�[or{�m*t�w[q���q�wy��¹dlNq(�r~]orqN�[lN~Þm*k�s�~m>¢yk�i� y�xm%�y�]��k��xtuq v��rt��[m<orvx{�k��rvx{�m*tVm*{�k��x�¡k6�rme�¥k����r~[mCm<¢[k�i� y�xm*tVk�l�orm<m�º�v��r~¥�r~[m��]m<orvx{Nk6�rvx{�m*tq�²^�r~ym�l�vx{�m*w�²Á�yw[s<�rvxqNwFµ

E¡A�ÓÁA�?69>=ÁQOÏõ8�Ï]Z¡ZGKßѶ8�H�=ÁÓI×�� � ! � ! ��#��� -dZ�Ñ�ÑÔUOQF?e9>=ÁZ]QFÐ�=@Ð 8·KG= J¥?6UOÓI9 9*8]Ð>`^MÐ�Z[H(A�9>=ÁH�A�Ð�`¡QOZ:Ý�Q°8]Ð�ö>H(ZGKGA�ÓPÐ>A�ÓÁA�?e9<=IZ[QYø E|åNæOM6å$� H±D��±Ñ�� ?6Z[Qy9<8�=ÁQYÐ�8ÞÓ@8�;<Ï]A(Ѷ8�H�=ÁÓI×ßZ�ÑÑÔUOQY?e9<=IZ[QYÐ�MF=I9�=@ÐCH(Z[;>A ÓI=Á`]A�ÓI×Þ9>\F8:9C=I9CÝ�=ÁÓÁÓ�?�Z]Qy9<8]=IQ·8dÏ]Z¡ZGK)8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]Q�Z�Ñ � äÔ9>\OAÑÔUOQY?e9<=IZ[Q ÝbAP8�;<A�9<;>ס=ÁQOÏ�9<ZC8�ÕYÕO;>Z�áG=ÁHJ8:9>A�çeM�ÒOUG9�=|9=ÁÐ8�Ó@Ð�Z/H(Z[;>AVÓI=Á`]A�ÓI×C9>\F8:99>\OAbÐ�A�ÓIAN?e9>ANK?�8�QFKG=ÁKY8:9>A�äÔUYÐ>=IQYÏ�9>\YA%9>;*8�=ÁQO=ÁQOÏ Ð>A69eç�Ý�=ÁÓIÓWÏ]A�QOA�;*8�ÓÁ=Iã�A�ÕFZ¡Z[;>ÓÁ×�ÒWA�?�8�UYÐ>A/HJ8]Qy×�ÑÔUOQY?69>=ÁZ]QYÐ=IQ�� Ý�=IÓÁÓ�8�Ï[;>A�A Ý�=I9>\ß9>\OA(9<;<8]=IQO=ÁQOÏÞKO8:9*8Ü8]QYK)9*8�`]A(Z]UO9>;*8�Ï]A�Z]UYÐ/Û:8�ÓÁUOA�Ð�ÒWA69rÝbA�A�Qß9>\YA9>;*8�=ÁQO=IQYÏ�Ð>8]H(ÕYÓIANÐ�D(�±ÑrM[Z]Q(9>\YA%Z�9<\OA�;�\F8�QYKpM��õ?�Z]Qy9<8]=IQFÐ�8�Ð�HJ8]ÓIÓGѶ8]H�=IÓÁ×�Z]Ñ^ÑÔUOQY?e9<=IZ[QYÐ�M]=I9=ÁÐ$H(Z[;>APÓI=Á`]A�ÓI×/9>\Y8�98�ÑÔUOQF?e9>=ÁZ]Q �"!ÞÝ�\O=@?*\½âO9*Ðp9>\YAVKO8�9<8�Ý�=ÁÓIÓyÒFAb8�Ï]Z¡ZGK�8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]QZ�Ñ ��D�Ù�\OA¥?�8]ÕY8]?�=|9r×·Z]ÑP9<\OA�Ѷ8�H�=ÁÓI×ßZ�ÑbÑÔUOQY?e9<=IZ[QYÐ�� =ÁнZ]ѳ9>A�Q°;<A6ÑÔA�;>;<A�K·9>Z·8]Ð�9<\OAJcCTKG=IH�A�QYÐ�=ÁZ]Q E G KOM7G MPH±D��±Ñ�8ôÓ@8�;<Ï]AÜ8]H�Z]UOQy9JZ�ÑCKY8:9<8ô=@ÐJ8�Û:8�=ÁÓÁ8]ÒOÓIA[M��4Ð�\YZ]UOÓ@K�?�Z]Qy9<8]=IQ 8ÓÁ8];>Ï[A�Ѷ8]H�=IÓÁ×ßZ]Ñ�ÑÔUOQY?e9<=IZ[QYÐdä¶\O=ÁÏ]\�cCT4KG=ÁH(A�QYÐ>=IZ[QFçeM�Ð�Z�9<\Y8:9�H(Z[;>A�ÑÔUYQY?e9<=IZ[QYÐ ?�8�Q¼ÒWA8�ÕOÕO;<Z�áG=IHJ8�9>A�K�M¡8]QYKÞ=IQ�ÕY8];�9<=Á?�UOÓ@8�;NM ��D��±ÑrMYZ]QÜ9>\YA�Z�9>\YA�;�\Y8]QYKpMG9<\OA�KO8:9*8�=ÁÐ�Ð<?�8];<?�A]M��Ð�\OZ[UOÓ@K�ÒWAV;<A�Ð�9>;<=Á?69>ANK½9>Z�8CÐ>HJ8�ÓÁÓ]Ѷ8]H(=ÁÓÁ×�Z�ÑOÑÔUOQF?e9>=ÁZ]QFÐbä¶ÓIZ:ݼcCT¼KG=IH�A�QYÐ�=ÁZ]QWçeMN9>Z�?6Z]Qy9<;>Z[Ó9>\OA½Û:8�ÓÁUOA�Ð�ÒWA69rÝbA�A�Qd9<\OAJäÔH�Z];<A½KG=ÁÐ�9<8]Qy9*çbÐ>8]H�ÕOÓIANÐ*L:DOÙ�\OA½cCT�KG=ÁH�A�QYÐ>=ÁZ]QÜ?�8�Q�8�Ó@Ð�Z(ÒWA?6Z]Qy9<;>Z[ÓIÓÁA�K�Ò¡×�ÕOUG9>9>=ÁQOÏÞ8¥`yQYZ]Ò·Z[Q·\OZ:Ý H UF?*\·A6à^A�?69C=@ÐCÏ[=IÛ[A�Q�9<ZÜÐ>Z]H�A ;<A�Ï[UOÓÁ8];>=Áã�A�;<Ð�D�OZ];�=ÁQYÐ�9<8�QF?6A�=|9�=@Ð�ÕFZyÐ>Ð>=IÒYÓIAC9>ZJ?6Z[Qy9>;<Z]ÓF9>\OA�?�8]ÕY8]?�=|9r×�Z]Ñ8�QOA�UO;*8�Ó^QOA�9rÝVZ[;>`�Ò¡×¥8]KYKG=IQYÏö�ÝbA�=ÁÏ]\y9CKGAN?�8�×GøJ8]ÐC8J;>A�Ï]UOÓ@8�;<=Áã�A�;ND*�ôA�=ÁÏ]\y9�KGAN?�8�×Þ=ÁÐC8JÓÁZyZyÐ�A�Ð�H�Z¡Z�9<\OQOA�Ð<Ð/?�Z]QYÐ�9>;*8�=ÁQy9Z]Q�9<\OA�?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]Q�ÑÔUOQY?e9<=IZ[Q�Ý�\O=Á?*\�ÝVZ[;>`GÐ�Ò¡×�KGA�?�;>AN8]Ð>=IQOÏ � # ��f�8]ÐÝVA�ÓIÓG8[Ð$9<\OA�A�;<;>Z[;Z]Q�9<\OAb9>;*8�=ÁQO=ÁQOÏ�Ð>A69ND�E¡=IQF?6Ab9>\OA�?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q�ÑÔUOQY?69>=ÁZ]Q(=ÁÐQOZ]9QYA�?6ANÐ>Ð<8�;<=ÁÓI×�Ð>H(Z¡Z]9>\�M�ÑÔZ[;=IQYÐ�9<8]QY?6A(8:9C8ÜKGA�?�=ÁÐ>=ÁZ]Q)ÒWZ]UOQYKY8�;<×]MO9<\OA ÝbA�=ÁÏ]\y9CKOA�?�8�×Þ;<A�Ï]UYÓÁ8];>=Áã�A�;/?�8]Q)\Y8�Û]A 8[KGÛ]A�;<Ð>AA6à^A�?e9*Ð�DR/Ð%H�A�Q[9<=IZ[QOA�K�AN8�;<ÓI=ÁA�;NMG9<\OA�;<A�Ï]UYÓÁ8];>=Áã�A�;%Ð�\OZ[UOÓ@K�;>A��YA�?69/=IQy9<A�;<A�Ð�9>=ÁQOÏJÕO;<Z]ÕWA�;>9>=ÁA�Ð�ä¯8

ÕO;>=ÁZ];<=$`¡QOZ:Ý�ÓÁA�KOÏ]ANç�Z�Ñ�9>\YA ÑÔUOQF?e9>=ÁZ]Q·9>ZdÒWA(ÓIAN8�;<QOA�KpD��±Ñ�9>\YA�ÑÔUOQY?69>=ÁZ]QYÐ�� 8]QYK � ! 8�;<A8]Ð<Ð�UOH�ANK�9>Z/ÒWAVKO=|à^A�;<A�Qy9>=@8�ÒYÓIA[M�Ý�\O=@?*\�=@Ð$Ï[A�QOA�;<8]ÓIÓÁ×%9<\OAb?�8]Ð>A]M�9>\OAbÐ�AN8�;*?*\CÑÔZ[; � ! ?�8]Q�ÒWA

g��bq��rm/�r~yk��V�r~yvxtb²@q�ori�k��xvxtui s*q��y�|�dk��xtuq(�Gm/�ytume��²Áq�o�i�m*i�q�ou¹��yk�tume�Jt±¹]t±�rm*i�µGh¯w¥�r~[mCs*k�tumº�~ym>orm�k��x���r~ym��uo�k�v|w[vxwylJ�[k���k�s*k�wÞ�Gm���m* [�%vxwÜi�m*i�q�ou¹:�G~[qeº�m*{�m>oe�Y�r~[m��V� �[vxi�m*w[tuv|q�wÜvxtvxw[¸ywyv��rmN�:k�w¡�C�r~[m�²@q�ori�k��xv|tui vxt�i�m*k�wyvxw[lN�xm*tut*µ���~[m��V�)�[vxi�m*w[tuvxqNw�vxtk��xmek6orwyvxwyl% yk�o�kN�]vxlNik�wy��vxt�wyq��V�[tum<²@�[�F�[wy�xm*tut��xmek�orw[vxwyl�vxt�v|w�{�q��x{�me�Yµ

Page 8: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ï];<A�8:9<ÓI× =IH�ÕO;<Z:Û]ANK Ò¡× ;<A�ùyUO=Á;<=IQOÏ�9<\Y8:9 � ! Ð�KGA�;>=ÁÛ:8:9>=ÁÛ]ANÐ�A�Û:8�ÓÁUY8�9>A�K�8�9�9>\OA%ÕWZ]=ÁQy9<Ð ! # � -8�;<AbH�Z[;>A�Z[;�ÓIANÐ>Ð�ANùyUY8�Ó$äÔ9>\O=@Ð�=@Ð�9<\OA�;>A�Ï]UOÓ@8�;<=Áã�A�;�`¡QOZ[ÒFç�9>Z½9<\OA/KGA�;>=ÁÛ:8:9>=ÁÛ]ANÐ�Z�Ñ �÷8:9�9>\YAÐ>8]H(A(ÕWZ]=ÁQ[9*Ð�ä��=IÏ[UO;<A��Þ;<=IÏ[\[9eçeDpÙ�\O=@н;>ANÐ�UYÓ|9�?�8]QõÒWA(A�á¡9>A�QFKGA�Kõ9>ZÞH UOÓI9>=@KG=ÁH�A�QYÐ>=IZ[QY8�Ó=IQOÕYUG9<Ð�D��uQÞ9>\Y=ÁÐC?�8]Ð>A]MGÝbA�?�8]Q�=IH�ÕWZ[Ð>A½9>\OA�ANù[UF8�ÓÁ=|9r×ÜZ�Ñ9>\OA KOA�;<=IÛ:8:9<=IÛ[A�Ð�Z]Ñ$� 8�QYK � !=IQ�é1'()eïuë:í³ñ 0:í ) '*é�ï¯í¶ð�ñOìeMYQYZ�9�QOA�?�A�Ð<Ð>8];>=ÁÓÁ×J=IQ)8�ÓÁÓ�KG=Á;>AN?e9<=IZ[QYÐ�Z�Ñ9>\OA�=ÁQOÕOUO9/Ð�ÕY8[?6A[DE¡UY?*\)?6Z[QYÐr9<;<8]=IQy9*Ð�âYQYK�=ÁH�H(ANKG=@8:9>A UYÐ�A =IQ�9<;<8[KG=|9<=IZ[QY8�Ó$ÓIAN8�;<QO=IQYÏ(ÕF8:9�9<A�;<Q);>AN?6Z[Ï]QO=IÖ

9>=ÁZ]Q(ÕO;>Z[ÒOÓÁA�HJÐ�D �±9�=@Ð�Z�ѳ9>A�Q 9<\OA%?�8[Ð�AV9>\F8:9/ë>8$)eí¶ð*)eí¡`¡QOZ:Ý�ÓIANKGÏ]AV=ÁÐ�8�Û:8]=IÓ@8�ÒOÓÁAbZ]Q \YZ:Ý�9>\YAKGA�Ð>=I;<A�K)ÑÔUOQY?e9<=IZ[QõÛ:8�;<=ÁA�Ð%Ý�=I9>\ß;>ANÐ�ÕWA�?69/9<ZÜÐ>Z]H�A�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[QYÐ%Z]Ñ�9>\OA�=IQYÕOUG9�D��±9�=ÁÐÐr9<;<8]=IÏ[\[9>ÑÔZ];<Ýb8];<K�9>Z KOA�;<=IÛ[A�9>\OA�?6Z];<;<A�Ð>ÕFZ[QYKG=ÁQOÏ�?6Z]QFÐr9<;<8]=IQy9�Z]Q(9<\OACKG=Á;<A�?e9<=IZ[QY8�ÓFKGA�;<=IÛ:8�Ö9>=ÁÛ]A�Ð$Z�Ñ¡9>\YA�âO9>9>A�K�ÑÔUOQY?e9<=IZ[Q � ! =IQ�9<\OAPKG=Á;<A�?e9<=IZ[QYÐ�Z]Ñ¡9>\OAP9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�äÔÕO;<A�Û¡=ÁZ]UYÐ>ÓI×QY8�H�A�K)9<8�QYÏ]A�Qy9�Û]A�?69>Z[;<Ð*çeDWÙPסÕO=@?�8]Ó$A6áO8]H(ÕYÓIANÐ/?�8]Q·ÒWA ÑÔZ[UOQYK)=ÁQ·ÕY8�9�9>A�;>Qõ;>AN?6Z]Ï[QO=I9>=ÁZ]QÝ�\OA�;<A(9>\OAJKGANÐ�=Á;>ANKß?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q·ÑÔUYQY?e9<=IZ[Qô=@н`¡QOZ:Ý�Qß9>Z�ÒFA�=ÁQ¡Û�8];>=@8�Qy9½Ý�=|9<\°;>ANÐ�ÕWA�?699>ZÜÐ>Z]H�A�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q�Z�Ñ�9<\OA(=IQOÕYUG9�Ð>UY?*\õ8[Ð�9>;*8�QFÐ�Ó@8:9<=IZ[Q�MY;<Z�9*8:9>=ÁZ]Q$MFÐ<?�8�ÓÁ=ÁQOÏYMFA69<?]DIM=IQ�Z]9>\OA�; ÝbZ];*KOÐ�M�9>\YAÜKG=Á;>AN?e9<=IZ[QY8�ÓbKGA�;<=IÛ:8�9>=ÁÛ]A�Ð Z�Ñ�9<\OAÜ?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[QôÑÔUOQY?e9<=IZ[Q�=ÁQ¼9>\YAKG=I;<A�?69>=ÁZ]QYÐ�Z]Ñ$9<\OA�Ð>A½9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐb=ÁÐ�ã�A�;<ZYDÙ�\O=ÁÐ =ÁÐ�=IÓÁÓIUFÐr9<;<8�9>A�K°=ÁQ �=IÏ[UO;<A��YD$Ù�\YA¥;>=ÁÏ]\y9 ÕF8�;>9�Z�Ñ�9>\OA¥âYÏ]UO;<A¥Ð>\OZ:Ý%Ð�\OZ:Ý 9>\YA

8]KOKG=I9>=ÁZ]QF8�Ó]?�Z]QYÐ�9>;*8�=ÁQy9<ÐpZ[Q��"!Ü\OA�ÓIÕ�Ï[A�QOA�;<8]ÓI=Áã�8�9>=ÁZ]Q�Òy×½?6Z[QYÐ�9>;*8�=ÁQO=IQYÏV9<\OAPÛ:8]ÓIUOANÐpZ�Ñ��"!Z]UG9*Ð�=@KGA�9<\OA�9<;<8]=IQY=IQOÏÜÐ�A�9�D��OZ];CA�Û[A�;<×d9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Q�Ý�\O=@?*\·\Y8]ÐC8¥`¡QOZ:Ý�Q)A�àWAN?e9CZ[Q9>\OA/?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q(ÑÔUOQY?69>=ÁZ]Q�M¡8½;>A�Ï]UOÓ@8�;<=Áã�A�;�?�8�Q�ÒWAC8]KYKGA�K(=ÁQJ9>\OA%ÑÔZ];<HÚZ]Ñp8�?�Z]QYÐ�9>;*8�=ÁQy9Z]Q�9>\YA�KG=Á;>AN?e9<=IZ[QY8�ÓKGA�;<=IÛ:8�9>=ÁÛ]A�Z�Ñ � ! =ÁQ)9>\OA(KG=Á;>AN?e9>=ÁZ]Q)Z]Ñ�9>\OA 9<8�QYÏ]A�Qy9/Û[A�?69>Z];�ä¶Ð>UY?*\8]Ðb9>\YA�Z]QOA�KGA�ÕO=Á?69>ANKÜ=ÁQ0�=IÏ[UO;>AOG[çeMY?�Z]H�ÕOUG9<A�KdÑÔ;>Z[H 9>\YA�?6UO;<Û]A�Z�Ñ9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q�DÙ�\OAVQYA6á¡9�Ð>A�?69>=ÁZ]Q�Ý�=IÓÁÓG8�QY8]ÓIסã�AV=IQ�KGA�9<8�=ÁÓ¡\OZ:Ý°9>Z�UYÐ>Ab8�KG=@Ðr9*8�QY?�AVÒF8]Ð>A�K�Z[Q�9<8]QOÏ]A�Qy9

Û]A�?69>Z[;V=ÁQdH�A�H�Z[;>×�ÒY8[Ð�ANKd8�ÓÁÏ]Z[;>=I9>\YH�Ð�D[Ù�\YA½Ð>UOÒYÐ>A�ùyUOA�Qy9bÐ>A�?e9<=IZ[QdÝ�=ÁÓIÓpKG=@Ð<?6UYÐ<ÐV9>\OA�UYÐ>AZ�ÑP9*8�QOÏ[A�Qy9½Û]A�?69>Z[;<Ð�=ÁQßQOA�UO;*8�Ó�QOA69rÝbZ];<`^M^Ý�=|9<\õ9<\OA�9<8]QOÏ]A�Q[9½ÕO;<Z]ÕY8]Ï[8�9>=ÁZ]Qõ8]ÓIÏ[Z];<=|9<\OH�DÙ�\OA�ÓÁ8[Ðr9%Ð>A�?69>=ÁZ]QÞÝ�=IÓÁÓ$?6Z[H�ÕY8�;<A�KO=|à^A�;<A�Qy9/8�ÓÁÏ]Z[;>=I9>\YH�ÐV9<Z¥?6Z]H�ÕOUO9>A�9<8]QOÏ]A�Q[9�Û[A�?e9<Z];*Ð�D

� ����Æ�����ÆPÇ� Í�[Ç���Æ/Ì �

Ù�\OA3#�UY?�ÓI=@KGA�8]Q.KG=@Ð�9<8�QF?6A�ÒFA�9rÝVA�A�Q.9rÝVZôÕY8�9�9>A�;>QFÐ � 8�QYK�� =ÁÐ�=ÁQ.Ï[A�QOA�;<8]Ó�QOZ�9¥8�ÕOÖÕO;>Z[ÕO;<=Á8�9>A½ÒFAN?�8]UYÐ�A�=I9/=@Ð%Ð>A�QYÐ>=|9<=IÛ[A½9>ZJ=Á;>;<A�ÓÁA�Û:8]Q[9�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐbZ�Ñ � 8�QYKÞZ]Ñ��JD��uQ?6Z]Qy9<;<8[Ðr9NMG9>\OA(KG=@Ðr9*8�QY?�A��Üä�������ç%KOA6âYQOANK�9>ZdÒFA 9>\OA�H(=ÁQO=ÁHJ8�Ó�KO=ÁÐ�9<8]QY?6A�ÒWA69rÝbA�A�Q)9>\YA9rÝVZ¥HJ8]QO=|ÑÔZ[ÓÁKYÐ ��� 8]QYK ��� =@Ð�9<;>UYÓI×�=ÁQyÛ:8];>=@8�Qy9%Ý�=I9>\õ;<A�Ð>ÕWA�?e9/9>Z¥9<\OA�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QUYÐ�ANK�9>Z�Ï]A�QOA�;*8:9<A ��� 8�QFK ��� ä¯Ð�A�A��=IÏ[UO;<A [ç6D��/QGÑÔZ];>9>UOQF8:9>A�ÓI×[M]9<\OA�Ð>A/HJ8]QO=|ÑÔZ[ÓÁKYÐP\Y8�Û[AQOZõ8�QF8�ÓÁ×[9<=Á?¥A6áGÕO;<A�Ð<Ð�=ÁZ]Q°=ÁQ�Ï]A�QOA�;*8�ÓjM$8�QFKôâYQYKG=ÁQOÏ·9<\OAdKG=@Ð�9<8�QF?6AdÒWA69rÝbA�A�Q°9>\OA�H =ÁÐ�8KG=9J¥?�UOÓ|9�Z]ÕG9<=IH�=Áã�8�9>=ÁZ]Q)ÕO;<Z]ÒOÓÁA�H Ý�=|9<\·H UYÓ|9<=IÕOÓÁA�ÓÁZG?�8�Ó�H�=IQO=ÁHJ8ODWaVANÐ�=@KGANÐ�MG9<;>UYA�=IQ¡Û:8�;<=IÖ8�QY?�A¥=ÁÐ QOZ�9�QOA�?�A�Ð<Ð>8];>=ÁÓI×ßKGA�Ð>=I;*8�ÒYÓIA¥Ð>=IQF?6Ad8);<Z�9*8:9<=IZ[QôZ]Ñ%8 ö>æ]ø�=ÁQy9>Zõ8 ö��]ø�KGZ¡ANÐ�QOZ]9ÕO;>ANÐ�A�;>Û[A/9<\OA�?6Z[;>;<A�?69�?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q�D

=CUO;$8�ÕYÕO;>Zy8]?*\½?6Z[QYÐ�=@Ð�9<ÐpZ]ÑG?�Z]H�ÕOUG9<=IQYÏ�9>\OAPH(=ÁQO=ÁH UOH KG=@Ð�9<8�QF?6A�ÒWA69rÝbA�A�Q½9<\OAPÓÁ=ÁQOA�8];Ð�UO;>Ѷ8]?�A�Ð�9>\Y8�9VÒWA�Ð�9b8�ÕYÕO;>Z�áG=ÁHJ8:9>A�9<\OA/QYZ]QGÖ±ÓI=ÁQOAN8�;PHJ8]QO=|ÑÔZ[ÓÁKYÐ � � 8]QYK � � D¡Ù�\O=@ÐPÐ>Z]ÓÁÛ]ANÐ9>\O;<A�AJÕO;<Z]ÒYÓIA�H�Ð�8�9�Z]QY?�A]û�å�ç�ÓI=ÁQOAN8�;�HJ8�QY=|ÑÔZ[ÓÁKOн\F8�Û]AJÐ�=ÁH�ÕOÓÁAd8�QY8]ÓI×y9<=Á?�8�Ó�A6áGÕO;<A�Ð<Ð>=IZ[QYÐÝ�\O=Á?*\#?�8]Q ÒFAôA�8[Ð�=ÁÓÁ× ?�Z]H�ÕOUG9<A�K 8�QYK Ðr9<Z];<A�KpM G]ç¥âYQYKG=ÁQOÏ.9>\OAôH�=IQY=IH�UOH KG=@Ðr9*8�QY?�AÒFA�9rÝVA�A�Q�ÓI=ÁQOAN8�;%HJ8�QY=|ÑÔZ[ÓÁKOÐ%=ÁÐ%8�Ð>=IH�ÕOÓÁA�ÓÁA�8[Ðr9%Ð<ù[UF8�;<A�Ð�ÕO;<Z]ÒOÓÁA�H Ý�\O=@?*\�?�8�Q�ÒWA�Ð�Z[ÓIÛ[A�KA(J¥?6=ÁA�Qy9>ÓÁ×�8�QFKpM� yç½9<\O=ÁÐ�KG=@Ð�9<8�QF?6Ad=@Ð(ÓIZG?�8�ÓÁÓI×°=ÁQyÛ:8];>=@8�Qy9 ÒOUO9(QOZ]9(Ï[ÓIZ[ÒY8�ÓÁÓÁ×ß=ÁQ¡Û�8];>=@8�Qy9NDÙ�\yUFÐ(9>\OA)KG=@Ðr9*8�QY?�A�ÒFA�9rÝVA�A�Q.8#ö<æ]øô8�QYK.8ôÐ>ÓÁ=IÏ[\[9<ÓI×¼;<Z�9*8:9<A�K ö<æ]øß=ÁÐJÐ>HJ8�ÓÁÓ�ÒOUG9J9>\YA

Page 9: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

KG=ÁÐ�9<8]QY?6AdÒFA�9rÝVA�A�Q�8�ö<æ]ø�8�QYK¼8 ö��]ø�=@Ð�Ó@8�;<Ï]A[D$Ù�\OAdKG=Ià^A�;<A�Qy9(KG=@Ð�9<8�QF?6A�Ð�ÒFA�9rÝVA�A�Q �8�QYK � 8�;<A�;<A�ÕY;>ANÐ�A�Q[9<A�KÜÐ>?*\OA�HJ8:9>=@?�8]ÓIÓÁ×¥=ÁQ0��=ÁÏ]UY;>A ODÙ�\OA�âYÏ[UO;<A�;>A�ÕO;<A�Ð>A�Qy9<Ð^9rÝbZ/ÕF8:9�9<A�;<QYÐ���8]QYK ��=ÁQ �Ö±KG=ÁH�A�QYÐ>=ÁZ]QY8]Ó[Ð>ÕY8]?�A]D�Ù�\YAPHJ8�QGÖ

=|ÑÔZ[ÓÁKOÐ�Ï[A�QOA�;<8�9>ANK�Ò¡× �%8];>Ab;<A�ÕO;<A�Ð>A�Qy9>ANK�Ò¡×�Z[QOA6ÖuKG=ÁH(A�QYÐ>=IZ[QY8�ÓG?6UY;>Û[A�ÐÏ]Z[=IQOÏC9>\Y;>Z[UOÏ]\ �8�QYK/�4;<A�Ð>ÕWA�?e9<=IÛ[A�ÓÁ×]DGÙ�\OA�ÓÁ=IQYA�8�;�8]ÕOÕO;<Z�á¡=ÁHJ8:9<=IZ[QYÐV9>Z(9>\OA�HJ8]QO=|ÑÔZ[ÓÁKYÐ�8�;<AC;<A�ÕO;<A�Ð>A�Qy9<A�KÒy×ÜÓI=ÁQOANÐ�9<8]QOÏ]A�Q[9�9<ZJ9>\YA�?6UO;<Û]ANÐ�8:9 � 8�QFK/��DWÙ�\OA�Ð>A�ÓÁ=IQOANÐ%KGZJQOZ]9/=ÁQ[9<A�;*Ð�AN?e9�=ÁQ JKG=|ÖH(A�QYÐ>=IZ[QYÐ�8�QYK�9<\OA/Ð>\OZ];>9>ANÐr9�KG=@Ðr9*8�QY?�A�ÒFA�9rÝVA�A�Q(9>\OA�H ä¶UOQO=@ùyUOA�ÓÁ×�KGA�âYQOANKYç�=@Ð��·ä�� �,��çeDÙ�\OA�KG=@Ðr9*8�QY?�A�ÒFA�9rÝVA�A�QÞ9>\OA�9rÝVZJQOZ[QGÖ±ÓI=ÁQOA�8];b9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q�?6UY;>Û[A�Ð �Üä � ����ç�=@Ð%8]ÓÁÐ>ZÐ�\OZ:Ý�QÞZ]QÜ9>\YA½âYÏ]UO;<A]DR%Q�A�JJ?�=IA�Qy9 =IH�ÕOÓÁA�H�A�Q[9*8:9<=IZ[Q�Z�Ñ�9>\YA¥9<8]QOÏ]A�Qy9�KG=@Ðr9*8�QY?�A��·ä�������ç�Ý�=ÁÓÁÓVÒWAdÏ[=IÛ[A�Q

=IQõ9>\OA�QOA�áy9�Ð>A�?e9<=IZ[Q�DpR%ÓI9>\OZ[UOÏ]\ß9>\YA 9*8�QOÏ[A�Qy9½KO=ÁÐ�9<8]QY?6A�?�8]QõÒWA(8]ÕOÕOÓÁ=IANK)9>Z�8�Q¡×)`¡=ÁQYKZ�Ñ�ÕY8:9>9>A�;<QYÐ�;<A�ÕY;>ANÐ�A�Q[9<A�KÜ8]Ð�Û[A�?e9<Z];*Ð�M¡ÝbA�\Y8�Û]A�?�Z]QY?�A�Qy9>;*8:9<A�KÜZ]UO;�A�àWZ[;�9*Ð�Z]Q)8�ÕOÕYÓI=@?�8�Ö9>=ÁZ]QYÐ�9>Z�=ÁH�8]Ï]A�;<A�?�Z]Ï]QY=|9<=IZ[Q�DyTVZ]H�ÕY8];>=@Ð>Z]Q(Z�ÑW9<8�QYÏ]A�Qy9VKO=ÁÐ�9<8]QY?6A�Ý�=I9>\�9<\OA/ÒWA�Ð�9P`¡QOZ:Ý�Q?6Z]H�ÕWA69<=IQOÏdH(A�9>\OZGK�Ý�=ÁÓIÓ�ÒFA�KGA�Ð<?6;<=IÒWA�K�D��=IQY8]ÓIÓÁ×ÞÝVA�Ý�=ÁÓIÓKG=@Ð>?�UYÐ>Ð%ÕFZyÐ>Ð>=IÒYÓIA�Û:8�;<=@8:9>=ÁZ]QFÐZ]Q·9<\OA�9<8�QYÏ]A�Qy9�KG=@Ðr9*8�QY?�A(8]QYK·\OZ:Ý =I9�?�8]QõÒWA�Ï[A�QOA�;<8]ÓI=Áã�ANK�9>ZÞÕO;<Z]ÒOÓÁA�HJÐCZ�9>\YA�;�9>\Y8]QÕY8:9>9>A�;<QÞ;>AN?6Z]Ï[QO=I9>=ÁZ]Q�D

��� �(������;� �$!P�6!� � �

�uQ 9<\O=ÁÐPÐ�AN?e9>=ÁZ]Q�ÝVA�KGANÐ>?�;>=ÁÒWAVÑÔZ[;>HJ8]ÓIÓÁ×½9>\OA�?�Z]H�ÕOUG9*8:9>=ÁZ]Q(Z�ÑF9>\OAb9*8�QOÏ[A�Qy9�KG=ÁÐ�9<8]QY?6A[D�S�A�99>\OAVÑÔUOQY?69>=ÁZ]Q ��9<;<8]QYÐ�ÑÔZ];<H÷8]Q�=ÁH�8]Ï]A � 9>Z �yä�������ç8]?�?6Z];*KG=ÁQOÏ�9<Z/9<\OAVÕF8�;*8�H�A69<A�; �VD �ôA;>ANù[UY=I;<A ��9<ZJÒFA KO=|à^A�;<A�Qy9>=@8�ÒYÓIA�Ý�=I9>\);<A�Ð>ÕFAN?e9�9<Z���8�QFK/��MW8�QYKÞ;<A�ùyUO=Á;>A �yä����<è[ç � ��D�±Ñ�� =ÁÐ�8 GdKG=IH�A�QYÐ�=ÁZ]QF8�Ó=IHJ8�Ï[A�ÑÔZ];�=ÁQYÐr9*8�QY?�A]M��[ä2������ç�?6Z]UYÓÁK)ÒWA�8d;>Z]9<8�9>=ÁZ]Q)Z�Ñ � Ò¡×9>\OA½8�QOÏ[ÓIA �PD�±Ñ�ÝbA�8];>A/=IQy9>A�;>ANÐr9<A�KJ=ÁQÜ8]ÓIÓF9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[QYÐ�Z]Ñ�=IHJ8]Ï]A�ÐPÝ�\O=@?*\Ü?6Z[QYÐ>A�;<Û]AKG=ÁÐ�9<8]QY?6ANÐ%äÔ=@Ð>Z]H�A69<;>×OçeM �[ä2������ç�ÝbZ]UOÓ@K(ÒFA/8�;<Z�9*8:9>=ÁZ]Q�Ò¡× ���bÑÔZ]ÓÁÓÁZ:ÝVANK�Ò¡× 8�9<;<8]QYÐ>ÓÁ8�9>=ÁZ]QÒy× � �,����Z]Ñp9>\OA�=ÁH�8]Ï]A ��D �uQd9>\O=@Ð�?�8[Ð�A1� � ä2� � ��� �,����çP=ÁÐ�8 Û]A�?69>Z[;PZ�Ñ�ÕY8�;*8�H�A69<A�;*ÐZ�Ñ�KG=IH�A�QYÐ�=ÁZ]Q OD*�uQ�Ï[A�QOA�;<8]Ó¯M*� � ä�� L ������� ��� ½çb=ÁÐ�Z]Ñ�KG=ÁH�A�QYÐ>=ÁZ]Q��ßDE¡=IQF?6A&�)=@Ð¥KG=Ià^A�;<A�Qy9>=@8�ÒOÓÁA]M�9<\OAõÐ>A69 � � � ! # %�' �÷ÑÔZ[;%Ý�\O=@?*\ # � �yä����,�ç.-õ=ÁÐ8�KG=IàWA�;>A�Qy9>=@8�ÒOÓÁAõHJ8�QY=|ÑÔZ[ÓÁK�Ý�\Y=Á?*\ ?�8�Q ÒWA°8�ÕOÕO;<Z�áG=IHJ8�9>A�K 9<Z�9<\OA·âY;*Ð�9ÞZ[;<KGA�;¥Ò¡×�8\yסÕWA�;<ÕOÓ@8�QOA � � DFÙ�\Y=ÁÐ�\¡×¡ÕWA�;<ÕOÓÁ8]QOA�=@Ð�9<8]QOÏ]A�Qy9�9>Z � � 8�91� 8]QYKÞ=ÁÐ/Ï]A�QOA�;*8:9<A�KÜÒy×Ü9>\YA?6Z]ÓÁUOH�QYÐ�Z]ÑHJ8:9<;>=Iá

� � � � �yä����,�ç� �

���� � ��� ��� � �yä2������ç� � L

�������$� � �yä����,�ç� � � � ��� ä yç

Ý�\O=Á?*\°8�;<A(Û]A�?69>Z[;<ÐC9<8]QOÏ]A�Qy9�9<ZÞ9>\OAJHJ8�QY=|ÑÔZ[ÓÁKpD��±Ñ �18�QYK � 8];>A�9rÝVZ�ÕY8�9�9<A�;<QYÐ�9>Z�ÒWA?6Z]H�ÕY8];>ANKpM�9<\OA/;<A�Ð>ÕWA�?e9<=IÛ[A�9<8]QOÏ]A�Qy9VÕOÓ@8�QOANÐ�� � 8]QYK � � ?�8�QdÒFACUYÐ�ANK�9>Z�KGA6âYQYAC8�QYA�ÝKG=ÁÐ�9<8]QY?6A�� ÒWA69rÝbA�A�QÜ9>\YA�Ð>A½9rÝVZ(ÕY8:9>9>A�;>QYÐ�DOÙ�\OA½9*8�QOÏ[A�Qy9%KG=ÁÐ�9<8]QY?6A��·ä � ����çbÒWA69rÝbA�A�Q� 8]QYK/�4=@Ð�KGA6âFQOA�KÞÒ¡×

�·ä � ����ç � H�=IQ������ � ������� � #! #" � f ä �¡ç

Ù�\OA�A�ùyUY8�9>=ÁZ]QÞZ�Ñ$9<\OA½9<8]QOÏ]A�Qy9�ÕYÓÁ8]QOA�Ð�� � 8�QYK � � =@Ð�Ï]=ÁÛ]A�QÜÒ¡×^û�%$jä2� � ç � � &� � � � ä�I[ç� $ ä2� � ç � � &� � � � ä¶æyç

Page 10: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ý�\OA�;<A � � 8�QYK&� � 8�;<A¥9<\OAdHJ8�9>;<=Á?�A�Ð(?6Z]Qy9*8�=ÁQO=IQYÏ)9>\OAÜ9<8]QOÏ]A�Qy9�Û]AN?e9<Z];*Ðdä¶Ð>A�AÜA�ùyUY8�Ö9>=ÁZ]Q ä yç�ç�8]QYKß9>\OAdÛ]A�?69>Z[;<Ð � � 8�QYK � � 8�;<A�9>\OAÜ?6Z¡Z];*KG=ÁQY8:9<A�нZ�Ñ�� $ 8�QYK � $ äÔUYÐ>=IQYÏÒY8]Ð>A�Ð � � 8�QFK � � ç½=ÁQ¼9>\YAÜ?6Z[;>;<A�Ð>ÕWZ]QYKG=ÁQOÏÞ9<8]QOÏ]A�Qy9 ÕOÓ@8�QOANÐ�D�B%Z�9<AJ9>\F8:9�� $ M��JM�� �8�QYK/� � KGA�QOZ�9<A½Û]A�?69>Z[;<Ðb8�QYKÜHJ8:9<;>=@?6ANÐ�=IQÞÓI=ÁQOAN8�;�A�ùyUY8�9>=ÁZ]QYнä I]çeD �YZ];�A�áG8]H�ÕOÓIA[MG=|Ñ9>\YAÕO=|áGA�ÓPÐ�ÕF8]?6AJÝ�8]нZ]Ñ%KG=IH�A�QYÐ�=ÁZ]Q�IOM$8�QYKß9<\OA�;<AJÝVA�;>A�9rÝbZÞ9<8]QOÏ]A�Q[9�Û[A�?69>Z];*Ð�MpÝVA¥?�Z]UOÓ@K;>A�Ý�;>=I9>A½A�ùyUY8�9>=ÁZ]Q°ä�I[ç�8]Ð �����

�� $L� $f� $�� $�� $�

������� �

������� L� f� �� �� �

�������

������� L>L � Lrf� f*L � f>f� � L � � f� � L � � f� � L � � f

������� � � L� f � ä M]ç

Ù�\OA�ùyUY8�Qy9>=I9>=ÁA�Ð � � 8�QYK � � 8];>A�8:9>9>;<=IÒYUG9>ANÐ�Z�Ñ9>\OA�ÕF8:9�9<A�;<QYÐ%Ð>Z�=IQ�HJ8�Q¡×Ü?�8]Ð>A�Ðb9<\OA�×?�8�QÞÒWA�ÕO;>AN?6Z[H(ÕYUG9>ANKÞ8]QYKÞÐr9<Z];<A�KpDTVZ]H�ÕOUG9<=IQOÏ(9<\OA½9<8]QOÏ]A�Q[9%KG=@Ð�9<8�QF?6A�·ä � ����ç � H�=ÁQ� � � � � � � $¯ä2� � ç ��$jä�� � ç � f ä�Kyç

8�H�Z]UOQy9*Ð%9<Z�Ð>Z]ÓÁÛ¡=IQYÏÜ8¥ÓÁ=ÁQOA�8];�ÓÁA�8[Ðr9�Ð<ù[UF8�;<A�Ð%ÕY;>Z[ÒOÓIA�H�D^Ù�\YA�Z[ÕG9>=ÁHJ8�ÓÁ=|9r×·?�Z]QYKG=I9>=ÁZ]Qß=ÁÐ9>\Y8�9 9>\YAdÕY8];�9<=Á8]ÓVKGA�;>=ÁÛ:8:9>=ÁÛ]ANÐ�Z]Ñ �·ä�� �,��ç�Ý�=I9>\.;>ANÐ�ÕWA�?69�9>Z � � 8]QYK � � Ð�\YZ]UOÓ@K¼ÒWAã�A�;<ZYû

� �õä�� �,��ç� � �

� GYä�� $jä�� � ç ��$¯ä2� � ç�ç � � � è ä��yç

� �·ä�� �,��ç� � �

� GYä�� $ ä�� � ç � $ ä2� � ç�ç � � � è ä�å�èyç

E¡UOÒYÐ�9>=I9>UG9<=IQYÏ � $ 8�QYK � $ Ò¡× 9<\OA�=Á;PA�á¡ÕY;>ANÐ>Ð>=IZ[QYÐ�ס=IA�ÓÁKYÐ�9>Z�9>\OA/ÑÔZ]ÓÁÓIZ:Ý�=ÁQOÏ�ÓÁ=IQOAN8�;PÐ>×GÐr9<A�HZ�Ñ�A�ùyUY8:9<=IZ[QYÐ�MyÝ�\Y=Á?*\�ÝbA�H�UYÐ�9/Ð�Z[ÓIÛ[A/ÑÔZ[; � � 8�QYK�� � û� � ä�� � � � � � &� � � � ç � è ä�å]å�ç��� ä�� � � � � � &� � � � ç � è ä�åPG[çÙ�\OA�Ð�Z[ÓIUO9>=ÁZ]QÞZ�Ñ$9<\O=@Ð%Ð�×GÐ�9>A�H =@Ð

ä � � � � � L� � � � � � ç6ä � ��ç � ä � � � ��� L� � � � � � ��� ç � � ä�å yçä � � � � � L��� ��� ��� ç6ä � ��ç � ä � � � � ��� � � L��� � � � ç � � ä�å �¡ç

Ý�\OA�;<A � � � � � � � � M � � � � � � � � M � � � � � � � � 8�QYK � ��� � � � � � DPS �KGA�?�Z]H�ÕFZyÐ�=I9>=ÁZ]QFÐPZ]Ñ � � � 8�QYK � ��� ?�8�Q¥ÒWACÕY;>AN?6Z]H�ÕOUO9>A�K�DyÙ�\OACH�Z[Ð�9bA�á¡ÕWA�QFÐ�=ÁÛ]A%ÕF8�;>9=IQõÐ>Z]ÓÁÛ¡=IQYÏJ9>\O=@нÐ�×GÐ�9>A�H =@ÐCA�Û:8�ÓÁUY8:9<=IQYÏ�� � � ä � � � ?�8]Q·ÒFA(Z]ÒO9<8�=ÁQOANK)Ò¡×d9<;<8]QYÐ�ÕWZ[Ð>=ÁQOÏ� ��� ç6D��±9C;<A�ùyUO=Á;>ANÐ � ��� � � KGZ�9½ÕO;<Z¡KOUY?e9*Ð�MWÝ�\OA�;<A%� � =@Ð%9<\OA(QyUYH ÒWA�;�Z�Ñ�9<8]QOÏ]A�Qy9Û]A�?69>Z[;<Ð�ÑÔZ[; �18�QYK � � =@Ð�9<\OA¥Q¡UOH�ÒFA�;�Z�Ñb9<8]QOÏ]A�Q[9�Û[A�?69>Z];*Ð�ÑÔZ]; ��D�=CQY?6A � ��� \Y8[ÐÒFA�A�Q ?�Z]H�ÕOUG9<A�KpM � � 8]QYK � � ?�8�Q ÒWA°?6Z]H�ÕOUO9>A�K Ò¡×�Ð�Z[ÓIÛ¡=ÁQOÏ�9rÝbZ#ä¯Ð�HJ8]ÓIÓ³ç¥ÓÁ=ÁQOA�8];Ð�×GÐ�9>A�HJÐPZ�Ñp;>ANÐ�ÕWA�?69>=ÁÛ]A�ÓI× � � 8]QYK � � A�ùyUY8:9<=IZ[QYÐ�D[Ù�\OA�9*8�QOÏ[A�Qy9VKG=@Ð�9<8�QF?6A%=@Ð�Z[ÒG9<8]=IQOANKÒy×�?6Z[H�ÕOUG9>=ÁQOÏ!� � $ ä2� � ç � $ ä2� � ç ��UYÐ>=IQYϽ9>\OA%Û:8]ÓIUOA/Z�Ñ�� � 8]QYK � � =IQJANù[UF8:9>=ÁZ]QFÐ%ä�I[ç8�QYK ä¶æ[ç6D��±Ñ ��=ÁÐ�9<\OAÞKO=IH�A�QFÐ�=ÁZ]Q.Z�Ñ%9<\OAÜ=ÁQOÕOUO9�Ð>ÕY8]?�Aßä¶=¯D A]D�9<\OAÜÓÁA�QYÏ�9>\.Z�Ñ%Û[A�?e9<Z]; �8�QYK ��ç6MY9<\OA�8�ÓÁÏ]Z];<=I9>\OH1KGA�Ð<?6;<=ÁÒFANK·8�ÒWZ:Û]A�;<A�ùyUO=Á;<A�Ð�;<Z]UOÏ[\OÓÁ× ��ä � � å�ç6ä � � å�ç Yä � �

� � �� ç�H�UOÓI9>=ÁÕOÓI×yÖu8]KOKYÐ�D:R%ÕYÕO;>Z�áG=ÁHJ8:9>=ÁZ]QFÐp9>ZC9>\YAP9*8�QOÏ[A�Qy9�KO=ÁÐ�9<8]QY?6A�?�8]Q�\OZ:ÝbA�Û]A�;ÒFA�?�Z]H�ÕOUG9<A�KÞH�Z];<A�A�J¥?6=ÁA�Qy9>ÓÁ×]D

Page 11: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

�� � �� ���������!��*�6!� ���3�*���� ��!��� �� ��� � �����6�* ������ Ù�\YA.ö>ÓIZG?�8]ÓV=ÁQ¡Û:8�;<=Á8]QY?6ANf6ø·Z]Ñ/9<8]QOÏ]A�Qy9(KG=@Ðr9*8�QY?�AÞ?�8�Q.ÒFAÞ=IÓÁÓÁUYÐrÖ9>;*8:9>ANK¥Ò¡×�9>;*8�QFÐrÑÔZ[;>H�=ÁQOÏ�8�;>A�ÑÔA�;<A�QY?�A/=ÁHJ8�Ï]A�Ò¡×JÛ:8�;<=IZ[UYÐV8]H�Z]UOQy9<Ð�8]QYK¥H�A�8[Ð�UY;>=ÁQOÏ =I9<ÐKG=ÁÐ�9<8]QY?6A�9<Z¥8(Ð>A69%Z]Ñ�ÕO;<Z�9<Z�9rסÕWA�Ð�D

−6 −5 −4 −3 −2 −1 0 1 2 3 4 5 6−6 −5 −4 −3 −2 −1 0 1 2 3 4 5 60

2

4

6

8

10

12

0

2

4

6

8

10

12

0 1

2

3

4

5

6

7

89

0

1

2

3

4

5

6

7

89

Euclidean Distance Tangent Distance

QSRUT ®��Y® ���ys*�xv|�[m*k�w�k�w¡����k�w[lNm*w:���[vxt±��k�w[s*m*t��Gm<�¯º�m*m<wd�*�%�¯¹[ [vxsek��G~yk�w¡��º�orv��u�rm*w�vxi�k�lNm*t�q�²F�r~[m�*���[vxl�vx�rtd¾±�e�Üs*�[or{�m<t�¿½k�w¡�ßk��[vxl�vx� vxi�k�l�m��uo�k�w[tu�Ik6�rme�ô~[q�orvx�*qNw:��k��|��¹ß�:¹ß{�k�orvxq��yt k�i�q��yw:�rt¾³vxwy�[vxsek��rm*�õvxwõk��[t±¹[tutrk��$v|w· [vx¢]m*�I¿>µ� bw)�r~ym�¸¡l��[orm��r~ym��uo�k�w[tu�Ik6�rme�ßvxi�k�lNm v|t½�r~[m(�]vxlNv��)±��òIorqNi �r~ymC�e�½�]v|l�v��rtbk��Gq�{NmNµ

=CQ �=IÏ[UO;>A IGMWå�è 9rסÕO=@?�8�ÓW\Y8]QYKGÝ�;<=|9>9>A�QÞKG=ÁÏ]=I9b=ÁHJ8�Ï]ANÐV8];>A�Ð�A�ÓIAN?e9<A�Kõä¶ÒFZ]9�9<Z]H¥çeMO8]QYKZ]QOAÞZ�ÑC9>\OA�H��ô9<\OA)KG=ÁÏ]=I9ôö� ]ø��¼=@ÐJ?*\OZ[Ð>A�Q¼9<Z°ÒFAÜ9>\OA�;<A6ÑÔA�;>A�QY?6A[D�Ù�\YAÞ;<A6ÑÔA�;>A�QY?6AÜ=ÁÐ9>;*8�QYÐ>ÓÁ8�9>ANK¥ÑÔ;<Z]H æ�ÕO=|áGA�ÓÁÐ�9<Z�9>\OA�ÓÁA6ѳ9NMO9<Z¥æ�ÕO=|áGA�ÓÁÐ�9<Z�9>\OA�;<=ÁÏ]\y9�MY8]QYKÜ9>\OA½9<;<8]QYÐ>ÓÁ8�9>=ÁZ]Q8�H�Z]UOQy9�=@Ð�=ÁQYKG=@?�8:9<A�K°=ÁQ�8�ÒYÐ>×GÐ>Ð<8OD #V8]?*\°?6UO;<Û]A�;<A�ÕO;<A�Ð>A�Qy9*Ð�9>\OA #PUY?6ÓÁ=@KGA�8]Q°_C=@Ðr9*8�QY?�AäÔZ];�9<\OA Ù8�QOÏ[A�Qy9%_/=@Ð�9<8�QF?6ANç�ÒFA�9rÝVA�A�QÞ9>\OA¥äÔ9>;*8�QYÐ>Ó@8:9>ANKYçb;<A6ÑÔA�;<A�QF?6A�8�QYKÞZ[QOA½Z�Ñ9>\OA¥åNèKG=IÏ[=|9*Ð�DE¡=IQF?6A�9>\YA¥;>A�ÑÔA�;<A�QY?�A�Ýb8[Ð�?*\OZ[Ð>A�QôÑÔ;<Z]H 9<\OA)å�è)KO=IÏ[=|9*Ð�M�=I9�=@Ð�QOZ]9 Ð�UO;<ÕO;<=ÁÐ>=ÁQOÏÞ9>\Y8�9

9>\OA)?6UY;>Û[AÞ?�Z];<;>ANÐ�ÕWZ]QYKO=IQOÏõ9>Zß9>\OA)KG=ÁÏ]=I9¼ö [øõÏ[ZyANÐ�9>Z°èõÝ�\OA�Q�9<\OA);<A6ÑÔA�;<A�QF?6Ad=@Ð�QOZ]99>;*8�QYÐ>ÓÁ8�9>ANK�ä¶èÜÕO=IáGA�Ó9>;*8�QYÐ>Ó@8:9>=ÁZ]QWçeD��±9�=Áн?6ÓÁA�8];/ÑÔ;<Z]H19<\OA(âYÏ]UY;>A�9>\Y8�9�=IÑP9<\OA�;>A�ÑÔA�;<A�QY?�A� �Fq:s*k���vxw]{�k�orv|k�wys*m orm<²@m<ort½�rqdvxw�{Nk6orvIk�wys*m�º�vx�r~·orm*tu Gm*s>�½�rqÜtui�k��x���uo�k�wyt±²Áq�ori�k��rvxq�wytJ¾³v³µ mNµ�korq���k6�rvxqNwõq�²�k¥{�m<ou¹Þtui�k��x��k�w[lN�xm6¿>µ$hjw·s*q�w:�uo�k�t±�e��lN�xq��¡k���vxw�{Nk6orv|k�wys<m�orm<²Ám<ort½�rqÜvxw�{�k�orv|k�w[s*mº�v��r~°orm*tu Gm*s<���rq·k�or�[vx�uo�k6orvx�x¹°�Ik6orlNmd�uo�k�wyt±²Áq�ori�k6�rv|q�wytܾ³vÔµ mNµ�k)orq���k��rvxqNw�q�² ���N�·�[m<l�orm*m*t�¿>µ� �xqN�yk���vxw]{�k�orv|k�wys*m/vxt�wyq����[m*tuv�o�k��y�xm½vxwd�[vxlNv���orm*s<qNlNw[v��rv|q�wF�Ytuvxwys*mCº�m�w[m*me���rq �[vxt±�rvxwyl��yvxtu~Â���ÃNtVk�wy�ßÂ���ÃNt*µ

Page 12: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

ä³9>\YAC=ÁHJ8�Ï]Adö ]øyç�=@Ð�9<;<8]QYÐ�Ó@8:9<A�K�Ò¡×�H�Z];<A�9>\Y8]Q G�ÕY=|áGA�Ó@Ð�M]9<\OA�#PUY?6ÓÁ=ÁKOA�8�QÜ_/=@Ðr9*8�QY?�A/Ý�=ÁÓIÓ?6Z]QOÑÔUYÐ�A½=I9�Ý�=|9<\ÜZ]9>\OA�;�KO=IÏ[=|9*Ð�MGQY8]H�A�ÓÁ×¼ö K]ø�Z];�ö1I�øOD �uQ�?6Z]Qy9<;<8[Ðr9NM[9<\OA�;<A�=ÁÐ�QOZ(ÕWZ[Ð<Ð�=ÁÒOÓÁA?6Z]QOÑÔUYÐ�=ÁZ]Q�Ý�\OA�Q·Ù�8]QOÏ]A�Qy9/_/=@Ðr9*8�QY?�A�=@Ð�UYÐ>A�KpDWR/Ð/8�HJ8�9�9>A�;%Z�Ñ�Ѷ8[?e9�MF=IQÞ9<\O=ÁÐ/A6áO8�H�ÕOÓÁA]M9>\OAJÙ8�QOÏ[A�Qy9�_C=ÁÐ�9<8]QY?6A�?6Z[;>;<A�?69>ÓÁ×�=ÁKGA�Qy9>=IâYA�Ð�9<\OA�;<A6ÑÔA�;>A�QY?6A(UOÕõ9>Z�8¥9<;<8]QYÐ�Ó@8:9<=IZ[Q·Z�Ñ IÕO=|áGA�ÓÁÐ��]E¡=ÁH(=ÁÓ@8�;�?6UO;<Û]ANÐÝVA�;>AbZ]ÒG9*8�=ÁQOA�K Ý�=|9<\J8�ÓÁÓ[9<\OA�Z�9<\OA�;9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐbäÔ;<Z�9*8:9>=ÁZ]Q$MÐ>?�8�ÓÁ=IQOÏFMGA69*?�DÁDID�çeDÙ�\OAJö�ÓÁZG?�8�Ó@ø�=IQ¡Û:8�;<=@8�QY?�AbZ]Ñ�Ù�8]QOÏ]A�Q[9PKO=ÁÐ�9<8]QY?6A�Ý�=I9>\J;<A�Ð>ÕFAN?e9�9<Z Ð�HJ8]ÓIÓG9<;<8]QYÐ�ÑÔZ];<H�8�Ö

9>=ÁZ]QYÐ/Ï]A�QYA�;*8�ÓÁÓI×¥=ÁH�ÕOÓÁ=IANÐ%H�Z];<A�8]?�?6UO;*8:9<A�?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]QÜÑÔZ];%H�UY?*\)Ó@8�;<Ï]A�;�9<;<8]QYÐ�ÑÔZ];<H�8�Ö9>=ÁZ]QYÐ�DYÙ�\O=@Ð�=ÁÐ�9>\OA�Ð>=IQYÏ]ÓÁA�H�ZyÐr9�=ÁH(ÕWZ];>9<8]Qy9bÑÔAN8:9<UO;>A�Z]ÑÙ8�QOÏ[A�Qy9%_/=@Ð�9<8�QF?6A]D

Original

Tangent vectors Points in the tangent plane

QSRUT ®��Y® �Fm>²Á� � VorvxlNvxw¡k���vxi�k�lNmNµ �(v|�y�]�xm�������k�w[lNm*w:��{Nm*s<�rq�ortbs*q�ouorm*tu Gq�w¡�]v|w[l�orm*tu Gm<s<�rvx{�m*��¹¥�rq�r~[m��J�uo�k�wyt±²Áq�ori�k6�rv|q�wyt ��tusek��xvxwyl]�orq���k��rvxqNwY��m>¢[ yk�w[tuv|q�wôq�²��r~ym�� k6¢[vxt½º�~[v|�xmJs*q�i� [orm*tutuvxw[l�r~[m��k6¢[vxt*�ym<¢] ¡k�w[tuvxqNw�q�²^�r~ym�¸[ort±�b�]v|k�lNq�w¡k��Fº�~yvx�xm%s*qNi� ]orm*tutuvxwyl��r~[m�tum*s*q�w¡���]vIk�lNq�w¡k��Wk�wy��r~[vxs>��m<wyvxwyl]µG}Vvxl�~�� �Y�N¤� Gq�vxw��rtbv|w¥�r~ym%��k�w[lNm*w:�btu ¡k�s<m�lNm<wym<o�k6�rme�d�:¹¥k��y�]v|w[l�q�o�tu�y�]�uo�k�s<�rvxw[lm*k�s�~�q�²^�r~[m�/��k�w[lNm*w:��{�m<s<�rq�ort*µ

Ù�\OAPÓÁZG?�8�ÓÁ=I9r×�Z�ÑG9>\YAV=ÁQ¡Û�8];>=@8�QF?6AP\F8]Ð8�QOZ]9>\OA�;�=ÁH�ÕFZ[;�9*8�Qy9ÒWA�QYA6âO9Nû�S$ZG?�8�Óy=ÁQyÛ:8];>=@8�QY?�A?�8�Q�ÒWAbA�QGÑÔZ[;<?�A�K½Ý�=I9>\��*'(),+�ò('� J9<8]QOÏ]A�Q[9�Û]A�?69>Z[;<Ð�D�Ù�\OAP;>AN8]Ð>Z]Q�=@Ðp9>\F8:9$ÑÔZ[;�=ÁQGâYQY=|9<A�Ð>=IHJ8�ÓäÔÓÁZ¡?�8�Ó³çC9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ�MF9>\YA�;<A(=@Ð�8�KG=Á;>AN?e9�?6Z[;>;<A�Ð>ÕWZ]QYKGA�QY?6A � ÒWA69rÝbA�A�Qô9>\OA(9<8]QOÏ]A�Qy9Û]A�?69>Z[;<ÐpZ]ÑG9>\OAV9<8]QOÏ]A�Qy9$ÕOÓ@8�QOAb8�QYK½9<\OAPÛ:8];>=ÁZ]UYÐ�?6Z[H(ÕWZ[Ð>=I9>=ÁZ]QYÐ$Z�ÑG9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[QYÐ�D �OZ[;A6áO8�H�ÕOÓÁA]M�9>\OAb9<\O;>A�A�9*8�QOÏ[A�Qy9�Û[A�?e9<Z];*Ð�ÑÔZ[;��/Ö¯9<;<8]QYÐ>ÓÁ8�9>=ÁZ]Q�M�B/Öj9>;*8�QYÐ>ÓÁ8�9>=ÁZ]Q 8]QYK�;<Z�9*8:9>=ÁZ]QFÐZ�Ñ$?6A�Qy9>A�; (M¡Ï]A�QOA�;*8:9<A/8�9*8�QOÏ[A�Qy9VÕOÓ@8�QYAC?6Z[;>;<A�Ð>ÕWZ]QYKG=ÁQOϽ9<Z�8�ÓÁÓF9>\OA�ÕFZyÐ>Ð>=ÁÒOÓIA�?6Z[H(ÕWZ[Ð>=IÖ9>=ÁZ]QYÐCZ�Ñ�\OZ[;>=Áã�Z[Qy9<8�Ó$9>;*8�QYÐ>Ó@8:9>=ÁZ]QFÐ�MFÛ]A�;>9>=@?�8]Ó�9<;<8]QYÐ�Ó@8:9<=IZ[QYÐ/8]QYK);<Z�9*8:9>=ÁZ]QFÐ�Z�ÑV?�A�Qy9>A�;� (DÙ�\OAP;<A�Ð>UOÓI9>=ÁQOÏ%9*8�QOÏ[A�Qy9KG=@Ð�9<8�QF?6AP=@Ð$9>\OA�Q ÓÁZ¡?�8�ÓÁÓI×�=ÁQyÛ:8];>=@8�Qy9�9<Z(ë:ê³ê�9>\YA�9>;*8�QFÐ�Ó@8:9<=IZ[QYÐ$8]QYKë:ê³ê:9<\OA/;<Z�9*8:9>=ÁZ]QFÐ�äÔZ�Ñ�8]Qy×�?6A�Qy9>A�;eçeD ��=ÁÏ]UY;>A/æ�ÑÔUY;�9<\OA�;V=IÓÁÓIUFÐr9<;<8�9>A�Ð�9>\O=@ÐPÕY\OA�QOZ[H�A�QOZ[Q(Ò¡×KG=ÁÐ>ÕOÓ@8�ס=IQOÏ)ÕWZ]=ÁQy9<Ð =IQ°9<\OA¥9*8�QOÏ[A�Qy9�ÕOÓ@8�QOA¥Ï[A�QOA�;<8�9>ANKõÑÔ;<Z]H2Z]QYÓI× IÞ9<8]QOÏ]A�Q[9�Û[A�?69>Z];*Ð�D��k�w�vxtuqNi�q�or [~yvxtui k�s>�r�¡k��x��¹:�¡tum<mJ±�Fvxm%k��|l�m*�[o�k�Ã�v|w�� ���

Page 13: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

#P8]?*\õZ�Ñ�9<\OA�Ð>A�=IHJ8�Ï[A�ÐCÓIZ¡Z[`¡Ð�ÓÁ=I`[A =I9�\Y8]Ð�ÒWA�A�QôZ]ÒO9<8�=ÁQOANK·Ò¡×·8�ÕOÕYÓIס=ÁQOÏÜÛ:8];>=ÁZ]UYÐ�?6Z[H(ÖÒO=IQF8:9>=ÁZ]QFÐ�Z�Ñ%Ð<?�8�ÓÁ=ÁQOÏYM�;<Z�9*8:9<=IZ[Q�Mp\OZ];<=Áã�Z]Qy9*8�Ó�8�QFK°Û]A�;>9>=@?�8]Ó�Ð�`[A�Ý�=ÁQOÏYM�8�QYKô9<\O=Á?*`[A�QO=ÁQOÏFDB�A69�MY9>\OA½9*8�QOÏ[A�Qy9%KG=@Ðr9*8�QY?�A�ÒFA�9rÝVA�A�Q)8�Q¡×¥Z]Ñ�9<\OA�Ð>A�ÕFZ[=IQy9*Ð%8�QYKd9>\YA�Z];<=IÏ[=IQF8�Ó^=IHJ8]Ï]A½=ÁÐèOD

� ��� ���5�* !�!P ��� ��� �!�� � �� �� !P � � #�áGÕWA�;<=IH�A�Qy9*нÝVA�;>A�?6Z[QYKGUY?69>A�Kß9<ZÞA�Û:8�ÓÁUY8:9<A9>\OACÕFA�;�ÑÔZ[;>HJ8�QF?6A�Z�Ñ^9*8�QOÏ[A�Qy9VKG=@Ðr9*8�QY?�A�ÑÔZ];V\Y8�QFKGÝ�;>=I9�9<A�Q¥KG=ÁÏ]=I9V;<A�?�Z]Ï[QO=|9<=IZ[Q�D]R/Q¥=IQy9<A�;>ÖA�Ð�9>=ÁQOÏJ?*\Y8�;*8]?69>A�;<=@Ðr9<=Á?/Z�ÑKO=IÏ[=|9�=ÁH�8]Ï]ANÐ�=ÁÐb9<\Y8:9�ÝbA½?�8�QÞ;>AN8]KG=ÁÓÁ×J=ÁKGA�Qy9>=IÑÔ×Ü8(Ð�A�9�Z]ÑÓÁZG?�8]Ó9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�Ý�\O=@?*\dKGZ QYZ�9b8�àWAN?e9V9>\OAC=ÁKGA�Qy9>=I9r×JZ�Ñp9>\YAC?*\Y8];<8[?e9>A�;�M]Ý�\O=IÓÁAC?�Z:Û]A�;>=ÁQOÏ8JÓÁ8];>Ï[A½ÕFZ[;�9<=IZ[Q)Z�Ñ�9>\OA�Ð�A�9/Z�Ñ�ÕWZ[Ð<Ð�=ÁÒOÓÁA¥í³ñOìeïuë:ñWé1'6ìbZ]Ñ�9>\OA(?*\Y8�;*8]?69>A�;NDYE¡A�Û]A�Q)Ð>UY?*\)=ÁH�Ö8�Ï]A%9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�ÝVA�;>A/=ÁKGA�Qy9>=IâYA�Kpû �÷8�QYK(B#9>;*8�QFÐ�Ó@8:9<=IZ[QYÐ�M[;>Z]9<8�9>=ÁZ]Q�MyÐ>?�8�ÓÁ=IQOÏFM�9rÝbZ\yסÕWA�;<ÒFZ[ÓI=@?d9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐÜä¶Ý�\O=Á?*\ ?�8]Q�Ï[A�QOA�;<8�9>AÜÐ>\OAN8�;<=IQOÏ°8�QFK�Ð<ù[UYA�A�ã�=IQYÏyçeM�8]QYKÓI=ÁQOA¥9<\O=@?*`]A�QY=IQOÏõZ];�9<\O=IQYQO=IQYÏYD�Ù�\OA¥âY;*Ð�9�Ð�=Iáô9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�ÝVA�;>A¥?*\YZ[Ð>A�Q°9>ZßÐ�ÕY8]Q9>\OA�Ð�A�9VZ�Ñ�8]ÓIÓFÕFZyÐ>Ð>=IÒYÓIACÓI=ÁQOA�8];P?6Z¡Z[;<KG=ÁQY8�9>A�9<;<8]QYÐrÑÔZ[;>HJÐ�=ÁQJ9<\OA/=ÁHJ8�Ï]A�ÕYÓÁ8]QOA�ä¶QOA�Û[A�;>9>\OA�ÖÓIANÐ>Ð�M]9<\OA�×J?�Z];<;>ANÐ�ÕWZ]QYK(9<Z \O=ÁÏ]\YÓI×�QOZ[QGÖjÓÁ=ÁQOA�8];�9>;*8�QFÐrÑÔZ[;>HJÐ�=ÁQdÕY=|áGA�Ó^Ð>ÕY8]?�ANçeD¡R/KYKG=|9<=IZ[QY8�Ó9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�\Y8�Û]ACÒFA�A�QJ9<;>=ÁA�KdÝ�=|9<\¥ÓÁA�Ð<ÐVÐ>UY?�?�A�Ð<Ð�D¡Ù�\O;<A�A�KO8:9*8�ÒY8[Ð�ANÐ�ÝVA�;>ACUYÐ>A�K�9<Z9>A�Ð�9%Z]UY;%8�ÓÁÏ]Z];<=I9>\OH�û� ��� ���!���� � ;��� � ���6!P�6������ Ù�\OACKO8:9*8�ÒY8[Ð�A�?6Z[QYÐ>=ÁÐ�9>ANK�Z]ÑåNæ � åNæ/ÕO=IáGA�ÓWÐ�=Áã�A�Ö

QOZ];<HJ8�ÓÁ=Iã�A�K)=ÁHJ8�Ï]ANÐ/Z]Ñ�\Y8�QFKGÝ�;>=I9�9<A�QôKG=IÏ[=|9*Ð�Mp?�Z]H�=IQYÏ¥ÑÔ;<Z]H �CEõHJ8�=ÁÓ�A�QyÛ[A�ÓÁZ]ÕWA�Ð�DWÙ�\YA9>;*8�=ÁQO=IQYÏ�8]QYKd9>A�Ð�9>=ÁQOÏ¥Ð>A69%\Y8[Kd;<A�Ð>ÕWA�?e9<=IÛ[A�ÓÁ× �;M�è ��8�QYK3G�è[è;M�A6áO8�H�ÕOÓÁA�Ð�D ����� � � ��!��6� ���P Ù�\OA Ð>A�?6Z[QYKÞA6áGÕWA�;<=IH�A�Qy9%Ý�8]Ð%8J?�Z]H�ÕWA69>=I9>=ÁZ]Q)Z[;>Ïy8�QO=Áã�ANKdÒ¡×

9>\OADB/8:9<=IZ[QY8�Ó��uQYÐ�9>=I9>UG9<A/Z�Ñ�E¡9<8�QFKO8�;*KOÐ�8]QYKJÙ$AN?*\OQOZ[ÓIZ[Ï]×Üä�B ��EGÙ%çP=ÁQdE¡ÕO;<=IQOÏdå$� �;GGD]Ù�\YAZ]ÒGØrAN?e9�Z�Ñ�9<\OA�?�Z]H�ÕWA69>=I9>=ÁZ]QÞÝ�8]ÐP9>Z�?6Ó@8]Ð<Ð�=IÑÔ×J8�9<A�Ð�9�Ð>A69�Z�Ñ�I �OM è]è[è�\Y8�QFKGÝ�;>=I9�9<A�QÞKG=IÏ[=|9*Ð�MÏ]=ÁÛ]A�Q�8�9>;*8�=ÁQO=ÁQOÏJÐ�A�9�Z]Ñ G G YM è[è]è�ÕF8:9�9<A�;<QYÐ�D ����� ���6!��������P Ù�\YAV9<\O=Á;<K�A6áGÕWA�;<=IH�A�Qy9�Ýb8[ÐÕFA�;�ÑÔZ[;>H�A�K Z]Q(8CKO8�9<8]ÒY8]Ð>APHJ8]KOA

Z]UG9�Z]ÑV9<\OA�9>;*8�=ÁQO=ÁQOÏ)8�QYKß9>ANÐr9<=IQYÏ·KO8:9*8�ÒY8[Ð�A�ÕO;<Z:Û¡=ÁKGANKõÒ¡× B1��EGÙ1ä¶Ð>A�A¥8]ÒFZ:Û[ANçeD B1��EGÙ\Y8]KôKG=ÁÛy=@KGANK·9<\OA¥KO8:9*8Þ=IQy9>ZÞ9rÝbZÞÐ>A69*нÝ�\O=@?*\ßUOQOÑÔZ];>9>UOQY8�9>A�ÓI×·\F8]KôKG=|à^A�;<A�Qy9�KG=@Ðr9<;>=ÁÒOUGÖ9>=ÁZ]QYÐ�D[Ù�\OA%9>;*8�=ÁQO=IQYÏ�Ð�A�9Cä�G G OM è]è[è/ÕF8:9�9<A�;<QYÐ*ç�Ýb8[Ð�A�8[Ð�=ÁA�;�9>\Y8]Q�9>\OA�9<A�Ð�9>=ÁQOÏ Ð>A69Cä I(�YM è[è]èÕY8:9>9>A�;<QYÐ*çeD �uQ�Z]UY;A�áGÕFA�;>=ÁH�A�Qy9<Ð�ÝbA�?6Z]H�ÒO=ÁQOA�K 9>\OANÐ�A�9rÝbZ½Ð�A�9<ÐSI�è� I]è�9>Z½HJ8�`[A�8/9<;<8]=IQOÖ=IQOÏ�Ð�A�9PZ�Ñ�æ]èYM è[è]è�ÕY8�9�9>A�;>QFÐP8�QFK(9>A�Ð�9>=ÁQOÏ*��Û:8�ÓÁ=ÁKY8:9>=ÁZ]QJÐ>A69*Ð�Z�Ñ�åNèOM è]è]è�ÕY8�9�9<A�;<QYÐ�AN8]?*\�M]8]ÓIÓ\Y8�Û¡=IQOÏ(9<\OA�Ð>8]H�A½?*\F8�;*8]?e9<A�;<=ÁÐ�9>=@?�Ð�D�OZ];PA�8]?*\(Z�Ñ^9<\OA�Ð>A�9>\O;<A�ACKO8:9*8�ÒY8[Ð�ANÐ�ÝVA�9<;>=ÁA�K(9<Z�A�Û:8�ÓÁUY8:9<A�\¡UOHJ8�QJÕWA�;>ÑÔZ];<H�8]QY?6Ab9<Z

ÒFA�QY?*\OHJ8�;<`�9>\OA�KG= J¥?6UOÓI9r×¥Z]Ñ$9>\YA½KY8:9<8]ÒY8]Ð>A]D*�YZ]; �CEG7bE^M¡9rÝbZ�H�A�H�ÒFA�;<ÐbZ]Ñ�Z]UY;bÏ[;>Z[UOÕÝVA�Q[9J9<\O;<Z]UOÏ[\.9<\OA)9<A�Ð�9ÜÐ�A�9d8�QYK ÒWZ�9<\�Z]ÒG9*8�=ÁQOA�K 8�GGD I� ;<8�Ý A�;<;<Z];�ÕFA�;�ÑÔZ[;>HJ8]QY?6A[DÙ�\OA¥\¡UOHJ8�Q�ÕWA�;>ÑÔZ];<HJ8�QY?�AJZ]Q�B1��EGÙ½å¥Ý�8]Ð�ÕO;<Z:Û¡=ÁKOA�KôÒ¡×õ9<\OA BC8:9>=ÁZ]QF8�Ó �uQFÐr9<=|9<UG9>AÜZ�ÑEy9<8]QYKO8�;*Kd8]QYKdÙ�A�?*\YQOZ]ÓÁZ]Ï[×]D¡Ù�\OAC\¡UOHJ8]QÜÕWA�;>ÑÔZ];<HJ8�QY?�A/Z]Q B ��EGÙ G Ý�8]ÐbH�A�8]Ð>UO;<A�KJZ[Q8)Ð�HJ8�ÓÁÓVÐ>UOÒYÐ<8�H�ÕOÓÁAJZ�Ñ�9<\OAdKO8�9<8�ÒF8]Ð>A¥8�QYK°H�UYÐr9 9>\OA�;>A�ÑÔZ];<AJÒFAJ9*8�`[A�Q°Ý�=|9<\�?�8]UG9>=ÁZ]Q�DE¡A�Û[A�;*8�Ó^Z�Ñ9>\OA�ÓÁA�8[KG=ÁQOÏJ8�ÓÁÏ]Z];<=I9>\OHJÐbÝ�\OA�;<A�9>ANÐr9<A�KÞZ]QÞA�8[?*\ÜZ]Ñ�9<\OA�Ð>A�KO8:9*8�ÒY8[Ð�ANÐ�DÙ�\OA âF;<Ð�9�A�áGÕFA�;>=ÁH�A�Qy9�UFÐ�ANK)9>\YA4A�ÖCB%AN8�;<A�Ð�9DB%A�=IÏ[\¡ÒFZ[;�8]ÓIÏ[Z];<=|9<\OH�MFUYÐ>=ÁQOÏd9<\OA(Z];>Ö

KG=IQF8�;<× #�UF?6ÓÁ=ÁKGAN8�Q·KG=@Ð�9<8�QF?6A]DWÙ�\OA�ÕO;<Z�9<Z�9rסÕWA Ð�A�9C?6Z[QYÐ>=ÁÐ�9>ANK�Z�Ñ�8�ÓÁÓ�8�Û:8�=ÁÓ@8�ÒOÓÁA½9>;*8�=ÁQO=ÁQOÏA6áO8�H�ÕOÓÁA�Ð�DFR å6ÖCB%AN8�;<A�Ð�9NB%A�=ÁÏ]\¡ÒWZ];C;>UOÓÁA Ïy8�Û]A�Z[ÕG9>=ÁHJ8�ÓÕFA�;�ÑÔZ[;>HJ8�QF?6A�=ÁQ �CEO7VE�Ý�\O=ÁÓÁA8� �Ö B/A�8];>ANÐr9 B%A�=IÏ[\yÒWZ];*Ð�;<UOÓIA½ÕWA�;>ÑÔZ];<H�A�KÜÒFA�9�9>A�;%=ÁQ B1��EGÙ GGDÙ�\OAÞÐ>A�?�Z]QYK�A6áGÕWA�;<=IH�A�Qy9JÝ�8]Ð�Ð�=ÁH�=IÓ@8�;(9<Zõ9<\OAÞâY;<Ð�9�M�ÒYUG9�9>\OA)KG=@Ðr9*8�QY?�AdÑÔUYQY?e9<=IZ[Q

Ýb8[Ð¥?*\Y8�QYÏ]A�K 9<Z¼9<8]QOÏ]A�Qy9ÜKG=@Ðr9*8�QY?�A·Ý�=|9<\ Mß9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐ�D �OZ[;¥9<\OA ��EG7VE 8]QYKB ��EGÙ G KO8:9*8�ÒY8[Ð�ANÐ�M[9>\OACÕO;<Z�9>Z]9rסÕFA�Ð�A�9bÝ�8]ÐV?6Z]QFÐr9<;>UY?69>ANK¥8]ÐVÒFA�ÑÔZ];<A]MyÒOUG9bÑÔZ]; B ��EGÙ½å�=I9

Page 14: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ýb8[Ð�?�Z]QYÐ�9>;<UY?69>A�KßÒ¡×·?6×G?6ÓÁ=ÁQOÏÞ9>\O;<Z]UOÏ[\·9<\OA�9>;*8�=ÁQO=ÁQOÏÞÐ>A69ND�R%Q¡×·ÕY8�9�9<A�;<QYнÝ�\O=@?*\ßÝbA�;<AH(=@Ð<?6Ó@8]Ð<Ð�=IâYA�K�ÝbA�;<A 8[KOKGA�KÞ9<Z¥9<\OA ÕO;<Z�9<Z�9rסÕWA Ð�A�9�D^R�ѳ9<A�;½8�ÑÔA�Ý ?�ס?�ÓIANÐ�MWQOZ¥H�Z[;>A�ÕY;>Z]Ö9>Z�9rסÕWA�ÐV8�;<A%8]KOKGANK·ä³9<\OA%9<;<8]=IQY=IQOÏ�A�;>;<Z];PÝb8[Ð�è[ç6D[Ù�\O=@Ð�;<A�Ð>UOÓI9>A�K�=ÁQ·åNèOM è]è]è�ÕO;<Z�9<Z�9rסÕWA�Ð�DR :ÖCB%A�8];>ANÐr9 B%A�=IÏ[\¡ÒFZ[;<Ðb;<UOÓIA�Ïy8�Û]ACZ]ÕG9<=IHJ8�Ó$ÕFA�;�ÑÔZ[;>HJ8�QF?6A�Z]QÞ9>\O=@Ð%Ð>A69ND

=/9>\OA�;b8]ÓIÏ[Z];<=|9<\OHJÐbÐ>UY?*\Þ8]Ð>B%A�UY;<8]ÓWQYA69<Ð EIå�KYM G�è HjM�=CÕG9<=IHJ8�Ó 2õ8�;<Ï]=ÁQÞTVÓÁ8[Ð>Ð>=IâYA�; E M�HjMS�ZG?�8]Ó�S$A�8];>QO=ÁQOÏ E H�8�QYKôabZyZyÐr9<=IQYÏ E �*H�ÝbA�;<A(8]ÓÁÐ>ZdUYÐ>A�KßZ]Qß9>\YA�Ð>A(KO8�9<8]ÒY8]Ð>A�Ð�DpR ?�8]Ð>AÐr9<UYKG×Ü?�8�QÞÒFA½ÑÔZ[UOQYKÞ=ÁQ E G�è HjDÙ�\OA�;>ANÐ�UOÓI9<ÐP8�;<A�Ð>UOH�HJ8�;<=Iã�A�K�=IQJÙ8�ÒOÓÁANGGD GGD�RCÐ�=ÁÓIÓÁUYÐ�9>;*8:9>ANK =ÁQ(9>\OA�9*8�ÒOÓÁA]M�9>\OA/Ù8�QOÖ

� �yi�k�w ���¯�V� �bµ � µ �Fm<wym<�e� �Fm*w[m<�r£ �J� �W� �q:qNt±���z]nz ¤�µ � �:µ � ¤�µ � £[µ ¤ £[µ � �]µ � ¤:µ ��Vhuz]�b� �Nµ � �]µ ¤ �]µ � £]µ|��Vhuz]��¤ �]µ ¤ ¤:µ £ �Nµx� �Nµ � �Nµx� �Nµx� �Nµx� ��µ �

� ¬[¨���� V¡® }Pm*tu�y���rt �]n^m<ou²Áq�ori�k�w[s*m*t�vxw�� q�²Wm<ouorq�ort�²Áq�o%¾³vxw�q�o��]m<o>¿ � �[i�k�wF����¶wymek6orm*t±��wym<v|l�~[��Gq�oe�]�^k�w[lNm*w:���]vxt±��k�wys<mN�]�Fm*w[m<�e�C¾³tuv|i� [�xmVw[m*�[o�k��¡wym<�¯º�q�or�y¿>�]�Ym*wym>�r£�¾³�|k�orl�mbw[m*�[o�k��Gwym<�¯º�q�or�[¿>� b ]�rvxi�k�� ��k�orl�v|wd��|k�tutuvx¸ym<oC¾� �J��¿>�G�Fq:s*k��W�xmek6orwyvxwyl(¾Ô�W�^¿�k�w¡����q:qNt±�rvxwylJ¾Ô�q:q�t±�>¿>µ

Ï]A�Qy9P_C=ÁÐ�9<8]QY?6AC8�ÓÁÏ]Z];<=I9>\OHÚA�ùyUY8�Ó@Ð�Z[;�Z]UG9<ÕFA�;�ÑÔZ[;>HJÐ�8]ÓIÓFZ�9>\YA�;P8]ÓIÏ[Z];<=|9<\OHJÐ�ÝVA�9<A�Ð�9>ANKpM]=ÁQ8�ÓÁÓ�?�8]Ð>A�Ð/A6áO?6A�ÕG9�Z[QOA]ûpabZ¡Z[Ð�9>ANK·S�APB%A69 �ÞÝb8[Ð%9<\OA�Ý�=ÁQOQOA�;CZ[Q·9<\OA4B1��EGÙ GÜKO8:9*8�ÒY8[Ð�A[DÙ�\O=ÁÐ�=@Ð�QYZ�9Ð>UO;<ÕO;<=ÁÐ>=IQYÏYD�Ù�\YALA�Ö±QOA�8];>ANÐr9$QYA�=ÁÏ]\¡ÒFZ[;�8]ÓIÏ[Z];<=|9<\OH ä¶Ý�=|9<\�QYZ/ÕO;<A�ÕO;<ZG?6ANÐ>Ð>=IQYÏyç=ÁÐ/Û]A�;<×ÞUOQGÖuÐ�Z[ÕO\O=@Ðr9<=Á?�8:9>ANKÞ=IQõ?�Z]H�ÕY8];>=@Ð�Z[QÞ9>Z¥ÓÁZG?�8�Ó�ÓIAN8�;<QO=ÁQOÏYMFZ]ÕG9<=IHJ8�Ó�HJ8�;<Ï]=ÁQ�?�ÓÁ8[Ð>Ð>=IÖâYA�;NMy8]QYKJÒWZ¡Z[Ð�9>=ÁQOÏYDyÙ�\OAC8[KGÛ:8�Qy9<8]QOÏ]A%Z]Ñp9<8]QOÏ]A�Q[9�KG=@Ðr9*8�QY?�A%=ÁÐP9>\YA�8�ÕY;>=ÁZ];<=Y`¡QOZ:Ý�ÓÁA�KGÏ[AZ�Ñ�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q =IQ¡Û:8�;<=Á8]QY?6A)A�H ÒWA�KOKOA�K�=ÁQy9>Z¼9<\OAßKG=@Ð�9<8�QF?6A]D ��\OA�Q�9>\OAõ9>;*8�=ÁQO=ÁQOÏKO8:9*8�=@ÐbÐ>U6J¥?6=ÁA�Qy9<ÓI×JÓ@8�;<Ï]A[M[8]ÐV=@ÐP9<\OA½?�8]Ð>A/=ÁQ B1��EGÙ GGMGÐ>Z]H�A/Z]Ñ�9>\Y=ÁÐb`¡QOZ:Ý�ÓÁA�KGÏ[A/?�8]QdÒWAÕO=Á?*`[A�K�UOÕ·ÑÔ;>Z[H 9<\OA�KO8�9<8JÒ¡×Þ9>\YA�H�Z];<A Ð�Z[ÕO\O=@Ðr9<=Á?�8:9<A�K)8�ÓÁÏ]Z[;>=I9>\YH�Ð�D��uQ)Z]9>\OA�;/ÝbZ];*KOÐ�M9>\OA�8�ÖjÕO;<=ÁZ];<=W`¡QOZ:Ý�ÓÁA�KOÏ]A½8]KGÛ:8�Qy9*8�Ï]A½KGAN?6;<A�8]Ð>A�ÐbÝ�=I9>\Þ9>\YA�Ð�=Áã�A½Z�Ñ9>\YA½9>;*8�=ÁQO=IQYÏ�Ð>A69ND

�� � � !P ���� �� !P������ �$! �� ���!������ �@ ����

Ù�\O=ÁÐ�Ð>A�?69>=ÁZ]Qd=@Ð�KGA�KO=Á?�8:9>ANKJ9<Z 9<\OA�9>A�?*\YQOZ]ÓÁZ]Ï[=Á?�8�ÓPö>`¡QOZ:Ý�\OZ:Ý%ø�Ý�\Y=Á?*\Ü=ÁÐbQOAN?6A�Ð<Ð<8�;<× 9<ZH�8]`]AP9<8�QYÏ]A�Qy9�KG=@Ðr9*8�QY?�APÝbZ];<`�Ý�=I9>\�Û:8�;<=IZ[UYÐ�8]ÕOÕOÓÁ=Á?�8:9<=IZ[QYÐ�D�Ù�\OANÐ�A(ör9<;>=@?*`GÐ>ø%8];>APUYÐ�UF8�ÓÁÓI×QOZ�9�ÕOUOÒOÓÁ=@Ð�\OANKJÑÔZ[;VÛ:8�;<=ÁZ]UYÐV;>AN8]Ð>Z]QYÐ/ä³9>\YA�×d8];>A%QYZ�9�8�ÓÁÝb8�×GÐ�9<\OA�Z[;>A�9>=@?�8]ÓIÓÁ×(Ð>Z]UYQYKpM¡ÕY8�Ï[AA�Ð�9<8:9<A½=ÁÐV9>Z¡Z�Û:8�ÓÁUY8�ÒYÓIA[M¡9>\OA�9>;<=@?*`¡Ð�8];>A½Ð�ÕWA�?�=|âW?/9<Z�Z]QOA�ÕY8�;>9>=@?6UYÓÁ8];�8]ÕOÕOÓÁ=Á?�8:9<=IZ[Q�M¡=IQy9>A�Ó|ÖÓIAN?e9>UF8�ÓPÕY;>Z[ÕFA�;�9r×õÑÔZ];<ÒO=ÁKYÐ�9>A�ÓIÓÁ=ÁQOÏõ8�Q¡×[Z]QOAJ\OZ:Ý 9<Z·;>A�ÕO;<Z¡KOUY?6A�9<\OAd;<A�Ð>UOÓI9�M$A�9<?]D ç6M$ÒOUG99>\OA�×Ü8�;<A�Z�ѳ9>A�Q)8�KGA�9>A�;>H�=ÁQO=IQYÏ�Ѷ8]?e9<Z];�=ÁQÞHJ8]`y=ÁQOÏ�9>\OA½9<A�?*\OQYZ]ÓÁZ]Ï]×¥8�Ð>UY?�?�A�Ð<Ð�DOE¡A�Û]A�;<8]ÓZ�Ñ9>\OANÐ�A�9>AN?*\OQO=@ùyUOA�Ð�Ý�=ÁÓÁÓpÒFA�KG=@Ð<?6UYÐ<Ð�ANKÞ\OA�;<A]D

� �� �!P" � � !P" ������ ! � � � � Ù�\Y=ÁÐC=ÁÐ/9>\YA(Ð>=IQYÏ]ÓÁA H�Z[Ð�9C=ÁH�ÕFZ[;�9*8�Qy9%Ѷ8]?69>Z[;C=ÁQZ]ÒG9*8�=ÁQO=IQYÏ)Ï]Z¡ZGK°ÕFA�;�ÑÔZ[;>HJ8�QF?6AJÝ�=I9>\°9<8]QOÏ]A�Qy9 KG=ÁÐ�9<8]QY?6A[D�ab×°KGA6âYQY=|9<=IZ[Q�M�9<\OAJ9<8]QOÏ]A�Qy9Û]A�?69>Z[;<Ð�8];>A�9>\OAbS�=ÁAVKGA�;>=ÁÛ:8:9>=ÁÛ]ANÐpZ�ÑG9<\OA�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q�ÑÔUOQY?69>=ÁZ]Q �[ä2������ç�Ý�=I9>\ ;>ANÐ�ÕWA�?699>Z��VDYÙ�\OA�×Ü?�8�QÞÒWA�Ý�;>=I9�9<A�Q)8]Ð�û

� � � � �yä����,�ç� �

���� � ÓÁ=IH�����

�yä������eç �yä����<è[ç�

ä�åPI[ç

Page 15: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

�±9½=@Ð�9>\YA�;<A6ÑÔZ];<A Û[A�;<×)=IH�ÕWZ];>9<8]Q[9�9>\F8:9 ��ÒWA�KO=|à^A�;<A�Qy9>=@8�ÒYÓIA�ä¶8]QYKõÝbA�ÓÁÓ�ÒFA�\Y8�Û]ANKYç/Ý�=|9<\;>ANÐ�ÕWA�?69�9>Z �VD��uQ°ÕY8�;>9>=@?6UYÓÁ8];�Mp=I9 =@Ð�?6ÓÁA�8];½ÑÔ;>Z[H A�ùyUY8�9>=ÁZ]Q ärå�I]çC9>\F8:9 �yä���� �*ç�H UYÐ�9 ÒWA?6Z]H�ÕOUO9>A�KßÑÔZ[; � 8�;<ÒO=I9>;*8�;<=IÓÁ×)Ð�HJ8]ÓIÓjD��OZ[;�9<UOQY8�9>A�ÓÁ×]MpA�Û]A�QßÝ�\OA�Q � ?�8�QôZ]QOÓÁ×)9<8]`]AJKG=@ÐrÖ?6;<A69>A�Û:8]ÓIUOANÐ�MY=|9/=@Ð%AN8]Ð>×J9>ZdH�8]`]A � KG=|à^A�;<A�Qy9<=Á8]ÒOÓIA[DFÙ�\OA½9<;>=@?*`d=@Ð�9<ZJUYÐ>A�8¥Ð�H�Z¡Z�9<\O=ÁQOÏ=IQy9>A�;>ÕWZ]Ó@8:9<=IQYÏ�ÑÔUOQY?69>=ÁZ]Q ��� 8[ÐV8 ÕO;<A�ÕO;<ZG?6A�Ð<Ð>=IQOϽÑÔZ[; ��MGÐ>UY?*\J9<\Y8:9 �yä ��� ä���ç ����ç�=@Ð�KG=|ѳÖÑÔA�;<A�Qy9>=@8�ÒOÓÁAJäÔÝ�=I9>\);<A�Ð>ÕFAN?e9�9<Z ��� ä2��ç�8�QYK��VMOQOZ]9%Ý�=|9<\Þ;<A�Ð>ÕWA�?e9�9<Z���çeD��YZ];�=ÁQYÐ�9<8�QF?6A]M=|Ñ�9<\OA�=IQOÕYUG9%Ð�ÕY8[?6A/ÑÔZ];�� =ÁÐbÒO=ÁQY8];>×�=ÁH�8]Ï]ANÐ�M � � ä���çV?�8�QdÒFA½8�?�Z]Q¡Û]Z[ÓIUO9>=ÁZ]QJZ�Ñ �÷Ý�=|9<\8��½8]UYÐ>Ð>=@8�Q·ÑÔUYQY?e9<=IZ[QßZ]ÑbÐ�9<8�QFKO8�;*K·KGA�Ûy=@8:9<=IZ[Q���D��±Ñ �yä � � ä���ç ����ç�=Áн8Ü9>;*8�QYÐ>Ó@8:9>=ÁZ]QõZ�Ñ��ÕO=IáGA�ÓjMO9<\OA KGA�;>=ÁÛ:8:9>=ÁÛ]A�Z]Ñ �[ä � � ä���ç ����ç%?�8]Q)A�8[Ð�=ÁÓI×ÜÒFA�?6Z]H�ÕOUO9>A�KõÐ�=ÁQY?6A �yä � � ä���ç.� �eç?�8�Q°ÒWAJZ]ÒG9*8�=ÁQOA�KôÒ¡×·9<;<8]QYÐ>ÓÁ8�9>=ÁQOÏ�½8�UYÐ<Ð�=@8�QßÑÔUOQY?69>=ÁZ]QYÐ�D$Ù�\O=@Ð�ÕO;<A�ÕY;>ZG?6ANÐ>Ð>=ÁQOÏÜÝ�=ÁÓÁÓPÒWAKG=ÁÐ<?6UFÐ>Ð>A�KÜ=IQ�H�Z];<A�KGA69*8�=ÁÓÁÐ�=ÁQ)Ð�AN?e9<=IZ[Q+�FDÙ�\OA/Ð>H(Z¡Z]9>\O=ÁQOϽѶ8]?69>Z[;�)?6Z[Q[9<;>Z[ÓÁÐ9<\OA%ÓÁZG?�8�ÓÁ=I9r×�Z]ÑW9<\OA%=ÁQ¡Û�8];>=@8�QF?6A]D[Ù�\OA/Ð>H(Z¡Z]9>\OA�;

9>\OAÜ9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q°?�UO;<Û]AÜKOA6âYQOANK¼Ò¡× �Þ=ÁÐ�M$9<\OAÞÓIZ[QOÏ]A�;�9>\YAdÓÁ=IQOAN8�;�8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]QÝ�=IÓÁÓ�ÒFA(Û:8�ÓÁ=ÁK�D��uQßÏ[A�QOA�;<8]Ó$9<\OA�ÒWA�Ð�9�Ð�H�Z¡Z�9<\O=ÁQOÏÜ=@ÐC9>\OA�HJ8:áG=ÁH UOH Ð>H�Z¡Z�9>\Y=IQOÏÞÝ�\O=@?*\KGZ¡A�Ð�QYZ�9 ÒOÓÁUO; 9>\OAJÑÔAN8:9<UO;>ANÐ�D �OZ[;�A6áO8�H�ÕOÓÁA]M�=IQ¼\F8�QYKGÝ�;<=I9�9>A�Q�?*\Y8�;*8]?69>A�;�;>AN?6Z]Ï[QO=I9>=ÁZ]QÝ�=|9<\ åNæ:ápå�æ)ÕY=|áGA�Ó�=ÁHJ8�Ï[A�Ð�M8��½8�UYÐ<Ð>=Á8]Q¼ÑÔUOQY?69>=ÁZ]Q Ý�=|9<\ 8ôÐr9*8�QYKY8�;*K�KGA�Û¡=@8:9<=IZ[Q�Z�Ñ�åÕO=|áGA�Ó�×y=ÁA�Ó@KGANK·9<\OA�ÒFANÐr9�;<A�Ð>UOÓI9<Ð�D �uQF?6;<A�8]Ð>A�KßÐ�H�Z¡Z�9<\O=ÁQOÏ�ÓIANK·9<Z�?�Z]QGÑÔUFÐ�=ÁZ]Q ä¶Ð>UY?*\ô8[н8ö7I�ø H�=ÁÐ�9<8]`]A�Q¥ÑÔZ];�ö>æ[ø�ÒWA�?�8]UYÐ>A%9>\YA�ÓÁZ:ÝbA�;PÓÁZ¡Z]ÕÜ\Y8]KdÒFA�A�Q�?6ÓÁZ[Ð>A�KJÒ¡×�9>\YA�Ð>H�ZyZ]9>\O=ÁQOÏ¡ç8�QYKÞKGA�?�;>AN8]Ð>A�KÞÐ�H�Z¡Z�9<\O=ÁQOÏJKG=ÁKOQ 9%HJ8]`]A�ÑÔUOÓÁÓ�UYÐ>A�Z�Ñ$9<\OA�=ÁQyÛ:8];>=@8�QY?�A½ÕO;>Z[ÕFA�;�9<=IANÐ�D�±Ñp?�Z]H�ÕOUG9*8:9<=IZ[Q�8]ÓIÓÁZ:Ý%Ð�=I9�M[9>\OA%ÒWA�Ð�9VÐ�9>;*8:9<A�Ï]× =ÁÐ�9>Z�A�áy9<;<8[?e9�ÑÔAN8:9>UY;>ANÐ�âY;*Ðr9NM]Ð�H�Z¡Z�9<\

Ð�\Y8]H�A�ÓÁA�Ð<Ð�ÓÁ×]MO8]QYK¥9<\OA�Q)?�Z]H�ÕOUG9<A½9>\OA½9*8�QOÏ[A�Qy9%KG=ÁÐ�9<8]QY?6A½Z[QÜ9<\OA�Ð>H(Z¡Z]9>\OANK¥ÑÔAN8:9>UY;>ANÐ�D� �$!��* �����;� ��� ��*���6!P � � Ù�\OA%ÓÁ=ÁQOA�8];PÐ�×GÐ�9>A�HÚÏ[=IÛ[A�Q�=IQJANùyUY8:9<=IZ[Q·ä�K[ç�=ÁÐVÐ�=ÁQOÏ]UOÖ

ÓÁ8];�=|ÑYÐ>Z]H�A�Z]ÑG9>\OA�9*8�QOÏ[A�Qy9$Û[A�?e9<Z];*ÐWÑÔZ]; � Z];�� 8�;<A�ÕY8];<8]ÓIÓÁA�ÓjD�R%ÓI9>\YZ]UOÏ[\�9>\OAPÕO;>Z[ÒY8�ÒY=IÓÁ=|9r×Z�Ñ�9>\Y=ÁÐC\Y8�ÕYÕFA�QO=IQYÏ¥=ÁÐCã�A�;>ZJÝ�\YA�Q)9<\OA�KO8�9<8J=@Ð/9<8�`[A�Q�ÑÔ;<Z]H18¥;<A�8]Ó�Û:8�ÓÁUOA�K·?�Z]Qy9>=ÁQ¡UOZ]UFÐKG=ÁÐ�9>;<=ÁÒOUG9>=ÁZ]Q.ä¶8]ÐC=@Ð%9<\OA�?�8]Ð>A�=IQß\Y8�QFKGÝ�;>=I9�9<A�Qõ?*\F8�;*8]?e9<A�;%;<A�?�Z]Ï[QO=|9<=IZ[QFçeMF=|9�=ÁÐCÕWZ[Ð<Ð�=ÁÒOÓÁA9>\Y8�9b8�ÕY8�9�9<A�;<QJH�8�×(ÒWACKGUOÕYÓI=@?�8�9>A�K�=ÁQ¥ÒWZ�9<\J9>\OA/9>;*8�=ÁQO=ÁQOÏ�8�QYKJ9>\OA�9<A�Ð�9bÐ>A69NM[;<A�Ð>UOÓI9>=ÁQOÏ=IQÞ8 KG=ÁÛ¡=ÁÐ>=IZ[QJÒ¡×(ã�A�;<Z A�;>;<Z];ND]Ù�\OA/âOá¥=ÁÐbùyUO=I9>A½Ð�=ÁH�ÕOÓIA�8�QYK¥A�ÓÁA�Ï[8]Qy9�D�#PùyUY8:9<=IZ[Q·ä�Kyç�?�8]QÒFA�;<A�ÕOÓ@8]?�A�KÜÒ¡×¥A�ùyUY8�9>=ÁZ]Q�û�·ä�� �,��ç � H�=IQ� � � � � � � �� � � � � � � � � � f ? � � � � � � f ? � � � � � � f ä�å�æyçÙ�\OA�ÕO\¡×GÐ�=@?�8]Ó�=IQy9>A�;>ÕY;>A�9<8:9<=IZ[QõZ]ÑP9>\Y=ÁнANù[UF8:9>=ÁZ]Q$M^KOA�ÕO=@?e9<A�Kô=IQ �=IÏ[UO;<A&MGMp=ÁÐC9>\F8:9�9>\YAÕFZ[=IQy9�� $ ä�� � çZ]Q 9>\OA�9*8�QOÏ[A�Qy9�ÕOÓÁ8]QOA�� � =@Ð�8:9>9<8]?*\YA�K�9<Z�� Ý�=I9>\J8�Ð�ÕO;<=ÁQOϽZ�ÑFA�ÓÁ8[Ðr9<=Á?�=|9r×?½8�QYKC9>Z � $ ä2� � ç�äÔZ[Q½9>\OA�9<8�QYÏ]A�Qy9�ÕOÓ@8�QYA � � çpÝ�=|9<\�8�Ð�ÕO;<=ÁQ[9�Z�Ñ¡A�Ó@8]Ð�9>=@?6=I9r× å]M�8�QYK � $ ä�� � ç=ÁÐ8�Ó@Ð�ZC8:9�9*8]?*\OANK½9>Z ��Ý�=I9>\ 8CÐ�ÕO;<=ÁQOÏ/Z]ÑOA�Ó@8]Ð�9>=@?6=I9r× ?pDNÙ�\OAVQOA�Ý�9<8�QYÏ]A�Qy9�KO=ÁÐ�9<8]QY?6AV=ÁÐ�9>\YA9>Z�9*8�ÓGÕWZ�9<A�Qy9>=@8�ÓGA�ÓÁ8[Ðr9<=Á?bA�QYA�;<Ï]×�Z�Ñ^Ðr9<Z];<A�K�8�ÓÁÓ¡9>\O;<A�A�Ð>ÕO;>=ÁQOÏyÐ8�9�A�ùyUO=ÁÓI=ÁÒO;<=IUOH�D�RCÐÑÔZ];�9>\YAÐr9*8�QYKO8];<KÞ9*8�QOÏ[A�Qy9CKG=@Ð�9<8�QF?6A]MF9>\YA�Ð>Z]ÓÁUG9>=ÁZ]Q·?�8�Q·AN8]Ð>=IÓÁ×ÜÒWA�KOA�;<=IÛ[A�K�Ò¡×ÞKO=|à^A�;<A�Qy9>=@8:9<=IQYÏA�ùyUY8:9<=IZ[Qôä�å�æyçPÝ�=I9>\�;<A�Ð>ÕFAN?e9�9<Z � � 8�QYK�� � DYÙ�\OA�KG=IàWA�;>A�Qy9>=@8:9>=ÁZ]Q�ס=ÁA�Ó@KOÐ�û��� ä�� � � � ä�å ?Oç�� � &� � � � ç � è ä�å�M]ç� � ä�� � � � � � �� � ärå ?Oç � � ç � è ä�å�KyçÙ�\OA�Ð�Z[ÓIUO9>=ÁZ]QÞZ�Ñ$9<\O=@Ð%Ð�×GÐ�9>A�H =@Ð

ä � � � � � L� � ��� ärå ?Oç � � ç�ä�� ��ç � ä � � � � � L� � � ���# ä�å ?Yç f � ��� ç � � ä�å$�yçä � ��� ��� L��� � � ärå ?Oç � � ç6ä � ��ç � ä�ä�å ?Oç f � � �# � � � � � L��� � � � ç � � ä G�èyç

Page 16: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

k

1

k

P

E

E’

P’

QSRUT ®��O® ��~[m���k�wyl�m*w:���]vxt±��k�wys<mC�Gm<�¯º�m*m<w � k�wy� vxtP�r~ym%m*�|k�t±�rvxs�m<wym<orl�¹�t±�rq�orm*��v|w¥mek�s>~Jq�²�r~[m��r~[orm<m�tu [orvxwyl�t/s<qNwyw[m*s<�rvxwyl������Ô� � �k�wy� � µ���k�wy� � �sek�w�i�q6{�mCº�v��r~[qN�[�%²Iorv|s>�rv|q�wk��xqNwyl½�r~[mb��k�w[lNm*w:�P y�|k�wym*t*µy��~ym�tu ]orv|w[l½s*q�wyt±��k�w:�rtPk�orm�vxwy�[vxsek��rm*�(qNw(�r~[mb¸¡l��[ormNµ

Ý�\OA�;<A � � � � � � � � M�� � � � � � � � M�� � � � � � � � 8]QYK&� ��� � � � � � D�Ù�\YAÐ�×GÐ�9>A�H \Y8]Ð(9<\OAõÐ<8�H�A)?6Z[H(ÕYÓIA�á¡=I9r×�8[Ð�9>\OA)Û:8]QO=IÓÁÓ@8ô9<8�QYÏ]A�Qy9¥KG=@Ðr9*8�QY?�A)A6áO?6A�ÕG9J9>\F8:9�M=|9�8�ÓÁÝb8�×Gн\Y8[Ð 8·Ð>Z]ÓÁUG9>=ÁZ]Q°ÑÔZ[; ?�� èOM8�QYK°=@Ð�H�Z];<A¥Q¡UOH�A�;<=@?�8�ÓÁÓÁ×ßÐ�9<8�ÒYÓIA[D B%Z]9>Ad9>\Y8�9ÑÔZ];@? � èOMp=I9 =@нA�ùyUO=ÁÛ:8�ÓÁA�Qy9�9>Z�9<\OA¥Ðr9*8�QYKY8�;*K·9*8�QOÏ[A�Qy9�KG=ÁÐ�9<8]QY?6A[M�Ý�\O=ÁÓIA�ÑÔZ[; ? ��� MÝVA·\F8�Û]A)9<\OA #PUY?6ÓÁ=ÁKOA�8�Q KO=ÁÐ�9<8]QY?6A[DbÙ�\O=@Ðd8]ÕOÕO;<Z[8[?*\ =@ÐÞ8�Ó@Ð�Z¼Û[A�;<×.UYÐ>A6ÑÔUYÓCÝ�\OA�Q�9>\YAQyUYH ÒWA�;�Z�Ñ$9<8�QYÏ]A�Qy9�Û]AN?e9<Z];*ÐP=@ÐbÏ];<A�8:9<A�;bZ];�A�ùyUY8]ÓW9>\Y8]Qd9<\OA½QyUYH ÒWA�;�Z�Ñ�KG=ÁH(A�QYÐ>=IZ[QYÐbZ�Ñ9>\OA�Ð�ÕF8]?6A[DWÙ�\YA�Ð�9<8�QFKO8�;*KÞ9<8]QOÏ]A�Q[9½KG=@Ðr9*8�QY?�A�ÝbZ]UOÓ@K�H�Z[Ð�9�ÓÁ=Á`]A�ÓÁ×�ÒFA�ã�A�;>ZôäÔÝ�\OA�Q·9>\YA9<8�QYÏ]A�Qy9�Ð�ÕF8]?6ANÐP=ÁQy9>A�;<Ð>A�?69*çeMGÒYUG9�9>\OA�ö>Ð>ÕO;<=IQYÏ[ø½9*8�QOÏ[A�Qy9�KO=ÁÐ�9<8]QY?6A½Ðr9<=IÓÁÓpס=IA�ÓÁKYÐVÛ:8]ÓIUY8]ÒOÓÁA=IQGÑÔZ[;>HJ8�9>=ÁZ]Q�D�±Ñ�9>\OA°Q¡UOH�ÒFA�;ÞZ]Ñ(KG=ÁH�A�QYÐ>=ÁZ]Q Z]Ñ�9>\YAô=ÁQOÕOUO9)Ð>ÕY8[?6Aô=ÁÐÞÓ@8�;<Ï]Aß?�Z]H�ÕY8];>ANK 9>Z 9>\YA

QyUYH ÒWA�;�Z�Ñ%9*8�QOÏ[A�Qy9�Û]A�?69>Z[;�M`]A�A�ÕO=ÁQOÏ ?�8]Ð(Ð�HJ8]ÓIÓ�8]Ð(ÕWZ[Ð<Ð�=ÁÒOÓÁAd=@Ð(ÒFA�9�9>A�;JÒFAN?�8]UYÐ�AÞ=I9KGZ¡A�Ð>Q 9%=ÁQy9>A�;�ÑÔA�;>A�Ý�=I9>\�9<\OA)ö>Ð>ÓI=@KG=ÁQOÏ[ø�8]ÓIZ[QOÏ(9>\OA�9*8�QOÏ[A�Qy9%ÕOÓ@8�QOA¥ä � $ 8]QYK�� $ 8�;<A½ÓIANÐ>Ð?6Z]QFÐr9<;<8]=IQOANKYçCDTVZ]Qy9>;*8�;<×/9<Z/=ÁQy9>UO=I9>=ÁZ]Q�M�9<\OA�;<Ab=ÁÐ�QOZ½KO8�QOÏ[A�;Z�ÑFÐ>ÓÁ=ÁKG=ÁQOÏ/9>Z¡Z/Ѷ8];�=ÁQ(\O=IÏ[\�KG=ÁH�A�QYÐ>=ÁZ]QY8]Ó

Ð�ÕY8[?6AVÒFAN?�8�UFÐ�AP9<8�QYÏ]A�Qy9Û]A�?69>Z[;<Ð�8�;<AV8�ÓÁÝ�8�סÐ�;>Z[UOÏ]\OÓÁ×½Z];>9>\OZ[Ï]Z[QY8�Ó]8]QYK½9<\OA�×�?�Z]UOÓ@K�Z]QYÓI×Ð�ÓÁ=ÁKOA½Ñ¶8�;�=IÑ�9<\OA�×ÜÝVA�;>A�ÕY8�;*8�ÓÁÓÁA�ÓjD� �;�*�6� �P"$�� �� ���!������ �� �±Ñ$Ð�A�Û]A�;<8]ÓG=IQ¡Û:8�;<=Á8]QY?6ANÐ�8�;<A%UYÐ>A�KpMy?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]Q(UYÐ>=IQYÏ

9<8�QYÏ]A�Qy9$KG=@Ð�9<8�QF?6A�8�ÓÁZ]QYA�ÝVZ[UOÓÁK�ÒWAPùyUO=I9>A�A�á¡ÕWA�QFÐ�=ÁÛ]A[D �OZ[;�9<UOQY8:9<A�ÓÁ×]M6=IÑO8b9rסÕO=Á?�8�Ó�H�A�H(Z[;>×ÒY8]Ð>A�K�8]ÓIÏ[Z];<=|9<\OH =@Ð�UYÐ�ANKpM�ÑÔZ];A6áO8�H�ÕOÓÁA]MPA�ÖjQOAN8�;<A�Ð�9�QOA�=ÁÏ]\¡ÒWZ];*Ð�M�=|9=ÁÐùyUO=I9>AbUOQOQOAN?6A�Ð<Ð<8�;<×9>Zß?�Z]H�ÕOUG9<A¥9<\OAdÑÔUYÓIÓb9<8]QOÏ]A�Qy9�KG=@Ðr9*8�QY?�AdÒWA69rÝbA�A�Q�9>\OAÞUOQY?�ÓÁ8[Ð>Ð>=|âFA�K°ÕY8�9�9>A�;>Q.8�QYK�8]ÓIÓ9>\OA¥Ó@8�ÒWA�ÓÁA�K¼Ð<8�H�ÕOÓÁA�Ð�D��uQ¼ÕY8];�9<=Á?�UOÓ@8�;NMp=|Ñ%8·?�;>UYKOAJA�Ð�9>=ÁH�8�9>AJZ]Ñb9<\OAJ9*8�QOÏ[A�Qy9 KG=@Ðr9*8�QY?�A=IQYKO=Á?�8:9>ANÐ$Ý�=I9>\(8/Ð>U6J¥?6=ÁA�Qy9�?6Z]QOâFKGA�QF?6A�9<\Y8:9�8/Ð<8�H�ÕOÓÁAP=@Ð�Û]A�;<×CѶ8�;�ÑÔ;>Z[H 9<\OAbÕY8:9>9>A�;<Q�9<ZÒFA/?�ÓÁ8[Ð>Ð>=IâYA�KpM:QYZ�H(Z[;>A�?�Z]H�ÕOUG9*8:9>=ÁZ]Q�=@Ð�QOA�A�KGANK(9>Z�`¡QOZ:Ý�9<\Y8:9P9>\O=@Ð�Ð<8�H�ÕOÓÁA�=ÁÐ�QYZ�9�Z]QYAZ�Ñ¡9>\OA>A�ÖjQYA�8�;<A�Ð�9pQOA�=ÁÏ]\¡ÒWZ];*Ð�D�ab8[Ð�ANK½Z]Q�9>\Y=ÁÐ�Z]ÒYÐ>A�;<Û:8:9<=IZ[Q�Z[QOAP?�8�Q½ÒOUO=ÁÓÁK 8�\O=ÁA�;*8�;*?*\y×/Z�ÑKG=ÁÐ�9<8]QY?6ANÐ�Ý�\O=@?*\�?�8�QÞÏ];<A�8�9>ÓÁ×J;>ANKGUY?6A�9<\OA�?6Z[H(ÕYUG9<8�9>=ÁZ]QÞZ�Ñ�A�8]?*\�?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q�DGS�A69� Ð8]Ð<Ð�UOH�A[M�ÑÔZ];�=IQFÐr9*8�QY?�A]M�9>\F8:9�ÝbAb\F8�Û]A � 8�ÕOÕO;<Z�áG=IHJ8�9>=ÁZ]QYÐ� � Z�ÑY9<\OAb9<8]QOÏ]A�Q[9�KG=@Ð�9<8�QF?6A]MZ];*KGA�;<A�KÞÐ�UF?*\�9>\Y8�9 � L =ÁÐ%9>\OA ?�;>UFKGA�Ð�9/8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]QÞZ�Ñ�9>\YA�9<8]QOÏ]A�Q[9/KO=ÁÐ�9<8]QY?6A�8]QYK

Page 17: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

� =@Ð�A6áO8]?69>ÓÁ×Ü9*8�QOÏ[A�Qy9�KO=ÁÐ�9<8]QY?6A�ä³ÑÔZ[;C=ÁQYÐ�9<8�QF?6A � L 9>Z � � ?�Z]UOÓ@K·ÒWA 9<\OA&#�UY?�ÓI=@KGAN8�QKG=ÁÐ�9<8]QY?6ACÝ�=|9<\d=ÁQY?6;<A�8[Ð�=ÁQOÏ�;<A�Ð>Z]ÓÁUG9>=ÁZ]Q$My8]QYK ����9<Z � L � A�8[?*\d8]KOKd8�9<8]QOÏ]A�Qy9PÛ]AN?e9>Z[;V8�9ÑÔUOÓIÓ$;>ANÐ�Z[ÓIUG9<=IZ[QFçeDÙ�\OAÜÒF8]Ð>=Á?d=ÁKGAN8õ=@Ð�9>Zß`]A�A�Õ.8ßÕFZ¡Z[ÓVZ]ÑC8�ÓÁÓb9>\OAÞÕO;<Z�9>Z]9rסÕFANÐ Ý�\O=@?*\.?�Z]UOÓ@K�ÕWZ�9<A�QGÖ

9>=@8�ÓÁÓI×·ÒWA�9>\YA5A�Ö±QOAN8�;<A�Ð�9CQOA�=IÏ[\yÒWZ];*Ð�Z�ÑV9>\OAJUOQF?6Ó@8]Ð<Ð�=IâYA�KõÕY8:9>9>A�;>Q�D��uQY=|9<=Á8]ÓIÓÁ×·9<\OA�ÕFZ¡Z]Ó?6Z]Qy9*8�=ÁQYÐ�8�ÓÁÓ^9>\OA�Ð<8�H�ÕOÓÁA�Ð�D�#P8]?*\ÜZ�Ñ�9>\OA�KG=@Ðr9*8�QY?�A�Ð � ��?6Z[;>;<A�Ð>ÕFZ[QYKOÐP9>ZJ8�Ðr9*8�Ï[ACZ]Ñ$9>\YA?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Q�ÕO;<ZG?6A�Ð<Ð�D[Ù�\OA/?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]QJ8�ÓÁÏ]Z[;>=I9>\OH \Y8[Ð �Ð�9>A�ÕYÐP8�9VA�8[?*\¥Ðr9*8�Ï[A]M]8]QYKÕO;>ZG?�A�A�KYÐ�ÑÔ;>Z[H Ð�9<8]Ï]A�å½9>ZdÐ�9<8]Ï]A � Z[;�UOQy9>=ÁÓ�9<\OA ?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[QÜ=@Ð/?�Z]H�ÕOÓÁA69>A[ûFEy9<A�Õ¼å]û9>\OA�KG=ÁÐ�9<8]QY?6A � � ÒWA69rÝbA�A�Qd8�ÓÁÓY9<\OACÐ<8�H�ÕOÓÁA�ÐV=ÁQJ9>\OACÕFZ¡Z[ÓW8]QYK�9>\YACUOQF?6Ó@8]Ð<Ð�=IâYA�K�ÕF8:9�9<A�;<Q=ÁÐ�?�Z]H�ÕOUG9<A�KpD�Ey9>A�Õ GGû�R ?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q°8�QYK°8)?6Z[QGâFKGA�QY?6A¥Ð<?6Z[;>A�=@Ð�?6Z[H�ÕOUG9>ANKôÝ�=|9<\9>\OANÐ�AJKG=@Ðr9*8�QY?�A�Ð�D��±ÑV9<\OA¥?6Z[QGâFKGA�QY?6A�=@нÏ]Z¡ZGK·A�QOZ]UOÏ[\�MpÓÁA69� Ð�Ð<8�×)ÒFA�9�9<A�;½9>\F8�Q � � ä³ÑÔZ];=IQYÐ�9<8]QY?6A[MY=IÑP8�ÓÁÓp9>\YA�Ð<8�H�ÕOÓÁA�Ð�ÓÁA6ѳ9�=IQ)9<\OA�ÕWZ¡Z]Ó�8];>A�=ÁQ)9<\OA Ð<8�H�A ?6Ó@8]Ð<Ð*çV9>\OA(?6Ó@8]Ð<Ð�=IâF?�8�Ö9>=ÁZ]Q¼=@Ð ?6Z[H�ÕOÓIA�9>A[M$Z�9<\OA�;<Ý�=@Ð�AJÕO;<ZG?6A�A�Kô9>ZßÐr9<A�Õ OD�Ey9>A�Õ OûÙ�\OA�� �%?�ÓIZyÐ�ANÐr9(Ð>8]H(ÕYÓIANÐ=IQÞ9>\YA�ÕWZ¡Z]ÓjMY8]?�?6Z[;<KG=ÁQOÏ 9>Z¥KG=@Ðr9*8�QY?�A�����8];>A�`]A�ÕG9�MGÝ�\Y=IÓÁA½9>\OA�;<A�HJ8]=IQO=ÁQOÏ�Ð>8]H�ÕOÓIANÐ�8�;<AKG=ÁÐ<?�8];<KOA�KpD��=ÁQYKG=ÁQOÏ�9<\OA�� ��?6ÓÁZ[Ð>A�Ð�9%Ð>8]H�ÕOÓIANÐ�?�8]QÞÒWA�KGZ[QOA�=IQ �ä��Wç�ä¶Ý�\OA�;<A���=ÁÐ�9<\OA�Q¡UOH ÒWA�;

Z�ÑbÐ<8�H�ÕOÓÁA�Ð�=ÁQõ9<\OA�ÕFZ¡Z[Ó@ç�Ð>=IQY?�A(9>\OANÐ�A(A�ÓÁA�H�A�Qy9*нQOA�ANKõQOZ]9½9>ZÞÒFA¥Ð>Z];>9>ANK�E G GGM G*HjD�Ù�\YA;>ANKGUY?6ANKdÕWZ¡Z]Ó�=@Ðb9>\YA�Q�ÕY8]Ð<Ð>A�Kd9>Z¥Ð�9<8�Ï[A� �å[DÙ�\OA%9rÝbZ ?6Z[QYÐ�9<8�Qy9*Ð � ��8]QYK� �$H UYÐ�9VÒWA½KGA69<A�;<H(=ÁQOANKJ=IQÜ8]KGÛ:8�QF?6A/UFÐ�=ÁQOÏ 8 Û�8]ÓI=@KO8�Ö

9>=ÁZ]Q)Ð�A�9�DYÙ�\Y=ÁÐ%?�8�Q�A�8[Ð�=ÁÓÁ×¥ÒFA�KOZ]QOA�Ï[;<8]ÕO\O=@?�8�ÓÁÓÁ×JÒy×ÜÕOÓÁZ�9�9<=IQYÏ�9>\OA�A�;>;<Z];%8[Ð�8 ÑÔUYQY?e9<=IZ[QZ�Ñ�� � 8�QYK � � 8:9%A�8[?*\ÜÐ�9<8]Ï]AJä¶Ð�9<8];�9<=IQOÏ(Ý�=I9>\)8�ÓÁÓ�� � A�ùyUY8]Ó^9>Z(9>\YA�QyUYH ÒWA�;�Z�ÑÓÁ8]ÒFA�ÓIANKÐ>8]H(ÕYÓIANÐ�8�QYK � � � å�ÑÔZ[;�8�ÓÁÓPÐ�9<8�Ï[A�Ð*çeDpRb9�A�8[?*\ôÐ�9<8]Ï]A�9>\OA�;>AJ=@Ð�8ÞH�=IQO=ÁH UYH � � 8]QYKH(=ÁQO=ÁH UOH � � Ý�\O=@?*\ßÏ[=IÛ[A�Z[ÕG9>=ÁHJ8�Ó�ÕWA�;>ÑÔZ];<HJ8�QY?�A�Z[Qõ9<\OAJÛ:8�ÓÁ=ÁKO8�9>=ÁZ]Q°Ð�A�9�D�ab×·9*8�`¡=ÁQOÏÓÁ8];>Ï[A�;�Û:8�ÓÁUOA�Ð�MOÝbA½?�8�Q)KGAN?6;<A�8]Ð>AC9>\OA�ÕO;<Z]ÒF8�ÒO=ÁÓI=I9r×¥Z]ÑHJ8�`¡=ÁQOÏ�A�;<;>Z[;<ÐbZ[QÜ9<\OA½9>ANÐr9/Ð>A69*Ð�DÙ�\OA�Ð�ÓÁ=ÁÏ]\y9>ÓÁ×¥ÝbZ];*Ð�A½ÕWA�;>ÑÔZ];<H�8]QY?6A½Z]ÑUYÐ>=ÁQOÏJ8�\O=IA�;<8];<?*\¡×JZ]Ñ�KG=@Ð�9<8�QF?6A�Ð�8];>A½ÝbA�ÓÁÓ�ÝbZ];>9>\9>\OAdÐ>ÕWA�A�KôUOÕ$D�Ù�\OA¥?�Z]H�ÕOUG9*8:9>=ÁZ]QF8�ÓP?�Z[Ð�9½Z]Ñ%8�ÕY8:9>9>A�;<Q¼?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]Qô=@н9>\OA�Q¼A�ùyUY8]Ó9>ZYû

s*q�i� y�]��k��rvxqNwyk��Ys*qNt±� ��� �w]�[i��Gm<o�q�² [orq��rq��j¹] Gm*tk��Pt±��k�l�m�� � �[vxt±��k�w[s*m

s*qNi� [�|m>¢[v��¯¹k��Pt±��k�l�m�� �  [orq��¡k��[vx�|v��¯¹

�rq�ormek�s>~t±��k�l�m�� ä GGå�ç

R%ÓÁÓO9>\Y=ÁÐV=ÁÐPÒFA�9�9<A�;P=ÁÓÁÓIUYÐ�9>;*8:9<A�K�Ý�=I9>\d8]QJA6áO8�H�ÕOÓÁA/8[Ð�=IQ+��=ÁÏ]UY;>A KYD[Ù�\O=@ÐPÐ>×GÐr9<A�H Ýb8[ÐUYÐ�ANK�ÑÔZ];C9>\YA �CEO7VE�A�á¡ÕWA�;<=ÁH(A�Qy9�KGANÐ>?�;>=ÁÒFANK)=ÁQõ8¥ÕO;<A�Û¡=ÁZ]UYÐCÐ�AN?e9<=IZ[Q�D��uQõ?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]QZ�Ñ�\Y8]QYKGÝ�;<=|9>9>A�QÜKG=IÏ[=|9*ÐCä�å�æ�á^åNæ½ÕO=Iá¡A�ÓF=ÁHJ8�Ï[A�Ð*çeM � L M � f My8]QYK � � MyÝVA�;>A%9>\OA #�UY?�ÓI=@KGAN8�QKG=ÁÐ�9<8]QY?6ANÐ�8:9�;>ANÐ�Z[ÓIUO9>=ÁZ]Q G � GOM � � ��8�QYK K � K ;<A�Ð>ÕWA�?e9<=IÛ[A�ÓÁ×]D � � Ýb8[ÐP9<\OA½Z]QOA½Ð>=ÁKGANK9<8�QYÏ]A�Qy9PKG=@Ð�9<8�QF?6A%Ý�=I9>\��/Ö¯9<;<8]QYÐ�Ó@8:9<=IZ[Q�M:Z]Q�9<\OA/Ð<8�H�ÕOÓÁA%Ð�=@KGA%Z[QOÓÁ×]M]8�9P;>ANÐ�Z[ÓIUO9>=ÁZ]Q5K � KYD� � Ýb8[ÐW9>\OAPKOZ]UOÒOÓÁAPÐ>=ÁKOA�KC9<8]QOÏ]A�Q[9KG=ÁÐ�9<8]QY?6A�Ý�=I9>\ �/Ö¯9<;<8]QYÐ>ÓÁ8�9>=ÁZ]Q½8:9$;<A�Ð>Z]ÓÁUG9>=ÁZ]Q(å�æ � å�æYD#P8]?*\�Z]Ñ�9>\OA°Ð>UOÒYÐ>A�ùyUOA�Qy9ÜKO=ÁÐ�9<8]QY?6Aô8]KOKGANK�Z]QYA·9<8]QOÏ]A�Qy9ÜÛ]AN?e9<Z];¥Z[Q A�8[?*\ Ð�=@KGA ä³B/Ö9>;*8�QYÐ>ÓÁ8�9>=ÁZ]Q�MyÐ>?�8�ÓÁ=IQYÏYM];<Z�9*8:9<=IZ[Q�M�\¡×¡ÕFA�;>ÒWZ]ÓÁ=Á?CKGA6ÑÔZ[;>HJ8�9>=ÁZ]Q�å[M�\¡×yÕWA�;<ÒWZ]ÓÁ=Á?%KOA6ÑÔZ];<HJ8:9<=IZ[Q�G8�QYKd9>\O=@?*`¡QOA�Ð<Ð*çbUOQy9>=ÁÓp9>\OA½ÑÔUYÓIÓp9*8�QOÏ[A�Qy9%KG=@Ðr9*8�QY?�A�Ýb8[Ðb?�Z]H�ÕOUG9<A�K°ä � L<L ç6DÙ�8]ÒOÓIANG�Ð�\OZ:Ý%Ð�9>\OA%A�áGÕFAN?e9>ANK�Q¡UOH ÒWA�;VZ�ÑpH UOÓI9>=ÁÕOÓÁ×yÖ±8[KOKOÐP8�9�A�8[?*\JZ�ÑW9>\OA�Ðr9*8�Ï]ANÐ�D �±9Ð�\OZ[UOÓ@K�ÒWA%QOZ]9>ANK 9<\Y8:9�9<\OA�ÑÔUOÓÁÓO9*8�QOÏ[A�Qy9�KG=@Ðr9*8�QY?�A�QOA�ANK�Z[QOÓÁ× ÒWA/?6Z[H(ÕYUG9>ANK ÑÔZ[;%å�=IQ G]èUOQO`¡QOZ:Ý�Q(ÕY8:9>9>A�;<QYÐ%ä¶ÕO;<Z]ÒY8]ÒO=IÓÁ=I9r× è�� è I[çeM:8�QYK(Z[QOÓI× Ý�=|9<\ ICÐ<8�H�ÕOÓÁA�Ð�Z[UG9�Z�ÑF9>\OA/Z];<=IÏ[=IQY8]Óå�è �<è]è[èOD:Ù�\OA%QOA�9VÐ>ÕFA�A�K�UOÕJÝ�8]Ð�=ÁQ�9>\OA/Z];*KGA�;�Z]Ñ I]è]èYM�?6Z]H�ÕY8];>ANK(Ý�=|9<\¥?6Z]H�ÕOUO9>=ÁQOÏ�9>\YAÑÔUOÓIÓ$9<8]QOÏ]A�Q[9�KG=ÁÐ�9<8]QY?6A�ÒWA69rÝbA�A�Q)A�Û[A�;<×¥UOQO`¡QOZ:Ý�Q)ÕY8�9�9<A�;<Q�8]QYK�A�Û[A�;<×dÐ<8�H�ÕOÓÁAdä³9<\O=ÁÐ/=ÁÐæ 9<=IH�ANÐ�Ѷ8]Ð�9>A�;�9>\Y8]Q�?�Z]H�ÕOUG9<=IQYÏ�9<\OA½9>\YAO#�UY?�ÓI=@KGA�8]QÞKO=ÁÐ�9<8]QY?6A�8�9bÑÔUYÓIÓ�;<A�Ð>Z]ÓÁUG9<=IZ[QFçeD

Page 18: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Category

ConfidenceConfidence

...Euc. Dist

2x2

Euc. Dist

Cost: 16

4x4

Tang.Dist

Unknown Pattern

Cost: 416x16

...

10,000 3,500 500 5

Prototypes

Cost:20000

Confidence

14 vectors

QSRUT ®��Y® npk��u�rm>orwõorm*s<qNlNw[v��rv|q�w°�ytuvxw[l�k¥~yvxm<o�k�ors�~:¹·q�²%�]vxt±��k�wys<m*t*µ���~ym(¸y�x�rm>o� [orq:s*m<me�[t½²IorqNi�xm<²I�/¾³t±��k�ou�rvxwyl/º�vx�r~ �r~ymbº�~[qN�xm��[k���k��yk�tum6¿�rq/orv|l�~:�%¾@º�~ym<orm�q�wy��¹�k/²Ám<º� [orq��rq��j¹] Gm*torm<i�k�vxwG¿>µ���bmek�s�~Üt±��k�lNm��[vxt±��k�w[s*m*t��Gm<�¯º�m*m<wÜ ]orq��rq��¯¹] Gm*t�k�w¡���r~ym/�yw[�]w[qeº�wd yk��u�rm<orwdk6orm�s*q�i� y�[�rm*�Y�tuq�ou�rme�Y�yk�wy���r~[mb�Gm*t±��s*k�w¡�]v|�yk��rmb [orq��rq��j¹] Gm*t�k6orm�tum*�xm*s<�rme� ²@q�o��r~[m�wym>¢]��t±��k�lNm�µ¡�Pt�r~ym�s<qNi½� [�xm<¢]vx�¯¹(q�²p�r~ym/�[vxt±��k�wys*m%vxwys<orm*k�tum*t*�¡�r~[m%w]�[i��Gm<o�q�²$ [orq��rq��j¹] Gm*tP�]m*s<ormek�tum*t*�Oi�k���vxwylCs*qNi� [�[���k6�rvxqNw)²@mek�tuvx�y�xmNµ�����mek�s>~·t±��k�lNm�kJs*�|k�tutuvx¸ysek��rvxq�wôvxt½k6�u�rm*i� [�rme�)k�w¡�)kJs*q�w[¸G�]m*wys<m�tus*q�orm�vxts<qNi� y�]�rme�Yµ]h¶²W�r~ym�s<qNw[¸¡�[m*w[s*m�tus*q�orm�vxtV~[v|l�~(m*w[qN�[lN~F�]�r~ymVorm*i�k�vxwyvxw[l½t±��k�l�m*tbk6orm�tu��v| [ Gme�Yµ

� � ��!� ���� �!�;�*�6!� � ��� Ù�8]QOÏ]A�Q[9%KG=@Ð�9<8�QF?6A�?�8]Q�ÒFA�Û¡=ÁA�ÝbA�K�8]Ð�Z[QOA�=I9>A�;*8:9<=IZ[Q�Z�Ñ�8B%A�Ý�9<Z]QGÖj9rסÕFA½8�ÓÁÏ]Z[;>=I9>\YH Ý�\Y=Á?*\ÜâYQYKOÐV9<\OA½ÕFZ[=IQy9<ÐbZ]Ñ$H�=IQO=ÁH UYH KG=@Ðr9*8�QY?�ACZ[Q¥9>\YAC9<;>UYA9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q�HJ8�QO=IÑÔZ]Ó@KOÐ�D�Ù�\OA�Û]AN?e9>Z[;<Ð�� � 8�QYK�� � 8];>AV9>\OA%?�Z¡Z];*KG=IQF8:9>ANÐZ�ÑW9>\OA�9rÝbZ?6ÓÁZ[Ð>A�Ð�9bÕWZ]=ÁQy9<Ð�=ÁQd9<\OA�;>ANÐ�ÕWA�?69>=ÁÛ]AC9<8]QOÏ]A�Q[9%Ð>ÕY8]?�A�Ð�M¡ÒOUG9�9>\OA�×Ü?�8�QÜ8]ÓÁÐ>Z(ÒFA½=ÁQ[9<A�;<ÕO;<A69>ANK8]Ð�9<\OAdÛ:8]ÓIUOA¥ÑÔZ];�9<\OAd;<A�8]ÓCä¶QOZ]QGÖ±ÓÁ=IQOAN8�;eç�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[QYÐ�D��uQ�Z�9<\OA�;�ÝVZ[;<KOÐ�M�ÝbAd?�8]QUYÐ�A � � 8]QYK � � 9<Z�?6Z[H(ÕYUG9>A)9<\OA·ÕWZ]=ÁQy9<Ð+�yä � ��� � çJ8]QYK �yä����,� � ç6M�9<\OAõ;<A�8]Ó�QOZ]QGÖÓI=ÁQOA�8]; 9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q°Z�Ñ � 8�QYK ��D��Y;>Z[H 9<\OA�Ð>AÞQOA�Ý ÕWZ]=ÁQy9<Ð�MÝbAÜ?�8�Q¼;<A�?�Z]H�ÕOUG9<A9>\OA�9*8�QOÏ[A�Qy9�Û[A�?69>Z];*Ð�M^8]QYKõ9>\OA(9<8]QOÏ]A�Qy9�KO=ÁÐ�9<8]QY?6AJ8�QFKõ;<A�=I9>A�;<8�9>A�9>\OA�ÕO;<ZG?6A�Ð<Ð�D �±Ñb9>\YA8�ÕOÕO;<Z]ÕY;>=@8:9<A%?6Z]QFKG=|9<=IZ[QYÐ�8];>A�H�A69NM]9<\O=@ÐPÕO;<ZG?6ANÐ>Ð�?�8�QJ?6Z[Q¡Û]A�;<Ï]Ab9<Z 8�ÓÁZ¡?�8�ÓFH(=ÁQO=ÁH UOH =ÁQ9>\OA�KG=@Ð�9<8�QF?6A½ÒFA�9rÝVA�A�QÞ9>\YA½9rÝVZ(9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[QdHJ8�QY=|ÑÔZ[ÓÁKÜZ�Ñ �48]QYK �JDÙ�\O=ÁÐ�ÕY;>ZG?6ANÐ>Ð�KG=@K#QOZ�9)=ÁH�ÕO;>Z:Û[A�K \Y8�QYKOÝ�;>=I9�9<A�Q÷?*\Y8�;*8]?69>A�;�;<A�?�Z]Ï]QY=|9<=IZ[Q�M�ÒOUG9·=I9

×y=ÁA�Ó@KGANK�=IH�ÕO;<A�Ð<Ð>=IÛ[A·;>ANÐ�UYÓ|9*Ðd=ÁQ�Ѷ8]?�Aõ;<A�?�Z]Ï]QY=|9<=IZ[QE G �*H±D �uQ 9<\Y8:9�?�8[Ð�A[MVAN8]?*\ Ð>UY?�?�A�Ð�ÖÐ�=ÁÛ]AÜ=|9<A�;*8:9>=ÁZ]Q�Ýb8[Ð�KGZ[QOAÜ8�9�=ÁQY?6;<A�8[Ð�=ÁQOÏß;>ANÐ�Z[ÓIUG9<=IZ[Q äÔ\OA�QY?6A�?6Z[H ÒO=ÁQO=ÁQOÏõ\Y=IA�;<8];<?*\O=@?�8]ÓKG=ÁÐ�9<8]QY?6ANÐ�8]QYK�H�UOÓI9>=ÁÕOÓIAÞ=I9>A�;*8:9<=IZ[QYÐ*çeM�HJ8�`¡=IQYÏ·9>\YAÜÝ�\OZ[ÓIAÜÕO;<Z¡?�A�Ð<Ð�?�Z]H�ÕOUG9*8:9<=IZ[QY8�ÓÁÓI×A(J¥?6=ÁA�Qy9�D

� ����Æ�����ÆPÇ��õÈOÉ�� �����ÇOÍrÉPÆ

�uQYÐr9<A�8[KßZ]Ñ�KOA6âYQO=ÁQOÏ·8�H�A69<;>=@?(ÑÔZ];�?6Z]H�ÕY8];>=ÁQOÏ�Ó@8�ÒWA�ÓÁA�K°Ð>8]H�ÕOÓIANÐ�8�QFKßUYQO`¡QOZ:Ý�QßÕF8:9�Ö9>A�;<QYÐ�M:=|9�=@Ð�8�Ó@Ð�Z�ÕFZyÐ>Ð>=ÁÒOÓIAb9<ZC=ÁQY?6Z[;>ÕWZ];*8:9<A�9<\OA�=IQ¡Û:8�;<=Á8]QY?6A�KG=Á;>AN?e9<ÓI×�=ÁQy9>Z�8C?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]QÑÔUOQY?e9<=IZ[Q�DpÙ�\O=@ÐC?�8�Q·ÒWA(KOZ]QOA�Òy×�ÓÁA�8];>QO=ÁQOÏ ',(8Wêxí¶é�í³ïjê +�9<\OA 9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q�=ÁQyÛ:8];>=@8�QY?�A=IQy9>Z�9<\OA�?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q¥ÑÔUOQY?69>=ÁZ]Q�D��uQÞ9>\O=@Ð%Ð>A�?e9<=IZ[Q�MOÝbA½ÕO;>ANÐ�A�Qy9/8�H�ZGKG=IâF?�8�9>=ÁZ]Q�Z�Ñ9>\YAÒY8]?*`¡ÕO;<Z]ÕY8]Ï[8�9>=ÁZ]Qõ8]ÓIÏ[Z];<=|9<\OH�M^?�8�ÓÁÓIANK ö�9<8]QOÏ]A�Q[9�ÕO;<Z]ÕF8�Ï[8�9>=ÁZ]QYøYMF=ÁQ°Ý�\O=@?*\õ9<\OAJ=IQ¡Û:8�;<=IÖ8�QY?�A½8];>A½ÓÁA�8];>QOANKdÒ¡×¥Ï[;<8[KG=ÁA�Qy9�KOA�Ð<?6A�Qy9ND�ßA�8�Ïy8�=ÁQ�8]Ð<Ð�UYH(A �Jä # ç�9<ZßÒWAÞ9>\OA°äÔUYQO`¡QOZ:Ý�QFç ÑÔUOQY?69>=ÁZ]Q�9<ZßÒWA�ÓIAN8�;<QOA�K�M � �

! ä # L ���Jä # L ç.������� ��ä #�� ���Jä #�� ç -�9>Z(ÒWA½8�âYQO=I9>A�Ð>A69�Z�Ñ�9<;<8]=IQO=ÁQOÏ(KO8:9*8�9*8�`]A�QJÑÔ;<Z]H 8(?6Z]QOÖÐr9*8�Qy9ÜÐ�9<8:9<=ÁÐ�9>=@?�8]Ó/KG=@Ðr9<;>=ÁÒOUG9<=IZ[Q �¥MV8]QYK �"!�ä # çJ9>Z�ÒWAõ8�Ð>A69ÜZ�ѽÑÔUOQY?69>=ÁZ]QYÐ�MP=ÁQYKGA�á¡ANK

Page 19: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

� �#q�²p�bµ �/µ }Vm<tuq �#q�²p [orq��rq�¾����¶¿ �#q�²^ [orq�� n�orq��¡k�� �#q�²pi��[����kN�[�� � £ �N����� � ��µ ��� £���� ���N�¤ � ��� � ���N� � ��µ ��� ����� ���N�� � ��£ ���N� � ��µ ��� ��¤:� ���N�£ � ��£ �6¤ � ¤ ��µ ��� �*£]� ���N�£ ¤ ¤ � � ��� � ��µ ��� £���� ���N�� £ ¤ � � £ � � ��µ £N� ��¤:� ���N�� � ¤ � � ¤ � � ��µ ¤�� �N��� ���N�� � ¤ � � � � �N� ��µ|�*� £]� ���N�� �e� ¤ � � �e� �e� ��µ|�*� ��� ���N��*� �6¤ ¤ � � � � � ��µ � � ��� ���N���� �*£ ¤ � � � �6� ��µ � � ��� ���N�

� ¬[¨���� �Y® z��yi�i�k6ou¹�s*q�i� y�]��k��rvxqNw½²Áq�o��r~ymPs*�|k�tutuv�¸ysek��rvxqNw�q�²$�b yk��u�rm<orw�����~ymP¸yort±�s*qN�x�yi�w�vxt�r~[m��[vxt±��k�wys*m�vxw¡�]m<¢O�6�r~ym�tum*s<qNw¡�/s*qN�x�[i�w�vxw¡�]v|s*k��rm*tp�r~ymw]�[i��Gm<oWq�²¡��k�wyl�m*w:��{�m*s<�rq�oV¾Ô��²@q�o$�r~[m���ys<�|v|�]mek�w �[vxt±��k�wys*m6¿>�:k�w¡���r~[m��r~yv�o��½s*qN�x�[i�w½vxw¡�[vxsek6�rm*t��r~[m�orm*tuqN�x�[�rvxq�w�vxw� yv�¢]m*�xt*���r~[m�²ÁqN�]ou�r~vxt�� � q�o%�r~[m�w��yiC�Gm<o�q�²� [orq��rq��¯¹[ Gm<t�qNwܺ�~[vxs>~d�r~ym½�]vxt±��k�wys<m�� � i��[t±���Gm½s*q�i� y�]�rme�Y�¡�r~[m¸[²Á�r~�s*q��|�[i�w�vxw¡�[vxsek6�rm*t�r~[mPw��[i��Gm<o$q�²WkN�[�[v��rvxqNwyk��O�[q��� ]orq��[�ys>�rt�º�~yvxs�~�i��yt±�$�GmPs*qNi� [�[�rme��rqJm<{Nk��|�yk��rm��[vxt±��k�w[s*m� � �W�r~[m tuv�¢��r~�s*q��x�yi�w)vxw¡�]v|s*k��rm*t%�r~ym� [orq��¡k��yvx�xvx�¯¹Þ�rqJw[q��Ctu�]vx ��r~yk��t±��k�lNm%k6²Á�rm<o��r~[m�s*qNw]¸G�]m*wys*mVtus*q�ormb~¡k�t��Gm<m*w(�[tume�Y�]k�w¡� �r~[mb�|k�t±�Ps*q��x�yi�w�vxw¡�[vxsek6�rm*t��r~ymP�rq���k��ke{�m>o�k�lNmVw��yi��Gm>oq�²WiC�y���rvx y��¹��jk��y�]t�º�~[vxs>~ i��[t±��GmP Gm<ou²@q�ori�me�J¾³ [orq��]�ys<�q�²Fs*qN�x�yi�w����rq ��¿k6�Pmek�s�~�t±��k�l�mNµ

Òy×Þ9<\OA�Û[A�?69>Z];�Z�Ñ�ÕF8�;*8�H�A69<A�;*Ð#�D��ôA�Ý�=@Ð�\õ9>ZdâYQYK # Ý�\O=@?*\·H�=IQY=IH�=Áã�A�ÐC9<\OA�A�QOA�;<Ï]×ÑÔUOQY?e9<=IZ[Q &��� � � ! ä # ç �Jä # ç � f� �dä # ç ä G G[çÏ]=ÁÛ]A�Q�8�âYQO=I9>A Ð<8�H�ÕOÓÁA"� 9<8]`]A�QÜÑÔ;<Z]H 9<\OA�KG=@Ðr9<;>=ÁÒOUG9<=IZ[Q �¥DFÙ�\O=ÁÐ�?�8�Q�ÒWA�8]?*\O=ÁA�Û[A�K¥Ò¡×H(=ÁQO=ÁH�=Iã�=IQOÏ

� � ��� � � L

��� ! ä # �±ç �Jä # �jç � ä G yçZ:Û]A�;^9<\OA�9<;<8]=IQO=ÁQOÏ%Ð>A69 �dD�R/Q�A�Ð�9>=ÁH�8�9>A�Z]Ñ #.?�8�Q½ÒWAP?6Z[H�ÕOUG9>ANK½Ò¡×%ÑÔZ]ÓÁÓÁZ:Ý�=IQOÏ%8�Ï[;<8[KG=ÁA�Qy9KGA�Ð<?6A�Q[9�UYÐ>=ÁQOÏ�9>\OA�ÝbA�=ÁÏ]\y9>ÖjUOÕ^KO8�9>A½;>UYÓIA[û

� # � �� � � �� #

ä G �¡ç

S�A69� ÐÜ8]Ð<Ð�UYH(A·9>\Y8�9Ü=ÁQ 8]KOKO=|9<=IZ[Q 9>Z¼9<\OA·9<;<8]=IQO=ÁQOÏ�Ð>A69 �¥MbÝVAõ8]ÓÁÐ>Z¼`¡QOZ:Ý 8�Q�=ÁQOÕOUG99>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q �[ä # �,�çCÝ�=I9>\ô;>ANÐ�ÕWA�?69C9<ZÞÕF8�;*8�H�A69<A�; �PMpÐ>UY?*\·9<\Y8:9 �yä # �>è[ç � # D��ôA8�Ó@Ð�Z)8]Ð<Ð>UOH�A�9<\Y8:9���� � ��� �� � � � �� � =@н`¡QOZ:Ý�Q°8:9 � � èYD$Ù$Z�=ÁQY?6Z[;>ÕWZ];*8:9<A 9<\OAJ=ÁQyÛ:8];>=@8�QY?�AÕO;>Z[ÕFA�;�9r×d=IQy9>Z �"!�ä # çeMFÝVA�8]KYK¥9<\Y8:9�9<\OA½ÑÔZ]ÓÁÓIZ:Ý�=ÁQOÏJ?6Z[QYÐ�9>;*8�=ÁQ[9�Z]QÜ9>\YA�KGA�;<=IÛ:8�9>=ÁÛ]A]û

��� � �� � � L

���� � �"!Vä �yä # � �,�ç>ç� � � �Jä �[ä # � ����ç�ç

� ����� f � � � ä G I[ç

Ð�\OZ[UOÓ@K¥ÒWA½Ð>HJ8�ÓÁÓp8:9 � � èYD�uQdHJ8�Q¡×JÕY8�9�9>A�;>QÞ?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]QdÕO;>Z[ÒOÓÁA�HJÐ�M¡ÝbAC8];>A�=IQy9<A�;>ÖA�Ð�9>A�KÜ=ÁQÜ9<\OA½ÓIZG?�8�Ó�?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q¥=IQ¡Û:8�;<=@8�QY?�ACÕO;<Z]ÕWA�;>9r×�ÑÔZ];�Jä # çbÝ�=I9>\�;<A�Ð>ÕFAN?e9�9>Z(9>\YA

Page 20: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q �)ä³9<\OA¥?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[QßKGZ¡A�нQYZ�9�?*\Y8�QYÏ]A(Ý�\OA�Qô9>\YA(=ÁQOÕOUO9�=ÁÐ�Ð>ÓÁ=IÏ[\[9<ÓI×9>;*8�QYÐ�ÑÔZ];<H�A�KYç6MOÐ�Z�ÝbA½?�8�Q�Ð�=ÁH�ÕOÓÁ=|ÑÔ×ÜA�ùyUY8:9<=IZ[Qôä G I[çV9>ZYû

� � � �� � � L

���� � �"!bä �yä # � ����ç�ç� ����� f � � � ä G�æyç

Ð�=ÁQY?6A ��� � ��� �� � ��� �� � � èYDyÙ�Z(H(=ÁQO=ÁH�=Iã�A/9<\O=ÁÐV9<A�;<H ÝbA½?�8�QdH(ZGKG=IÑÔ×�9>\YACÏ[;<8[KG=IA�Qy9VKGA�Ð<?6A�Qy9;>UOÓÁA�9>ZJUYÐ>A�9>\OA�A�QOA�;<Ï]×¥ÑÔUOQY?e9<=IZ[Q

� � � � � � ��� ä G M]çÝ�=|9<\Ü9<\OA�ÝbA�=ÁÏ]\y9�UOÕWKY8:9>A�;<UOÓÁA]û � # � � �

� #ä G Kyç

Ù�\OAßÓÁA�8];>QY=IQOÏ�;<8�9>A�ÐôäÔZ[;Ü;<A�Ï[UOÓ@8�;<=IãN8:9>=ÁZ]Q ÕY8];<8]H�A69>A�;<Ð*ç � 8]QYK � 8];>Aõ9>;<A�H�A�QYKOZ]UYÐ>ÓI×=IH�ÕWZ];>9<8�Qy9NM�ÒFAN?�8]UYÐ�A�9<\OA�×°KGA�9>A�;<H�=IQYAJ9>\OA�9<;<8[KGA�Z]à�ÒWA69rÝbA�A�Q¼H�=ÁQO=ÁH(=Áã�=ÁQOÏ�9<\OA¥UYÐ>UY8�ÓZ]ÒGØrAN?e9>=ÁÛ]A�ÑÔUOQF?e9>=ÁZ]Q)8]QYKdH�=ÁQO=ÁH(=Áã�=ÁQOÏ�9<\OA�KG=Á;>AN?e9>=ÁZ]QF8�Ó�KGA�;>=ÁÛ:8:9>=ÁÛ]A½A�;>;<Z];NDÙ�\OA�ÓÁZ¡?�8�Ó�Û:8];>=@8:9<=IZ[Q Z�Ñ�9>\OA.?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Q#ÑÔUOQY?69>=ÁZ]Q�M�Ý�\O=@?*\ 8�ÕOÕWA�8];·=ÁQ A�ùyUY8�Ö

9>=ÁZ]Q¼ä�G]æ[çb?�8]QÜÒWA�Ý�;>=I9�9<A�Q)8]Ð�û

� � ! ä2�yä # ����ç�ç� �

���� � � � � � � ! ä2�yä # ����ç�ç� �yä # ����ç

� �[ä # �,�ç� �

���� � � � ��� �"!�ä # ç � � �yä # ����ç� �

���� � � �ä G(�yçÐ�=ÁQY?6A �yä # ����ç � # =IÑ�� � èYDyÝ�\YA�;<A � � ! ä # çb=@ÐP9<\OA�X[8[?6Z[ÒO=Á8]Q¥Z�Ñ � ! ä # çVÑÔZ[;bÕF8:9�9<A�;<Q# MP8]QYK � �yä���� # ç�� � � =@Ð(9>\YA¼ï±ë�ñ;:;'6ñFï ��'*é�ï±ð )(8[Ð>Ð>ZG?6=@8:9>ANK�Ý�=I9>\.9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q �)8[ÐKGA�Ð<?6;<=IÒWA�K�=IQ 9>\OAVÕO;>A�Û¡=IZ[UYÐ�Ð>A�?69>=ÁZ]Q�D�2)UOÓI9>=ÁÕOÓÁס=IQOÏ/9>\OAV9<8]QOÏ]A�Qy9�Û]AN?e9<Z];$Ò¡×�9>\YAbXy8]?�Z]ÒO=@8�Q=IQ¡Û]Z[ÓIÛ[A�ÐPZ]QOA/ÑÔZ];<Ýb8];<K�ÕO;<Z]ÕF8�Ï[8�9>=ÁZ]Q�9<\O;>Z[UOÏ]\¥8ßö�ÓÁ=ÁQOA�8];>=Áã�ANKOø�Û]A�;<Ð>=ÁZ]QJZ�Ñp9<\OA�QOA69rÝbZ];<`^D�±Ñ1� =@Ð�H UOÓI9>=IÖ±KO=IH�A�QFÐ�=ÁZ]QY8]Ó¯M�9>\YA¥ÑÔZ];<Ý�8�;*KßÕY;>Z[ÕY8�Ïy8:9>=ÁZ]Q°H�UYÐr9(ÒFAÞ;<A�ÕWA�8:9<A�KôÑÔZ[;�AN8]?*\9<8�QYÏ]A�Qy9�Û]AN?e9<Z];NDÙ�\OAd9<\OA�Z[;>×¼Z]ÑCS�=ÁA)8�ÓÁÏ]A�ÒO;<8[Ð EIå]å�H�A�QYÐ>UO;>ANÐ�9>\Y8�9J?�Z]H�ÕWZ[Ð>=|9<=IZ[QYÐ Z]ÑCÓÁZG?�8�Ó�ä¯Ð�HJ8�ÓÁÓ³ç

9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐd?6Z[;>;<A�Ð>ÕWZ]QYK 9<Z�ÓÁ=IQOAN8�;�?6Z[H ÒO=ÁQY8:9<=IZ[QYÐ¥Z]Ñ�9>\YAô?�Z];<;>ANÐ�ÕWZ]QYKO=IQOÏ°9*8�QGÖÏ]A�Qy9%Û[A�?e9<Z];*нä³9>\Y=ÁÐ�;<A�Ð>UOÓI9%Ý�=IÓÁÓ�ÒWA�KG=@Ð>?�UYÐ>Ð>A�KÜÑÔUO;>9>\YA�;%=ÁQ)Ð�AN?e9>=ÁZ]Q/�yç6DWTVZ]QFÐ�ANù[UYA�Qy9>ÓÁ×]MG=IÑ� � ä # ç � è�=ÁÐ�Û[A�;<=|âFA�KpM�9<\OA%QOA�9rÝVZ[;>`�KGA�;>=ÁÛ:8:9<=IÛ[Ab=ÁQ�9>\YA%KG=I;<A�?69>=ÁZ]Q(Z�Ñp8½ÓI=ÁQOAN8�;�?6Z[H ÒO=ÁQY8�Ö9>=ÁZ]Q�Z�Ñ^9>\YA�9<8�QYÏ]A�Qy9�Û[A�?e9<Z];*Ð�=ÁÐ�A�ùyUY8�Ó¡9<Z�9>\YA%Ð>8]H(A�ÓÁ=ÁQOA�8];�?6Z]H�ÒO=ÁQY8:9<=IZ[Q(Z]ÑF9<\OA/KGANÐ�=Á;<A�KKGA�;<=IÛ:8�9>=ÁÛ]A�Ð�D��uQ°Z�9<\OA�;�ÝbZ];*KOÐ�Mp=|Ñ�9>\YAJQOA69rÝbZ];<`·=@Ð�Ð�UY?�?6ANÐ>Ð�ÑÔUOÓÁÓI×õ9>;*8�=ÁQOA�Kõ9>Z)ÒWA¥ÓÁZ¡?�8�ÓÁÓI×=IQ¡Û:8�;<=Á8]Qy9CÝ�=I9>\õ;<A�Ð>ÕWA�?e9C9>ZFM^Ð<8�×]MF\OZ];<=Áã�Z]Qy9*8�Ó�9<;<8]QYÐ>ÓÁ8�9>=ÁZ]QYÐC8�QYKõÛ]A�;>9>=@?�8]Ó�9<;<8]QYÐ�Ó@8:9<=IZ[QYÐ�M=|9%Ý�=ÁÓIÓ$ÒFA�=ÁQ¡Û�8];>=@8�Qy9�Ý�=|9<\�;>ANÐ�ÕWA�?69�9>ZJ?6Z[H(ÕWZ[Ð>=I9>=ÁZ]QYÐb9<\OA�;<A�Z]ÑrD�±9V=@ÐPÕWZ[Ð<Ð�=ÁÒOÓÁA%9<Z�KGA�Ûy=@Ð>AC8�QdA(J¥?6=ÁA�Qy9b8]ÓIÏ[Z];<=|9<\OH�M$ö�9<8�QYÏ]A�Qy9PÕY;>Z[ÕYøOM]ÑÔZ];bÕFA�;�ÑÔZ[;>H�=ÁQOÏ

9>\OA�ÝVA�=IÏ[\y9VUOÕ^KO8:9<A(ä¶A�ùyUY8�9>=ÁZ]Qõä G K[ç>çeD �±9b=@Ð�8�QY8]ÓIZ[Ï]Z[UYÐ�9>Z�Z];*KG=ÁQY8�;<×�ÒF8]?*`¡ÕO;<Z]ÕY8]Ï[8:9<=IZ[Q�MÒOUG9�=ÁQ(8[KOKG=I9>=ÁZ]Q�9<Z½ÕO;>Z[ÕY8�Ïy8:9<=IQOÏCQOA�UY;>Z[Q�8]?69>=ÁÛ:8:9>=ÁZ]QFÐ�M�=I9P8�Ó@Ð�Z�ÕO;<Z]ÕY8]Ï[8:9<A�Ð�9>\OAb9<8]QOÏ]A�Qy9Û]A�?69>Z[;<Ð�DGÙ�\OA�A�ùyUY8�9>=ÁZ]QYÐ�?�8�QÞÒFA�AN8]Ð>=IÓÁ×dKGA�;>=ÁÛ]ANKJÑÔ;<Z]H �=ÁÏ]UO;<A �OD

��� �L �� ���>� ����Ù�\OA½ÑÔZ];<Ýb8];<KdÕO;<Z]ÕY8]Ï[8:9<=IZ[Q¥A�ùyUY8�9>=ÁZ]QÞ=ÁÐ�û

��� � � �

# �� # � � L # � � � ��ä �� � ç ä ]èyç

Page 21: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

�Vm<�jºq�or� ��k�s*qN�[v|k�wJwym>�jº�q�or�

����� g� �

� �� � ��

� �� ��

��

� ��

� � g�

� ��

� ��

� � � g�� � � g� � � � g�

� � �

����� g� �

� �� � � �� �

QSRUT ®��Y® ¦yq�ouº�k�o��Þ ]orqN yk�l�k6�rme��{�k�orv|k��[�|m<t ¾ ��� � � � � ¿>�^k�w¡�d�¡k�s��:º�k6o��Þ [orq� ¡k�lNk��rme�Ü{Nk6orvIk��y�xm*t¾ ����� � � � ¿�vxw��r~ym�orm*l��y�|k�obwym<�¯º�q�or�d¾@orqNi�k�w�t±¹]i��Gq��|t�¿�k�wy���r~ym/��k�s*qN�[v|k�w·¾³�xv|w[mek�orvx�*m*�¡¿Pw[m<�u�ºq�or�½¾³l�orm*m*�Vt±¹[iC�GqN�xt�¿>µ��q�w]{Nm<orlNvxwyl�²Áq�or��t¾³vxw��r~ym$�]vxorm<s<�rvxqNw%vxw%º�~[vxs>~��r~ym�tuvxlNwyk��:vxtF�uo�ke{�m<�|vxw[l:¿k6orm�tu�yi�t*�[�[vx{�m>orlNvxwyl�²Áq�or��t��±�yt±�V�]�y y�xvxsek6�rmb�r~ym�{�k��x�ym<t*µ

��\OA�;<A��¥=ÁÐ�8CQOZ]Q�ÓI=ÁQOA�8];KO=|à^A�;<A�Qy9>=@8�ÒYÓIAVÑÔUOQY?69>=ÁZ]Q�ä³9rסÕO=@?�8�ÓÁÓÁ×½8�Ð>=IÏ[H(Z[=ÁKFçeD:Ù�\OAVÑÔZ];<Ýb8];<KÕO;>Z[ÕY8�Ïy8:9<=IZ[QÞÐ�9<8];�9*Ð/8:9/9>\OA âY;<Ð�9CÓ@8�×]A�;�ä�� � åNç6MYÝ�=I9>\ # � ÒWA�=ÁQOÏJ9<\OA =ÁQOÕOUO9CÓ@8�×]A�;NMY8]QYKA�QYKOÐ�8:9�9<\OA¥Z[UG9>ÕOUO9�Ó@8�×]A�;¥ä�� � �VçeD�E¡=ÁH(=ÁÓ@8�;<ÓI×[M$Ù�\OAJ9*8�QOÏ[A�Qy9�ÑÔZ];<Ý�8�;*KõÕO;<Z]ÕF8�Ï[8�9>=ÁZ]Qä³9<8]QOÏ]A�Qy9�ÕY;>Z[ÕFçb=ÁÐ�KOA6âYQOANKÞÒy×^û

� �� � �

# �� �� � � L � �� � � $ ä � � � ç � �� ä Oå�ç

Ù�\OA½9<8]QOÏ]A�Q[9�ÑÔZ[;>Ý�8�;*KÜÕO;>Z[ÕY8�Ïy8:9<=IZ[QÜÐ�9<8�;>9<Ð�8�9�9>\OA�âY;<Ð�9%Ó@8�×]A�;�ä�� � åNçeMYÝ�=|9<\ � � ÒFA�=IQYÏ9>\OA�9*8�QOÏ[A�Qy9�Û[A�?69>Z]; ����� � ���� � M$8�QFKõA�QYKOÐ�8:9½9<\OAJZ]UG9<ÕOUG9�Ó@8�×]A�;Jä�� � �bçeD�Ù�\OA(9<8]QOÏ]A�Qy9Ï];*8]KG=ÁA�Qy9�ÒY8]?*`¡ÕO;<Z]ÕY8]Ï[8�9>=ÁZ]Qd?�8�QÞÒFA�?�Z]H�ÕOUG9<A�KÞUYÐ>=IQOÏ(9<\OA�?*\Y8�=ÁQÞ;>UOÓÁA]û

� �� � ��� �

�� �� � � � L�

� � � � L�� � ��

� �� � �� � �

� � ��� � ��� � �� ä G[ç

! �� � ��

" � � L� # � � L� � " �� �#! �� � $±ä �� � ç ä yç

Ù�\OA�9*8�QOÏ[A�Qy9CÒY8[?*`¡Ýb8];<K�ÕO;<Z]ÕF8�Ï[8�9>=ÁZ]Q)Ð�9<8�;>9<ÐC8:9C9>\OA(Z]UG9<ÕOUG9½ÓÁ8�×[A�;(ä�� � �bçeMWÝ�=|9<\ �%$ÒFA�=IQOÏd9>\OA�QOA69rÝbZ];<`dÛ:8];>=@8:9<=IZ[Q ��&(' � � � � ��� �� � MW8�QYK·A�QYKOÐ�8:9C9>\OA�=IQYÕOUG9CÓ@8�×]A�;�DWE¡=IH�=ÁÓÁ8];>ÓÁ×]M

Page 22: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

9>\OA�Ï[;<8[KG=IA�Qy9bÒY8[?*`¡ÕO;>Z[ÕY8�Ïy8:9<=IZ[Q¥=ÁÐ�û

� �� # � �� �

�� �� � � � L�

� � � � L�� # � �

� �� � � �� � �� # � �

� # � �� � � �

� �� � ��

� � ��� � � �

ä (�¡ç

� � � � ��" � � L� # � � L� � " �� � � � � � $¯ä ��� � ç ! � � $ $¯ä ��� � ç � �� ä I[ç

Ù�\OAÞÐ�9<8]QYKO8];<K¼ÒF8]?*`¡Ýb8];<K°ÕY;>Z[ÕY8�Ïy8:9>=ÁZ]Q�Ð�9<8];�9*Ð�8�9�9<\OAÞZ]UG9<ÕOUG9JÓÁ8�×[A�;Þä�� � �bçeMÝ�=|9<\# $ � � ! ä # � ç�ÒFA�=IQYÏ�9>\OAÜQOA69rÝbZ];<`·Z]UO9>ÕOUG9NM�8]QYK°A�QYKYÐ 8:9�9<\OA¥=ÁQOÕOUG9(ÓÁ8�×[A�;ND��=IQY8]ÓIÓÁ×]M9>\OA�ÝbA�=ÁÏ]\y9�UOÕ^KO8:9<A�=ÁÐ�û

� # �� � � �� � � � � � � �� # �� � �� � �� �

� ��� # ��

ä ]æyç

� # �� � " �� # � � L " �� � � � L ä ;M]çÙ�\OA�?6Z]H�ÕOUO9<8:9<=IZ[Qd;<A�ùyUO=Á;>ANÐbZ]QOACÑÔZ];<Ýb8];<K¥ÕO;>Z[ÕY8�Ïy8:9<=IZ[Qd8�QYKdZ]QOA½ÒY8[?*`¡Ýb8];<K�ÕO;<Z]ÕF8�Ï[8�Ö9>=ÁZ]Q¼ÕWA�;(ÕY8:9>9>A�;<Q�8�QFK°ÕFA�;�9<8]QOÏ]A�Qy9 Û]AN?e9>Z[; KGUO;<=ÁQOÏ�9>;*8�=ÁQO=ÁQOÏYDR�ѳ9>A�; 9<\OA¥QOA�9rÝVZ[;>`ß=ÁÐ9>;*8�=ÁQOA�KpM¡=I9b=@Ð�Ð�=ÁH�ÕOÓÁ×¥ÓIZG?�8�ÓÁÓI×�=ÁQyÛ:8];>=@8�Qy9VÝ�=I9>\Þ;<A�Ð>ÕFAN?e9b9>Z�9>\OA½?*\OZ[Ð>A�Qd9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q�DÙ�\OA�?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Qd?�Z]H�ÕOUG9*8:9>=ÁZ]QÜ=ÁÐb=ÁQÞ8]ÓIÓ^Ý�8�×GÐP=@KGA�Qy9<=Á?�8�ÓW9>Z�8 QYA69rÝbZ];<`(Ý�\Y=Á?*\Ü=ÁÐbQOZ]99>;*8�=ÁQOA�KdÑÔZ];�=ÁQ¡Û�8];>=@8�QF?6AJäÔA�áG?�A�ÕG9�ÑÔZ[;�9>\OA�ÝbA�=ÁÏ]\y9*ÐVÝ�\Y=Á?*\�\Y8�Û[A�KG=Ià^A�;<A�Qy9%Û:8�ÓÁUOA�Ð*çeD

�� � ;����� !��Ù�ÝVZ½A6áGÕWA�;<=IH�A�Q[9*Ð�=ÁÓIÓÁUYÐ�9>;*8:9>Ab9<\OA%8]KGÛ:8]Q[9*8�Ï[A�ÐZ�ÑW9<8]QOÏ]A�Q[9�ÕO;<Z]Õ$D]Ù�\OA�âY;*Ðr9�A�á¡ÕWA�;<=ÁH(A�Qy9=ÁÐ�8ô?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Q¼9*8]Ð>`^M�UYÐ>=ÁQOÏô8ôÐ>H�8]ÓIÓ�ä¶ÓI=ÁQOA�8];>ÓÁ×¼Ð�A�ÕY8�;*8�ÒYÓIA�ç�Ð>A69JZ�Ñ1�;K]èõÒO=IQF8�;<=Iã�A�K\Y8�QYKOÝ�;>=I9�9<A�Q°KG=IÏ[=|9*Ð�D�Ù�\YA 9<;<8]=IQO=ÁQOÏ�Ð�A�9<Ð�?�Z]QYÐ>=ÁÐ�9½Z�ÑCåNèOM G�èYM��[èOM K[èOM�å�æ[è¥Z]; ;G�è¥ÕF8:9�Ö9>A�;<QYÐ�M�8]QYK°9>\YA¥9>ANÐr9�Ð�A�9(?�Z]Qy9<8]=IQYÐ�9>\OAÞ;>A�HJ8�=ÁQO=IQYϼå�æ]è·ÕY8:9>9>A�;<QYÐ�D�Ù�\OAÜÕY8:9>9>A�;>QYÐ�8�;<AÐ�H�Z¡Z�9<\OA�K�UYÐ>=ÁQOÏ�8 �½8]UYÐ<Ð�=@8�QJ`[A�;<QOA�ÓOÝ�=I9>\ÞÐr9*8�QYKY8�;*K(KOA�Û¡=Á8�9>=ÁZ]QJZ]Ñ�Z]QOA%\F8�ÓIÑ�ÕO=|áGA�Ó¯D�OZ[;A�8]?*\ÜZ�Ñ�9>\OA�9>;*8�=ÁQO=ÁQOÏ�Ð�A�9bÕY8�9�9<A�;<QYÐ�M¡9<\OA�9<8]QOÏ]A�Q[9�Û]AN?e9<Z];*Ð�ÑÔZ[;�\OZ];<=Iã�Z]Qy9<8]ÓW8]QYK¥Û[A�;>9>=@?�8]Ó9>;*8�QYÐ>ÓÁ8�9>=ÁZ]Q)8�;<A�?6Z[H�ÕOUG9>ANKpDFÙ�\YA�QOA69rÝbZ];<`d\F8]Ð�9rÝbZJ\O=@KOKGA�QõÓÁ8�×[A�;*Ð�Ý�=|9<\)ÓIZG?�8]ÓIÓÁ×�?6Z]QOÖQOA�?69>A�K�Ð�\Y8];>ANKôÝVA�=IÏ[\[9*Ð�M�8�QYK°Z[QOA¥Z]UO9>ÕOUG9(ÓÁ8�×[A�;�Ý�=I9>\ åNè�UOQO=I9<ÐÞä�IOå$�(�·?�Z]QOQYA�?e9<=IZ[QYÐ�Må�è]æ[è�ÑÔ;<A�A�ÕF8�;*8�H�A69<A�;*Ð<ç�E|å �*H±DYÙ�\OA�Ï[A�QOA�;<8]ÓI=Áã�8�9>=ÁZ]QdÕWA�;>ÑÔZ];<HJ8�QY?�A�8[Ð%8�ÑÔUOQY?69>=ÁZ]Q)Z�Ñ9>\YA9>;*8�=ÁQO=IQYÏ Ð�A�9bÐ>=Iã�A�ÑÔZ];P9>;*8]KG=I9>=ÁZ]QF8�ÓYÒF8]?*`¡ÕO;<Z]ÕJ8�QYKJ9<8�QYÏ]A�Qy9PÕY;>Z[Õ¥8�;<A/?6Z[H(ÕF8�;<A�K�=ÁQ���=ÁÏ�ÖUO;>A�åNèOD �ôA�\Y8�Û]A�?�Z]QYKGUF?e9>ANK�8]KOKG=I9>=ÁZ]QF8�ÓOA�á¡ÕWA�;<=ÁH(A�Qy9<Ð�=ÁQJÝ�\O=Á?*\�ÝbA�=IH�ÕOÓÁA�H�A�Qy9<A�K�QOZ]9Z]QOÓÁ×¼9>;*8�QFÐ�Ó@8:9<=IZ[QYÐ ÒYUG9d8�Ó@Ð�Zô;<Z�9<8�9>=ÁZ]QYÐ�M�A6áGÕY8]QYÐ�=ÁZ]QFÐ(8]QYK�\¡×¡ÕWA�;<ÒFZ[ÓI=@?�KGA6ÑÔZ[;>HJ8:9<=IZ[QYÐ�DÙ�\O=ÁÐ�Ð>A69�Z]Ñ�æÞÏ]A�QOA�;*8:9<Z];*Ð%=@Ð�8ÞÒY8]Ð>=ÁÐ�ÑÔZ[;�8�ÓÁÓ�ÓI=ÁQOA�8];½9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ/Z]Ñb?�ZyZ[;<KO=IQY8�9>ANÐÑÔZ];b9rÝVZ(KG=ÁH(A�QYÐ>=IZ[QY8�Ó^=ÁH�8]Ï]ANÐ�D �±9�=ÁÐ�Ð�9>;*8�=ÁÏ]\y9�ÑÔZ[;>Ý�8�;*K�9>Z(=IH�ÕOÓÁA�H�A�Qy9�Z�9<\OA�;bÏ]A�QOA�;*8:9<Z];*Ð=IQY?�ÓIUFKG=IQYϼÏ];*8�×[Ö±ÓÁA�Û]A�Ó|ÖuÐ>\O=|ѳ9<=IQYÏYM/ö<Ð�H�Z¡Z�9<\Yø°Ð�A�Ï]H�A�Qy9*8:9>=ÁZ]Q$MÓÁZG?�8]Ó%?6Z]Qy9<=IQ¡UOZ[UYÐ�?6Z¡Z];*KG=IÖQY8:9<A½9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐV8]QYKÞ=IQYKOA�ÕWA�QYKGA�Qy9%=IHJ8�Ï[A�Ð�A�Ï]H�A�Qy9�9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ�DÙ�\OA�QOA�áy9CA6áGÕFA�;>=ÁH�A�Qy9/=@Ð/KGANÐ�=ÁÏ]QYA�KÜ9>ZdÐ>\OZ:Ý 9>\Y8�9C=ÁQ·8�ÕOÕYÓI=@?�8�9>=ÁZ]QYÐ�Ý�\YA�;<A KO8:9*8�=ÁÐ

\O=IÏ[\OÓÁ×°?�Z];<;>A�ÓÁ8�9>ANKpM�9*8�QOÏ[A�Qy9�ÕO;<Z]ռס=ÁA�Ó@KOÐ(8·Ó@8�;<Ï]AdÐ>ÕFA�A�K�8[KGÛ:8�Qy9<8]Ï]A[DE¡=ÁQY?6AÜ9>\YAÜKG=@ÐrÖ9>Z];>9>=ÁZ]Q.H�Z¡KOA�Ó�=ÁH(ÕYÓI=ÁA�ÐJ8[KOKG=ÁQOÏßÓÁZ�9*Ð(Z�ÑC\Y=IÏ[\OÓI×�?�Z];<;>A�ÓÁ8�9>ANK�KO8:9*8OM9>\YA�8[KGÛ:8�Qy9<8]Ï]AÜZ�Ñ9<8�QYÏ]A�Qy9�ÕO;<Z]ÕÞZ:Û]A�;V9<\OA�KG=@Ðr9<Z];>9>=ÁZ]QÞH�Z¡KOA�Ó�ÒWA�?�Z]H�A�Ð�?�ÓIAN8�;NDÙ�\OA�9*8]Ð>`Þ=@ÐC9>ZÞ8�ÕOÕO;<Z�áG=IHJ8�9>A 8¥ÑÔUOQY?e9<=IZ[Q·9<\Y8:9½\Y8]ÐCÕOÓ@8:9<A�8�UFÐC8�9/9>\Y;>A�A�ÓÁZG?�8:9<=IZ[QYÐ�D

�ßA½Ý�8�Qy9V9<Z�A�QGÑÔZ[;<?�ACÓÁZG?�8�Ó^=ÁQ¡Û�8];>=@8�QF?6A�QOA�8];bAN8]?*\ÜZ�Ñ�9<\OA½9>;*8�=ÁQO=ÁQOÏ�ÕWZ]=ÁQy9<нä��=ÁÏ]UO;<A�å]å[M

Page 23: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

0

10

20

30

40

50

60

Backprop

Tangent Prop

10 20 40 80 160 320

%Error onthe test set

Training set size

QSRUT ®*V��F® � m*wym<o�k��xv|�*k��rvxqNw� Gm<ou²@q�ori�k�wys<mVs*�]or{�mVk�t�k�²Á�ywys>�rv|q�w�q�²Y�r~ym��uo�k�vxwyvxw[lCtum<�tuvx�*mV²Áq�o�r~[m��k�wyl�m*w:�V ]orqN (k�w¡���r~ym��yk�s��� [orqN (k��xlNq�orv��r~yi�t

ÒFZ]9�9>Z[H¥çeDOÙ�\OA�QOA69rÝbZ];<`(\Y8]ÐbZ]QYAC=ÁQOÕOUG9�UOQO=I9�M�G]è�\O=@KOKGA�QÜUYQO=|9*Ð�8�QYKdZ]QOA�Z]UG9<ÕOUG9�UOQO=I9�DÙ�ÝVZ/Ð�9>;*8:9<A�Ï[=IANÐ$8�;<A�ÕWZ[Ð<Ð�=ÁÒOÓÁA]ûNA�=I9>\OA�;�Ï[A�QOA�;<8�9>AP8CÐ�HJ8]ÓIÓ¡Ð�A�9�Z]ÑO9>;*8�=ÁQO=ÁQOÏ%ÕWZ]=ÁQ[9*Ð�?�Z:Û]A�;>=ÁQOÏA�8]?*\�Z]Ñ¡9>\OAVÕOÓÁ8�9>AN8�UYÐVäÔZ]ÕWA�Q�Ð<ùyUY8�;<A�Ð�Z]Q �=ÁÏ]UO;<A�å]å�ÒWZ�9>9>Z[HJç6M�Z[;$Ï[A�QOA�;<8�9>A�Z]QYA�9>;*8�=ÁQO=ÁQOÏÕFZ[=IQy9ÑÔZ];�A�8[?*\ ÕOÓ@8:9<A�8�UÜä¶?6ÓÁZ[Ð>A�K�Ð>ùyUY8];>ANÐ<ç6MN8]QYK�A�QGÑÔZ];*?6AbÓÁZ¡?�8�Óy=IQ¡Û:8�;<=@8�QY?�AV8�;<Z]UYQYK½9>\YA�HäÔÒ¡×·Ð�A�9�9<=IQOÏd9>\OA�KGANÐ�=Á;>ANK·KGA�;>=ÁÛ:8:9>=ÁÛ]A 9>ZÞè[çeDpÙ�\OA 9>;*8�=ÁQO=ÁQOÏÜÐ>A69�Z�Ñ�9<\OA ÑÔZ[;>H�A�;/H�A69<\OZGK=ÁÐdUYÐ�ANK 8]Ð¥8¼H�AN8]Ð>UO;>A�Z]ѽÕWA�;>ÑÔZ];<HJ8�QY?�A�ÑÔZ];¥ÒWZ�9<\�H�A69>\YZ¡KYÐ�DbR%ÓÁÓ/ÕY8];<8]H(A�9>A�;<Ð�ÝbA�;<A8]K�ØrUYÐ�9>ANK�ÑÔZ[;�8]ÕOÕO;<Z�áG=IHJ8:9<A�ÓÁ×CZ[ÕG9>=ÁHJ8�Ó¡ÕWA�;>ÑÔZ];<H�8]QY?6AV=IQ(8�ÓÁÓy?�8]Ð>A�Ð�DNÙ�\YAVÓÁA�8];>QO=ÁQOÏ�?6UO;<Û]ANÐÑÔZ];�ÒWZ�9>\�H�ZGKGA�Ó@Ð�8�;<A½Ð�\OZ:Ý�QÞ=ÁQ/�=ÁÏ]UO;<A�å]åJä³9<Z]ÕFç6D�#P8[?*\ÜÐ>ÝVA�A�Õ¥9<\O;<Z]UOÏ[\d9<\OA½9>;*8�=ÁQO=ÁQOÏÐ�A�9�ÑÔZ[;$9*8�QOÏ[A�Qy9�ÕO;<Z]Õ�=ÁÐ8CÓI=I9�9<ÓIAVѶ8]Ð�9>A�;Ð>=IQF?6Ab=|9�;>ANùyUO=I;<A�Ð�Z]QYÓI×�æ�ÑÔZ[;>Ý�8�;*K½ÕO;<Z]ÕY8]Ï[8�9>=ÁZ]QYÐ�MÝ�\O=IÓÁA%=I9V;<A�ùyUO=Á;>ANÐ ��=ÁQ�9>\OA/KO=ÁÐ�9>Z[;�9<=IZ[QJH(ZGKGA�Ó¯DyR/ÐV?�8�Q�ÒWACÐ>A�A�Q�M[Ð�9<8]ÒOÓIA�ÕWA�;>ÑÔZ];<HJ8�QY?�A�=ÁÐ8]?*\O=ÁA�Û[A�K)8:ѳ9<A�;(å [è]è¥Ð>ÝVA�A�ÕYÐ%ÑÔZ];C9>\OA 9<8]QOÏ]A�Q[9�ÕO;<Z]Õ�MWÛ]A�;<Ð>UYÐNK[è]è[è(ÑÔZ];C9>\OA�KG=@Ðr9<Z];>9>=ÁZ]QH(ZGKGA�Ó¯DFÙ�\OA�Z:Û]A�;<8]ÓIÓ^Ð�ÕWA�ANKGUOÕ�=@Ðb9>\OA�;>A�ÑÔZ];<A½8]ÒFZ[UG9�å�èYDÙ�8]QOÏ]A�Q[9�ÕO;<Z]ÕÞ=IQÞ9<\O=ÁÐ%A6áO8�H�ÕOÓÁA�?�8�QÜ9<8]`]A�8]KOÛ�8]Qy9<8�Ï[ACZ]Ñ�8(Û[A�;<×JÓÁ8];>Ï[A�;<A�Ï[UOÓ@8�;<=IãN8:Ö

9>=ÁZ]Q(9>A�;>H�D]Ù�\OA%KO=ÁÐ�9>Z[;�9<=IZ[Q�H�ZGKGA�ÓY=ÁÐ�8�9P8½KG=ÁÐ<8]KOÛ�8]Qy9<8�Ï[AVÒWA�?�8�UYÐ>A�9>\YA�Z]QOÓÁ×�ÕY8];<8]H(A�9>A�;9>\Y8�9�A�àWAN?e9<=IÛ[A�ÓÁ× ?6Z[Q[9<;>Z[ÓÁÐ�9<\OA�8�H�Z]UOQy9�Z�ÑW;>A�Ï]UOÓ@8�;<=IãN8:9<=IZ[Q�=ÁÐ9<\OA�HJ8�Ï]QY=|9<UYKGA�Z�ÑF9>\OA�KG=@ÐrÖ9>Z];>9>=ÁZ]QFÐ�MW8�QYK�9<\O=@Ð�?�8�QOQYZ�9/ÒWA�=ÁQY?�;>AN8]Ð>A�KÞ9>ZÜÓÁ8];>Ï[A�Û:8�ÓÁUOA�ÐCÒFAN?�8]UYÐ�A 9>\OA�;>=ÁÏ]\y9C8]QYÐ>ÝVA�;=ÁÐ�Z[QOÓÁ×¥=IQ¡Û:8�;<=@8�Qy9bUYQYKGA�;�ì1%Jë:ê³êGKG=@Ð�9>Z];>9>=ÁZ]QFÐ�D

�� � � !P ���� �� !P������ �$!!���� � �@ ���� �6�*�� � ;! �@ ���� � � ��� � !1� B/Z�9�HJ8�Q¡×�A�á¡ÕWA�;<=ÁH(A�Qy9<Ð�\Y8�Û[AbÒWA�A�QJKGZ]QOA�Ý�=I9>\(9<8]QOÏ]A�Qy9ÕO;>Z[ÕY8�Ïy8:9<=IZ[Q�D �±9Ü=@Ðd?�ÓIAN8�;NM�\OZ:ÝbA�Û]A�;�M�9<\Y8:9Ü9>\OAõ=IQ¡Û:8�;<=Á8]QY?6A·?�Z]QYÐ�9>;*8�=ÁQy9¥=ÁÐÜA6á¡9<;>A�H(A�ÓI×Ðr9<;>Z[QOÏYD��±Ñ�9>\YA¥QOA69rÝbZ];<`õKGZ¡ANÐ�QOZ�9�\Y8�Û]A�A�QYZ]UOÏ[\°?�8�ÕY8[?6=I9r×]Mp=I9 Ý�=ÁÓIÓVQOZ�9�ÒWA�QOA�âO9�ÑÔ;<Z]H9>\OA�A�áy9<;<8(`¡QOZ:Ý�ÓÁA�KGÏ[AC=ÁQy9>;<ZGKGUY?6ANKdÒ¡×d9>\OA�=ÁQ¡Û�8];>=@8�QF?6A]D

Page 24: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

output output

Average NMSE vs ageAverage NMSE vs age

−1.5 −1 −0.5 0 0.5 1 1.5−1.5

−1

−0.5

0

0.5

1

1.5

−1.5 −1 −.5 0 .5 1 1.5−1.5

−1

−.5

0

.5

1

1.5

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

.1

.15

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

0.1

0.15

Tangent propDistortion model

QSRUT ®*V�V¡® �q�i� ¡k�orvxtuq�wßq�²b�r~ym(�[vxt±�rq�ou�rvxq�wôi�q��[m<��¾³�|m>²Á��s*q��x�yi�wG¿/k�wy�·��k�wylNm<w��½ [orqN �¾@orvxlN~:�s<qN�x�yi�w¡¿>µN��~[m��rqN �orqeºßlNvx{�m*tp�r~[m��|m*k�orwyvxw[l�s*�[or{Nm*t�¾³m<ouorq�o${�m<ortu�[t�w]�[i��Gm<o^q�²Ot±º�m<m* ytp�r~[orq��yl�~�r~[m��uo�k�vxwyvxwyl%tum<�>¿>µ:��~ym��Gq��u�rq�i orqeºôlNvx{�m*t$�r~ym�¸¡wyk��yvxw[ y�[�u�¶q��[�r y�]�p²@�[wys<�rvxqNw�q�²Y�r~[m�wym<�¯º�q�or����r~[m��yk�tu~yme���|vxw[m�v|t��r~[mborm*tu�y����²@q�ob�[w¡k��[q�orw[me���yk�s��� [orqN Yµ

� �$!�;�*���� � ���� !�"� !P������ �$! �� � !� ���� E¡=ÁQY?6A�9<\OA�9<8]QOÏ]A�Qy9�Û[A�?69>Z];*Ð�=IQy9>;<ZGKGUY?�AA�Û]A�Q�H�Z];<Aõ?6Z[;>;<A�Ó@8:9<=IZ[Q =ÁQYÐ>=ÁKOA·9>\YA·9>;*8�=ÁQO=ÁQOÏ�Ð>A69NMb8�Ð>UOÒYÐ�9<8]Qy9>=@8�ÓCÐ>ÕWA�A�K UOÕ ?�8]Q ÒWAZ]ÒG9*8�=ÁQOA�K Ò¡×�8�ÓI9>A�;>QY8�9>=ÁQOÏ�8¼;<A�Ï[UOÓÁ8];JÑÔZ];<Ý�8�;*K 8]QYK ÒY8[?*`yÝ�8�;*K ÕO;<Z]ÕY8]Ï[8:9<=IZ[Q.Ý�=I9>\ 89<8�QYÏ]A�Qy9(ÑÔZ];<Ýb8];<K¼8]QYK�ÒF8]?*`¡Ýb8];<K°ÕY;>Z[ÕY8�Ïy8:9>=ÁZ]Q ä¶A�Û[A�Q�=IÑ%9<\OA�;<AÞ8];>AÞÐ�A�Û]A�;<8]Ó�9<8]QOÏ]A�Qy9Û]A�?69>Z[;<Ð�MGZ]QOÓÁ×dZ[QOA�=@Ð�UYÐ>A�K)8:9%AN8]?*\ÞÕY8:9>9>A�;>QFç6D��YZ];%=ÁQYÐ�9<8�QF?6A]MY=|Ñ�9>\YA�;<A�ÝVA�;>A �9<8]QOÏ]A�Qy9Û]A�?69>Z[;<Ð�My9>\OA½9<;<8]=IQO=ÁQOÏJÐ>A�ùyUOA�QF?6A�?6Z[UOÓÁKÜÒWA]û

# L ��� L ä # L ç.� # f ��� f ä # f ç.� # � ��� � ä # � ç.� # � ��� L ä # � ç.� # � ��� f ä # � ç.������� ä Kyç

Ý�\OA�;<A # � H(AN8�QYÐ 8ÞÑÔZ];<Ýb8];<Kô8�QYKôÒY8[?*`¡Ýb8];<KßÕO;<Z]ÕY8]Ï[8:9<=IZ[Q·ÑÔZ[;�ÕY8:9>9>A�;>Q ��8�QYK�� ä # � çH(AN8�QYÐ�8(9*8�QOÏ[A�Qy9�ÑÔZ];<Ýb8];<KÜ8�QYKÞÒY8[?*`¡Ýb8];<KdÕO;>Z[ÕY8�Ïy8:9<=IZ[Q¥Z�Ñ9<8]QOÏ]A�Q[9�Û[A�?e9<Z];��¥Z�Ñ�ÕF8:9�Ö9>A�;<Q �*D ��=|9<\(Ð>UY?*\�=IQy9>A�;>ÓÁA�8�Û¡=ÁQOÏYM�9>\OAbÓÁA�8�;<QO=ÁQOÏC?�Z]Q¡Û]A�;>Ï[A�Ð^Ѷ8[Ðr9<A�;9>\F8�Q Ï[;>Z[UOÕO=ÁQOÏ/8]ÓIÓy9>\YA9<8�QYÏ]A�Qy9PÛ[A�?69>Z];*Ð�9<Z]Ï[A69>\YA�;ND�=/Ñ$?6Z[UO;*Ð�A[M:9>\O=@ÐbZ]QOÓÁ×(H�8]`]ANÐPÐ>A�QYÐ>A%Ý�=I9>\dZ[QGÖ±ÓI=ÁQOA%ÓÁA�8];>QO=ÁQOÏFD

Page 25: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

� ����Æ�����ÆPÇ�� �$ÌWÇGÉ�È �uQ�9>\O=@ÐCÐ>A�?69>=ÁZ]Q�MYÝVA ?�Z]QYÐ>=@KGA�;�9<\OA�Ï[A�QOA�;<8]Ó�ÕY8�;*8]KO=IÏ[H ÑÔZ];/9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QÞ=ÁQyÛ:8];>=@8�QY?�A8�QYKÞÑÔZ[;%9>\YA�9<8]QOÏ]A�Q[9CÛ]A�?69>Z[;<Ð�Ý�\O=@?*\)\Y8�Û[A�ÒFA�A�Q)UYÐ>A�K)=ÁQ�9>\YA�9rÝVZdÕO;>A�Û¡=IZ[UYÐ%Ð>A�?e9<=IZ[QYÐ�DaVA�ÑÔZ];<A�ÝbA�=IQy9<;>ZGKGUY?�A�AN8]?*\õ9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Qô8�QYKõ9>\YA�=Á;½?�Z];<;>ANÐ�ÕWZ]QFKG=IQYÏ¥9<8]QOÏ]A�Qy9½Û[A�?6Ö9>Z];*Ð�M$8�ÒO;<=IA�Ñ�A�áGÕOÓÁ8]QY8:9<=IZ[Qß=@Ð�Ï[=IÛ[A�QôZ�Ñ�9>\YAJ9>\OA�Z];<×õÒWA�\O=ÁQYKô9<\OA¥ÕO;*8]?69>=@?6A]D�Ù�\OA�;<A¥8�;<A9rÝVZJ8]Ð>ÕWA�?e9*Ðb9>Z�9<\OA�ÕO;<Z]ÒOÓÁA�H�D��=Á;<Ð�9�=|9/=@Ð�ÕWZ[Ð<Ð�=ÁÒOÓÁA½9>ZJA�Ð�9<8]ÒOÓÁ=ÁÐ>\Þ8�ÑÔZ[;>HJ8�Ó�?�Z]QOQYA�?e9<=IZ[QÒFA�9rÝVA�A�QdÏ[;>Z[UOÕYÐPZ�Ñ�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐ�Z]Ñ�9>\YAC=ÁQOÕOUG9�Ð>ÕY8]?�A�ä¶Ð>UY?*\Þ8]ÐP9>;*8�QYÐ>ÓÁ8�9>=ÁZ]Q�M[;>Z]9<8�Ö9>=ÁZ]Q�M�A�9<?]D�Z�Ñ � f ç�8]QYK�9>\OA�=I;�A�àWAN?e9PZ[QJ8/ÑÔUYQY?e9<=IZ[QY8�ÓOZ]ÑF9<\Y8:9PÐ>ÕY8[?6A�ä¶Ð>UY?*\�8]Ð�8�HJ8�ÕYÕO=IQYÏZ�Ñ ��f�9<Z ��MWÝ�\O=@?*\·HJ8�×Þ;<A�ÕY;>ANÐ�A�Q[9/8]Q·=ÁH�8]Ï]A[MY=ÁQß?6Z[Qy9>=ÁQyUYZ]UYÐ%ÑÔZ];<HJç6DWÙ�\YA�9>\YA�Z];<×ÞZ�ÑS�=ÁAbÏ[;>Z[UOÕYÐ�8�QFK�S$=IA�8�ÓÁÏ]A�ÒO;*8@E æ*HY8�ÓÁÓÁZ:Ý%Ð�UFÐ9>Z�KGZ�9>\O=@Ð�D�Ù�\OA%Ð�AN?6Z[QYK�ÕO;<Z]ÒYÓIA�H \Y8]Ð�9<Z½KOZÝ�=|9<\ß?6ZGKG=ÁQOÏFD^TVZ]H�ÕOUO9>A�;�=ÁH�8]Ï]ANÐ/8�;<A�âYQY=|9<A Û]AN?e9<Z];*Ð%Z�ÑPKO=ÁÐ<?6;<A69<A�Û:8�;<=Á8]ÒOÓIANÐ�D��%Z:Ý ?�8]Q8�9>\YA�Z];<×(Ý�\O=Á?*\ÜÝb8[ÐPKGA�Û]A�ÓIZ[ÕFANK�ÑÔZ];�KG=IàWA�;>A�Q[9<=Á8]ÒOÓÁA%ÑÔUOQY?69>=ÁZ]QY8]ÓFZ]Ñ ��f%9<Z � ÒWA�8]ÕOÕOÓÁ=IANK9>Z�9>\YA�Ð>A�Û[A�?69>Z];*Ð���ßA�âY;*Ð�9½Ï[=IÛ[AJ8�ÒO;>=ÁA6ÑbA6áGÕOÓ@8�QY8�9>=ÁZ]QôZ�Ñb9>\OA�9<\OA�Z[;>A�HJнZ�Ñ�S�=ÁAJÏ];<Z]UOÕYн8]QYK°S�=ÁA¥8�ÓIÖ

Ï]A�ÒY;<8[ÐCÝ�\Y=Á?*\°8�;<AJ8�ÕOÕOÓÁ=@?�8�ÒYÓIA(9>Z)ÕF8:9�9<A�;<Qß;<A�?�Z]Ï[QO=|9<=IZ[Q�D B%A�áy9NM�ÝbA(A�áGÕOÓIZ[;>AJÐ>Z]ÓÁUG9>=ÁZ]QFÐ9>Z�9>\OA½?6ZGKG=ÁQOÏ ÕY;>Z[ÒOÓIA�H�D �=ÁQY8�ÓÁÓI×¥Ð>Z]H�ACA6áO8�H�ÕOÓÁA�ÐVZ�Ñ�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q¥8]QYKd?�Z¡KO=IQOÏ(8�;<AÏ]=ÁÛ]A�QÜÑÔZ];�ÕF8�;>9>=@?6UOÓ@8�;�8�ÕYÕOÓI=@?�8�9>=ÁZ]QYÐ�D

� ��� �L �3���� �����5����� � �3������;� �*���TVZ]QYÐ>=ÁKOA�;�8�Qd=ÁQOÕOUO9�Ð>ÕY8[?6A�#ä³9<\OA�ÕOÓÁ8]QOA ��f/ÑÔZ[;VA�áO8�H�ÕOÓÁANçV8�QYKÜ8�KG=Ià^A�;<A�Qy9>=@8�ÒOÓÁACÑÔUOQY?eÖ9>=ÁZ]Q�)Ý�\O=@?*\ÞH�8]ÕYÐ�ÕWZ]=ÁQy9<Ð�Z�Ñ��.9>Z ��D

õû� ��� �� �ä� ôç� � ä �yçÙ�\OA ÑÔUYQY?e9<=IZ[Q��ä� ßç � �ä # � " ç�?�8�Q·ÒWA�=IQy9>A�;>ÕY;>A�9>A�Kõ8]Ð/9>\OAJ?6Z[Qy9>=ÁQyUYZ]UYÐ(ä¶KGA�âYQOANK·ÑÔZ[;8�ÓÁÓpÕFZ[=IQy9<Ð%Z�Ñ���f�çbA�ùyUO=ÁÛ:8�ÓÁA�Qy9�Z�Ñ9>\OA�KG=@Ð<?6;<A69>A�?�Z]H�ÕOUG9<A�;�=ÁHJ8�Ï]A �&E � � � HjD

B%A6á¡9NM�?6Z]QFÐ�=@KGA�;J8ßѶ8�H�=ÁÓI×�Z�Ñ/9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QYÐ � � M�ÕF8�;*8�H�A69<A�;<=Iã�A�K�Ò¡× �VM�Ý�\O=@?*\H�8]ÕYÐ�ÒO= ØrA�?69>=ÁÛ]A�ÓÁ×Ü8(ÕWZ]=ÁQy9%Z�Ñ��.9<Z¥8�ÕWZ]=ÁQy9%Z�Ñ���D

� � û� ���� �� � � ä� ßç �� ä��[èyç

�ßA�8[Ð>Ð>UOH�A½9>\Y8�9 � � =ÁÐ/KG=|à^A�;<A�Qy9<=Á8]ÒOÓIA�Ý�=I9>\);<A�Ð>ÕWA�?e9�9<Z���8�QFK� �MO8�QYKÜ9>\F8:9 ��=@Ð�9>\YA

=ÁKGA�Qy9>=I9r×]D��YZ];�A6áO8]H(ÕYÓIA � � ?6Z[UOÓ@KdÒWA½9>\OA�Ï[;>Z[UOÕÞZ�Ñ8 JJQOA½9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐPZ]Ñ ��f]û� � û�� # "�� � �� � # � L # � f " � �� � # " � � " ��� � Ý�=|9<\ ���� å � L � f� � å � �

������� è ä��Yå�ç

�±9� Ð%8�S�=ÁA�Ï];<Z]UOÕ � Z]Ñæ�ÕY8];<8]H(A�9>A�;<Ð�DGR%QOZ]9>\OA�;�A6áO8�H�ÕOÓÁA½=ÁÐ�9<\OA�Ï];<Z]UYÕÜZ]ÑKG=Á;>AN?e9%=@Ð�Z[H(ÖA69>;<×^û

� � û�� # "�� � � � # ?6ZyÐ"! #" Ð>=IQ#! �# Ð�=ÁQ$! " ?�Z[Ð"! � � ä��;G[ç

�±9� Ð%8�S�=ÁA�Ï[;>Z[UOÕÞZ�Ñ �ÕF8�;*8�H�A69<A�;*Ð�D� ���Fvxm/l�orq��y ¥vxt�k�l�orq��y ��r~¡k6�bvxt�k��xtuq�k��]vx´Om>orm*w:�rvIk��y�xm/i�k�wyv�²ÁqN�|�Jtu�ys�~��r~¡k��P�r~ymC�[v�´Om<orm<w[��rv|k��[�|m�t±�uor�[s<�r�[ormbv|tPs*q�i� ¡k��rvx�[�|mVº�vx�r~��r~ym�l�orqN�y (t±�uor�ys<�r�]ormNµ

Page 26: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

�ßA�QOZ:Ý#?6Z[QYÐ�=@KGA�;V9<\OA½ÑÔUOQY?69>=ÁZ]QY8]Ó �yä �,�ç6MYKGA6âFQOA�KÞÒ¡×�yä �,�ç � � � � L� ä��� yç

Ù�\O=ÁÐÜÑÔUOQF?e9>=ÁZ]QF8�Ó �]M�Ý�\O=@?*\ 9*8�`[A�Ðd8]QOZ�9<\OA�;ÜÑÔUOQY?69>=ÁZ]QY8]Ó ÷8]ÐÞ8�Q 8];>Ï[UOH�A�Qy9�MbÐ�\YZ]UOÓ@K;>A�H(=ÁQYK 9>\OAô;>AN8]KGA�;ÜZ]Ñ �=IÏ[UO;>A�G¼Ý�\YA�;<A ��Mb9>\OA°KG=@Ð>?�;>A�9>AôA�ùyUO=ÁÛ:8�ÓÁA�Qy9ÜZ]Ñ M�=ÁÐÜ9>\YA8�;<Ï]UOH�A�Qy9�Z�Ñ �[DÙ�\OA�S$=IA�8]ÓIÏ[A�ÒO;*8 8[Ð>Ð>ZG?6=@8:9>ANK¥Ý�=I9>\Ü9>\OA�8[?e9>=ÁZ]QÜZ�Ñ � � Z[Q )=ÁÐb9<\OA�Ð�ÕY8[?6A�Ï]A�QOA�;*8:9<A�K

Òy×¥9>\OA�� ÓÁZG?�8]Ó^9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ � � �PZ�Ñ ·KGA�âYQOANKÜÒ¡×^û� � �eä pç � � �yä ����ç

� � ����� � � � ä�� �¡ç

�ßA�?�8�QÞQOZ:Ý Ý�;<=|9<A½9>\OA�ÓÁZG?�8]Ó�8�ÕOÕO;<Z�áG=IHJ8�9>=ÁZ]Q¥Z]Ñ ��8[Ð�û�yä ����ç � � L � � � ä pç � f � � � ä pç ����� � � � � ä pç��Gä�� ��� f ç�ä pç ä��;I[ç

Ù�\O=ÁÐ�ANùyUY8:9<=IZ[Qd=@ÐP9<\OA�?6Z[Q[9<=IQ¡UOZ[UYÐbA�ùyUO=ÁÛ:8�ÓÁA�Qy9bZ]Ñ$A�ùyUY8�9>=ÁZ]Qôä�G]çVUYÐ>A�KÜ=ÁQd9<\OA½=IQy9>;<ZGKGUY?6Ö9>=ÁZ]Q�DÙ�\OA(ÑÔZ]ÓÁÓIZ:Ý�=ÁQOÏdA�áO8�H�ÕOÓÁA(=ÁÓÁÓIUYÐ�9>;*8:9<A�Ð�\OZ:Ý � � �C?�8]QßÒWAJ?6Z]H�ÕOUO9>A�KõÑÔ;<Z]H � � D�S�A69� Ð

?6Z]QFÐ�=@KGA�;/9>\OA�Ï];<Z]UOÕ�Z]ÑPKG=I;<A�?69/=@Ð�Z[H(A�9>;<×ÞKOA6âYQOANK)=ÁQ·A�ùyUY8�9>=ÁZ]Q�ä ��G]ç�ä¶Ý�=|9<\·ÕY8];<8]H(A�9>A�;� � ä�! � � � � ç�8]Ð�ÒWA6ÑÔZ[;>A[MO8�QYK � ä # � " ç>çeD�[ä ����ç6ä� ßç � �ä>ä # � çG?6ZyÐ"! ä "% � çGÐ>=ÁQ$! � ä # � çGÐ>=ÁQ�! ä " � çG?6ZyÐ ![ç ä��[æyç�±ÑÝbA�KG=|à^A�;<A�Qy9<=Á8�9>A�8�;<Z]UYQYK � � ä¶è��>è��>è[çVÝ�=I9>\�;<A�Ð>ÕFAN?e9�9>Z�!OMYÝVA½Z]ÒO9<8�=ÁQ

� �yä ����ç� ! ä� ßç � " �

� #ä # � " ç ä # ç �

� " ä # � " ç ä���M]ç

9>\Y8�9%=ÁÐ � � � " �� #

ä # ç �� " ä�� Kyç

Ù�\OA)9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q � � � �� 8�QFK �� � �� � ?�8�Q�ÒWA·Z]ÒO9<8�=ÁQOANK.=ÁQ 8¼Ð>=IH�=ÁÓÁ8];Ѷ8]Ð>\O=IZ[Q�DOR%ÓÁÓ�ÓÁZG?�8�Ó^9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QÜZ�Ñ9>\OA�Ï[;>Z[UOÕÜ?�8�QÞÒFA�Ý�;<=|9>9>A�Q�8[Ð

�yä �,�ç � !Yä " � � #

ä # ç � � " ç � � � # � �

� " �Gä � � � f ç�ä pç ä�� �yç

Ý�\O=Á?*\ß?6Z];<;<A�Ð>ÕFZ[QYKOÐ�9<Zd8¥ÓÁ=IQYA�8�;�?6Z[H ÒO=ÁQY8:9<=IZ[Q)Z�Ñ�9<\OA JÒF8]Ð>=Á? Z]ÕWA�;*8:9>Z[;<Ð � � M � � 8]QYK�� � D�Ù�\OA�ÕO;<Z]ÕWA�;>9r×�Ý�\O=@?*\�=@Ð�H�Z[Ð�9�=ÁH�ÕFZ[;�9*8�Qy99>Z½UYÐ�=@Ð9>\Y8�99<\OA� CZ]ÕWA�;*8:9>Z[;<Ð�Ï]A�QOA�;*8:9<A9>\OA%Ý�\YZ]ÓÁA%Ð�ÕY8[?6A�Z]ÑWÓÁZG?�8�ÓO9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QFÐ�D:Ù�\OA%;<A�Ð>UOÓI9�Z�Ñp8�ÕOÕYÓIס=ÁQOϽ9>\OA/Z]ÕWA�;*8:9>Z[;<Ð�9<Z8�ÑÔUOQY?e9<=IZ[Q M�Ð�UF?*\�8[Ð�8OG�_ =IHJ8�Ï[AbÑÔZ];�A�áG8]H�ÕOÓIA[M�=@Ð�9>\OA%Ð>A69�Z]ÑpÛ]A�?69>Z[;<ÐÝ�\O=@?*\�ÝVA�\Y8�Û[AÒFA�A�Q¥?�8]ÓIÓÁ=ÁQOÏ�ör9<8]QOÏ]A�Qy9�Û]A�?69>Z[;<øC=IQ(9<\OA%ÕO;<A�Û¡=ÁZ]UYÐ�Ð>A�?69>=ÁZ]QYÐ�D #P8[?*\�ÕWZ]=ÁQy9�=IQ�9<\OA�9<8]QOÏ]A�Qy9Ð�ÕY8[?6A/?�Z];<;>ANÐ�ÕWZ]QFK(9>Z�8�UOQO=@ù[UYA%9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q¥8�QYKd?6Z]Q¡Û[A�;*Ð�A�ÓI× 8]Q¡×�9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QZ�Ñ�9>\YA�S�=ÁA�Ï[;>Z[UOÕ�äÔ=ÁQ)9>\YA A6áO8]H(ÕYÓIA 8]ÓIÓ;<Z�9<8�9>=ÁZ]QYÐ%Z�ÑP8�Q¡×Þ8�QOÏ[ÓIA 8]QYK·?�A�Qy9>A�;%9>Z[Ï]A�9>\OA�;Ý�=|9<\�8]ÓIÓ^9<;<8]QYÐ>ÓÁ8�9>=ÁZ]QYÐ*çb?6Z];<;<A�Ð>ÕFZ[QYKOÐV9>ZJ8(ÕFZ[=IQy9%=ÁQÞ9>\OA½9*8�QOÏ[A�Qy9�ÕOÓÁ8]QOA]D����~ym<tum½qN Gm<o�k6�rq�ort%k6orm�trk�v|�¥�rq�l�m*wym<o�k6�rm�k(�Yvxm½k��xl�m*�[o�k��W�Gm*s*k��ytumCqNw¥�rqN dq�²��r~ym½k��y�]vx�rvxq�wk�wy�Þi��[���rv| [�xv|s*k��rvxqNw��:¹Þk(tusek��Ik6oe�^º�m�sek�w·�[m<¸ywyme�Ük�tu Gm*s*v|k���i��[�x�rvx [�|vxsek6�rvxqNw)sek��|�xme� Âj�Fvxm�[o�k�s>�Nm<�rÃ�k�w¡���[m<¸ywyme���:¹�� � g � � � �� �� g�� � ��� � ��� � g µ]h¯w �r~[m�k��Gq6{�mbm<¢[k�i� y�xm�º�m�~yke{�m� ��� � ������ ����e� � ��� � ��� �� ¼�]�yk�wy� � ��� � ��� �� ����[µ

Page 27: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

� �� �>������ �$! �� � !P ��*�

Ù�\OA�ÓÁ8[Ðr9�ÕO;<Z]ÒOÓÁA�H Ý�\O=@?*\(;>A�H�8]=IQFÐ$9<ZCÒWA�Ð�Z[ÓIÛ[A�K =ÁÐ�9>\YAbÕY;>Z[ÒOÓIA�H Z�Ñ^?6ZGKG=ÁQOÏYD]TVZ]H�ÕOUO9>A�;=IHJ8�Ï[A�Ð�M^ÑÔZ];�=ÁQYÐ�9<8�QF?6A]M�8�;<A(?�Z¡KOA�K°8]Ð�8ÜâYQO=I9>AÜÐ�A�9�Z�ÑbKO=ÁÐ<?6;<A69<A·äÔA�Û[A�QôÒO=ÁQY8�;<×Oç/Û:8]ÓIUOANÐ�DÙ�\OA�Ð>A·8�;<A)\Y8�;*KGÓÁ×�9<\OAõKG=IàWA�;>A�Qy9>=@8�ÒOÓÁA·HJ8�ÕOÕY=IQOÏyÐJZ�Ñ � 9<Z � Ý�\Y=Á?*\�ÝbA)\Y8�Û]A)ÒWA�A�Q8]Ð<Ð�UOH�=ÁQOÏ�=IQÞ9<\OA�ÕO;<A�Û¡=IZ[UYÐ�Ð�UYÒYÐ�AN?e9<=IZ[Q�DÙ$Z Ð>Z]ÓÁÛ]A%9>\O=@ÐPÕY;>Z[ÒOÓIA�H ÝVA%=ÁQy9>;<ZGKGUY?6A�8�Ð>H(Z¡Z]9>\¥=ÁQy9>A�;<ÕWZ]Ó@8:9>=ÁQOÏ�ÑÔUYQY?e9<=IZ[Q � Ý�\O=@?*\

H�8]ÕYÐP9>\OA½KG=@Ð>?�;>A�9>A/Û[A�?69>Z];*Ð�9<Z�?�Z]Qy9>=ÁQ¡UOZ]UYÐVHJ8�ÕOÕY=IQOÏ�Z�Ñ ��9>Z ��D�YZ];bA6áO8�H�ÕOÓÁA]My=IÑ �÷=ÁÐ8�=IHJ8�Ï[A�Z�Ñ �°ÕO=|áGA�ÓÁÐ�MO=I9C?�8]QÞÒWA�HJ8�ÕYÕFANKÞ9>Z¥8¥?�Z]Qy9>=ÁQ¡UOZ]UYÐ>ÓÁ×¥Û:8�ÓÁUOA�KÞÑÔUOQF?e9>=ÁZ]Q õZ:Û]A�;��fJÒ¡×ß?6Z[Q¡Û]Z]ÓÁÛ¡=IQYÏÜ=I9 Ý�=I9>\¼8Þ9rÝVZ·KO=IH�A�QFÐ�=ÁZ]QY8]Ó��½8�UFÐ>Ð>=Á8]QõÑÔUYQY?e9<=IZ[Q�� � Z]Ñ�Ð�9<8]QYKO8�;*KKGA�Û¡=@8:9>=ÁZ]Q���D$Ù�\O=@Ð�=@Ð�ÒWA�?�8]UYÐ>A�� � =ÁÐ 8ÞKO=|à^A�;<A�Qy9>=@8�ÒYÓIA�HJ8�ÕOÕY=IQOÏ)Z]Ñ ��f(9<Z ��M$8�QYK �?�8�QßÒFA�=ÁQy9>A�;<ÕO;<A69<A�Kô8]н8ÜÐ>UOH Z]ÑV=ÁH�ÕOUOÓ@Ð�A(ÑÔUOQY?69>=ÁZ]QYÐ�D��uQß9>\OA(9rÝbZÞKG=ÁH�A�QYÐ>=ÁZ]QY8]Ó�?�8]Ð>AÝVA�?�8]QÜÝ�;<=I9>A½9>\YA�QOA�Ý#=IQy9<A�;<ÕO;>A�9<8�9>=ÁZ]QdZ]Ñ��Ú8]Ð�û

��$¯ä # � " ç � � � � �&E � HCE � H��Gä # �rç��Gä "% �yç ä I�èyç

Ý�\OA�;<A �&E � HCE � HYKGA�QOZ�9<A�Ð9>\YA�âYQO=I9>A�Û]AN?e9<Z];�Z�Ñ^KG=ÁÐ<?6;<A69<A�Û�8]ÓIUYA�Ð�M�8]Ð�Ð�9>Z];<A�K�=ÁQ(8½?6Z[H�ÕOUG9>A�;�DÙ�\OAJ;>ANÐ�UYÓ|9 Z]ÑV9<\OAd?�Z]Q¡Û]Z[ÓIUG9<=IZ[Qß=@Ð�Z]Ñ�?�Z]UO;*Ð�AJKG=Ià^A�;<A�Qy9>=@8�ÒOÓÁA¥ÒWA�?�8�UYÐ>A�=|9 =@Ð 8)Ð>UOH Z�Ñ�½8�UYÐ<Ð�=@8�QÜÑÔUOQY?69>=ÁZ]QYÐ�DYÙ�\OA �½8]UYÐ>Ð>=@8�QÞH�8]ÕOÕO=ÁQOÏ�=ÁÐ%Ï]=ÁÛ]A�QÞÒ¡×^û

� � û � � � � � $�� � � ä IGå�ç

�uQÜ9<\OA½9rÝbZ�KO=IH�A�QFÐ�=ÁZ]QY8]Ó�?�8[Ð�A[M[9<\OA½ÑÔUOQY?69>=ÁZ]Q õ?�8�QÞÒWA�Ý�;>=I9�9<A�Q)8]Ð�û�ä # � " ç � � � � �&E � H E � H�� � ä #� � � " �¡ç ä I G[ç

=/9>\OA�;�?�ZGKG=IQYÏ�ÑÔUOQY?69>=ÁZ]Q � ?�8�Q ÒWA°UYÐ�ANKpM�Ð�UF?*\ 8]ÐÞ?6UYÒO=Á?ôÐ>ÕOÓI=ÁQOA°Z];ÞA�Û[A�Q ÒO=IÓÁ=ÁQOA�8];=IQy9>A�;>ÕWZ]Ó@8:9<=IZ[Q�D�ab=IÓÁ=ÁQOA�8];=ÁQy9>A�;>ÕWZ]Ó@8:9<=IZ[Q ÒWA69rÝbA�A�Q(9<\OA�ÕO=|áGA�ÓÁÐ�ס=ÁA�Ó@KOÐ�8CÑÔUOQY?e9<=IZ[Q JÝ�\O=@?*\=ÁÐ�KG=Ià^A�;<A�Qy9>=@8�ÒOÓÁA�8�ÓÁH�Z[Ð�9A�Û]A�;<סÝ�\OA�;<A]D�Ù�\OAbѶ8]?69�9>\Y8�9�9>\OA�KGA�;>=ÁÛ:8:9>=ÁÛ]ANÐ\Y8�Û]AV9rÝVZ�Û:8�ÓÁUOA�Ð8:9�9>\OA�=ÁQ[9<A�Ï[A�;½ÓIZG?�8:9>=ÁZ]QFÐ�ä¶ÒFAN?�8�UFÐ�A(9>\YA(ÒY=IÓÁ=IQYA�8�;½=ÁQy9>A�;<ÕWZ]Ó@8:9>=ÁZ]Qõ=ÁÐ�KG=Ià^A�;<A�Qy9�Z[QßÒWZ�9<\Ð�=@KGA)Z�ѽA�8[?*\.ÕY=|áGA�Ó@Ð*ç(=ÁÐJQYZ�9d8ôÕO;<Z]ÒOÓÁA�H =IQ ÕO;*8]?69>=@?6A¼ä�ØrUYÐ�9Ü?*\OZ¡ZyÐ�A�Z]QYA)Z�Ñ/9<\OA)9rÝbZÛ�8]ÓIUYA�Ð*çeDÙ�\OA �½8�UYÐ<Ð>=Á8]Q HJ8]ÕOÕO=ÁQOϼ=ÁÐ¥ÕO;>A�ÑÔA�;<;>ANK�ÑÔZ[;J9rÝbZ¼;>AN8]Ð>Z]QYÐ�û ��=Á;*Ðr9NM�9>\YAõÐ�H�Z¡Z�9<\O=ÁQOÏ

ÕY8�;*8�H�A69<A�; �Þ?�8�Q(ÒWA�UYÐ�ANK�9>Z�?6Z[Q[9<;>Z[Ó¡9>\OA�ÓÁZG?�8]ÓI=I9r×�Z�ÑF9>\OA�=ÁQ¡Û:8�;<=Á8]QY?6A[D�Ù�\Y=ÁÐ�=@Ð�ÒWA�?�8]UYÐ>AÝ�\OA�Q��=@ÐÐ>H�Z¡Z�9>\YA�;NM�9<\OA�ÓIZG?�8�ÓG8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]Q�Z]ÑFA�ùyUY8�9>=ÁZ]QÞä �;I[ç�=@ÐÒFA�9�9<A�;%äÔ=I9�=@Ð�Û:8]ÓI=@KÑÔZ];�Ó@8�;<Ï]AC9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QWçeDGR%QYK�Ð>A�?�Z]QYKpMGÝ�\YA�Q)?6Z[H ÒO=ÁQOA�KÜÝ�=I9>\Þ9>\OA½9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]QZ]ÕWA�;*8:9>Z[;%�%M�9>\YAÜKGA�;>=ÁÛ:8:9>=ÁÛ]AÜ?�8�Q¼ÒWA�8�ÕOÕOÓÁ=ÁA�K°Z]Q�9>\OAÞ?6ÓÁZ[Ð>A�KôÑÔZ];<H Z�Ñ�9<\OA��½8]UYÐ>Ð>=@8�QÑÔUOQY?e9<=IZ[Q�D��YZ];½=IQFÐr9*8�QY?�A]Mp=IÑP9>\YA��%Öj9>;*8�QYÐ>ÓÁ8�9>=ÁZ]Q·Z[ÕFA�;<8�9>Z[; � � �� =@Ð�8�ÕYÕOÓI=ÁA�Kõ9>Z �� $ � � � MG9>\YA�8]?e9<UY8�Ó�?�Z]H�ÕOUG9*8:9<=IZ[QÜÒWA�?�Z]H�A�Ð�û

��ä pç � �� #ä2��$ � � � ç � ��$ � � � �

� #ä I yç

ÒFAN?�8]UYÐ�AVZ�ÑO9<\OA�KG=|à^A�;<A�Qy9<=Á8�9>=ÁZ]Q ÕY;>Z[ÕFA�;�9<=IANÐ�Z�ÑW?6Z]Q¡Û[Z]ÓÁUG9>=ÁZ]Q�Ý�\OA�Q�9<\OAbÐ>UOÕOÕWZ];>9�=ÁÐ�?6Z[H(ÖÕY8]?69�DOÙ�\O=@Ð�=ÁÐbAN8]Ð>=IÓÁ×¥KGZ]QYA�Ò¡×¥?�Z]Q¡Û]Z[ÓIÛ¡=ÁQOÏ�9<\OA½Z];<=IÏ[=IQY8]ÓF=ÁHJ8�Ï[ACÝ�=I9>\Þ9>\YA �/Ö±KOA�;<=IÛ:8:9<=IÛ[A

Page 28: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

= *

= *

= *

QSRUT ®*V �Y® � o�k� y~[v|s�v|�x�x�yt±�uo�k6�rvxqNwÞq�²�r~[mCs*q�i� y�]��k��rvxqNw¥q�²��Þk�wy�¥�¯º�q���k�w[lNm*w:��{�m<s<�rq�ort�s<q�ouorm<�tu Gq�w¡�]v|w[l/�rq ��� ��������Þ¾ �V�Ô�uo�k�wytu�|k��rvxqNw¡¿�k�wy� �� ����� � ¾ �b�Ô�uo�k�wytu�|k��rvxqNw¡¿>��²ÁorqNi÷k/�[v|wyk�ou¹vxi�k�lNm��¡µp��~[m � k��ytutuv|k�w·²@�[wys<�rvxq�w� O¾�� � � ¿ m>¢[ Y¾ �

� � ��� ���� � ¿�~¡k�t�kJt±��k�wy�yk6o��)�[m*{�v|k��rvxqNw)q�²� ¼��� ��v|w½�r~[vxt�m<¢[k�i� y�xm�k����r~yq��yl�~�vx�rt$l�o�k� y~[v|s�orm* ]orm*tum*w:��k��rvxqNwJ¾³tui�k��x�yvxi�k�l�m*tq�w��r~ym�orvxlN~:�>¿~yke{�m��Gm*m*w�orm*tusek��xme� ²@q�oVs*�|k�orv��¯¹:µ

Z�Ñ^9>\OA �½8]UYÐ<Ð�=@8�Q(ÑÔUOQY?69>=ÁZ]Q � � D]Ù�\Y=ÁÐPZ]ÕWA�;*8:9>=ÁZ]Q(=@Ð�=ÁÓIÓÁUYÐ�9>;*8:9>ANK(=IQ��=ÁÏ]UO;<A�åPGOD]E¡=IH�=ÁÓÁ8];>ÓÁ×]M9>\OA½9*8�QOÏ[A�Qy9�Û]A�?69>Z[;VÑÔZ[;%Ð>?�8�ÓÁ=IQYÏ�?�8�QÞÒFA�?�Z]H�ÕOUG9<A�KÞÝ�=I9>\

����ä pç � � # �� #

" �� " � ä�� � � � ç � # ä�� � � � �� # ç " ä�� � � � �� " ç ä I �¡ç

Ù�\O=@Ð�Z]ÕWA�;*8:9<=IZ[QÜ=@Ð�=ÁÓIÓÁUYÐ�9>;*8:9>ANKÞ=IQ��=IÏ[UO;>A�å OD

� �� �(��� ���!P���$! !��*��� � ��*����!� � ���4 �� ��������� �* �� ����P ����Ù�\O=ÁÐbÐ>A�?e9<=IZ[QdÐ>UOH�H�8];>=Áã�ANÐ�\OZ:Ý 9<Z�?6Z[H�ÕOUG9>A/9>\OAC9<8]QOÏ]A�Qy9PÛ]AN?e9>Z[;<Ð�ÑÔZ];V=IHJ8�Ï[A/ÕO;<ZG?6ANÐ>Ð�Ö=IQOÏ.äÔ=ÁQ G�_½çeD�#P8[?*\�KG=ÁÐ<?6;<A69<A¥=ÁH�8]Ï]A�� �/=@Ð ?�Z]Q¡Û]Z[ÓIÛ[A�KôÝ�=|9<\�8 �½8]UYÐ>Ð>=@8�Q°Z�Ñ/Ð�9<8]QYKO8�;*KKGA�Û¡=@8:9>=ÁZ]Q � � 9>Z(Z]ÒG9*8�=ÁQd8�;<A�ÕO;<A�Ð>A�Qy9*8:9>=ÁZ]QJZ]Ñ�9>\YA�?�Z]Qy9>=ÁQ¡UOZ]UYÐP=IHJ8]Ï]A �rMG8]?�?6Z];*KG=ÁQOÏ�9<ZA�ùyUY8:9<=IZ[Q�û � � � � � � � � ä I I[çÙ�\OA½;>ANÐ�UOÓI9>=ÁQOÏ(=IHJ8]Ï]A �Ý�=IÓÁÓpÒWA�UFÐ�ANKd=ÁQ)8�ÓÁÓF9<\OA�?6Z[H�ÕOUG9<8�9>=ÁZ]QYÐb;<A�ùyUO=Á;>=ÁQOÏ�� ��äÔA�áO?6A�ÕO9ÑÔZ]; ?�Z]H�ÕOUG9<=IQOÏ�9<\OA�9<8]QOÏ]A�Qy9�Û]A�?69>Z[;*ç6D��OZ];�AN8]?*\°=ÁH�8]Ï]A�� � Mp9>\YAJ9<8]QOÏ]A�Q[9�Û[A�?e9<Z];*Ð�8�;<A?6Z]H�ÕOUO9>A�K Ò¡×�8]ÕOÕOÓÁ×y=ÁQOÏô9>\YA)Z]ÕWA�;*8:9>Z[;<Ð�?�Z];<;>ANÐ�ÕWZ]QYKO=IQOÏß9>Zô9<\OA�9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[QYÐ(Z�Ñ=IQy9>A�;>ANÐr9C9>ZÜ9>\OA(A6áGÕO;<A�Ð<Ð>=IZ[Q�� � � � � DpÙ�\OA�;<A�Ð>UOÓI9�MWÝ�\O=@?*\ß?�8]Q·ÒWA�ÕO;<A�?�Z]H�ÕOUG9<A�KpM^=@ÐC8]Q=IHJ8�Ï[A½Ý�\O=Á?*\Þ=@Ðb9>\OA½9*8�QOÏ[A�Qy9�Û]A�?69>Z[;�DGÙ�\OA�ÑÔZ]ÓÁÓÁZ:Ý�=IQOÏ(ÓÁ=ÁÐ�9/?6Z[Q[9*8�=ÁQYÐ�Ð�Z[H�A�Z]Ñ$9<\OA�H�Z[Ð�9UYÐ�A�ÑÔUOÓp9*8�QOÏ[A�Qy9�Û]A�?69>Z[;<Ð�û

Page 29: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

=

=x

x

QSRUT ®*V �Y® � o�k� y~[v|sCvx�|�x�[t±�uo�k��rvxqNwÜq�²�r~[mCs<qNi� y�]��k��rvxqNw¥q�²��r~[m/��k�wylNm<w���{�m*s<�rq�o ��� � � � �������¾³�Gq��u�rqNi÷vxi�k�lNm6¿>µ]hjw��r~yvxt�m<¢[k�i� y�xmV�r~[m��[vxtu y�|k�s*m*i�m*w:��²Áq�oPmek�s�~  yv�¢]m*�Ov|t� ]orqN Gq�ou�rvxq�w¡k��

�rqP�r~ym�[vxt±��k�w[s*m�q�²y�r~ym [v�¢[m<���rqP�r~ym�s*m*w:�rm<o�q�²[�r~ym�v|i�k�lNm�¾�� � ¾�� ��� ¿� � � ����k�w¡� � � ¾�� � � ¿� � � � �e¿>µ���~ym��jºq�i��[�x�rvx [�|vxsek6�rvxqNwyt�¾³~[q�orvx�*qNw:��k��¡�xv|w[m*t�¿�k�t�º�m*�x�Gk�t$�r~[m�kN�y�]v��rv|q�wJ¾³{�m<ou�rvxsek��[orvxlN~:�s<qN�x�yi�w¡¿�k�orm%�[q�wym� [v�¢[m<�Y�rq½ [v�¢[m<�Ôµ

��� !��*��� �P����!� � � Ù�\O=@н9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Qô=ÁÐ�UFÐ�A�ÑÔUOÓPÝ�\OA�Qô9>\OAd?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q·ÑÔUYQY?e9<=IZ[Q=ÁÐ%`yQYZ:Ý�9>ZJÒWA�=IQ¡Û:8�;<=@8�Qy9bÝ�=I9>\�;<A�Ð>ÕFAN?e9�9<Z�9<\OA�=ÁQOÕOUG9%9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q�û

� � û�� # "� � �� � # �" � ä I�æyç

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

� � �� #

ä I M]ç� !��*��� �P����!� � � Ù�\O=@н9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Qô=ÁÐ�UFÐ�A�ÑÔUOÓPÝ�\OA�Qô9>\OAd?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]Q·ÑÔUYQY?e9<=IZ[Q=ÁÐ%`yQYZ:Ý�9>ZJÒWA�=IQ¡Û:8�;<=@8�Qy9bÝ�=I9>\�;<A�Ð>ÕFAN?e9�9<Z�9<\OA�=ÁQOÕOUG9%9<;<8]QYÐ�ÑÔZ];<H�8�9>=ÁZ]Q�û

� � û�� # "�� � �� � #" � � ä I Kyç

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

��� � �� " ä I(�yç

� �!��6!P � � Ù�\O=ÁÐ$9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q½=ÁÐ$UYÐ�A�ÑÔUOÓyÝ�\OA�Q½9<\OAV?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q�ÑÔUOQY?e9<=IZ[Q�=@Ð$`¡QOZ:Ý9>ZJÒWA�=IQ¡Û:8�;<=@8�Qy9bÝ�=I9>\�;<A�Ð>ÕFAN?e9�9<Z�9<\OA�=ÁQOÕOUG9�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Q�û

� � û�� # "� � � � # ?�Z[Ð�� " Ð�=ÁQ �# Ð>=IQ � " ?6ZyÐ � � ä¯æ]èyç

Page 30: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

��� � " �� #

ä # ç �� " ä¯æOå�ç

��� ���� � �� Ù�\Y=ÁÐb9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Qd=@Ð�UYÐ>A6ÑÔUOÓ$Ý�\OA�QÞ9>\YA½?�ÓÁ8[Ð>Ð>=IâF?�8�9>=ÁZ]QÜÑÔUOQY?e9<=IZ[Q�=ÁÐ�`¡QOZ:Ý9>ZJÒWA�=IQ¡Û:8�;<=@8�Qy9bÝ�=I9>\�;<A�Ð>ÕFAN?e9�9<Z�9<\OA�=ÁQOÕOUG9�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Q�û

� � û � # "� � � � # � #" � "$� ä¯æ G[ç

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

� � � # �� #

" �� " ä¯æ yç

�N�6�*�������� "$� � ;�P�L ��� �� !��*����� ���� �6!� � � Ù�\O=@Ð�9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q�=ÁÐ�UYÐ>A6ÑÔUOÓ¡Ý�\OA�Q�9>\YA?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]Q½ÑÔUYQY?e9<=IZ[Q =@Ð�`¡QOZ:Ýô9<ZCÒWAV=ÁQ¡Û:8�;<=Á8]Q[9�Ý�=|9<\ ;<A�Ð>ÕFAN?e9$9<Z/9<\OAb=IQOÕYUG9�9<;<8]QYÐ�ÖÑÔZ];<HJ8:9>=ÁZ]Q$û

� � û � # " � � � � # � #"% � " � ä¯æ(�¡ç

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

� � � # �� #

" �� " ä¯æ I[ç

� ����� �����L"$� �L ��� ��� � !��*����� ���� �6!� � � Ù�\O=@ÐÞ9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q =@Ð�UYÐ�A�ÑÔUOÓ�Ý�\OA�Q9>\OA)?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q�ÑÔUOQF?e9>=ÁZ]Q =@Ð�`yQYZ:Ý 9>Z°ÒFA�=ÁQ¡Û�8];>=@8�Qy9�Ý�=|9<\.;<A�Ð>ÕWA�?e9(9>Zô9<\OA�=IQGÖÕOUG9�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]Q�û

� � û � # " � � � � # � "" � # � ä¯æ]æyç

Ù�\OA�S�=ÁA½Z]ÕWA�;*8:9<Z];�=ÁÐ/KGA6âYQYA�KÞÒ¡×Wû

� � � " �� #

# �� " ä¯æ;M]ç

Ù�\OA�;>ANÐ�UYÓ|9<=IQOÏõ9<8]QOÏ]A�Q[9�Û]AN?e9<Z];(=ÁÐ�=@Ð�9>\OA�QOZ[;>H Z�ÑC9>\OAÞÏ];*8]KO=IA�Q[9�Z�Ñ/9>\OA�=ÁH�8]Ï]A[MÝ�\O=@?*\Þ=@Ð�Û]A�;>×dA�8]Ð>×�9>Z¥?�Z]H�ÕOUG9<A]D

�O" ��� � � �� Ù�\O=@Ð�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q°=ÁÐ UYÐ�A�ÑÔUOÓPÝ�\YA�Q¼9<\OAd?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[QßÑÔUYQY?e9<=IZ[Q�=ÁÐ`¡QOZ:Ý 9>ZôÒFA�=ÁQ¡Û�8];>=@8�Qy9(Ý�=|9<\�;<A�Ð>ÕFAN?e9(9>ZôÛ�8];>=@8:9<=IZ[Q�Z�Ñ/9>\O=@?*`¡QOA�Ð<Ð�D�Ù�\O=@Ð(=ÁÐ(`¡QOZ:Ý=IQ.H�Z];<ÕO\OZ]ÓÁZ]Ï[×ß8[Ð�KG=ÁÓ@8:9>=ÁZ]Q$MZ[; A�;>ZyÐ�=ÁZ]Q�D �±9(=@Ð�Û]A�;<×°UYÐ>A6ÑÔUOÓ�=ÁQ�?6A�;�9*8�=ÁQ�KGZ]HJ8]=IQ�MÐ�UF?*\õ8]Ð/\Y8�QYKOÝ�;>=I9�9<A�Q·?*\Y8];<8[?e9>A�;%;<A�?6Z[Ï]QO=I9>=ÁZ]Q�ÒWA�?�8�UYÐ>A�9<\O=Á?*`[A�QO=ÁQOÏd8]QYK�9<\O=IQYQO=IQYÏ8�;<A%QY8�9>UO;*8�ÓWÛ:8�;<=Á8�9>=ÁZ]QYÐ�Ý�\Y=Á?*\d?�Z];<;>ANÐ�ÕWZ]QFK(9>Z�9<\OA/ÕY;>ANÐ>Ð>UO;<A/8�ÕOÕYÓI=ÁA�KJZ[Qd8�ÕWA�Q�M¡Z[;9>Z·KG=Ià^A�;<A�Qy9�8�ÒFÐ�Z[;>ÒG9<=IZ[QßÕY;>Z[ÕFA�;�9r×·Z]Ñ�9>\OA¥=ÁQO`ßZ]Qô9>\OAdÕY8�ÕWA�;ND$R KG=ÁÓ@8:9>=ÁZ]Q�ä¶;>ANÐ�Õ�DA�;<Z[Ð>=IZ[QFç�?�8�Q�ÒWAÞKOA6âYQOANK�8]Ð 9>\YAdZ]ÕWA�;*8:9<=IZ[Q�Z�Ñ%;<A�ÕYÓÁ8[?6=ÁQOÏ·A�8[?*\¼Û:8�ÓÁUOA¥Ñ�äÔá^M ×Oç�Ò¡×9>\OA ÓÁ8];>Ï[A�Ð�9 äÔ;<A�Ð>Õ�DYÐ>H�8]ÓIÓÁA�Ð�9*ç�Û�8]ÓIUYA�Z�Ñ��ä # $ � " $ çbÑÔZ[;�ä # $ � " $ ç�Ý�=|9<\O=IQ·8�;<A�Ï[=IZ[Q�Z�Ñ�8?6A�;�9*8�=ÁQ¥Ð�\Y8]ÕFA[M[?�A�Qy9>A�;>ANKJ8:9Cä # � " ç6D¡Ù�\YA%;>A�Ï]=ÁZ]Q�=@ÐP?�8]ÓIÓÁA�K(9<\OACÐ�9>;<UY?69>UO;*8�ÓFA�ÓÁA�H�A�Qy9�D�ßA�Ý�=ÁÓÁÓO8]Ð<Ð�UOH�Ab9>\F8:9�9>\OA%Ð�9>;<UY?69>UO;*8�ÓGA�ÓIA�H(A�Qy9�=@Ð�8½Ð�ÕY\OA�;<A�Z�ÑF;*8]KO=IUYÐ��VD(�ôA%KGA6âYQYA

Page 31: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

9>\OAb9<\O=Á?*`[A�QO=ÁQOÏC9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q 8]Ð�9>\YAVÑÔUYQY?e9<=IZ[Q�Ý�\O=@?*\�9*8�`[A�Ð�9>\OAbÑÔUOQF?e9>=ÁZ]Q�J8]QYKÏ]A�QOA�;*8:9<A�ÐV9>\OA½ÑÔUYQY?e9<=IZ[Q $� KGA6âYQYA�KÞÒ¡×Wû $� ä� ôç � HJ8:á

� � ��� � �ä� ���ç ÑÔZ]; ���.è ä¯æ Kyç

$� ä� ßç � H�=IQ� � ��� � � �ä� ����ç ÑÔZ]; � �.è ä¯æ �yç

Ù�\OA�KGA�;>=ÁÛ:8:9>=ÁÛ]A½Z]Ñ$9>\YA½9>\O=@?*`]A�QO=IQYÏ�ÑÔZ[; � � � è(?�8�QÞÒFA�Ý�;<=I9�9>A�Q)8]Ð�û

ÓÁ=IH� � � �

$ ä� ôç �ä� ßç� � ÓÁ=IH� � � �

HJ8:á � � ��� � �ä� ���ç �ä� ßç� ä M:èyç

�ä� ßç%?�8�Q)ÒWA ÕYUG9/Ý�=I9>\O=ÁQ)9>\YA HJ8:á�A6áGÕO;<A�Ð<Ð�=ÁZ]Q�ÒWA�?�8�UYÐ>A�=|9½KGZ¡A�Ð/QOZ�9CKOA�ÕWA�QYK)Z[Q��� �:DWE¡=IQF?6A � ���C9>A�QYKOÐb9>Z:Ý�8�;*KdèYMOÝbA½?�8�QÞÝ�;>=I9>A[û�ä� ���ç �ä� ßç � � � � �ä� ôç �ä���� � f ç��� � � �ä� ßç ä M¡å�çÙ�\OA�HJ8:áG=ÁH UOH Z]Ñ

HJ8:á� � ��� � �ä� ���ç �ä� ßç � H�8�á

� � ��� � � � � �ä� ßç ä M G[ç

=ÁÐ/8:9�9*8�=ÁQOA�KÜÝ�\OA�Q�(8�QYK � �ä� ôçV8];>A�?�Z�Ö±ÓI=ÁQOA�8];�My9<\Y8:9%=@Ð�Ý�\OA�Q� � � � �ä� ßç

� � �ä� ßç � ä M yç8]Ð<Ð�UYH(=ÁQOÏ � �#èOD��±9�?�8]Q·A�8[Ð�=ÁÓÁ×ÜÒWA�Ð>\OZ:Ý�Q�9>\F8:9/9<\O=@Ð/ANù[UF8:9>=ÁZ]Q)\YZ]Ó@KOÐ/Ý�\YA�Q&�.=ÁÐQOA�Ïy8:9<=IÛ[A]M�ÒWA�?�8�UYÐ>A%ÝbA�9>\OA�QJ9>;<×�9>Z�H�=ÁQO=IH�=Áã�ACA�ùyUY8:9<=IZ[Q)ä¶æ �yçeD �ßA/9>\OA�;>A�ÑÔZ];<A�\Y8�Û]A[û

ÓI=ÁH� � � �

$� ä� ôç �ä� ßç� � � � �ä� ßç � ä M �¡ç

��\O=@?*\Þ=@Ðb9>\YAC9*8�QOÏ[A�Qy9�Û]AN?e9>Z[;�Z�Ñ�=ÁQy9>A�;>ANÐr9ND�B%Z�9<A½9>\Y8�9�9>\O=@Ð�=@Ðb9>;<UOA�ÑÔZ]; �°ÕWZ[Ð>=|9<=IÛ[AZ];½QOA�Ï[8:9<=IÛ[A]D�Ù�\OA(9*8�QOÏ[A�Qy9�Û[A�?69>Z];*ÐCÑÔZ];½9<\O=Á?*`[A�QO=ÁQOÏ�8�QYKõ9>\Y=IQOQY=IQOÏ·8];>A(=@KGA�Qy9>=@?�8]Ó¯DR%ÓI9>A�;>QY8�9>=ÁÛ]A�ÓI×[MGÝVA�?�8]QÜUFÐ�A½Z]UY;%?6Z]H�ÕOUO9<8:9<=IZ[QÜZ]Ñ$9<\OA�KG=ÁÐ>ÕOÓ@8]?�A�H�A�Qy9��(8]QYKÞKGA6âYQYA9>\OA½ÑÔZ[ÓIÓÁZ:Ý�=ÁQOÏ(9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]QÜZ�Ñ�9>\OA�=ÁQOÕOUG9Nû

� � ä pçbû�� # "�� � �� � # ��� " ��� � � ä M I[ç

Ý�\OA�;<Aä � ��� � ç � � � � � �ä� ßç

� � �ä� ßç � ä M:æyçÙ�\O=@Ð�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q°Z�Ñ�9>\OAÜ=IQYÕOUG9(Ð>ÕY8[?6AJ=@Ð�KG=Ià^A�;<A�Qy9�ÑÔZ];�A�8[?*\ôÕF8:9�9<A�;<Q #äÔÝbAKGZ(QOZ�9b\Y8�Û]A�8 S�=ÁA�Ï];<Z]UOÕJZ]Ñ�9>;*8�QYÐ�ÑÔZ];<HJ8:9<=IZ[Q�M]ÒOUO9V9<\OA/âFA�Ó@KdÐ�9>;<UY?e9<UO;>ACÏ]A�QOA�;*8:9<A�KÒ¡×õ9<\OAõäÔÕFÐ�A�UYKGZõS$=IA�ç½Z]ÕWA�;*8:9<Z];�=@Ð Ð�9>=ÁÓIÓVUYÐ�A�ÑÔUOÓjD�Ù�\OA¥Z[ÕFA�;<8�9>Z[;�UYÐ>A�Kß9>Z)âFQYK°9>\YA9<8]QOÏ]A�Q[9�Û[A�?e9<Z];�=@Ð�KGA6âFQOA�KÞÒ¡×^û � � � � � � ä M M]çÝ�\O=@?*\¼H(AN8�QYÐ�9>\Y8�9 9>\YAJ9<8]QOÏ]A�Q[9�Û[A�?69>Z]; =IHJ8�Ï[AJ=ÁÐ Z]ÒG9*8�=ÁQOA�KôÒ¡×°?6Z]H�ÕOUO9>=ÁQOÏ)9>\YAQOZ];<HJ8�ÓÁ=Iã�A�KßÏ];*8�×�ÓIA�Û]A�ÓPÏ];*8]KG=ÁA�Qy9½Z�Ñb9>\OAJ=ÁHJ8�Ï]A�8:9�AN8]?*\ôÕFZ[=IQy9¥ä³9<\OA¥Ï[;<8[KG=IA�Qy9½8�9A�8[?*\ÜÕWZ]=ÁQy9%=ÁÐ�QOZ[;>HJ8]ÓI=Áã�ANKYçeD

Page 32: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

Ù�\OA/Ó@8]Ð�9 I�9>;*8�QYÐ�ÑÔZ];<HJ8:9>=ÁZ]Q¥8];>A�KGA�ÕY=Á?69>A�Kd=IQ/�=IÏ[UO;>A�å ��Ý�=I9>\d9>\OAC9<8]QOÏ]A�Qy9VÛ]AN?e9<Z];NDÙ�\OA%Ó@8]Ð�9�Z]ÕWA�;*8:9>Z[;�?6Z[;>;<A�Ð>ÕFZ[QYKOÐ9<Z 8�9>\O=@?*`]A�QO=ÁQOÏ�Z];�9<\O=ÁQOQO=ÁQOÏ�Z�Ñ^9<\OA%=ÁH�8]Ï]A[D]Ù�\O=@Ð�UOQGÖUYÐ�UF8�Ó^9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Qd=@Ð�A6á¡9<;>A�H(A�ÓI×dUYÐ�A�ÑÔUOÓpÑÔZ[;�\Y8�QYKOÝ�;>=I9�9<A�Q)?*\Y8�;*8]?69>A�;V;<A�?�Z]Ï]QY=|9<=IZ[Q�D

Scaling Rotation Axis deformation Diagonaldeformation

Thicknessdeformation

QSRUT ®*V �W® h¯�x�|�[t±�uo�k��rvxqNwCq�² ����k�wyl�m*w:�^{�m*s>�rq�ort�¾@�rq� G¿>�eº�v��r~Cs<q�ouorm*tu Gq�w¡�[vxw[lb�]v|tu [�|k�s*m*i�m<w��rt�¾³i�v|�]��]�xm6¿^k�wy�/�uo�k�w[t±²@q�ori�k��rvxqNw½m>´Om*s<�rt�¾³�Gq��u�rq�i ¿>µ���~ym��]v|tu [�|k�s*m*i�m<w�� � � k�wy� �

�k�orm�orm* [orm*tum<w��rm*�

vxwd�r~ymC²Áq�ori q�²�{�m*s<�rq�o�¸¡m*�|�OµFh¶��sek�wÞ�Gm½w[q��rme�¥�r~¡k����r~[mC��k�wylNm<w��%{Nm*s<�rq�o�²Áq�o%�r~ym/�r~yvxs��]w[m*tut�]m<²@q�ori�k��rvxqNwô¾@orv|l�~:�%s*qN�x�yi�w¡¿Vs*q�ouorm*tu GqNwy�Ü�rq��r~[m½wyq�ori q�²��r~[m½l�o�k��[vxm*w:�%q�²��r~[m½l�o�k*¹Ü�xm*{�m*�vxi�k�lNmNµ

� �ßÉVÆ/Ì���Ë ¡ÍrÉPÆÙ�\OAJÒY8]Ð>=@?JÙ�8]QOÏ]A�Q[9 _C=@Ðr9*8�QY?�A¥8�ÓÁÏ]Z];<=I9>\OH =@Ð�ùyUO=I9>A¥AN8]Ð>×)9>Z)UYQYKGA�;*Ð�9<8�QFKô8]QYKô=IH�ÕOÓÁA6ÖH(A�Qy9�D�#�Û[A�QJ9<\OZ]UYÏ]\d\F8�;*KGÓI×¥8]Q¡×(ÕY;>A�ÕO;>ZG?�A�Ð<Ð�=ÁQOÏ�Z[;bÓÁA�8];>QY=IQOÏ�=ÁÐb;<A�ùyUO=Á;>ANKpM[9>\OA½ÕWA�;>ÑÔZ];>ÖH�8]QY?6A/=ÁÐ�Ð>UO;<ÕO;<=ÁÐ>=IQYÏ]ÓÁ× Ï]Z¡ZGK�8�QYK�?�Z]H�ÕY8�;<A�Ð�ÝVA�ÓIÓG9<Z�9>\YA%ÒFANÐr9�?6Z[H(ÕWA69<=IQYÏ�8�ÓÁÏ]Z];<=I9>\OHJÐ�D�ßA�ÒWA�ÓÁ=IA�Û]A(9>\Y8�9½9>\OAJHJ8�=ÁQô;>AN8]Ð>Z]Q)ÑÔZ[;C9<\O=@Ð�Ð�UF?�?6ANÐ>Ð�=@н=|9*Ð�8�ÒO=ÁÓÁ=|9r×�9<ZÞ=ÁQY?�Z];<ÕFZ[;<8�9>AÞë8�)eí¶ð*)eí[`¡QOZ:Ý�ÓIANKGÏ]AP=IQy9>Z/9>\OA�KG=@Ðr9*8�QY?�APH�A�8[Ð�UY;>A[DNÙ�\OAVZ]QOÓÁ×�8�ÓÁÏ]Z[;>=I9>\OH#Ý�\O=@?*\�ÕWA�;>ÑÔZ];<H(ANKÒFA�9�9>A�;�9<\Y8�Q¥Ù8�QOÏ[A�Qy9�_/=@Ð�9<8�QF?6A%Z[Q(Z[QOA�Z�Ñ^9<\OA�9>\Y;>A�A%KO8:9*8�ÒY8[Ð�ANÐ�Ýb8[Ð�ÒFZ¡ZyÐr9<=IQOÏFM�Ý�\O=@?*\\Y8]Ðb9<\OA�Ð�=ÁH�=IÓ@8�; ë 8�)eí¶ð )eíp`yQYZ:Ý�ÓIANKGÏ]A½8�ÒWZ]UG9�9<;<8]QYÐrÑÔZ[;>HJ8�9>=ÁZ]QYÐbÒOUO=ÁÓI9%=IQy9>ZJ=I9�D

2·8�Q¡×J=ÁH(ÕY;>Z:Û[A�H�A�Qy9<Ð�8];>A�Z�Ñ�?6Z[UO;*Ð�A�ÕFZyÐ>Ð>=ÁÒOÓIA[D �=I;*Ðr9NMY8�Ð�HJ8];�9�ÕO;<A�ÕO;<ZG?6ANÐ>Ð>=IQYÏ�?�8]Q8�ÓÁÓIZ:Ý�UFÐ$9<Z½H(AN8]Ð>UO;<A�9>\YA�Ù8�QYÏ]A�Qy9�_/=@Ð�9<8�QF?6A�=IQ�8CH(Z[;>A�8�ÕYÕO;>Z[ÕO;<=Á8�9>A�ö�ÑÔA�8:9<UO;<A�ø�Ð�ÕY8[?6A[M=IQYÐ�9>AN8]KÞZ�Ñ�9<\OA�Z];<=ÁÏ]=ÁQY8�ÓpÕO=IáGA�Ó$Ð>ÕY8[?6A]D��uQ)=IHJ8�Ï[A�?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Q�MOÑÔZ];�A6áO8]H(ÕYÓIA[MO9>\YA½ÑÔA�8:Ö9>UO;<A�ÐÞ?6Z[UOÓÁK ÒFAô\OZ];<=Áã�Z]Qy9*8�ÓC8]QYK�Û[A�;>9>=@?�8�Ó�A�KOÏ]A�Ð�DVÙ�\Y=ÁÐÞÝbZ]UOÓ@K�H�Z[Ð�9ÜÓÁ=I`[A�ÓÁ× ÑÔUO;�9<\OA�;=IH�ÕO;<Z:Û]AC9>\OA½ÕWA�;>ÑÔZ];<HJ8�QY?�A � Ù�\OA½Z]QOÓÁ×¥;<A�ùyUO=Á;>A�H�A�Qy9b=@Ðb9>\F8:9�9>\OA�ÕY;>A�ÕO;>ZG?�A�Ð<Ð�=ÁQOÏ H�UYÐ�9� ��~ym>orm�i�ke¹¥�Gm½k�w�k��y�]vx�rvxq�w¡k��$s*q�t±�/²Áq�o/s*q�i� y�[�rvxw[l��r~ym/��k�w[lNm*w:�/{�m*s<�rq�ort�vxwÜ�r~ymC²@mek6�r�[ormtu ¡k�s*m�v�²^�r~ym�²Ámek��r�]orm�tu ¡k�s*m�vxtV{�m>ou¹�s*qNi� [�|m>¢Yµ

Page 33: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

ÒFA/KO=|à^A�;<A�Qy9>=@8�ÒYÓIA[M]Ð�Z�9<\Y8:9�9<\OA�9<8]QOÏ]A�Qy9�Û]A�?69>Z[;<Ð�?�8�Q(ÒFA/?�Z]H�ÕOUG9<A�K�äÔÕY;>Z[ÕY8�Ïy8:9>ANKYç�=IQy9<Z9>\OA½ÑÔAN8:9>UY;>A�Ð>ÕY8]?�A]DE¡A�?�Z]QYKpM�9<\OA)Ù�8]QOÏ]A�Qy9�c�AN?e9<Z];*Ð(8]QYK�9>\OA)ÕO;<Z�9<Z�9rסÕWA�Ð�?�8]Q.ÒWA�ÓIAN8�;<QOA�K�ÑÔ;>Z[H 9>\YA

9>;*8�=ÁQO=IQYÏ KO8:9*8OMy;<8�9>\OA�;�9<\OA�Qd?*\YZ[Ð>A�Qßë 8�)eí¶ð )eí³D*�uQd8�ÕYÕOÓI=@?�8�9>=ÁZ]QYÐbÐ>UY?*\¥8]ÐbÐ�ÕWA�AN?*\�MyÝ�\OA�;<A=|9�=@Ð�QOZ]9�?�ÓIAN8�;�9<Z�Ý�\Y8:9�9>;*8�QFÐrÑÔZ[;>HJ8:9<=IZ[Q�9<\OA%?6Ó@8]Ð<Ð�=IâF?�8:9>=ÁZ]Q(=@Ð�=IQ¡Û:8�;<=Á8]Qy9�M�9>\YA%8�ÒO=ÁÓI=I9r×�9<ZÓIAN8�;<Q¥9>\YA�Ù�8]QOÏ]A�Q[9�c�A�?69>Z[;�=ÁÐ�8�;>ANùyUO=I;<A�H�A�Q[9ND*�±9�=ÁÐ�Ð�9>;*8�=ÁÏ]\y9�ÑÔZ[;>Ý�8�;*K(9>Z�H�ZGKG=IÑÔ×Ü8�ÓÁÏ]Z]Ö;>=I9>\OHJÐVÐ�UY?*\J8[Ð�S]c��Úä¯S�A�8];>QY=IQOϽc�AN?e9>Z[;VùyUY8]Q[9<=IãN8:9<=IZ[QFç�9>Z�UFÐ�A%8½9*8�QOÏ[A�Qy9PKG=@Ðr9*8�QY?�A]D �±9=ÁÐC8�Ó@Ð�ZJÐ�9>;*8�=ÁÏ]\y9�ÑÔZ[;>Ý�8�;*KJ9<ZdKGA�;<=ÁÛ]A�8�Q)ÒY8�9<?*\�EIå *H�Z];%Z[QGÖjÓÁ=ÁQOA E G *H�8]ÓIÏ[Z];<=|9<\OH 9<Z�9>;*8�=ÁQ9>\OA½9*8�QOÏ[A�Qy9�Û]A�?69>Z[;<Ð��=ÁQY8]ÓIÓÁ×]MOHJ8]Qy×ÜZ]ÕG9<=IH�=Áã�8�9>=ÁZ]QYÐ�Ý�\Y=Á?*\)8�;<A�?6Z[H(H�Z[QOÓI×ÜUYÐ>A�KÞ=IQ·KO=ÁÐ�9<8]QY?6A�ÒY8[Ð�ANK�8�ÓIÖ

Ï]Z];<=I9>\OHJн?�8]QßÒWAJUYÐ>A�Kô8]Ð�Ð>UY?�?�A�Ð<ÐrÑÔUOÓÁÓÁ×·Ý�=|9<\¼Ù�8]QOÏ]A�Q[9�_C=ÁÐ�9<8]QY?6A(9>Z·Ð>ÕWA�A�KßUOÕ¼?6Z[H(ÖÕOUG9<8�9>=ÁZ]Q�D�Ù�\OA·H�UOÓI9>=IÖj;<A�Ð>Z]ÓÁUG9>=ÁZ]Q 8]ÕOÕO;<Z[8[?*\ \F8�Û]A·8�ÓÁ;<A�8]KO×�ÒWA�A�Q 9>;<=IANK�Ð�UY?�?6ANÐ>Ð�ÑÔUOÓÁÓI×E G I�H±D;=/9>\OA�;PH�A69<\OZGKOÐ�ÓÁ=Á`]A¥ö>H UOÓI9>=IÖjANKG=I9�Öu?6Z]QFKGA�QYÐ>=ÁQOÏ[ø5EIå]M [è*HW8]QYK5A�Ö±K(9<;>A�AOE � HW8];>A%8]ÓÁÐ>ZÕFZyÐ>Ð>=IÒYÓIA[DÙ�\OAÜHJ8]=IQ 8]KGÛ:8]Q[9*8�Ï[AdZ�Ñ�Ù�8]QOÏ]A�Q[9J_C=ÁÐ�9<8]QY?6A�=@Ð 9<\Y8:9J=I9J=ÁÐ�8ôH�Z¡KO=|âF?�8:9<=IZ[Q�Z�Ñ�8

Ðr9*8�QYKO8];<K�KG=@Ð�9<8�QF?6A�H�A�8[Ð�UO;<A½9>ZÜ8�ÓÁÓIZ:Ý =|9C9>ZJ=ÁQY?6Z[;>ÕWZ];*8:9<A�8�ÕO;>=ÁZ];<=�`¡QOZ:Ý�ÓÁA�KGÏ[A�9>\Y8�9/=ÁÐÐ�ÕWA�?�=|âF?�9<Z�×]Z]UY;�ÕO;>Z[ÒOÓÁA�H�DOR%Q¡×Ü8]ÓIÏ[Z];<=|9<\OHJÐbÒY8]Ð>A�KÜZ]Q�8J?6Z[H(H�Z[QÞKG=@Ð�9<8�QF?6A½H�A�8]Ð>UO;<Aä¶8]Ð�=I9�=ÁÐ�Z�ѳ9<A�Qõ9>\OAJ?�8[Ð�A�=IQ°?6Ó@8]Ð<Ð>=|âF?�8:9<=IZ[Q�MYÛ[A�?e9<Z];½ùyUY8�Qy9>=Áã�8�9>=ÁZ]Q�M^ÕO;<A�KO=Á?69>=ÁZ]QYÐ�MFA�9<?]DIDÁD ç?�8�Q(ÕWZ�9>A�Qy9>=@8�ÓÁÓI×�ÒFA�QOA6âO9�ÑÔ;<Z]H 8�H(Z[;>A�ÕO;<Z]ÒYÓIA�H�ÖuÐ>ÕFAN?6=IâF?�KG=ÁÐ�9<8]QY?6A[D 2·8�Q¡×�Z]ÑF9<\O=@Ð�ö>KG=@Ð�Ö9<8�QF?6A%ÒY8[Ð�ANKOø½8�ÓÁÏ]Z[;>=I9>\OHJÐ�KGZ QOZ�9P;<A�ùyUO=Á;<A�8]Q¡× ÓÁA�8�;<QO=ÁQOÏYM]Ý�\O=Á?*\�H�A�8]QYÐ�9<\Y8:9P9>\OA�×�?�8]QÒFA(8]KO8]ÕG9>ANK�=IQYÐ�9<8]Qy9>ÓÁ×ÜÒ¡×�ØrUYÐr9�8]KOKG=ÁQOÏ¥QYA�Ý÷ÕY8:9>9>A�;>QYÐ/=IQ·9>\OA KY8:9<8]ÒY8]Ð>A]DFÙ�\OA�Ð>A 8]KYKG=|Ö9>=ÁZ]QYÐ�8];>A½ÓÁA�Û]A�;<8]Ï]ANK¥Ò¡×J9>\YA�8:Ö±ÕO;>=ÁZ];<=^`¡QOZ:Ý�ÓIANKGÏ]A�ÕOUG9%=ÁQÞ9>\OA½9*8�QOÏ[A�Qy9%KG=@Ðr9*8�QY?�A]DÙ�\OA(9rÝVZ)KG;*8�Ý�ÒY8]?*`GÐCZ�ÑV9<8�QYÏ]A�Qy9�KG=@Ðr9*8�QY?�AJ8�;<A�=I9<Ð�H�A�H(Z[;>×ß8�QYKô?6Z[H(ÕYUG9<8�9>=ÁZ]QY8]Ó

;>ANù[UY=I;<A�H�A�Qy9*Ð�DWÙ�\OA�H�ZyÐr9½?6Z]H�ÕOUO9<8:9<=IZ[QY8�ÓÁÓÁ×Þ8]QYK·H�A�H�Z[;>×ÞA(J¥?6=ÁA�Qy9�8�ÓÁÏ]Z[;>=I9>\OHJÐ/Ï]A�QOÖA�;*8�ÓÁÓI×J=ÁQ¡Û]Z]ÓÁÛ]ACÓIAN8�;<QO=ÁQOÏ E G�è HjD*�YZ];>9>UOQY8�9>A�ÓI×[M[9<\OA�?6Z[QY?6A�ÕG9�Z�Ñ$9*8�QOÏ[A�Qy9�Û]A�?69>Z[;<Ð�?�8]QÜ8]ÓÁÐ>ZÒFA)UYÐ>A�K.=ÁQ.ÓÁA�8];>QO=ÁQOÏFD�Ù�\Y=ÁÐ�=@Ð(9>\OA�ÒY8[Ð�=@Ð(ÑÔZ];(9<\OA�9<8]QOÏ]A�Q[9�ÕO;<Z]ÕY8]Ï[8�9>=ÁZ]Q�8]ÓIÏ[Z];<=|9<\OH�DÙ�\OAÜ?�Z]QY?�A�ÕG9(=ÁÐ�ùyUO=I9>AÜÐ>=ÁH(ÕYÓIA[û�=ÁQYÐ�9>A�8[K�Z]Ñ%ÓIAN8�;<QO=IQYÏõ8õ?�ÓÁ8[Ð>Ð>=|âW?�8:9<=IZ[Q°ÑÔUOQY?69>=ÁZ]Q¼ÑÔ;<Z]HA6áO8�H�ÕOÓÁA�Ð/Z�Ñ�=I9<ÐCÛ�8]ÓIUYA�Ð�MFZ[QOA ?�8]Qõ8�Ó@Ð>ZJUYÐ�A�=IQOÑÔZ];<H�8�9>=ÁZ]Q·8]ÒFZ[UG9C=I9<Ð�KGA�;<=ÁÛ�8�9>=ÁÛ]ANÐ�DWÙ�\O=ÁÐ=IQGÑÔZ[;>HJ8�9>=ÁZ]Qd=@ÐbÕO;<Z:Ûy=@KGANKJÒ¡×�9>\OA�9<8]QOÏ]A�Qy9bÛ[A�?e9<Z];*Ð�D �%QOÑÔZ];>9>UOQY8�9>A�ÓI×[MGQOZ�9�HJ8�Q¡×JA6áGÕWA�;>Ö=IH�A�Qy9*н\Y8�Û]A�ÒFA�A�Q°KGZ]QOA(=ÁQõ9<\O=ÁÐ�KO=I;<A�?69>=ÁZ]Q�DpÙ�\OA(9rÝbZÜHJ8�=ÁQôÕO;>Z[ÒOÓÁA�HJÐ/Ý�=I9>\ß9<8]QOÏ]A�Qy9ÕO;>Z[ÕY8�Ïy8:9<=IZ[Qô8];>AJ9<\Y8:9 9>\OAÞ?�8�ÕF8]?6=I9r×õZ]Ñ�9>\OAÜÓIAN8�;<QO=IQYÏ)H�8[?*\O=ÁQOA¥\Y8[Ð�9>ZõÒFAÞ8]K�ØrUYÐ�9>ANK9>ZJ=ÁQY?6Z[;>ÕWZ];*8:9<AC9<\OA 8[KOKG=I9>=ÁZ]QY8]Ó�=IQOÑÔZ];<H�8�9>=ÁZ]Q�ÕWA�;>9>=ÁQOA�Qy9%9<Z�9>\OA�9*8�QOÏ[A�Qy9%Û]AN?e9<Z];*Ð�MO8]QYK9>\Y8�9b9<;<8]=IQY=IQOÏ�9>=ÁH�A�H UYÐ�9�ÒFA½=ÁQY?6;<A�8[Ð�ANKpDGR�ѳ9<A�;�9>;*8�=ÁQO=ÁQOÏYMy9<\OA�?6Ó@8]Ð<Ð�=IâF?�8�9>=ÁZ]Qd9>=ÁH(A�8]QYK?6Z]H�ÕOÓÁA6áG=I9r×d8�;<A½UOQY?*\Y8]QOÏ]ANKpMGÒOUG9�9<\OA�?6Ó@8]Ð<Ð>=|âYA�;� ÐbÕFA�;�ÑÔZ[;>HJ8]QY?6A�=ÁÐ%=IH�ÕO;<Z:Û]ANKpDÙ$Z�8�âY;<Ð�9%8�ÕOÕY;>Z�áG=ÁH�8�9>=ÁZ]Q�MyUYÐ>=IQYÏ�Ù8�QYÏ]A�Qy9�_/=@Ð�9<8�QF?6A�Z];�Ù8�QOÏ[A�Qy9�ÕO;>Z[ÕY8�Ïy8:9<=IZ[Q¥=ÁÐ

ÓI=Á`]A¥\F8�Ûy=ÁQOÏ)8)H�UY?*\¼ÓÁ8];>Ï[A�;�KO8�9<8]ÒY8]Ð>A]D��±Ñ�9<\OAdKO8�9<8�ÒF8]Ð>AJÝb8[Ð�ÕOÓÁA�Qy9r×ßÓ@8�;<Ï]A(9<Z·ÒFA�Ï]=ÁQÝ�=|9<\�M�9<8]QOÏ]A�Q[9�KG=@Ðr9*8�QY?�AÜZ[; 9*8�QOÏ[A�Qy9(ÕY;>Z[ÕY8�Ïy8:9>=ÁZ]Q¼ÝbZ]UYÓÁK�QOZ�9�=IH�ÕO;<Z:Û]Ad9>\OAÞÕWA�;>ÑÔZ];>ÖH�8]QY?6A[D�Ù$Z¼8ôÒWA69�9<A�;¥8]ÕOÕO;<Z�á¡=ÁHJ8:9<=IZ[Q�M9<8�QYÏ]A�Qy9�Û]AN?e9>Z[;J8�;<AÜÓÁ=I`[A�UYÐ�=ÁQOϼ8°KG=@Ðr9<Z];>9>=ÁZ]QH(ZGKGA�Ó^9>Z(HJ8�Ï]QY=|ÑÔ×J9<\OA�Ð�=Áã�A�Z�Ñ�9>\OA�9<;<8]=IQO=ÁQOÏ�Ð�A�9�D*�uQÞHJ8�Q¡×¥?�8[Ð�ANÐ�MyUYÐ>=IQOÏ(9*8�QOÏ[A�Qy9bÛ[A�?6Ö9>Z];JÝ�=ÁÓÁÓ/ÒWA)ÕO;>A�ÑÔA�;*8�ÒOÓÁAÞ9>Z�?�Z]ÓÁÓIAN?e9>=ÁQOÏ ä¯8�QYK Ó@8�ÒWA�ÓÁ=IQOÏ � ç(Û:8[Ðr9<ÓI×.H(Z[;>AÞ9<;<8]=IQO=ÁQOϼKO8�9<8OM8�QYKßÕO;<A6ÑÔA�;*8�ÒYÓIA�äÔANÐ�ÕWA�?�=Á8]ÓIÓÁ×)ÑÔZ];½H�A�H�Z[;>×yÖ±ÒY8]Ð>A�K·?�ÓÁ8[Ð>Ð>=IâYA�;*Ð<ç%9>Z)KGAN8�ÓÁ=IQYÏÜÝ�=I9>\¼8�ÓÁÓ�9>\YAKO8:9*8�Ï]A�QOA�;*8:9<A�K(Ò¡×(9>\OA½KG=@Ðr9<Z];>9>=ÁZ]QJH�ZGKGA�ÓjDyÙ8�QYÏ]A�Qy9bÛ]A�?69>Z[;<Ð�ÕY;>Z:Û¡=@KGA/8 ?�Z]H�ÕY8[?e9V8]QYKÕFZ:ÝbA�;>ÑÔUOÓ$;<A�ÕY;>ANÐ�A�Q[9*8:9<=IZ[QÜZ]Ñ�8�ÖjÕY;>=ÁZ];<=p`¡QOZ:Ý�ÓIANKGÏ]A�Ý�\Y=Á?*\·?�8�Q)A�8[Ð�=ÁÓÁ×dÒWA =ÁQ[9<A�Ï[;<8�9>A�KÞ=ÁQ9>\OA�H�Z[Ð�9�ÕFZ[ÕOUOÓ@8�;%8]ÓIÏ[Z];<=|9<\OHJÐ�D

Page 34: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

� ��� �$È �$Æ/Ì �

�Nµ%n^µ6�%µ � m*{�v �±{�m<oWk�wy���[µ ��v��u�r�xm<oeµ�������� ������������������������������� �!��#"�������%$��'&(&)�*�+�(��,�>µ�nporm*w:�rvxs*m� k��x�³�¡��wyl��xm<º�q:q��d��xv�´Ot*�W� ����¤:µ¤�µ%�P��²Áorme� �Cµ�V~yq]���NqN~[w¼��µ � q� Gs<orq�²I�e��k�w¡�ß�Nm<´Gorm<¹ � µ$�b�x�xi�k�wYµ.-/��� �0�1�� 21�3�24��5�!�76

�8$ �(�!� �#��,�9�"rµ��b�y�]vxtuqNw[�Ô»Þm*tu�xm<¹:�O� �����]µ�]µ%�8:m*qNw��q��u�rq��Jk�wy� �b�|k��[vxi�v�o��Cµ �k� ywyvx�Gµ��Yq:sek��F�|m*k�orwyvxw[l�k��xlNq�orvx�r~[i�t*µ<;/ 24����$>=���98&)2@?

� �!������y�y£y¾ �N¿�� �����+A �N���]�p������¤:µ£[µ%�P�|k�w/�[µ6��orq��]m<oeµ]z:�uo�k��rm*l�v|m<tW²@q�opm3B�s*vxm*w:�Wv|w[s<orm*i�m*w:��k���wymek6orm*t±�^w[m*vxlN~��Gq�o^tumek�ors�~Fµ�������� ���

������������#����!�y�O¤�� �x�6�:� Ay�6������� �����]µ��µ � µ�zOµ���orq:q�i�~ymek��Ck�w¡� � µ��FqeºmNµ �(�y���rvx{Nk6orv|k��y�xm²Á�yw[s<�rvxqNw¡k���v|w:�rm<or Gq��|k��rvxqNwCk�wy�Ck��yk� ]�rvx{�mw[m<�jºq�or��t*µC=���98&7$D�EF�1G�"��� 9�"r�G¤ � ��¤:� A�� � �:�$�������]µ

�]µ�b{�qNw[wym��~yq�À:�ym<�u�¯��or�[~¡k6�e���m<s*vx�|m � m<»¼v��u�u� �(q�orm<�u�rmN�k�wy� � vx�x�|k�o����j���|m<v|s��GµH���7�%$ G!"��#"��I ����� J3�%$K6�"L���)6M�>,4G�"����3"rµ)�Vq�ou�r~]� � qN�x�|k�wy�Y���Pi�t±�rm<o��yk�i��� V¢�²@q�o��Y�p�Vm<º ��q�or�G���Wq��:¹:q[������N¤�µ��µ/�q�orvxwyw¡kb�q�ou�rm*t�k�w¡���b�|kN�]vxi�vxo �k� ywyvx�Gµ]z��y y Gq�ou�^{�m<s<�rq�opw[m<�jºq�or��t*µ I �(��,��#�N�OP��������#�����¤�� � ¤N���QA�¤����:��� ��� ��µ

�]µ%��m*�x�[o �Cµ � k�trk6o�k��r~:¹:µR;S����� "��T;S �D�Q,1U����WV#;8;<XY;C����9�"3Z[;�;\������� ���]�^$K��"�"�� _'���������` ���,�����a32� "rµhj���$�°�q�i� y�]�rm<oVz]q:s*vxm<�¯¹�n�orm*tut*�¡�Fq�tb�P�Ik�i�v��rqNt*�¡��k��xv�²@q�orw[v|k]�^�����]��µ

�]µ � k�ouorvxt � or�ys���m<oe�Y}PqN�Gm<ou��z�s>~yk� yv�ormN�Wk�wy��npk��uorvxs*m �Cµpz�v|i�k6o��Yµ �q:q�t±�rvxwyl¥ Gm<ou²@q�ori�k�w[s*mvxwõwym*�]o�k���w[m<�jºq�or��t*µcb����� ���7������!�7�%$�d4��24� �)�%$��*Je������� ����������������#�����f���76Y������ _>�3���%$b��1��3$#$ �D�� �)� >�¡���[�bq]µy£ � ��� �^A]�:� �]�������N��µ

�*�]µ ��m<v|w[qNtu�[��m�¦y�y���[w¡k�lNk�k�wy����~yq�i�k�t/��µp¦^�xvxs>�GµJ�Pw�qN ]�rvxi�k��$lN�xqN�yk���w[mek�orm*t±�Cwym<v|l�~��Gq�oi�m>�uorv|s�µgb*h>h�hi��*����" �(�3������"g���j������� ���c���)�%$ G!"���"F���76 I �%��,4�#�NTb�����3$�$ �D�� �7��>��]�^�Vq[µ� � �]�*£QA���� �]�p������£[µ

���Nµ�}%µ � vx�xi�q�ormNµkOk� Tl<�*��2Q&)"��mOm� ��8$ ��^U3���!"S���76�" ��9e/�nJ8��,� �#�8�'&(&7$ ����!�������"rµ^»°v|�xm<¹:���Vm<º��q�or�G�O� �N��£]µ

�e¤�µ%�Yorm*{�q�o � k�t±�rvxmN�:�p¹�k�� ��v|tu~[qNwY� ��k��xs*q��xi ��|k�or�G��k�w¡�½��k�tuqNw ¦¡k�wFµO�°i�q��]m*�]²Áq�otuv|l�w¡k6�r�[orm{Nm<orv�¸¡sek6�rv|q�wFµY�Wm<s>~[wyvxsek��y}Vm* Gq�ou�V���6¤:�*£�� ���e���:� �e�j�N�����Ü�O�$�����õ�m*�x�G�Wk��Gq�o�k6�rq�orvxm*t*�[�N�y��¹�������Nµ

�*�]µ%�Yorm*{�q�o � k�t±�rvxmk�w¡��npk6�uorv|s<m���µez]vxi�k�o��Oµ��(m<�uorvxs*t^k�w¡��i�q��]m*�xtY²Áq�o^~yk�wy�]º�orv��u�rm*w�s�~¡k�o�k�s<�rm<oorm<s*qNl�wyv��rvxqNwFµ��1� ����#"�������%$o�)�3� �7��>�W�*�]�F� �����]µ

�<£[µ%�bµ]�]µ � m*t±�rvxm�k�w¡��}/µ]�[µ]��vx�[tu~yv�o�k�w[vÔµ[l� �N �*�%$ �DpQ�6�Om�#�N���� I �+6�3$ "rµ��~yk� yi�k�w k�w¡� � k��x�Ô��YqNwy�[qNwY�O� �����]µ

� ��µ � m<q�´Gorm<¹���µ � vxw��rq�wF�p�~[orvxt±�rqN [~ym<o �½µ^h>µF»¼vx�x�|v|k�i�t*�$k�w¡� �(vxs�~¡k�m*� � µW}Vm*{Nq6º�µd�V�yk� [��rvx{�m�m<�Ik�t±�rvxs�i�q��[m<�|t�²Áq�o�~yk�wy�]�¶ [orvxw:�rme��s�~¡k6o�k�s<�rm>oborm*s*q�lNw[vx�rvxq�wFµ�hjwq��6�r!���)�� "T�#�M;S 24���%$b��^J^����9e�������s�t�*�+�� "�"��#���W�1G�"��� 9�"r�� yk�l�m*t �:�6¤^A �����]µ �(q�orl�k�w �%k��[²Ái�k�wywôn$�[�y�xvxtu~ym<ort*������N¤�µ

���]µ%�%µ$��µ � q:m<or��k�wy�õ}%µp» µ ��m*wywyk�o��Oµ·}Vv|�[l�m�orm*l�orm*tutuvxqNw����v|k�tum*�ßm*t±�rvxi�k��rvxq�wß²@q�o�wyq�w[�q�ou�r~yq�lNqNwyk��F [orqN�[�xm*i�t*µ ` ���,4�7�!9F ������3"r�^�6¤ � ���+A �����p�������]µ

�e��µ%�bµ ��qN~[qNwym<wFµ z�m*��²Á�¶q�orlNk�wyvx�ek6�rvxqNw.k�w¡�¼k�tutuq:s*v|k��rvx{�mÜi�m*i�q�ou¹:µ�hjwu��&)���#�4�� �q�N ��� "M���b��^J^����9e�������v�7�3� �)�� "��y{�qN�x�[i�m �]µ¡z] ]orv|w[lNm<ou� �$m<or�|k�l]�F������£]µ

���]µ�Cµp�Ym(��[wF�^�Vµp�q�tum<oe�$�[µ$zGµ � m<wy��m<oe� � µ � m*wy�[m<ortuq�wF�^}%µp��µ � qeº�k�o��O�^» µ � �y�[�¡k6o��Y�k�w¡���$µ � µ���k�s>��m<�Ôµ � k�w¡��º�orvx�u�rm<w��]vxlNv���orm*s*q�lNw[vx�rvxq�w º�v��r~�k%�yk�s��:�¯ ]orqN yk�l�k6�rvxqNw wym<�¯º�q�or�Gµh¯w � ke{]v|�d�WqN�]orm<�r�*�:¹:�Gme�]v��rq�oe�P��6�r!�!�7�� "��#�q;/ 24�*�%$mb��^J^����9e�������q�<���+�� "�"��#�4�v��G!"��� 9T"r�{NqN�x�yi�m�¤��^¾ � m*w�{�m<oe�G� �����N¿>�^� ���N�]µ��(q�orlNk�w �%k��]²@i�k�wFµ

���]µ�Cµ��Fm6��[wFµ � m<wym<o�k��|vx�ek6�rvxqNw k�wy��wym<�¯º�q�or�C�]m*tuvxlNw�t±�uo�k6�rm*lNvxm*t*µGh¯w�}%µ�np²@m<vx²Ám<oe�@w^µ:z�s>~]orm<�rm<oe�¦µ�¦yq�lNm*�xi�k�wF�Nk�w¡�C�$µNz��rm<m*�xt*��me�]vx�rq�ort*�x=��!���N��3����!���#"�9i�#�T�� � "�&1��3��#r�>�@wG�[orvxs�~F��z�º�v��r�*m<ou��|k�w¡�Y����������µ����|tum<{]vxm<oeµ�k�wÞm<¢��rm*wy�[me�¥{�m<ortuvxqNwܺ�k�t/ y�[�y�xvxtu~yme�dk�tCk �rm<s>~[wyvxsek���orm* Gq�ou�Cq�²�r~[m��bw[vx{�m<ortuv��j¹ q�²p�Wq�orq�w��rq]µ

Page 35: ÑÔZ]; =IQYÐ 9QY=IQOÏdÛ]A ?69>Z[;CùyUY8 ...yann.lecun.com/exdb/publis/pdf/simard-00.pdf · s(® * ! " # ! # . $ .- "! # $#

¤��]µ�Cµ]�Fm6��[wF�]��µ � µ]��k�s���m*�³�y�$µ]�q��u�rqN�Y�¡�Vµy�q�ou�rm*t*�[�[µyzOµ � m*w[��m<oe� � µ � or�ys���m<oe�:h�µ � �[¹:q�wF��/µ6�/µ��(�[�x�|m>oe�N��µ�z[k�s>��vxwyl�m<oe��n^µ�z]vxi�k�o��O��k�wy���/µ��k� yw[v|�Gµ��Fmek6orwyvxwylVk��xlNq�orv��r~yi�t^²@q�ops*�|k�t±�tuv�¸ysek��rvxqNw��G� s<qNi� ¡k6orvxtuqNw�q�w�~¡k�w¡�]º�orv��u�rm*w��[vxlNv���orm*s*q�lNwyv��rvxqNwYµ�h¯w��[µ � µ b~F�¡�Vµ ��ºqNwF�k�w¡��zOµN�~yq]��me�]vx�rq�ort*�4;S 24���%$4;/ ���>�����!"3Z ` ,�t��� ����#"�������%$ I ���,1������3">�8 ��"�&1��3���r�>�: ¡k�lNm*t¤ �]��A]¤N� �]µ¡»Þq�or�|�Jz�s*vxm*w:�rvx¸ysN�Y� ��� ��µ

¤:�Nµ%��µ]npk�or�<m*wFµ� bw�m*t±�rvxi�k��rvxqNw q�²Wk% [orqN�yk��[v|�xv��j¹��[m*w[tuvx�¯¹½²@�[wys<�rvxq�w(k�w¡��i�q��[mNµP���1��� I �!��,���1� �����|�y�N� �|�*��� �+A¡�*��� �]�$� ���N¤�µ

¤�¤�µ�» µ � µWnporm*tut*���Vµ^npµ^¦^�|k�wyw[m<ou¹:�p�Fm*�y��q��xtu��¹�zOµW�/µx�^k�w¡� ��m<�u�rm>or�|vxw[l�» µp�bµj;[249F �������%$����3� &1 "rµ��k�i��]orv|�[lNm��Vwyvx{�m<ortuv��¯¹ n�orm*tut*�G��k�iC�[orv|�[l�mN�F�������]µ

¤��]µ � qN�xlNm>o�z�s>~:ºm*wy�Gµ���~[m%�[v|k��Gq��|q�s<�Ik�tutuv�¸¡m<oeµt;/ 24�*�%$>=���98&)24� ��������[�G ¡k�lNm�vxw( ]orm*tut*�W�������]µ¤6£[µ�}PqN�[vxw)z]vx�[tuqNwFµ(z:�r�¡�[vxm*t%vxwÞ�r~[m½orqN�y�[t±�rwym*tut%q�²�i��y���rv|�[vxi�m*w[tuvxqNw¡k���tusek��xvxwyl ��n�orq:s>or�yt±�rm*tt±��k6�rvxt±�rv|s<m*t*µ[d�� ��o��� �����"����m�7�+��|�[£�� � ¤���£QA]¤����]����������µ

¤���µ%npk6�uorvxs*m�CµOz�v|i�k6o��Yµ��mB�s*vxm*w:�Vs*qNi� [�[��k6�rv|q�wJq�²$s*q�i� y�xm<¢(�[vxt±��k�w[s*mCi�m>�uorv|s<tV�[tuv|w[l�~yvxm<ou�k6ors>~[vxsek��O¸¡���rm<orvxwyl]µ�hjwM��6�r!�!�7�� "C�#�g;S 24���%$7b��+J^����9e�!������g�<���+�� "�"����4�L�1G�"��� 9�"rµ��(q�orl�k�w�%k��[²Ái�k�wyw�n��y�y�xvxtu~ym>ort*�W� ����£]µ

¤ �]µ%¦]o�k�wy�Jz]vxw¡�]m*wÞk�w¡� � q�o��]qNwÜ»°v|��²ÁqNwyl]µ bw]�¶�|vxw[mCorm*s*q�lNw[vx�rvxq�w�q�²~¡k�wy�]º�orv��u�rm*wdt±¹]i��GqN�xt*µ�Fm*s>~[wyvxsek��F}Pm* Gq�ou�����6¤�¤���� ���e�����N�6�j�N¤�h �Ü�^�$�����¼�m*�x�^�Fk��Gq�o�k��rq�orvxm*t*�G�N�yw[m½� ���N¤�µ

¤���µ��/µ��Cµ �k� ywyvx�Gµ�ht"���#9 ��������g�*J�6��&1 �)6� �7�� "[U��!"^�6T��� 9[&)�#�������%$�6(��� ��µGz] [orvxw[lNm<o �$m<or�|k�l[������N¤�µ

¤ �]µ��/µ��/µ���k� yw[vx�ôk�w¡�¼�/µ ��k�µ��~ym>or{�qNw[m*wy��vxt*µ bw°�r~ym¥�yw[v�²@q�ori s*q�w]{Nm<orlNm*w[s*m¥q�²%orm*�|k���rvx{�m�²IormeÀ:�ym*w[s*vxm*t½q�²Pm*{Nm*w:�rt/�rqJ�r~ym<vxo� [orq��¡k��[vx�|v��rvxm*t*µ ` ,��>�<���(U�8���76q�#�" �<&(&)$ ������������1"���e�]¾¶¤�¿�� ¤ ��£QA]¤����]��� �N����µ

¤ �]µ��V�yw[q �k�tus*qNw[s*m*�xqNt(k�wy�¼�Vw¡��orm<ºÚ�Fvx y [i�k�wYµ �(�y���rv�orm*tuq��|�]�rvxqNw���k�wyl�m*w:���]vxt±��k�wys<m¥²Áq�okQB�wym>�¯vxw�{�k�orv|k�w:�Vs*�|k�tutuv�¸¡sek6�rvxqNwFµ�hjwj��6!r!���7�� "����q;S 24���%$xb��^J^����9e������!�v�<���+�� "�"��#�4�M��G!"�?�� 9T"r�y{�q��x�yi�m��*�]�¡ yk�l�m*t ��£��+A ��£ �]µ �(q�orl�k�w �%k��[²Ái�k�w[w(n��y�[�|vxtu~[m<ort*�W��������µ

���]µ%��mek�w ��q�v|tuvxw�k�wy�(n�v|m>ouorm � m*{�v �±{�m>oeµp�Vw�k� [ y�xvxsek��rvxq�w�q�²F�r~ymbiC�y���rvxme�[v��u�¶s*q�w¡�[m<wytuvxwyl%�rm*s�~[�w[v|À��[m��rqd�r~[m�orm>²@m<orm*w[s*m(tum<�|m<s<�rvxqNwô ]orqN�[�|m<i vxwõkd ]orvxw��Corm*s*q�lNwyv��rvxqNwõt±¹]t±�rm*i�µ0������� ��������������������[�O¤��½�bq � � £ � �^A:£���£[���������:µ