ÿØÿà JFIF 

  
 
  ÿÛ „ 
 !.%+&8&+/1555$;@;4?.451
4,$,44444444444414444444444444444444444444444444444444ÿÀ  á á
" 

ÿÄ   



           
 ÿÄ ? 
    
 !1AQaq"2‘¡±ÁðBRbrÑá#‚’¢²3S CñÿÄ 
 

             
ÿÄ ! 
 
 
       
!1QAa‘2ÿÚ 
  ? 5˜Z¯V¦cø)›t/? z¨±>Õ5€¶‹Á¤·¼z¼Ü¬+ñ®v¤¨_ˆR­BFn©—˜ý®ç̝P8gýt·ÉSTŦˆìät?þé¼íìN/Þa)ì–í6ô…
Ï¿øÃj´¿KÇü]ÿ ªô¹-eKànëÕHTx}ýSÜ›ÿ ”7Ø×&µ<¦  ¥ÑO¶[Ù¯ä¨ÞÃÿ PZ-¬;#õ|•
oaÿ ©CìÞz3˜öː/¤­ñTûIØ}š^   mÓ%ªxˆ¥ÉŸu=Z+ISe¿45™¼u;ú&WØ÷€æßQ™®{|
íx*TC“#ZŠìZ§²‹ 6pv…³¿¡äª*áZÐ%ÒOáˆo"x«OHk
w±æ+¬V(kMúŸ5Vö«$ ÁrÏbàb57/luR ¸ÑÛj Òµì`Ðœq­û žICÀ
Ê•©4€Âcà¨Ï€O´<èÐ:›ù(Ë^L8þ‘ÍÌ#¸Ð_Ì©ÙK(Öz 4¬û+¸;ü’V’84‘¬ÃŽ:[â‡ÔÌáõp¢~§ªlæ£ö{®G>J¼"°‡7¯ÆÉèßû ‹É‹§ÁòÃýâßî
^ƾÙõ‹×óH#«LP½ïX=xÑÍ$|W?•~•îëÔ©ª‹
{ÝT…Kÿ ”hûâá)J*ö˜–ÔU;iÇ€/ ÆþjóZ\ýwØ=Ìm
ºèËL9 ýè
Æð/¨’¥öo=nË.%Îì   ŽÕ¯È|{Oj²ƒE6e/ßdÄõ²Ìâ1O®ò×TsəԸhOMýíMˆ¿¼H˜l²,7Â¥#MF/Úf°Ö½± ¸–dr‹NýÊ íjqx{œÉ ä-È    ¦
øÄër¨q°ð
†nцýÑÄÆ’mä…n<0È™;ÁÝá¯ÁZƒ7FÀmì­ É&9ˆîéi¶ùN§Y•ÃZãAâ?•‡©‰ , ó¾IŸŠc1 4â&y­&pŠ­6;M À 0¹qç»p.á …ŸÅáK@<u %Ü6ÓtIÌpÓr»söfeÁ0O…­´Kƒ&ñ™\©<+,~)‹
ºHÐ*×µ&†ž‘Æ÷ÚŸ
1ßjC5Oh´‡›

Ø×ë æâ“Þà

ÒT]§n{ºöB³ ¦ ̹Â8Êï¦MÉ<„¨s¹¶ì7Ì”æR$Ncà! /éþäRM÷Gƒ¿É$
¦oà>%6·y6ƒ‰3?”úºŽ‰éX5ªPT §µ!=Mž«Ú½‹ÅgÂSâÉaþÓoö–¯ÁÔìR>5éÿ üs¶ÆUcÌ  kÇR
]ÿ ù¬¼«VŽ;Â|‡~¢¦”ÏÅ°æ
{L™Õ°Óv¹ò¸írŽ¡×¢CÃ!íVÕ {¶»sŒNPg/
"uÕbkm²“$ďå¿é¹§°½æz¯6 †s¿!s–wÚÝ“™Œ°.ûj>·+™Òa…©Œ&rÝÎtÛë긪Ît’LAVp%c Úý[ÄzJ¾ÇàXXç@˜ó<êL]·T˜¾¥1Ó©V‡g´æ½¦Ý@¹óø!_@´ÞâSÁ —S3™•& ]@JHÚý©ZŽ €×æÔr»Áf!‡yÞ4Mv*èÓã_{‘åóUuљØ«Oïé*®EvÑ
Œ÷‡U \"㪒ÍK+À 4“M¡ï:0¥5í!'<@î´”>Ç»&Z–ïCCV˜Ì5Šo&îhè.žû
|ÓK©h$s6KìŒëã)¹hI¦GïOåóI;ììü#É$Š0…Ææ¥TØ.5­¾gn´ “ÂÖ\:hœ89G)J@„}œ:’Ò{/Š"¦_Æ×7Æ3VÇŠÊa]ÚŒÙ€Ä–=®uÁßâACZƒ§§£Qnâ:«,×{tyø¬iÛcœÜÄ€H½ÄÍCk´÷šß
.W'b¤Íåh]÷€=,Žv×cÚEÚHXJX¶îo¨FÒtèöŸ>ªª6[J®Fµ£sGÁeqõfe\íjÒÐïÄÐGˆe1Ø‹.Ø”‘Ëuø
Y­ˆÜŽG|zùªüMpDnQWÄ”%JŠ™)â*p@Örš«ÕT2Ð%ˆG#ª„
·¤!°ŸOTÂT¸aÚ%4&h™LµšØüÐ.F¿²ÐÞ_Ç‚¾ÅÃaÜ÷09Æ 
q€öy˜v‡85õN÷]¬äѼóS{°_MŽ¬úÔ#°Ç¸0åÞè2ëôPcvÆw9®ií1Ä8F™˜à‰´+‰Ik1òÝ7“Ñ×ÒsÝ\x‚h`ÞÑ`ó"|µEcý£n˜h`}GÞ  !±ù²Ápü²
ß6
0ïi󜵩SÈÇ7˜-ÕURO˜¦´f$ªž-Í6(œ}<„ éc øs]ŽŽ„*—¾
ìdŽ„)méª\¿êÎIg¾ØÞ~I#C/¼¼´EÁÈŽi8“©õådô·>euä ƒ'Ê×लR1ÉJE1ÐAát`t;ÇР%Ý<‡¥„ÍÆ`×Oyó)õiI€ñQaŸ4Ûù\áàaÃÔ¹HÃu¹*k€¦<„eS‡&õÏ
B!ŽhüÞ`yj}mªf×\¿
Ç~æ­9‡û\՞Ǖg²1Žû5V7 !àöšm°
c`ܬøÇìµÒ'P"?…´Ö,"§^•õŽ¨sÔ)6˜sæéÍR¼
ò|Sl”‹7 nPW

Gòú÷½§O¯‡„l¡kSÞŒr½PÊ@æ¢pŽ-mÿ #Ÿ˜Àº¶Áä¦;ïÔæ$1££`“Õ>„—·ž)ß
ð
³ñ#Ï Ô$¶œ‰ÊE‹À;÷º
¯«P:Ñ”8–IÊtpÞ3ª“>ê“þës4ò2OÏÕ­±zô†Õ§‰.÷ä¸;¿˜“'œ›žª}«Œ{ª±Ì 9ÔóÞÕ‡0
$íWV3Üì¬
—@kÝ4@¿r¼±½¬™›?øØæ´'Áé®CË3-g$˜ö‡×auÚi´Žp/êÛæF›Ú2v‹ã¿¿,nB1̨ƃqÞa5͝@&Æû“él÷ \C²½UÍc ¯k×¢U
ÖéQå™—-r    wô ÞÏ<Ò=&=ÿ Ôê Òêˈt,i—;LîÜ á¸*ÚÃ1$êL•LÍ <É)ýÐà’
;F™{ƒ™˜€&'}‚ãÄK`¡ÞT@I;®žZóè‚s’7®°›+§O­Åq©é»²9<Ô
J ¼9O’HL»Ùïì¸rk¼Ž_ý‘TŸu[²ßÚŒ·ü÷B%¯E ŸÔX5êO´Ç•€’I0 É<mÆÈJoúáõóQâØòÙaÙt:  ·=|Ôaz]a1cÉŸ1
4˜à3΍}GFøí7/B«vεŕL=Ðr1ÕšnwH›óÞ¢ºö{)´4€æZç^ÿ yÑ:_([¿ipqQÊ^âÆe;»½“¦º|¢ÀTçèrú1”óTÃ’u¦CG¤ ¡
Š­7ë

·i“¿z»n
ö¼±ù^ö<‹çu7¸ÈlvZðÛAUXje•
dLÞH ÚosÇr`È™‡pÊeÇ4@x7´)ö†Ótœ~i‘Ï[x@äUÃæ­m·e´x
°t¸·²5'žªža   =ÖÌåzϘ'-´ˆpž3§ò‡k  Ý'‰ýÑï¦3ºdè“{eiÔõNkîšECEΠ¥ÃT9ãqÃâŸîɐ8z \?€    >JX` 
ñ¹õ%;µŸD‘«´€àwÒ™UûئžÖö\×®×´8 ½‡ºÐÆÓ§?Àkmœ=;d5*@-ì0F Rªýš[Ü6âö̃ڸr*KA9· u*µæ£?U¸Âêí†8@¦X4 e-ò„0s{ HâUpU?¼mñRa°®a%Ð'tÉ×’\¾ÊÉ]t›h>·(Ë@R¼¡Ãt h}’O÷au<+nT…Ö…MӐ??Óe95 q>í/;&JSû °¯ÊéÞøƒ*Ã2½Ài&:nôUl=¾¿5eˆ3”ñc|Ú2V”>„»&eE;«ÚäC
p¢Ûúy 9š[ŒÌx¼擼A&DåÒ¯ˆ¤ÀÌ;"˜ÏQä¸åhÊ}Ûq«Û0WžÒ|»€ø®öCm5•\ÇÀ§Pe3£]0ÃàLDÉ‰1øªxjgwT‚÷¿LΨK‹›ùs—xˆÜ±µ
kæ¸f‰‰ÜGk/LÛØ6d9ò¶ùA{ƒA3š/¬D¬khÓk‰`˜"㯒r¿±Óã jx‡°e}<Ñø\3y:'À•/h½Í€Ç4~g ?Û(¼]v‘ªlKÎâ~?O‚W%{Ì:“'©úNq¾›úo(X’¥¯ˆ
nFê{Ç€ü?º'ë
ø‹ì
Þ09ŒÌç9Æ —ËC`j@ÓÄ(+a‹un¸#ÂꟋ{K`‘ÑÍÍ'à´»/Û,KW;Þ4²þð ï Nm|~fGÏ(…³Ã)«1ö­Õ
¥‡¨©ƒÃ™ü-s=à=U66Ï«Ýc蓦W¹íž®›nÔ%êÇìŒ<#Ü×84ån®Ð ÒåOC`  ñânÑs‡¢ç 1õ%Îhì½Ã½® e:ݼUZo
™`  
ÅZŸŒÊ«ê1ÏÄo$q¹Þ€©ˆhÐÉä¯ñ[!…Ú˜àJ:x2$Íß&PåT£6ç—
‡Í*4Ýšçjÿ ‰É nófÐ ó(L5C•åÆ\rMÒ@ò
}y-W}™üýVù—ú¢=Ù”c®‘< M ž
´Phr¦©TD ‘ù.$´÷O‡‘V2Æò.=IUŒ=ž‡â¬i™aþÓåÙ?òUø'ØÖ•.~* šTŒ!•-×áºTâ®ä#õü'´ eýlYÅÓeÕKÂrT"CÚ@u!Óxƒ{š3€}1¿(r}%«nËamjÑ%ÑNEò
v  ˜à  σöK³,*º.àzù¨™Ó ÚçâU¦*¿ 9{%Ö¹ njûdaXöb)  kÛƱûÓ\°M7ˆÂ=û›ç¿Ã‚­V»Cg–8ÙêE-j)k$º`Ã-ùEýeBÆÇ]c¡°ñty&Òd0nõ'¡W+ƒ*|–ø<K\%¦cÏÄ#N,Úú‘¤òÒÛõ ’¶§‰Ì
ú•dçb31Ð\7‡ëÚw82
‰KÝkÚñ'¢Ž…[&¼ÌAÔzHàP *اoßreÁY¥Ñ¾ãE|&QbKg]ã„þê*b;R³Öi‹  ±î$£ÈTm¶k÷ˆ
¦»Ù¹i¦¨6 h%Á'Ú
J   \F±â+‹b/Å9ñå5Àë­nˆRv«¯#„&î„$º’@:‘Ôrø+,àq‘秬*Úg´>µFa\GQªEAÔp5\Ǽ·¼Ç8·õ -â§Ú[
‡
uZeÖ 3}×d'+¹:ð+K†Û®s!Ï$úe€<Û”x)1»a­¡LC]¸µík…ÚàA»AYº{†ªS[¦5HÒ7ù
--,ísòDØ€èk   ÞÀîÜò@â(  ËNˆë›4ô½•/¦o‡€Û7
ê•ÆêòðÜy
'Án½µ á˜ݦ  ndeo…[ì¶Ê,¥R³Ä=À±—–ß;£™´ñSâ*g§”ïaið‘Jå~™ÓÞß³Õ¢»8x埒²52>AÊb&-÷\7´éÄù€T˜,w;3{ï˜k…à¹ÄqÀ«œ{
€\
ˆ¾[´¨јr &Úé„Ívˆ±8†¿]|¬ņ4I×pÞS1ÈÖz‰#Ìv‡G!YNògñ:màTz¢Ý1ô©^O=~ë|5Bã™ç•¼µõ•bÆ@úÕS¬ÈŒ#¬zünrŸ
û”
Z²•ï›èðV"ÁHÚý©wÝ €7¼Ìu1hÑa3Éä û f$o¿É ™Ú›ÝçnpÒ3äÌ3†Í§,Äï]$‰/pê†«À¼
¸e9­Æê_C]žƒ·ý·frÁN«,E=›Çq
-‰öŒ:aÏ¿±í&£Í:-}
84‘ÿ eƒQÑeëSsuiA
³g㟥ú£?ÿ ʼn*”“÷aühe:ÊWa@ÒÞk±eØ]
F Ô—r.åä˜@ö¥ªZoÐýYL·¥S²G/‡ñ
<~*ZÆ´è>JlòàÛƽÿ 窘ìGN¢:I®KšJp/`íIÁÀõ#Ä-€ö­šµŒoF4|ÆQØÆ@Ì|£Ô…¢À{9˜è½Üó›€ôYÒÎYsið;ís¤€à²ˆ‚4qÉVŒI$  ‰"°   æµ8cXGjœˏ¡Aâý•ËÜ¢ûïe·çLx']á"oÅÎê3¯Ç—¹”ó0nå‚âg{Œñ>
S´˜îè°g238‚ãköÝfÚd´6Ò€;ò÷±¢™¼›º¢Æ'¥Ðx'e¬ç
]bÈÆV¢ó‹kýBO
ðÊâ$Ÿ!×T
3Mýמ
žìٍàÌü‘8÷€àæØ8æ©6‰©L´«…oãpð„~Çk‰!ñ;‹”ÛžÍ àž±z Ÿôû øŸÝužÏ;ÿ #|u6™Þ¬ÚˆÐõA4¶â|ôl|Ê2ŽÇ¤ÝÅÇY.<#Aí.k§hóF‚”Y;
M½Ö4hŸ4&›­¿tès´%FìL¥£Ãk‰ÇT¤haÁ¤ÚxfÉ`ÑìË›>i 3t‚:,–+^÷´–{Û–Nxi"x‘Ûg
î¨>¥Õ܁ùZH,2Û“:8xÊ¢Çí9.É-Ìâã-=çjwµS˜dütžçwýGòú®®ûº_ˆýx$–¡ãøO
EÚÛÏ÷R„×w+3£Á£öUMyR²¹âŒ°š›¸Ñãò9§Ó_Dl+Ùßc›úšGÅÌc†Ž!Ko=¶.‘Îÿ c²(2®V mª.ÿ ¹B›¹å ù„öŸSV>™ü¯$y:G¢Z×àøúdî¹û­·ýÇ´:•c LÍõi_‹ö+ÎæGÊè>OŠ•äž´§Þ{X}¨1ÚTc›»Qþ•êô°t¿OP?eæ~É{5]•ÙR£r5†nZ\ã@
&îJõ ¾àC°þV>fé¥/ü5ñÊIº_é5;e­h<@
Ä&æÃëE%;X,ÒãÆÞ`Oò¦kŸm#˜!ÀyÄ¢|
óLšò¥Ä`
¶R=|ÈCâh5ò3DˆïF†ðÒ#ÅìÛœ?¸yhBãœí
ZxßÎÄhºRK„`Þödvײ™ÀÈÑÒgŒuYw³%†ƒÓzõ ÖÏp‚dH®¦A´ù§»ÓÇMæ~)ˆð‡û:ù&Ä •vGD´À 
n
Ý
‡¼Ö8Fö   óáà£~Ë¥x`oK|Ä?fxiØü%pìR>éò+Û±éÎ>núlFŤ'tq8LZÏvÃ?„¡ß±È⽆¯³íü@x|PöUäèØã¡ð‚ŒAìÏ"vÍwóŸÍ{ ý0.z    È•Ö{,N¡£¡ŸKÕÙž>Ýœþ ÍÀ<n^Àïe€Þgn¹”ÜiQê
9Ú±‡„è]ËA¿‚MJìiÓxŒn-ÄS6!ÓæÑû*jÎWøìšÇ뙦gj~+?\,Ö7ÁÕÊ”G™q6TF)óCç<ÓšÂUé†gW=àà“hóR³ÊQì<!÷ÇpR1¯u„De,è­vvâDß~žHå‰ú£Kìní¢ÐçÅÄÅ¥Ü÷
±ÃìÇžûƒz\þÞ¨-•‹
€

¸+¶n¿^
?†[ÖWóÒX¸,0Ø:l?‰Âý«ÿ ·Dc+fÔ[ƒ›ûªšx«kæž1KI…<#'nž¶\±íÎ8¤ìK@’@J¥v)º[Ž±Æå´¼
½Ý¡ç¯ÐO¥ø®è•Óȱ߿wša©“œt‡4îyªœ&+1ÏØ:‚æƒ=/ꎔ`hm
“=©‹]°g}ì€DgUƒ8¤qCˆãáÅ,‡Ô¬ñ10HÑDÂbáÂÿ Œ»v·:uòB6»áp‰>°<×EA!Å‚D™IúOÍ
¡>ôG}Â`ÍßkÜL™Ž  Þð™
{IøF²
¹òQ3&!ÃÂÞz.d&Ï-sH¸,Ôõ˜ŽP€
77ˆÝ¼ÊëÜw   =cÕ
Ú,ØÐ5ÎYÐ)ì´öœgŒ[¤
ßv㙑8心>h]§µháYš£²ºÑ.{Ï7Sð•?´~×SÃKýJÛ˜
™Íäiúu<µX¶1õ^kâçIÑ£sZ4h>j*ÔšD:4­¿_ ÷¸
Õxæÿ ¸?Mù _•­ÊÐ ä ÷ý ÑwL
œ­ïnTkÛUÍN©ë:¦fV ¶ÜÔÜMªÅâA½–¿R×TXš-%iTÊT•‡Ù‚JôϐZxWÑè‰f‰òG º
×Õû2aZ7OU3[“×AT–ÞŒ…-‘¤”Ì 
ì&(ˆ¿­•ƒkï’:ðY¦W‘Å)“†‘˜³Åtcø˜ñTÂwÚÇ4
|üLÇªí–v-
qˆèU
qPE.†â‘˜µ Æ,ÐÅs]8¾„oúÑ i>ÜxxÈó)ƒ ´æÁâØ$À‰vžŸf$Ž|ãw;ÀÁIJ»b`
{¦Ó¤Ú$©YÀ‘n@Óïž«9J¼êG   m¤
ܯ¹ÌW4€ÐÒÅÛ‡#褕Ÿn-?í|с¥÷Ú¹¬'´ÞÜ9ÓK`hê£SÄSà?7—Wí_´…óB›»:=Ãïq`<8ñÓŒÑlú2d¬ê³£hÖ[l|$vÝro~'R®‰§°ñmY ͧäP
|PUª¹·:3Œ[Û{Xÿ ºâ@‚W–Äé u‚ ¯´*=íή.pûÒdt @
G‰¬ s¸    ëÉücr ÞæѨÊ@>¤¢Ö±.Þ'¯°ÌME[YéïĵÂCå½
Ué©Áû'Ê9%eÔðNU”ë‘ÌsD3/®+UI˜9h.WC”
빓$#:pz:YÓ ¿xž* ³$Í +$kñAŠ‹†¢
Uê>¸)_š¬÷©ßAÂÔb9ÇU ¯¾
á•9¯ÏÏ÷O÷¼¼Fähal1‰3Ì[Ïr•´UCksNÐ]R‘¸¥H+§Šé†c©vÖÞ0iÓ76s†î!§=ß
¼~Ô'°Ãmäoäš³ªøi1úÉ)³yV8CLÄØÁ‘WYïi€H6ÖÑiámø^ÈY´°Ñ7¥Û*—Ñ©L«Qƒï—Ùrÿ ›£Ð*š¸ˆL©ˆ$ˆ ÷¾D§9È®«qbqC)–ˆïv´çñsÑVT­Ø, <àïºÀO«Jý·õ 
àfPìð  .wFšir´þ’2_Y
*Æ€x\«
ì€9š@ Ž|F⇥ˆkZ@hÖÄ0t¿-<“‹qµ¾*ZL¤Ú)&BJpÓF5=$„at*Zš$’ÑtdûÝRI1    2Ž‰$€$I$#‰SÞ’H
ë¬ï;Á$¡t$’`<(ñÇt)$‡Ð.Êf¢X’Kt=Éé$‚ˆªè¢oÝëòI%Rgcª÷ŠyI%¡‰ÿ !ñ)´õ $¤ Ô’IIGÿÙa

�3�g%��@sN
dd
lm
Z

ddlmZmZmZmZmZmZmZm	Z	m
Z
mZmZm
ZmZmZmZ
ddlmZmZ
ddlZddlmZmZ
ddlmZmZ
ddlZddlZdd�ZGd	d
�d
e�ZGdd�de�Zd
d�Z Gdd�de�Z!Gdd�de�Z"Gdd�de�Z#Gdd�de�Z$Gdd�de�Z%Gdd�de�Z&Gdd�de�Z'e(dk�
rJe�)�
dS)�)
�support)�tokenize�	_tokenize�
untokenize�NUMBER�NAME�OP�STRING�	ENDMARKER�ENCODING�tok_name�detect_encoding�open�Untokenizer�generate_tokens�NEWLINE)�BytesIO�StringION)�TestCase�mock)�VALID_UNDERSCORE_LITERALS�INVALID_UNDERSCORE_LITERALSc
Cs�g}t|
�
��
}|
d
dv
}|D]b\}}}}}	|tkr:
q�|rT|tkrT|d|krTq t|}|�d|d�d|d�d|�d|���

q |S)N�����z
r
�    Z10�
 Z13)�len�
splitlinesr
rr�append)
Ztoken_generatorZ
source_string�resultZ	num_linesZmissing_trailing_nl�type�token�start�end�line�r$�1/opt/python39/lib/python3.9/test/test_tokenize.py�stringify_tokens_from_sources







(r&c@s�eZ
dZd
d�Zdd�Zdd�Zdd�Zd	d
�Zdd�Zd
d�Z	dd�Z
dd�Zdd�Zdd�Z
dd�Zdd�Zdd�Zdd�Zdd �Zd!d"�Zd#d$�Zd%d&�Zd'd(�Zd)S)*�TokenizeTestcCs<t|
�
d
�
�
}tt|j�
|
�}|�|dg
|�����
dS)N�utf-8z*    ENCODING   'utf-8'       (0, 0) (0, 0))r�encoder&r�readline�assertEqual�rstripr��self�
s�expected�
frr$r$r%�check_tokenize*s



��zTokenizeTest.check_tokenizec
CsDtd
�
d�
�
}
tt|
j�
�
}|�|djt�
|�|djt�
dS)N�
xr(�����r)	rr)�listrr*r+rrr
)r.r1�tokensr$r$r%�test_implicit_newline4s


z"TokenizeTest.test_implicit_newlinec
Csd|�d
d�
|�dd�
d}
t
|
�
j}|�td��
t|�
D]}q<Wd�
n1sV0


Y
dS)Nz1 + 1z�    NUMBER     '1'           (1, 0) (1, 1)
    OP         '+'           (1, 2) (1, 3)
    NUMBER     '1'           (1, 4) (1, 5)
    z3if False:
    # NL
    
    True = False # NEWLINE
af    NAME       'if'          (1, 0) (1, 2)
    NAME       'False'       (1, 3) (1, 8)
    OP         ':'           (1, 8) (1, 9)
    NEWLINE    '\n'          (1, 9) (1, 10)
    COMMENT    '# NL'        (2, 4) (2, 8)
    NL         '\n'          (2, 8) (2, 9)
    NL         '\n'          (3, 4) (3, 5)
    INDENT     '    '        (4, 0) (4, 4)
    NAME       'True'        (4, 4) (4, 8)
    OP         '='           (4, 9) (4, 10)
    NAME       'False'       (4, 11) (4, 16)
    COMMENT    '# NEWLINE'   (4, 17) (4, 26)
    NEWLINE    '\n'          (4, 26) (4, 27)
    DEDENT     ''            (5, 0) (5, 0)
    sdef k(x):
    x += 2
  x += 5
z3unindent does not match any outer indentation level)r2rr*�assertRaisesRegex�IndentationErrorr)r.Zindent_error_filer*�tokr$r$r%�
test_basic<s
�


�
zTokenizeTest.test_basicc

Csp|�d
d�
|�dd�
|�dd�
|�dd�
|�d	d
�
|�dd�
|�d
d�
|�dd�
|�dd�
dS)Nz0xff <= 255z�    NUMBER     '0xff'        (1, 0) (1, 4)
    OP         '<='          (1, 5) (1, 7)
    NUMBER     '255'         (1, 8) (1, 11)
    z0b10 <= 255z�    NUMBER     '0b10'        (1, 0) (1, 4)
    OP         '<='          (1, 5) (1, 7)
    NUMBER     '255'         (1, 8) (1, 11)
    z0o123 <= 0O123z�    NUMBER     '0o123'       (1, 0) (1, 5)
    OP         '<='          (1, 6) (1, 8)
    NUMBER     '0O123'       (1, 9) (1, 14)
    z1234567 > ~0x15z�    NUMBER     '1234567'     (1, 0) (1, 7)
    OP         '>'           (1, 8) (1, 9)
    OP         '~'           (1, 10) (1, 11)
    NUMBER     '0x15'        (1, 11) (1, 15)
    z2134568 != 1231515z�    NUMBER     '2134568'     (1, 0) (1, 7)
    OP         '!='          (1, 8) (1, 10)
    NUMBER     '1231515'     (1, 11) (1, 18)
    z(-124561-1) & 200000000ac
    OP         '('           (1, 0) (1, 1)
    OP         '-'           (1, 1) (1, 2)
    NUMBER     '124561'      (1, 2) (1, 8)
    OP         '-'           (1, 8) (1, 9)
    NUMBER     '1'           (1, 9) (1, 10)
    OP         ')'           (1, 10) (1, 11)
    OP         '&'           (1, 12) (1, 13)
    NUMBER     '200000000'   (1, 14) (1, 23)
    z0xdeadbeef != -1z�    NUMBER     '0xdeadbeef'  (1, 0) (1, 10)
    OP         '!='          (1, 11) (1, 13)
    OP         '-'           (1, 14) (1, 15)
    NUMBER     '1'           (1, 15) (1, 16)
    z0xdeadc0de & 12345z�    NUMBER     '0xdeadc0de'  (1, 0) (1, 10)
    OP         '&'           (1, 11) (1, 12)
    NUMBER     '12345'       (1, 13) (1, 18)
    z0xFF & 0x15 | 1234z�    NUMBER     '0xFF'        (1, 0) (1, 4)
    OP         '&'           (1, 5) (1, 6)
    NUMBER     '0x15'        (1, 7) (1, 11)
    OP         '|'           (1, 12) (1, 13)
    NUMBER     '1234'        (1, 14) (1, 18)
    �
r2�
r.r$r$r%�test_intas
zTokenizeTest.test_intc

Cs4|�d
d�
|�dd�
|�dd�
|�dd�
dS)	Nzx = 0z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '0'           (1, 4) (1, 5)
    zx = 0xfffffffffffz�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '0xfffffffffff' (1, 4) (1, 17)
    zx = 123141242151251616110z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '123141242151251616110' (1, 4) (1, 25)
    zx = -15921590215012591z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    OP         '-'           (1, 4) (1, 5)
    NUMBER     '15921590215012591' (1, 5) (1, 22)
    r<r=r$r$r%�	test_long�szTokenizeTest.test_longc

CsX|�d
d�
|�dd�
|�dd�
|�dd�
|�d	d
�
|�dd�
|�d
d�
dS)Nzx = 3.14159z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '3.14159'     (1, 4) (1, 11)
    zx = 314159.z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '314159.'     (1, 4) (1, 11)
    zx = .314159z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '.314159'     (1, 4) (1, 11)
    zx = 3e14159z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '3e14159'     (1, 4) (1, 11)
    z	x = 3E123z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '3E123'       (1, 4) (1, 9)
    z
x+y = 3e-1230z�    NAME       'x'           (1, 0) (1, 1)
    OP         '+'           (1, 1) (1, 2)
    NAME       'y'           (1, 2) (1, 3)
    OP         '='           (1, 4) (1, 5)
    NUMBER     '3e-1230'     (1, 6) (1, 13)
    zx = 3.14e159z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '3.14e159'    (1, 4) (1, 12)
    r<r=r$r$r%�
test_float�szTokenizeTest.test_floatc
CsJd
d�}
tD]}d|vrq|�
|
|�
|�
qtD]}|�|
|�
|�
q0dS)Nc
Ss<t|�
d
�
�
}
t|
j�
D]\}}}}}|tkr|
SqdS)Nr(z
invalid token)rr)rr*r)r/r1Ztoktyper r!r"r#r$r$r%�number_token�s






z;TokenizeTest.test_underscore_literals.<locals>.number_token�
()rr+rZassertNotEqual)r.rAZlitr$r$r%�test_underscore_literals�s




z%TokenizeTest.test_underscore_literalsc

Cs<
|�d
d�
|�dd�
|�dd�
|�dd�
|�d	d
�
|�dd�
|�d
d�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd �
|�d!d"�
|�d#d$�
|�d%d&�
|�d'd(�
|�d)d*�
|�d+d,�
|�d-d.�
|�d/d0�
|�d1d2�
|�d3d4�
dS)5Nzx = ''; y = ""a5
    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     "''"          (1, 4) (1, 6)
    OP         ';'           (1, 6) (1, 7)
    NAME       'y'           (1, 8) (1, 9)
    OP         '='           (1, 10) (1, 11)
    STRING     '""'          (1, 12) (1, 14)
    zx = '"'; y = "'"a6
    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     '\'"\''       (1, 4) (1, 7)
    OP         ';'           (1, 7) (1, 8)
    NAME       'y'           (1, 9) (1, 10)
    OP         '='           (1, 11) (1, 12)
    STRING     '"\'"'        (1, 13) (1, 16)
    zx = "doesn't "shrink", does it"z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     '"doesn\'t "' (1, 4) (1, 14)
    NAME       'shrink'      (1, 14) (1, 20)
    STRING     '", does it"' (1, 20) (1, 31)
    zx = 'abc' + 'ABC'z�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     "'abc'"       (1, 4) (1, 9)
    OP         '+'           (1, 10) (1, 11)
    STRING     "'ABC'"       (1, 12) (1, 17)
    zy = "ABC" + "ABC"z�    NAME       'y'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     '"ABC"'       (1, 4) (1, 9)
    OP         '+'           (1, 10) (1, 11)
    STRING     '"ABC"'       (1, 12) (1, 17)
    z%x = r'abc' + r'ABC' + R'ABC' + R'ABC'a�
    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     "r'abc'"      (1, 4) (1, 10)
    OP         '+'           (1, 11) (1, 12)
    STRING     "r'ABC'"      (1, 13) (1, 19)
    OP         '+'           (1, 20) (1, 21)
    STRING     "R'ABC'"      (1, 22) (1, 28)
    OP         '+'           (1, 29) (1, 30)
    STRING     "R'ABC'"      (1, 31) (1, 37)
    z%y = r"abc" + r"ABC" + R"ABC" + R"ABC"a�
    NAME       'y'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    STRING     'r"abc"'      (1, 4) (1, 10)
    OP         '+'           (1, 11) (1, 12)
    STRING     'r"ABC"'      (1, 13) (1, 19)
    OP         '+'           (1, 20) (1, 21)
    STRING     'R"ABC"'      (1, 22) (1, 28)
    OP         '+'           (1, 29) (1, 30)
    STRING     'R"ABC"'      (1, 31) (1, 37)
    zu'abc' + U'abc'z�    STRING     "u'abc'"      (1, 0) (1, 6)
    OP         '+'           (1, 7) (1, 8)
    STRING     "U'abc'"      (1, 9) (1, 15)
    zu"abc" + U"abc"z�    STRING     'u"abc"'      (1, 0) (1, 6)
    OP         '+'           (1, 7) (1, 8)
    STRING     'U"abc"'      (1, 9) (1, 15)
    zb'abc' + B'abc'z�    STRING     "b'abc'"      (1, 0) (1, 6)
    OP         '+'           (1, 7) (1, 8)
    STRING     "B'abc'"      (1, 9) (1, 15)
    zb"abc" + B"abc"z�    STRING     'b"abc"'      (1, 0) (1, 6)
    OP         '+'           (1, 7) (1, 8)
    STRING     'B"abc"'      (1, 9) (1, 15)
    z%br'abc' + bR'abc' + Br'abc' + BR'abc'a;
    STRING     "br'abc'"     (1, 0) (1, 7)
    OP         '+'           (1, 8) (1, 9)
    STRING     "bR'abc'"     (1, 10) (1, 17)
    OP         '+'           (1, 18) (1, 19)
    STRING     "Br'abc'"     (1, 20) (1, 27)
    OP         '+'           (1, 28) (1, 29)
    STRING     "BR'abc'"     (1, 30) (1, 37)
    z%br"abc" + bR"abc" + Br"abc" + BR"abc"a;
    STRING     'br"abc"'     (1, 0) (1, 7)
    OP         '+'           (1, 8) (1, 9)
    STRING     'bR"abc"'     (1, 10) (1, 17)
    OP         '+'           (1, 18) (1, 19)
    STRING     'Br"abc"'     (1, 20) (1, 27)
    OP         '+'           (1, 28) (1, 29)
    STRING     'BR"abc"'     (1, 30) (1, 37)
    z%rb'abc' + rB'abc' + Rb'abc' + RB'abc'a;
    STRING     "rb'abc'"     (1, 0) (1, 7)
    OP         '+'           (1, 8) (1, 9)
    STRING     "rB'abc'"     (1, 10) (1, 17)
    OP         '+'           (1, 18) (1, 19)
    STRING     "Rb'abc'"     (1, 20) (1, 27)
    OP         '+'           (1, 28) (1, 29)
    STRING     "RB'abc'"     (1, 30) (1, 37)
    z%rb"abc" + rB"abc" + Rb"abc" + RB"abc"a;
    STRING     'rb"abc"'     (1, 0) (1, 7)
    OP         '+'           (1, 8) (1, 9)
    STRING     'rB"abc"'     (1, 10) (1, 17)
    OP         '+'           (1, 18) (1, 19)
    STRING     'Rb"abc"'     (1, 20) (1, 27)
    OP         '+'           (1, 28) (1, 29)
    STRING     'RB"abc"'     (1, 30) (1, 37)
    z"a\
de\
fg"z3    STRING     '"a\\\nde\\\nfg"' (1, 0) (3, 3)
    zu"a\
de"z/    STRING     'u"a\\\nde"'  (1, 0) (2, 3)
    zrb"a\
d"z/    STRING     'rb"a\\\nd"'  (1, 0) (2, 2)
    z
"""a\
b"""z0    STRING     '"""a\\\nb"""' (1, 0) (2, 4)
    zu"""a\
b"""z1    STRING     'u"""a\\\nb"""' (1, 0) (2, 4)
    zrb"""a\
b\
c"""z7    STRING     'rb"""a\\\nb\\\nc"""' (1, 0) (3, 4)
    zf"abc"z/    STRING     'f"abc"'      (1, 0) (1, 6)
    z	fR"a{b}c"z/    STRING     'fR"a{b}c"'   (1, 0) (1, 9)
    z
f"""abc"""z0    STRING     'f"""abc"""'  (1, 0) (1, 10)
    zf"abc\
def"z1    STRING     'f"abc\\\ndef"' (1, 0) (2, 4)
    zRf"abc\
def"z2    STRING     'Rf"abc\\\ndef"' (1, 0) (2, 4)
    r<r=r$r$r%�test_string�sT					
�
�
�
�
��
�
�zTokenizeTest.test_stringc

Cs(|�d
d�
|�dd�
|�dd�
dS)Nz!def d22(a, b, c=2, d=2, *k): passa    NAME       'def'         (1, 0) (1, 3)
    NAME       'd22'         (1, 4) (1, 7)
    OP         '('           (1, 7) (1, 8)
    NAME       'a'           (1, 8) (1, 9)
    OP         ','           (1, 9) (1, 10)
    NAME       'b'           (1, 11) (1, 12)
    OP         ','           (1, 12) (1, 13)
    NAME       'c'           (1, 14) (1, 15)
    OP         '='           (1, 15) (1, 16)
    NUMBER     '2'           (1, 16) (1, 17)
    OP         ','           (1, 17) (1, 18)
    NAME       'd'           (1, 19) (1, 20)
    OP         '='           (1, 20) (1, 21)
    NUMBER     '2'           (1, 21) (1, 22)
    OP         ','           (1, 22) (1, 23)
    OP         '*'           (1, 24) (1, 25)
    NAME       'k'           (1, 25) (1, 26)
    OP         ')'           (1, 26) (1, 27)
    OP         ':'           (1, 27) (1, 28)
    NAME       'pass'        (1, 29) (1, 33)
    zdef d01v_(a=1, *k, **w): passa�    NAME       'def'         (1, 0) (1, 3)
    NAME       'd01v_'       (1, 4) (1, 9)
    OP         '('           (1, 9) (1, 10)
    NAME       'a'           (1, 10) (1, 11)
    OP         '='           (1, 11) (1, 12)
    NUMBER     '1'           (1, 12) (1, 13)
    OP         ','           (1, 13) (1, 14)
    OP         '*'           (1, 15) (1, 16)
    NAME       'k'           (1, 16) (1, 17)
    OP         ','           (1, 17) (1, 18)
    OP         '**'          (1, 19) (1, 21)
    NAME       'w'           (1, 21) (1, 22)
    OP         ')'           (1, 22) (1, 23)
    OP         ':'           (1, 23) (1, 24)
    NAME       'pass'        (1, 25) (1, 29)
    z&def d23(a: str, b: int=3) -> int: passa�    NAME       'def'         (1, 0) (1, 3)
    NAME       'd23'         (1, 4) (1, 7)
    OP         '('           (1, 7) (1, 8)
    NAME       'a'           (1, 8) (1, 9)
    OP         ':'           (1, 9) (1, 10)
    NAME       'str'         (1, 11) (1, 14)
    OP         ','           (1, 14) (1, 15)
    NAME       'b'           (1, 16) (1, 17)
    OP         ':'           (1, 17) (1, 18)
    NAME       'int'         (1, 19) (1, 22)
    OP         '='           (1, 22) (1, 23)
    NUMBER     '3'           (1, 23) (1, 24)
    OP         ')'           (1, 24) (1, 25)
    OP         '->'          (1, 26) (1, 28)
    NAME       'int'         (1, 29) (1, 32)
    OP         ':'           (1, 32) (1, 33)
    NAME       'pass'        (1, 34) (1, 38)
    r<r=r$r$r%�
test_function�
s
zTokenizeTest.test_functionc

Cs|�d
d�
dS)NzXif 1 < 1 > 1 == 1 >= 5 <= 0x15 <= 0x12 != 1 and 5 in 1 not in 1 is 1 or 5 is not 1: passa�    NAME       'if'          (1, 0) (1, 2)
    NUMBER     '1'           (1, 3) (1, 4)
    OP         '<'           (1, 5) (1, 6)
    NUMBER     '1'           (1, 7) (1, 8)
    OP         '>'           (1, 9) (1, 10)
    NUMBER     '1'           (1, 11) (1, 12)
    OP         '=='          (1, 13) (1, 15)
    NUMBER     '1'           (1, 16) (1, 17)
    OP         '>='          (1, 18) (1, 20)
    NUMBER     '5'           (1, 21) (1, 22)
    OP         '<='          (1, 23) (1, 25)
    NUMBER     '0x15'        (1, 26) (1, 30)
    OP         '<='          (1, 31) (1, 33)
    NUMBER     '0x12'        (1, 34) (1, 38)
    OP         '!='          (1, 39) (1, 41)
    NUMBER     '1'           (1, 42) (1, 43)
    NAME       'and'         (1, 44) (1, 47)
    NUMBER     '5'           (1, 48) (1, 49)
    NAME       'in'          (1, 50) (1, 52)
    NUMBER     '1'           (1, 53) (1, 54)
    NAME       'not'         (1, 55) (1, 58)
    NAME       'in'          (1, 59) (1, 61)
    NUMBER     '1'           (1, 62) (1, 63)
    NAME       'is'          (1, 64) (1, 66)
    NUMBER     '1'           (1, 67) (1, 68)
    NAME       'or'          (1, 69) (1, 71)
    NUMBER     '5'           (1, 72) (1, 73)
    NAME       'is'          (1, 74) (1, 76)
    NAME       'not'         (1, 77) (1, 80)
    NUMBER     '1'           (1, 81) (1, 82)
    OP         ':'           (1, 82) (1, 83)
    NAME       'pass'        (1, 84) (1, 88)
    r<r=r$r$r%�test_comparison�
s
�zTokenizeTest.test_comparisonc

Cs|�d
d�
dS)Nzx = 1 << 1 >> 5a6
    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '1'           (1, 4) (1, 5)
    OP         '<<'          (1, 6) (1, 8)
    NUMBER     '1'           (1, 9) (1, 10)
    OP         '>>'          (1, 11) (1, 13)
    NUMBER     '5'           (1, 14) (1, 15)
    r<r=r$r$r%�
test_shift�
szTokenizeTest.test_shiftc

Cs|�d
d�
dS)Nz%x = 1 - y + 15 - 1 + 0x124 + z + a[5]a$    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '1'           (1, 4) (1, 5)
    OP         '-'           (1, 6) (1, 7)
    NAME       'y'           (1, 8) (1, 9)
    OP         '+'           (1, 10) (1, 11)
    NUMBER     '15'          (1, 12) (1, 14)
    OP         '-'           (1, 15) (1, 16)
    NUMBER     '1'           (1, 17) (1, 18)
    OP         '+'           (1, 19) (1, 20)
    NUMBER     '0x124'       (1, 21) (1, 26)
    OP         '+'           (1, 27) (1, 28)
    NAME       'z'           (1, 29) (1, 30)
    OP         '+'           (1, 31) (1, 32)
    NAME       'a'           (1, 33) (1, 34)
    OP         '['           (1, 34) (1, 35)
    NUMBER     '5'           (1, 35) (1, 36)
    OP         ']'           (1, 36) (1, 37)
    r<r=r$r$r%�
test_additive�
szTokenizeTest.test_additivec

Cs|�d
d�
dS)Nzx = 1//1*1/5*12%0x12@42a�    NAME       'x'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    NUMBER     '1'           (1, 4) (1, 5)
    OP         '//'          (1, 5) (1, 7)
    NUMBER     '1'           (1, 7) (1, 8)
    OP         '*'           (1, 8) (1, 9)
    NUMBER     '1'           (1, 9) (1, 10)
    OP         '/'           (1, 10) (1, 11)
    NUMBER     '5'           (1, 11) (1, 12)
    OP         '*'           (1, 12) (1, 13)
    NUMBER     '12'          (1, 13) (1, 15)
    OP         '%'           (1, 15) (1, 16)
    NUMBER     '0x12'        (1, 16) (1, 20)
    OP         '@'           (1, 20) (1, 21)
    NUMBER     '42'          (1, 21) (1, 23)
    r<r=r$r$r%�test_multiplicativesz TokenizeTest.test_multiplicativec

Cs|�d
d�
|�dd�
dS)Nz~1 ^ 1 & 1 |1 ^ -1a�
    OP         '~'           (1, 0) (1, 1)
    NUMBER     '1'           (1, 1) (1, 2)
    OP         '^'           (1, 3) (1, 4)
    NUMBER     '1'           (1, 5) (1, 6)
    OP         '&'           (1, 7) (1, 8)
    NUMBER     '1'           (1, 9) (1, 10)
    OP         '|'           (1, 11) (1, 12)
    NUMBER     '1'           (1, 12) (1, 13)
    OP         '^'           (1, 14) (1, 15)
    OP         '-'           (1, 16) (1, 17)
    NUMBER     '1'           (1, 17) (1, 18)
    z-1*1/1+1*1//1 - ---1**1aH    OP         '-'           (1, 0) (1, 1)
    NUMBER     '1'           (1, 1) (1, 2)
    OP         '*'           (1, 2) (1, 3)
    NUMBER     '1'           (1, 3) (1, 4)
    OP         '/'           (1, 4) (1, 5)
    NUMBER     '1'           (1, 5) (1, 6)
    OP         '+'           (1, 6) (1, 7)
    NUMBER     '1'           (1, 7) (1, 8)
    OP         '*'           (1, 8) (1, 9)
    NUMBER     '1'           (1, 9) (1, 10)
    OP         '//'          (1, 10) (1, 12)
    NUMBER     '1'           (1, 12) (1, 13)
    OP         '-'           (1, 14) (1, 15)
    OP         '-'           (1, 16) (1, 17)
    OP         '-'           (1, 17) (1, 18)
    OP         '-'           (1, 18) (1, 19)
    NUMBER     '1'           (1, 19) (1, 20)
    OP         '**'          (1, 20) (1, 22)
    NUMBER     '1'           (1, 22) (1, 23)
    r<r=r$r$r%�
test_unary(s
zTokenizeTest.test_unaryc

Cs|�d
d�
dS)Nz/import sys, time
x = sys.modules['time'].time()a�    NAME       'import'      (1, 0) (1, 6)
    NAME       'sys'         (1, 7) (1, 10)
    OP         ','           (1, 10) (1, 11)
    NAME       'time'        (1, 12) (1, 16)
    NEWLINE    '\n'          (1, 16) (1, 17)
    NAME       'x'           (2, 0) (2, 1)
    OP         '='           (2, 2) (2, 3)
    NAME       'sys'         (2, 4) (2, 7)
    OP         '.'           (2, 7) (2, 8)
    NAME       'modules'     (2, 8) (2, 15)
    OP         '['           (2, 15) (2, 16)
    STRING     "'time'"      (2, 16) (2, 22)
    OP         ']'           (2, 22) (2, 23)
    OP         '.'           (2, 23) (2, 24)
    NAME       'time'        (2, 24) (2, 28)
    OP         '('           (2, 28) (2, 29)
    OP         ')'           (2, 29) (2, 30)
    r<r=r$r$r%�
test_selectorMszTokenizeTest.test_selectorc

Cs|�d
d�
dS)Nz @staticmethod
def foo(x,y): passa    OP         '@'           (1, 0) (1, 1)
    NAME       'staticmethod' (1, 1) (1, 13)
    NEWLINE    '\n'          (1, 13) (1, 14)
    NAME       'def'         (2, 0) (2, 3)
    NAME       'foo'         (2, 4) (2, 7)
    OP         '('           (2, 7) (2, 8)
    NAME       'x'           (2, 8) (2, 9)
    OP         ','           (2, 9) (2, 10)
    NAME       'y'           (2, 10) (2, 11)
    OP         ')'           (2, 11) (2, 12)
    OP         ':'           (2, 12) (2, 13)
    NAME       'pass'        (2, 14) (2, 18)
    r<r=r$r$r%�test_methodcszTokenizeTest.test_methodc

Cs|�d
d�
dS)Nzdef f():
	if x
        	passa_    NAME       'def'         (1, 0) (1, 3)
    NAME       'f'           (1, 4) (1, 5)
    OP         '('           (1, 5) (1, 6)
    OP         ')'           (1, 6) (1, 7)
    OP         ':'           (1, 7) (1, 8)
    NEWLINE    '\n'          (1, 8) (1, 9)
    INDENT     '\t'          (2, 0) (2, 1)
    NAME       'if'          (2, 1) (2, 3)
    NAME       'x'           (2, 4) (2, 5)
    NEWLINE    '\n'          (2, 5) (2, 6)
    INDENT     '        \t'  (3, 0) (3, 9)
    NAME       'pass'        (3, 9) (3, 13)
    DEDENT     ''            (4, 0) (4, 0)
    DEDENT     ''            (4, 0) (4, 0)
    r<r=r$r$r%�	test_tabsts�zTokenizeTest.test_tabsc

Cs|�d
d�
dS)Nu!Örter = 'places'
grün = 'green'u7
    NAME       'Örter'       (1, 0) (1, 5)
    OP         '='           (1, 6) (1, 7)
    STRING     "'places'"    (1, 8) (1, 16)
    NEWLINE    '\n'          (1, 16) (1, 17)
    NAME       'grün'        (2, 0) (2, 4)
    OP         '='           (2, 5) (2, 6)
    STRING     "'green'"     (2, 7) (2, 14)
    r<r=r$r$r%�test_non_ascii_identifiers�sz'TokenizeTest.test_non_ascii_identifiersc

Cs|�d
d�
dS)Nu#Örter = u'places'
grün = U'green'u7
    NAME       'Örter'       (1, 0) (1, 5)
    OP         '='           (1, 6) (1, 7)
    STRING     "u'places'"   (1, 8) (1, 17)
    NEWLINE    '\n'          (1, 17) (1, 18)
    NAME       'grün'        (2, 0) (2, 4)
    OP         '='           (2, 5) (2, 6)
    STRING     "U'green'"    (2, 7) (2, 15)
    r<r=r$r$r%�test_unicode�szTokenizeTest.test_unicodec

Cs
|�d
d�
|�dd�
|�dd�
|�dd�
|�d	d
�
|�dd�
|�d
d�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd�
|�dd �
|�d!d"�
|�d#d$�
|�d%d&�
|�d'd(�
|�d)d*�
dS)+Nz	async = 1z�    NAME       'async'       (1, 0) (1, 5)
    OP         '='           (1, 6) (1, 7)
    NUMBER     '1'           (1, 8) (1, 9)
    za = (async = 1)a8
    NAME       'a'           (1, 0) (1, 1)
    OP         '='           (1, 2) (1, 3)
    OP         '('           (1, 4) (1, 5)
    NAME       'async'       (1, 5) (1, 10)
    OP         '='           (1, 11) (1, 12)
    NUMBER     '1'           (1, 13) (1, 14)
    OP         ')'           (1, 14) (1, 15)
    zasync()z�    NAME       'async'       (1, 0) (1, 5)
    OP         '('           (1, 5) (1, 6)
    OP         ')'           (1, 6) (1, 7)
    zclass async(Bar):passa<
    NAME       'class'       (1, 0) (1, 5)
    NAME       'async'       (1, 6) (1, 11)
    OP         '('           (1, 11) (1, 12)
    NAME       'Bar'         (1, 12) (1, 15)
    OP         ')'           (1, 15) (1, 16)
    OP         ':'           (1, 16) (1, 17)
    NAME       'pass'        (1, 17) (1, 21)
    zclass async:passz�    NAME       'class'       (1, 0) (1, 5)
    NAME       'async'       (1, 6) (1, 11)
    OP         ':'           (1, 11) (1, 12)
    NAME       'pass'        (1, 12) (1, 16)
    z	await = 1z�    NAME       'await'       (1, 0) (1, 5)
    OP         '='           (1, 6) (1, 7)
    NUMBER     '1'           (1, 8) (1, 9)
    z	foo.asyncz�    NAME       'foo'         (1, 0) (1, 3)
    OP         '.'           (1, 3) (1, 4)
    NAME       'async'       (1, 4) (1, 9)
    zasync for a in b: passa;
    NAME       'async'       (1, 0) (1, 5)
    NAME       'for'         (1, 6) (1, 9)
    NAME       'a'           (1, 10) (1, 11)
    NAME       'in'          (1, 12) (1, 14)
    NAME       'b'           (1, 15) (1, 16)
    OP         ':'           (1, 16) (1, 17)
    NAME       'pass'        (1, 18) (1, 22)
    zasync with a as b: passa<
    NAME       'async'       (1, 0) (1, 5)
    NAME       'with'        (1, 6) (1, 10)
    NAME       'a'           (1, 11) (1, 12)
    NAME       'as'          (1, 13) (1, 15)
    NAME       'b'           (1, 16) (1, 17)
    OP         ':'           (1, 17) (1, 18)
    NAME       'pass'        (1, 19) (1, 23)
    z	async.fooz�    NAME       'async'       (1, 0) (1, 5)
    OP         '.'           (1, 5) (1, 6)
    NAME       'foo'         (1, 6) (1, 9)
    �asyncz/    NAME       'async'       (1, 0) (1, 5)
    zasync
#comment
awaitz�    NAME       'async'       (1, 0) (1, 5)
    NEWLINE    '\n'          (1, 5) (1, 6)
    COMMENT    '#comment'    (2, 0) (2, 8)
    NL         '\n'          (2, 8) (2, 9)
    NAME       'await'       (3, 0) (3, 5)
    zasync
...
awaitz�    NAME       'async'       (1, 0) (1, 5)
    NEWLINE    '\n'          (1, 5) (1, 6)
    OP         '...'         (2, 0) (2, 3)
    NEWLINE    '\n'          (2, 3) (2, 4)
    NAME       'await'       (3, 0) (3, 5)
    zasync
awaitz�    NAME       'async'       (1, 0) (1, 5)
    NEWLINE    '\n'          (1, 5) (1, 6)
    NAME       'await'       (2, 0) (2, 5)
    z
foo.async + 1z�    NAME       'foo'         (1, 0) (1, 3)
    OP         '.'           (1, 3) (1, 4)
    NAME       'async'       (1, 4) (1, 9)
    OP         '+'           (1, 10) (1, 11)
    NUMBER     '1'           (1, 12) (1, 13)
    zasync def foo(): passa;
    NAME       'async'       (1, 0) (1, 5)
    NAME       'def'         (1, 6) (1, 9)
    NAME       'foo'         (1, 10) (1, 13)
    OP         '('           (1, 13) (1, 14)
    OP         ')'           (1, 14) (1, 15)
    OP         ':'           (1, 15) (1, 16)
    NAME       'pass'        (1, 17) (1, 21)
    zNasync def foo():
  def foo(await):
    await = 1
  if 1:
    await
async += 1
a�    NAME       'async'       (1, 0) (1, 5)
    NAME       'def'         (1, 6) (1, 9)
    NAME       'foo'         (1, 10) (1, 13)
    OP         '('           (1, 13) (1, 14)
    OP         ')'           (1, 14) (1, 15)
    OP         ':'           (1, 15) (1, 16)
    NEWLINE    '\n'          (1, 16) (1, 17)
    INDENT     '  '          (2, 0) (2, 2)
    NAME       'def'         (2, 2) (2, 5)
    NAME       'foo'         (2, 6) (2, 9)
    OP         '('           (2, 9) (2, 10)
    NAME       'await'       (2, 10) (2, 15)
    OP         ')'           (2, 15) (2, 16)
    OP         ':'           (2, 16) (2, 17)
    NEWLINE    '\n'          (2, 17) (2, 18)
    INDENT     '    '        (3, 0) (3, 4)
    NAME       'await'       (3, 4) (3, 9)
    OP         '='           (3, 10) (3, 11)
    NUMBER     '1'           (3, 12) (3, 13)
    NEWLINE    '\n'          (3, 13) (3, 14)
    DEDENT     ''            (4, 2) (4, 2)
    NAME       'if'          (4, 2) (4, 4)
    NUMBER     '1'           (4, 5) (4, 6)
    OP         ':'           (4, 6) (4, 7)
    NEWLINE    '\n'          (4, 7) (4, 8)
    INDENT     '    '        (5, 0) (5, 4)
    NAME       'await'       (5, 4) (5, 9)
    NEWLINE    '\n'          (5, 9) (5, 10)
    DEDENT     ''            (6, 0) (6, 0)
    DEDENT     ''            (6, 0) (6, 0)
    NAME       'async'       (6, 0) (6, 5)
    OP         '+='          (6, 6) (6, 8)
    NUMBER     '1'           (6, 9) (6, 10)
    NEWLINE    '\n'          (6, 10) (6, 11)
    z)async def foo():
  async for i in 1: passa�    NAME       'async'       (1, 0) (1, 5)
    NAME       'def'         (1, 6) (1, 9)
    NAME       'foo'         (1, 10) (1, 13)
    OP         '('           (1, 13) (1, 14)
    OP         ')'           (1, 14) (1, 15)
    OP         ':'           (1, 15) (1, 16)
    NEWLINE    '\n'          (1, 16) (1, 17)
    INDENT     '  '          (2, 0) (2, 2)
    NAME       'async'       (2, 2) (2, 7)
    NAME       'for'         (2, 8) (2, 11)
    NAME       'i'           (2, 12) (2, 13)
    NAME       'in'          (2, 14) (2, 16)
    NUMBER     '1'           (2, 17) (2, 18)
    OP         ':'           (2, 18) (2, 19)
    NAME       'pass'        (2, 20) (2, 24)
    DEDENT     ''            (3, 0) (3, 0)
    zasync def foo(async): awaitah
    NAME       'async'       (1, 0) (1, 5)
    NAME       'def'         (1, 6) (1, 9)
    NAME       'foo'         (1, 10) (1, 13)
    OP         '('           (1, 13) (1, 14)
    NAME       'async'       (1, 14) (1, 19)
    OP         ')'           (1, 19) (1, 20)
    OP         ':'           (1, 20) (1, 21)
    NAME       'await'       (1, 22) (1, 27)
    z@def f():

  def baz(): pass
  async def bar(): pass

  await = 2a�    NAME       'def'         (1, 0) (1, 3)
    NAME       'f'           (1, 4) (1, 5)
    OP         '('           (1, 5) (1, 6)
    OP         ')'           (1, 6) (1, 7)
    OP         ':'           (1, 7) (1, 8)
    NEWLINE    '\n'          (1, 8) (1, 9)
    NL         '\n'          (2, 0) (2, 1)
    INDENT     '  '          (3, 0) (3, 2)
    NAME       'def'         (3, 2) (3, 5)
    NAME       'baz'         (3, 6) (3, 9)
    OP         '('           (3, 9) (3, 10)
    OP         ')'           (3, 10) (3, 11)
    OP         ':'           (3, 11) (3, 12)
    NAME       'pass'        (3, 13) (3, 17)
    NEWLINE    '\n'          (3, 17) (3, 18)
    NAME       'async'       (4, 2) (4, 7)
    NAME       'def'         (4, 8) (4, 11)
    NAME       'bar'         (4, 12) (4, 15)
    OP         '('           (4, 15) (4, 16)
    OP         ')'           (4, 16) (4, 17)
    OP         ':'           (4, 17) (4, 18)
    NAME       'pass'        (4, 19) (4, 23)
    NEWLINE    '\n'          (4, 23) (4, 24)
    NL         '\n'          (5, 0) (5, 1)
    NAME       'await'       (6, 2) (6, 7)
    OP         '='           (6, 8) (6, 9)
    NUMBER     '2'           (6, 10) (6, 11)
    DEDENT     ''            (7, 0) (7, 0)
    zFasync def f():

  def baz(): pass
  async def bar(): pass

  await = 2a    NAME       'async'       (1, 0) (1, 5)
    NAME       'def'         (1, 6) (1, 9)
    NAME       'f'           (1, 10) (1, 11)
    OP         '('           (1, 11) (1, 12)
    OP         ')'           (1, 12) (1, 13)
    OP         ':'           (1, 13) (1, 14)
    NEWLINE    '\n'          (1, 14) (1, 15)
    NL         '\n'          (2, 0) (2, 1)
    INDENT     '  '          (3, 0) (3, 2)
    NAME       'def'         (3, 2) (3, 5)
    NAME       'baz'         (3, 6) (3, 9)
    OP         '('           (3, 9) (3, 10)
    OP         ')'           (3, 10) (3, 11)
    OP         ':'           (3, 11) (3, 12)
    NAME       'pass'        (3, 13) (3, 17)
    NEWLINE    '\n'          (3, 17) (3, 18)
    NAME       'async'       (4, 2) (4, 7)
    NAME       'def'         (4, 8) (4, 11)
    NAME       'bar'         (4, 12) (4, 15)
    OP         '('           (4, 15) (4, 16)
    OP         ')'           (4, 16) (4, 17)
    OP         ':'           (4, 17) (4, 18)
    NAME       'pass'        (4, 19) (4, 23)
    NEWLINE    '\n'          (4, 23) (4, 24)
    NL         '\n'          (5, 0) (5, 1)
    NAME       'await'       (6, 2) (6, 7)
    OP         '='           (6, 8) (6, 9)
    NUMBER     '2'           (6, 10) (6, 11)
    DEDENT     ''            (7, 0) (7, 0)
    r<r=r$r$r%�
test_async�s:




�,��%�zTokenizeTest.test_asyncN)�__name__�
__module__�__qualname__r2r7r;r>r?r@rCrDrErFrGrHrIrJrKrLrMrNrOrQr$r$r$r%r'#s*
%9('<&%r'c@seZ
dZd
d�ZdS)�GenerateTokensTestcCs0t|
�
}t
t|j�
|
�}|�||�����
dS�
N)rr&rr*r+r,rr-r$r$r%r2�s

z!GenerateTokensTest.check_tokenizeN)rRrSrTr2r$r$r$r%rU�s
rUc
Cs�g}
tt
|�d
�
�
j�
}|D]T\}}}}}|tkrbd|vrb|
�tdftdftt	|�
ftdfg�

q|
�
||f�

qt|
�
�d
�
S)Nr(�
.�DecimalrB�
))
rrr)r*r�extendrrr	�reprrr�decode)r/r�
g�toknum�tokval�
_r$r$r%�decistmt�s









�
rac@seZ
dZd
d�ZdS)�TestMiscc
CsPd
dlm
}

d}|�t|�
d�
|�tt|�
�
d�
|�tt|�
�
|
d�
�
dS)Nr
)
rXz+21.3e-5*-.1234/81.7z8+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')z-3.2171603427[0-9]*e-0+7z!-3.217160342717258261933904529E-7)�decimalrXr+raZassertRegexr[�eval)r.rXr/r$r$r%�
test_decistmt�s



�
�zTestMisc.test_decistmtN)rRrSrTrer$r$r$r%rb�srbc@s@eZ
dZd
Zdd�Zdd�Zdd�Zdd	�Zd
d�Zdd
�Z	dS)�TestTokenizerAdheresToPep0263zU
    Test that tokenizer adheres to the coding behaviour stipulated in PEP 0263.
    cCs,tj
�tj
�t�
|
�}t�|t|d
��
dS)N�rb)�os�path�join�dirname�__file__�
TestRoundtrip�check_roundtripr)r.�filenamerir$r$r%�	_testFile�s

z'TestTokenizerAdheresToPep0263._testFilec
Csd
}
|�|
�

dS)Nz9tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt�
rp�r.r1r$r$r%�'test_utf8_coding_cookie_and_no_utf8_bom�s

zETestTokenizerAdheresToPep0263.test_utf8_coding_cookie_and_no_utf8_bomc
Csd
}
|�t
|j|
�
dS)a�

        As per PEP 0263, if a file starts with a utf-8 BOM signature, the only
        allowed encoding for the comment is 'utf-8'.  The text file used in
        this test starts with a BOM signature, but specifies latin1 as the
        coding, so verify that a SyntaxError is raised, which matches the
        behaviour of the interpreter when it encounters a similar condition.
        z8tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txtN��assertRaises�SyntaxErrorrprrr$r$r%�&test_latin1_coding_cookie_and_utf8_bom�s
zDTestTokenizerAdheresToPep0263.test_latin1_coding_cookie_and_utf8_bomc
Csd
}
|�|
�

dS)Nz9tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txtrqrrr$r$r%�"test_no_coding_cookie_and_utf8_bom�s

z@TestTokenizerAdheresToPep0263.test_no_coding_cookie_and_utf8_bomc
Csd
}
|�|
�

dS)Nz6tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txtrqrrr$r$r%�$test_utf8_coding_cookie_and_utf8_bom�s

zBTestTokenizerAdheresToPep0263.test_utf8_coding_cookie_and_utf8_bomc

Cs$|�t
|jd
�
|�t
|jd�
dS)Nz
bad_coding.pyzbad_coding2.pyrtr=r$r$r%�test_bad_coding_cookies

z4TestTokenizerAdheresToPep0263.test_bad_coding_cookieN)
rRrSrT�__doc__rprsrwrxryrzr$r$r$r%rf�s
rfc@seZ
dZd
d�Zdd�ZdS)�
Test_Tokenizec
sPd
}
|
�d�
�
d���
fdd�}t
t|dd��
dd�}d	g
}|�||d
�
dS)N�"ЉЊЈЁЂ"r(Fc
s�sd
��
SdSdS�NT�r$r$��firstr#r$r%r*
s

zNTest_Tokenize.test__tokenize_decodes_with_specified_encoding.<locals>.readline�
�encoding�
r4��r})r�r
)r��r}zbytes not decoded with encoding)r)r5rr+)r.�literalr*r6�expected_tokensr$r�r%�.test__tokenize_decodes_with_specified_encoding	s




	


�z<Test_Tokenize.test__tokenize_decodes_with_specified_encodingc
sFd
�
d���
fdd�}
tt
|
dd��
dd�}dg
}|�||d�
dS)	Nr}Fc
s�sd
��
SdSdSr~r$r$�r�r�r$r%r*s

zQTest_Tokenize.test__tokenize_does_not_decode_with_encoding_none.<locals>.readliner�r4r�z*string not tokenized when encoding is None)r5rr+)r.r*r6r�r$r�r%�1test__tokenize_does_not_decode_with_encoding_nones


	


�z?Test_Tokenize.test__tokenize_does_not_decode_with_encoding_noneN)rRrSrTr�r�r$r$r$r%r|sr|c@s�eZ
dZd
d�Zdd�Zdd�Zdd�Zd	d
�Zdd�Zd
d�Z	dd�Z
dd�Zdd�Zdd�Z
dd�Zdd�Zdd�Zdd�Zdd �Zd!d"�Zd#d$�Zd%d&�Zd'd(�Zd)S)*�TestDetectEncodingcsd
���
fdd�}|S)Nr
c
s$�t�
�
krt
�
�
�}�d
7�|S�Nr�)r�
StopIteration)
r#��index�linesr$r%r*1s




z1TestDetectEncoding.get_readline.<locals>.readliner$�r.r�r*r$r�r%�get_readline/s

zTestDetectEncoding.get_readlinec
Cs>d
}
t|�
|
�
�
\}}|�|d�
|�|t|
dd��
�
dS)N)�# something
�print(something)
�do_something(else)
r(�)r
r�r+r5�r.r�r��consumed_linesr$r$r%�test_no_bom_no_encoding_cookie:s


z1TestDetectEncoding.test_no_bom_no_encoding_cookiec
Cs6d
}
t|�
|
�
�
\}}|�|d�
|�|ddg�
dS)N)s# something
r�r��	utf-8-sigr�r��r
r�r+r�r$r$r%�test_bom_no_cookieDs



�z%TestDetectEncoding.test_bom_no_cookiec
Cs4d
}
t|�
|
�
�
\}}|�|d�
|�|dg
�
dS)N)�# -*- coding: latin-1 -*-
r�r��
iso-8859-1r�r�r�r$r$r%�test_cookie_first_line_no_bomOs


z0TestDetectEncoding.test_cookie_first_line_no_bomc
Cs4d
}
t|�
|
�
�
\}}|�|d�
|�|dg
�
dS)N)s# coding=utf-8
r�r�r�s# coding=utf-8
r�r�r$r$r%�&test_matched_bom_and_cookie_first_lineYs


z9TestDetectEncoding.test_matched_bom_and_cookie_first_linec
Cs d
}
|�|
�
}|�
tt|�
dS)N)s## vim: set fileencoding=ascii :
r�r��r�rurvr
r�r$r$r%�<test_mismatched_bom_and_cookie_first_line_raises_syntaxerrorcs


zOTestDetectEncoding.test_mismatched_bom_and_cookie_first_line_raises_syntaxerrorc
Cs:d
}
t|�
|
�
�
\}}|�|d�
ddg}|�||�
dS)N)�
#! something
� # vim: set fileencoding=ascii :
r�r��asciir�r�r��r.r�r�r�r0r$r$r%�test_cookie_second_line_no_bomls




z1TestDetectEncoding.test_cookie_second_line_no_bomc
Cs6d
}
t|�
|
�
�
\}}|�|d�
|�|ddg�
dS)N)�#! something
�f# coding=utf-8
r�r�r�r�r�r�r�r$r$r%�'test_matched_bom_and_cookie_second_linexs



�z:TestDetectEncoding.test_matched_bom_and_cookie_second_linec
Cs d
}
|�|
�
}|�
tt|�
dS)N)r�r�r�r�r�r�r$r$r%�=test_mismatched_bom_and_cookie_second_line_raises_syntaxerror�s


zPTestDetectEncoding.test_mismatched_bom_and_cookie_second_line_raises_syntaxerrorc
Cs8d
}
t|�
|
�
�
\}}|�|d�
dg
}|�||�
dS)N)�print('£')
�%# vim: set fileencoding=iso8859-15 :
�
print('€')
r(r�r�r�r$r$r%�/test_cookie_second_line_noncommented_first_line�s




zBTestDetectEncoding.test_cookie_second_line_noncommented_first_linec
Cs:d
}
t|�
|
�
�
\}}|�|d�
ddg}|�||�
dS)N)�
#print('£')
r�r��
iso8859-15r�r�r�r�r$r$r%�,test_cookie_second_line_commented_first_line�s




z?TestDetectEncoding.test_cookie_second_line_commented_first_linec
Cs:d
}
t|�
|
�
�
\}}|�|d�
ddg}|�||�
dS)N)�

r�r�r�r�r�r�r�r$r$r%�(test_cookie_second_line_empty_first_line�s




z;TestDetectEncoding.test_cookie_second_line_empty_first_linec
	Csdd
}
|
D]V}dD]L}|�d|�}dd|�
d�
ddd	f}|�|�
}t|�
\}}|�|d
�
qqdS)N)zlatin-1r�ziso-latin-1zlatin-1-unixziso-8859-1-unixziso-latin-1-mac��
-r`r��#!/usr/bin/python
�
# coding: r�r�sprint(things)
sdo_something += 4
r���replacer)r�r
r+�	r.�	encodingsr�Zrep�encr�Zrl�foundr�r$r$r%�test_latin1_normalization�s





�


z,TestDetectEncoding.test_latin1_normalizationc
Cs d
}
|�|
�
}|�
tt|�
dS)N�
s
print("�")r�r�r$r$r%�test_syntaxerror_latin1�s

z*TestDetectEncoding.test_syntaxerror_latin1c
	Csbd
}
|
D]T}dD]J}|�d|�}dd|�
d�
ddf}|�|�
}t|�
\}}|�|d	�
qqdS)
N)r(z	utf-8-macz
utf-8-unixr�r�r�r�r�r�s1 + 3
r(r�r�r$r$r%�test_utf8_normalization�s





�


z*TestDetectEncoding.test_utf8_normalizationc
Cs�|�d
�
}
t
|
�
\}}|�|d�
|�|dg
�
t
|�d�
�
\}}|�|d�
|�|g�
|�d�
}
t
|
�
\}}|�|d�
|�|dg
�
|�d�
}
t
|
�
\}}|�|d�
|�|g�
|�d�
}
|�tt
|
�
dS)	N)
r�r(r�r$)
sprint(something)
r�)
s)
s# coding: bad
)r�r
r+rurv�r.r*r�r�r$r$r%�test_short_files�s"
















z#TestDetectEncoding.test_short_filesc
Cs4|�d
�
}
t
|
�
\}}|�|d�
|�|dg
�
dS)N)
�print("#coding=fake")r(r�)r�r
r+r�r$r$r%�test_false_encoding�s



z&TestDetectEncoding.test_false_encodingc
	Cs4
tj
d
}
|�tj|
�
dD]�}t|
d|d��,}td||d�
td|d�
Wd�
n1s`0


Y
t|
�
�,}|�|j|�
|�|j	d�
Wd�
q1s�0


Y
qt|
dd	d��}td|d�
Wd�
n1s�0


Y
t|
�
�,}|�|jd	�
|�|j	d�
Wd�
n1�
s&0


Y
dS)
Nz.py)ziso-8859-15r(�
wr�z# coding: %s�
�fileuprint('euro:€')�
rr�)
rZTESTFNZ
addCleanup�unlinkr�print�
tokenize_openr+r��mode)r.ror��fpr$r$r%�	test_open�s





*



.
*



zTestDetectEncoding.test_openc
s�d
}
d�G�f
dd�d�}|�t
�
�(
|�|
�}|`t|j�

Wd�
n1sR0


Y
|�t
d�|
�
��$
|�|
�}t|j�

Wd�
n1s�0


Y
dS)NZsome_file_pathr�cs eZ
dZd
d�Z�f
dd�ZdS)z;TestDetectEncoding.test_filename_in_exception.<locals>.BunkcSs||_|
|_
d
|_dS)Nr
)�nameZ_lines�_index)r.r�rir$r$r%�__init__s


zDTestDetectEncoding.test_filename_in_exception.<locals>.Bunk.__init__c
s.|jt
��
krt�
�|j}
|jd
7_|
Sr�)r�rr�)r.r#�
r�r$r%r*s






zDTestDetectEncoding.test_filename_in_exception.<locals>.Bunk.readlineN)rRrSrTr�r*r$r�r$r%�Bunk
s
r�z.*{})rurvr�r
r*r8�format)r.rir�Zinsr$r�r%�test_filename_in_exceptions




(



z-TestDetectEncoding.test_filename_in_exceptionc
CsTtd
�
}
t
jd|
d��
|�ttd�
Wd�
n1s:0


Y
|�|
j�

dS)Ns#coding:xxxztokenize._builtin_open)
Zreturn_valueZfoobar)rrZpatchrurvr�Z
assertTrue�closed)r.�
mr$r$r%�test_open_error#s

,
z"TestDetectEncoding.test_open_errorN)rRrSrTr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r$r$r$r%r�-s(


	


r�c@s<eZ
dZd
d�Zdd�Zdd�Zdd�Zd	d
�Zdd�Zd
S)�TestTokenizec
s�d
dl}
t
��
d��
f
dd�}�f
dd�}d
��f
dd�}|
j}|
j}||
_||
_z,t|�
}|�t|�
gd�
�
W||
_||
_n||
_||
_0|���
�
dS)	Nr
c

s�d
dgfS)N�first�secondr$)
r*r�r$r%�mock_detect_encoding1s
z8TestTokenize.test_tokenize.<locals>.mock_detect_encodingcs"|
�g}|�}|r|�|�

q|SrV)
r)r*r��outZ	next_line)
�
encoding_usedr$r%�mock__tokenize4s





z2TestTokenize.test_tokenize.<locals>.mock__tokenizecs �d
7��dkrdSt��
�
�S)Nr��r)�strr)r$)
�counterr$r%�
mock_readline@s


z1TestTokenize.test_tokenize.<locals>.mock_readline)r�r��
1�
2�
3�
4)r�objectr
rr+r5)r.Ztokenize_moduler�r�r�Zorig_detect_encodingZorig__tokenize�resultsr$)r�r�r�r%�
test_tokenize-s*












�
�
zTestTokenize.test_tokenizec
Csfg}
td
�
D]}|
�
dj|d�
�

q|
�
d�

d�|
�
}
ttt|
�d�
�
j�
�
}|�	|dj
d�
dS)Ni�
zdef i{i}(): return {i})
�
iZOK�

r(�����)�rangerr�rjr5rrr)r*r+�string)r.�bufr�Ztoksr$r$r%�test_oneline_defsUs







zTestTokenize.test_oneline_defscGs�tt
t|
�d
�
�
j�
�
}t|�
}|�t|�
d|�
|�t|djtt	�
t
|�
D]&}|�t||djt||�
qV|�t|d|jttj�
|�t|d|jttj
�
dS)Nr(r�r
r�r�)r5rrr)r*rr+r�
exact_typerr�r rr
)r.ZopstrZoptypesr6Znum_optypesr�r$r$r%�assertExactTypeEqualas 




�


�
�
�z!TestTokenize.assertExactTypeEqualc

Cs�|�d
t
jt
j�
|�dt
jt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j	�
|�dt
j
�
|�dt
j�
|�d	t
j�
|�d
t
j
�
|�dt
j�
|�dt
j�
|�d
t
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
jt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j�
|�dt
j �
|�dt
j!�
|�dt
j"�
|�dt
j#�
|�d t
j$�
|�d!t
j%�
|�d"t
j&�
|�d"t
j&�
|�d#t
j'�
|�d$t
j(�
|�d%t
j)�
|�d&t
j*�
|�d't
j+�
|�d(t
j,�
|�d)t
j-�
|�d*t
j.�
|�d+t
j/�
|�d,t
j0�
|�d-t1t
jt2t
j	t1t
jt2t
jt1t
jt2�
|�d.t
jt
j2t
jt
j2t
jt
j2t
j�
|�d/t
jt
jt
j1t
jt
j2t
j�
dS)0Nz()z[]�
:�
,�
;�
+r��
*�
/�
|�
&�
<�
>�
=rW�
%z{}z==z!=z<=z>=�
~�
^z<<z>>z**z+=z-=z*=z/=z%=z&=z|=z^=z<<=z>>=z**=z//z//=z:=z...z->�
@z@=za**2+b**2==c**2z	{1, 2, 3}z
^(x & 0x1))3r�r �LPAR�RPAR�LSQB�RSQB�COLON�COMMA�SEMI�PLUS�MINUS�STAR�SLASH�VBAR�AMPER�LESS�GREATER�EQUAL�DOT�PERCENT�LBRACE�RBRACE�EQEQUAL�NOTEQUAL�	LESSEQUAL�GREATEREQUAL�TILDE�
CIRCUMFLEX�	LEFTSHIFT�
RIGHTSHIFT�
DOUBLESTAR�	PLUSEQUAL�MINEQUAL�	STAREQUAL�
SLASHEQUAL�PERCENTEQUAL�
AMPEREQUAL�	VBAREQUAL�CIRCUMFLEXEQUAL�LEFTSHIFTEQUAL�RIGHTSHIFTEQUAL�DOUBLESTAREQUAL�DOUBLESLASH�DOUBLESLASHEQUAL�
COLONEQUAL�ELLIPSIS�RARROW�AT�ATEQUALrrr=r$r$r%�test_exact_typeos�

















































�




�



�zTestTokenize.test_exact_typec

Cs|�d
t
j�
dS)Nz@          )r�r r4
r=r$r$r%�%test_pathological_trailing_whitespace�sz2TestTokenize.test_pathological_trailing_whitespacec
Cs�d
}
tj
tjtjtjtjtjg}ttt	|
�
d�
�
j�
�
}|�t
|djt
t�
td�
D]&}|�t
||djt
||�
qZ|�t
|djt
tj�
dS)Nzb = 1

#testr(r
�r�r)r rr
rr�NL�COMMENTr5rrr)r*r+rr�rr�r
)r.�sourcer�r6r�r$r$r%�5test_comment_at_the_end_of_the_source_without_newline�s



$
zBTestTokenize.test_comment_at_the_end_of_the_source_without_newlineN)	rRrSrTr�r�r�r6
r7
r<
r$r$r$r%r�+s(Ar�c@s$eZ
dZd
d�Zdd�Zdd�ZdS)�UntokenizeTestc
Csnt�}
d
|
_
d
|
_|�t�
�}|
�d�

Wd�
n1s<0


Y
|�|jjdd�
|�t|
jd�
dS)Nr�)r�r�r
z'start (1,3) precedes previous end (2,2))r�r�)	r�prev_row�prev_colru�
ValueError�add_whitespacer+�	exception�args)r.�
u�cmr$r$r%�test_bad_input_order�s



(

�z#UntokenizeTest.test_bad_input_orderc
Csdt�}
d
|
_
d
|
_g|
_|
�d�

|�|
jdg
�
d|
_
|
�d�

|�|
jgd�
�
t�|d�
dS)Nr�)r�r
�\
r�)�rH
)rG
z\
\
rza
  b
    c
  \
  c
)rr>
r?
r6rA
r+rmrn)r.rD
r$r$r%�test_backslash_continuation�s










z*UntokenizeTest.test_backslash_continuationc
Cs�t�}
t
d
f}tdf|g}|
�|tg�
�
|�|
jdg
�
t�}
|�|
�t|g
�
�
d�
t�}
|�|
�t|�
�
d�
|�|
jd�
|�tt|�
�
d�
dS)NZHellor(zHello sHello )	rrr�compat�iterr+r6rr�)r.rD
r r6r$r$r%�test_iter_compat�s










zUntokenizeTest.test_iter_compatN)rRrSrTrF
rI
rL
r$r$r$r%r=
�s
r=
c@sLeZ
dZd
d�Zdd�Zdd�Zdd�Zd	d
�Zdd�Zd
d�Z	dd�Z
dS)rmcCs�t|
t
�r|
�d
�
}n|
��}|
��
t|jdd�
�
j}tt	|�
�
}dd�|D�
}t
|�
}t|jdd�
�
j}dd�t	|�
D�
}|�||�
t
|�
}	t|	jdd�
�
j}
dd�t	|
�
D�
}|�||�
dS)	a)
        Test roundtrip for `untokenize`. `f` is an open file or a string.
        The source code in f is tokenized to both 5- and 2-tuples.
        Both sequences are converted back to source code via
        tokenize.untokenize(), and the latter tokenized again to 2-tuples.
        The test fails if the 3 pair tokenizations do not match.

        When untokenize bugs are fixed, untokenize with 5-tuples should
        reproduce code that does not contain a backslash continuation
        following spaces.  A proper test should test this.
        r(T)
�keependsc
Ssg|]}
|
dd
��qS�Nr�r$��.0r:r$r$r%�
<listcomp>�rz1TestRoundtrip.check_roundtrip.<locals>.<listcomp>c
Ssg|]}
|
dd
��qSrN
r$rO
r$r$r%rQ
rc
Ssg|]}
|
dd
��qSrN
r$rO
r$r$r%rQ
rN)�
isinstancer�r)�read�closerK
r�__next__r5rrr+)r.r1�coder*Ztokens5Ztokens2Zbytes_from2Z	readline2Z
tokens2_from2Zbytes_from5Z	readline5Z
tokens2_from5r$r$r%rn�s











zTestRoundtrip.check_roundtripc
Cs�|�d
�

|�d�

|�d�

t
�d�
}
t|
d��}|�|�

Wd�
n1sR0


Y
|�d�

|�d�

|�d�

|�d	�

dS)
Nzif x == 1:
    print(x)
z # This is a comment
# This also
zif x == 1 : 
  print(x)
�tokenize_tests.txtrgzoif x == 1:
    # A comment by itself.
    print(x) # Comment here, too.
    # Another comment.
after_if = True
zPif (x # The comments need to go in the right place
    == 1):
    print('x==1')
z�class Test: # A comment here
  # A comment with weird indent
  after_com = 5
  def x(m): return m*5 # a one liner
  def y(m): # A whitespace after the colon
     return y*4 # 3-space indent
zttry: import somemodule
except ImportError: # comment
    print('Can not import' # comment2
)else:   print('Loaded')
)rnr�findfiler)r.�fnr1r$r$r%�test_roundtrip
s





(



zTestRoundtrip.test_roundtripc

Cs|�d
�

dS)Nzna = (3,4, 
5,6)
y = [3, 4,
5]
z = {'a': 5,
'b':15, 'c':True}
x = len(y) + 5 - a[
3] - a[2]
+ len(z) - z[
'b']
�
rnr=r$r$r%�test_continuation1szTestRoundtrip.test_continuationc

Cs|�d
�

|�d�

dS)Nz)x=1+\
1
# This is a comment\
# This also
z# Comment \
x = 0r[
r=r$r$r%rI
>s
z)TestRoundtrip.test_backslash_continuationc

Cs|�d
�

dS)Nz'' ''r[
r=r$r$r%�test_string_concatenationGsz'TestRoundtrip.test_string_concatenationc

Cs
d
dl}
d
dl
}t�d�
}tj�|�
p*tj}|
�tj�|
�	|�
d��
}|�
tj�|d��

dD]}|�
tj�|d�|�

q^t�d�
s�|�|d�}|D]z}tj
d	kr�td
|�
t|d��F}|j|d�
�
|�|�

Wd�
n1s�0


Y
Wd�
q�1�
s0


Y
q�dS)
Nr
rW
ztest*.pyztest_unicode_identifiers.py)�bufferZbuiltinZfileio�inspectrh�platform�sysz
test_%s.pyZcpu�
r�rrgr�)�glob�randomrrX
rhrirk�curdirrj�escape�removeZis_resource_enabled�sample�verboser�rZsubTestrn)r.rc
rd
rY
�tempdirZ	testfilesr1Ztestfiler$r$r%�test_random_filesKs














zTestRoundtrip.test_random_filescCs,t|
t
�r|
�d
�
}
ttt|
�
j�
�
�d
�
S)Nr()rR
r�r)rrrr*r\)r.rV
r$r$r%�	roundtriphs




zTestRoundtrip.roundtripc
Cs6d
}
|�|
�
�
d�
}|�|d|d�
|�|
�

dS)z�
        Ensure that although whitespace might be mutated in a roundtrip,
        the semantic meaning of the indentation remains consistent.
        zif False:
	x=3
	x=3
r�r�r�N)rl
�splitr+rn)r.rV
Z	codelinesr$r$r%�#test_indentation_semantics_retainedms


z1TestRoundtrip.test_indentation_semantics_retainedN)rRrSrTrnrZ
r\
rI
r]
rk
rl
rn
r$r$r$r%rm�s '
	rm�__main__)*�testrrrrrrrr	r
rrr
rr�rrr�iorrZunittestrrZtest.test_grammarrrrhr r&r'rUrarbrfr|r�r�r=
rmrR�mainr$r$r$r%�<module>
s>
D



	%&)