ÿØÿà JFIF    ÿÛ „ !.%+&8&+/1555$;@;4?.451 4,$,44444444444414444444444444444444444444444444444444ÿÀ  á á" ÿÄ     ÿÄ ?    !1AQaq"2‘¡±ÁðBRbrÑá#‚’¢²3S CñÿÄ   ÿÄ !    !1QAa‘2ÿÚ   ? 5˜Z¯V¦cø)›t/? z¨±>Õ5€¶‹Á¤·¼z¼Ü¬+ñ®v¤¨_ˆR­BFn©—˜ý®ç̝P8gýt·ÉSTŦˆìät?þé¼íìN/Þa)ì–í6ô… Ï¿øÃj´¿KÇü]ÿ ªô¹-eKànëÕHTx}ýSÜ›ÿ ”7Ø×&µ<¦  ¥ÑO¶[Ù¯ä¨ÞÃÿ PZ-¬;#õ|•oaÿ ©CìÞz3˜öː/¤­ñTûIØ}š^ mÓ%ªxˆ¥ÉŸu=Z+ISe¿45™¼u;ú&WØ÷€æßQ™®{|íx*TC“#ZŠìZ§²‹ 6pv…³¿¡äª*áZÐ%ÒOáˆo"x«OHk w±æ+¬V(kMúŸ5Vö«$ ÁrÏbàb57/luR ¸ÑÛj Òµì`Ðœq­û žICÀÊ•©4€Âcà¨Ï€O´<èÐ:›ù(Ë^L8þ‘ÍÌ#¸Ð_Ì©ÙK(Öz 4¬û+¸;ü’V’84‘¬ÃŽ:[â‡ÔÌáõp¢~§ªlæ£ö{®G>J¼"°‡7¯ÆÉèßû ‹É‹§ÁòÃýâßî ^ƾÙõ‹×óH#«LP½ïX=xÑÍ$|W?•~• îëÔ©ª‹ {ÝT…Kÿ ”hûâá)J*ö˜–ÔU;iÇ€/ ÆþjóZ\ýwØ=Ìm ºèËL9 ýèÆð/¨’¥öo=nË.%Îì ŽÕ¯È|{Oj²ƒE6e/ßdÄõ²Ìâ1O®ò×TsəԸhOMýíMˆ¿¼H˜l²,7Â¥#MF/Úf°Ö½± ¸–dr‹NýÊ íjqx{œÉ ä-È ¦ øÄër¨q°ð †nцýÑÄÆ’mä…n<0È™;ÁÝá¯ÁZƒ7FÀmì­ É&9ˆîéi¶ùN§Y• ÃZãAâ?•‡©‰ , ó¾IŸŠc1 4â&y­&pŠ­6;M À 0¹qç»p.á …ŸÅáK@%6·y6ƒ‰3?”úºŽ‰éX5ªPT §µ!=Mž«Ú½‹ÅgÂSâÉaþÓoö–¯ÁÔìR>5éÿ üs¶ÆUcÌ kÇR ]ÿ ù¬¼«VŽ;Â|‡~¢¦”ÏÅ°æ {L™Õ°Óv¹ò¸írŽ¡×¢CÃ!íVÕ {¶»sŒNPg/ "uÕbkm²“$ďå¿é¹§°½æz¯6 †s¿!s–wÚÝ“™Œ °.ûj>·+™Òa…©Œ&rÝÎtÛë긪Ît’LAVp%c Úý[ÄzJ¾ÇàXXç@˜ó<êL]·T˜¾¥1Ó©V‡g´æ½¦Ý@¹óø!_@´ÞâSÁ —S3™•& ]@JHÚý©ZŽ €×æÔr»Áf!‡yÞ4Mv*èÓã_{‘åóUuљØ«Oïé*®EvÑ Œ÷‡U \"㪒ÍK+À 4“M¡ï:0¥5í!'<@î´”>Ç»&Z–ïCCV˜Ì5Šo&îhè.žû |ÓK©h$s6KìŒëã)¹hI¦GïOåóI;ììü#É$Š0…Ææ¥TØ.5­¾gn´ “ÂÖ\:hœ89G)J@„}œ:’Ò{/Š"¦_Æ×7Æ3VÇŠÊa]ÚŒÙ€Ä–=®uÁßâACZƒ§§£ Qnâ:«,×{tyø¬iÛcœÜÄ€H½ÄÍCk´÷šß .W'b¤Íåh]÷€=,Žv×cÚEÚHXJX¶îo¨FÒtèöŸ>ªª6[J®Fµ£sGÁeqõfe\íjÒÐïÄÐGˆe1Ø‹.Ø”‘Ëuø Y­ˆÜ ŽG|zùªüMpDnQWÄ”%JŠ™)â*p@Örš«ÕT2Ð%ˆG#ª„ ·¤!°ŸOTÂT¸aÚ%4&h™LµšØüÐ.F¿²ÐÞ_Ç‚¾ÅÃaÜ÷09Æ q€öy˜v‡85õN÷]¬äѼóS{°_MŽ¬úÔ#°Ç¸0åÞè2ëôPcvÆw9®ií1Ä8F™˜à‰´+‰Ik1òÝ7“Ñ×ÒsÝ\x‚h`ÞÑ`ó"|µEcý£n˜h`}GÞ !±ù²Ápü²ß6 0ïi󜵩SÈÇ7˜-ÕURO˜¦´f$ªž-Í6(œ}<„ éc øs]ŽŽ„*—¾ ìdŽ„)méª\¿êÎIg¾ØÞ~I#C/¼¼´EÁÈŽi8“©õådô·>euä ƒ'Ê×लR1ÉJE1ÐAát`t;ÇР%Ý<‡¥„ÍÆ`×Oyó)õiI€ñQaŸ4Ûù\áàaÃÔ¹HÃu¹*k€¦<„e S‡&õÏ B!ŽhüÞ`yj}mªf×\¿ Ç~æ­9‡û\՞Ǖg²1Žû5V7 !àöšm° c`ܬøÇìµÒ'P"?…´Ö,"§^•õŽ¨sÔ)6˜sæéÍR¼ ò|Sl”‹7 nPW Gòú÷½§O¯‡„l¡kSÞŒr½PÊ@æ¢pŽ-mÿ #Ÿ˜Àº¶Áä¦;ïÔæ$1££`“Õ>„—·ž)ßð³ñ#Ï Ô$¶œ‰ÊE‹À;÷º ¯«P:Ñ”8–IÊtpÞ3ª“>ê“þës4ò2OÏÕ­±zô†Õ§‰.÷ä¸;¿˜“'œ›žª}«Œ{ª±Ì 9ÔóÞÕ‡0 $íWV3Üì¬ —@kÝ4@¿r¼±½¬™›?øØæ´'Áé®CË3-g$˜ö‡×auÚi´Žp/êÛ æF›Ú2v‹ã¿¿,nB1̨ƃqÞa5͝@&Æû“él÷ \C²½UÍc ¯k×¢U ÖéQå™—-r wô ÞÏ<Ò=&=ÿ Ôê Òêˈt,i—;LîÜ á¸*ÚÃ1$êL•LÍ <É)ýÐà’ ;F™{ƒ™˜€&'}‚ãÄK`¡ÞT@I;®žZóè‚s’7®°›+§O­Åq©é»²9<Ô J ¼9O’HL»Ùïì¸rk¼Ž_ý‘TŸu[²ßÚŒ·ü÷B%¯E ŸÔX5êO´ Ç•€’I0 ÉJX` ñ¹õ%;µŸD‘«´€àwÒ™U ûئžÖö\×®×´8 ½‡ºÐÆÓ§?Àkmœ=;d5*@-ì0F Rªýš[Ü6âö̃ڸr*KA9· u*µæ£?U¸Âêí†8@¦X4 e-ò„0s{ HâUpU?¼mñRa°®a%Ð'tÉ×’\¾ÊÉ]t›h>·(Ë@R¼¡Ãt h}’O÷au<+nT…Ö…MӐ??Óe95 q>í/;&JSû °¯ÊéÞ øƒ*Ã2½Ài&:nôUl=¾¿5eˆ3”ñc|Ú2V”>„»&eE;«ÚäC p¢Û úy 9š[ŒÌx¼擼A&DåÒ¯ˆ¤ÀÌ;"˜ ÏQä¸åhÊ}Ûq«Û0WžÒ|»€ø®öCm5•\ÇÀ§Pe3£]0ÃàLDÉ‰1øªxjgwT‚÷¿LΨK‹›ùs—xˆÜ±µ kæ¸f‰‰ÜGk/LÛØ6d9ò¶ùA{ƒA3š/¬D¬khÓk‰`˜"㯒r¿±Óã jx‡°e}<Ñø\3y:'À•/h½Í€Ç4~g ?Û(¼]v‘ªlKÎâ~?O‚W%{Ì:“'©úNq¾›úo(X’¥¯ˆ nFê{Ç€ü?º'ë ø‹ì Þ09ŒÌç9Æ —ËC`j@ÓÄ(+a‹un¸#ÂꟋ{K`‘ÑÍÍ'à´»/Û,KW;Þ4²þð ï Nm|~fGÏ(…³Ã)«1ö­Õ ¥‡¨©ƒÃ™ü-s=à=U66Ï«Ýc蓦W¹íž®›nÔ%êÇìŒ<#Ü×84ån®Ð ÒåOC` ñânÑs‡¢ç 1õ%Îhì½Ã½® e:ݼUZo™`  ÅZŸŒÊ«ê1ÏÄo$q¹Þ€©ˆhÐÉä¯ñ[!…Ú˜àJ:x2$Íß&PåT£6ç— ‡Í*4Ýšçjÿ ‰É nófÐ ó(L5C•åÆ\rMÒ@ò }y-W}™üýVù—ú¢=Ù”c®‘< M ž ´Phr ¦©TD ‘ù.$´÷O‡‘V2Æò.=IUŒ=ž‡â¬i™aþÓåÙ?òUø'ØÖ•.~* šTŒ!•-×áºTâ®ä#õü'´ eýlYÅÓeÕKÂrT"CÚ@u!Óxƒ{š3€}1¿(r}%«nËamjÑ%ÑNEò v ˜à  σöK³,*º.àzù¨™Ó ÚçâU¦*¿ 9{%Ö¹ njûdaXöb) kÛƱûÓ\°M7ˆÂ=û›ç¿Ã‚­V»Cg–8ÙêE- j)k$º`Ã-ùEýeBÆÇ]c¡°ñty&Òd0nõ'¡W+ƒ*|–øµFa\GQªEAÔp5\Ǽ·¼Ç8·õ -â§Ú[ ‡ uZeÖ 3}×d'+¹:ð+K†Û®s!Ï$úe€<Û”x)1»a­¡LC]¸µík…ÚàA»AYº{†ªS[¦5HÒ7ù --,ísòDØ€èk ÞÀîÜ ò@â( ËNˆë›4ô½•/¦o‡€Û7 ê•ÆêòðÜy'Án½µ á˜ݦ ndeo…[ì¶Ê,¥R³Ä=À±—–ß;£™´ñSâ*g§”ïaið‘Jå~™ÓÞ ß³Õ¢»8x埒²52>AÊb&-÷\7´éÄù€T˜,w;3{ï˜k…à¹ÄqÀ«œ{€\ ˆ¾[´¨јr &Úé„Ívˆ±8†¿]|¬ņ4I×pÞS1ÈÖz‰#Ìv‡G!YNògñ:màTz¢Ý1ô©^O=~ë|5Bã™ç•¼µõ•bÆ@úÕS¬ÈŒ#¬zünrŸ û” Z²•ï›èðV"ÁHÚý©wÝ €7¼Ìu1hÑa3Éä û f$o¿É ™Ú›ÝçnpÒ3äÌ3†Í§,Äï]$‰/pê †«À¼¸e9­Æê_C]žƒ·ý·frÁN«, E=›Çq -‰öŒ:aÏ¿±í&£Í:-} 84‘ÿ eƒQÑeëSsuiA ³g㟥ú£?ÿ ʼn*”“÷aühe:ÊWa@ÒÞk±eØ] F Ô—r.åä˜ @ö¥ªZoÐýYL·¥S²G/‡ñ <~*ZÆ´è>JlòàÛƽÿ 窘ìGN¢:I®KšJp/`íIÁÀõ#Ä-€ö­šµŒoF4|ÆQØÆ@Ì|£Ô…¢À{9˜è½Üó›€ôYÒÎYsið;ís¤€à²ˆ‚4qÉVŒI$ ‰"° æµ8cXGjœˏ¡Aâý•ËÜ¢ûï e·çLx']á"oÅÎê3¯Ç—¹”ó0nå‚âg{Œñ> S´˜îè°g238‚ãköÝfÚd´6Ò€;ò÷±¢™¼›º ¢Æ'¥Ðx'e¬ç ]bÈÆV¢ó‹kýBO ðÊâ$Ÿ!×T 3Mýמ žìٍàÌü‘8÷€àæØ8æ©6‰©L´«…oãpð„~Çk‰!ñ;‹”ÛžÍ àž±z Ÿôû øŸÝužÏ;ÿ #|u6™Þ¬ÚˆÐõA4¶â|ôl|Ê2ŽÇ¤ÝÅÇY.<#Aí.k§hóF‚”Y; M½Ö4hŸ4&›­¿tès´%FìL¥£Ãk‰ÇT¤haÁ¤ÚxfÉ`ÑìË›>i 3t‚:,–+^÷´–{Û–Nxi"x‘Ûg î¨>¥Õ܁ùZH,2Û“:8xÊ¢Çí9.É-Ìâã-=çjwµS˜dütžçwýGòú®®ûº_ˆýx$–¡ãøO EÚÛÏ÷R„×w+3£Á£öUMyR²¹âŒ°š›¸Ñãò9§Ó_Dl+Ùßc›úšGÅÌc†Ž!Ko=¶.‘Îÿ c²(2®V mª.ÿ ¹B›¹å ù„öŸSV>™ü¯$y:G¢Z×àøúdî¹û­·ýÇ´:•c LÍõi_‹ö+ÎæGÊè>OŠ•äž´§Þ{X}¨1ÚTc›»Qþ•êô°t¿OP?eæ~É{5]•ÙR£r5†nZ\ã@ &îJõ ¾àC°þV>fé¥/ü5ñÊIº_é5 ;e­h<@ Ä&æÃëE%;X,ÒãÆÞ`Oò¦kŸm#˜!ÀyÄ¢| óLšò¥Ä` ¶R=|ÈCâh5ò3DˆïF†ðÒ#ÅìÛœ?¸yhBãœí ZxßÎÄhºRK„`Þödvײ™ÀÈÑÒgŒuY w³%†ƒÓzõ ÖÏp‚dH®¦A´ù§»ÓÇMæ~)ˆð‡û:ù&Ä •vGD´À n ݇¼Ö8Fö óáà£~Ë¥x`oK|Ä?fxiØü%pìR>éò+Û±éÎ>núlFŤ'tq8LZÏvÃ?„¡ß±È⽆¯³íü@x|PöUäèØã¡ð‚ŒAìÏ"vÍwóŸÍ{ ý0.z È•Ö{,N¡£¡ŸKÕÙž>Ýœþ ÍÀ°<×EA!Å‚D™IúOÍ¡>ôG}Â` ÍßkÜL™Ž Þð™ {IøF²¹òQ3&!ÃÂÞz.d&Ï-sH¸,Ôõ˜ŽP€ 77ˆÝ¼ÊëÜw =cÕ Ú,ØÐ5ÎYÐ)ì´öœgŒ[¤ßv㙑8心>h]§µháYš£²ºÑ.{Ï7Sð•?´~×SÃKýJÛ˜ ™Íäiúu<µX¶1õ^kâçIÑ£sZ4h>j*ÔšD:4­¿_ ÷¸ Õxæÿ ¸?Mù _•­ÊÐ ä ÷ý ÑwL œ­ïnTkÛUÍN©ë:¦fV ¶ÜÔÜMªÅâA½–¿R×TXš-%iTÊT•‡Ù‚JôϐZxWÑè‰f‰òG º ×Õû2aZ7OU3[“×AT–ÞŒ…-‘¤”Ì ì&(ˆ¿­•ƒkï’:ðY¦W‘ Å)“†‘˜³Åtcø˜ñTÂwÚÇ4|üLÇªí–v- qˆèU qPE.†â‘˜µ Æ,ÐÅs]8¾„oúÑ i>ÜxxÈó)ƒ ´æÁâØ$À‰vžŸf$Ž |ãw;ÀÁIJ»b` {¦Ó¤Ú$©YÀ‘n@Óïž«9J¼êG m¤ ܯ¹ÌW4€ÐÒÅÛ‡#褕Ÿn-?í|с¥÷Ú¹¬'´ÞÜ9ÓK `hê£SÄSà?7—Wí_´…óB›»:=Ãïq`<8ñÓŒÑlú2d¬ê³£hÖ[l|$vÝro~'R®‰§°ñmY ͧäP |PUª¹·:3Œ[Û{Xÿ ºâ@‚W–Äé u‚ ¯´*=íή.pûÒdt @G‰¬ s¸ ëÉücr ÞæѨÊ@>¤¢Ö±. Þ'¯°ÌME[YéïĵÂCå½ Ué©Áû'Ê9%eÔðNU”ë‘ÌsD3/®+UI˜9h.WC”빓$#:pz:YÓ ¿xž* ³$Í +$kñAŠ‹†¢ Uê>¸)_š¬÷©ßAÂÔb9ÇU ¯¾á•9¯ÏÏ÷O÷¼¼Fähal1‰3Ì[Ïr•´UCksNÐ] R‘¸¥H+§Šé†c©vÖÞ0iÓ76s†î!§=ß ¼~Ô'°Ãmäoäš³ªøi1úÉ)³yV8 CLÄØÁ‘WYïi€H6ÖÑiámø^ÈY´°Ñ7¥Û*—Ñ©L«Qƒï—Ùrÿ ›£Ð*š¸ˆL©ˆ$ˆ ÷¾D§9È®«qbqC)–ˆïv´çñsÑVT­Ø, <àïºÀO«Jý·õ àfPìð .wFšir´þ’2_Y *Æ€x\« ì€9š@ Ž|F⇥ˆkZ@hÖÄ0t¿-<“‹qµ¾*ZL¤Ú)&BJpÓF5=$„at*Zš$’ÑtdûÝRI1 2Ž‰$€$I$#‰SÞ’Hë¬ï;Á$¡t$’`<(ñÇt)$‡Ð.Êf¢X’Kt=Éé$‚ˆªè¢oÝëòI%Rgcª÷ŠyI%¡‰ÿ !ñ)´õ $¤ Ô’IIGÿÙimport asyncio import contextlib from concurrent.futures import ALL_COMPLETED from async_timeout import timeout as async_timeout from ..log import sentinel_logger from ..pubsub import Receiver from ..pool import create_pool, ConnectionsPool from ..errors import ( MasterNotFoundError, SlaveNotFoundError, PoolClosedError, RedisError, MasterReplyError, SlaveReplyError, ) from ..util import CloseEvent # Address marker for discovery _NON_DISCOVERED = object() _logger = sentinel_logger.getChild('monitor') async def create_sentinel_pool(sentinels, *, db=None, password=None, encoding=None, minsize=1, maxsize=10, ssl=None, parser=None, timeout=0.2, loop=None): """Create SentinelPool.""" # FIXME: revise default timeout value assert isinstance(sentinels, (list, tuple)), sentinels # TODO: deprecation note # if loop is None: # loop = asyncio.get_event_loop() pool = SentinelPool(sentinels, db=db, password=password, ssl=ssl, encoding=encoding, parser=parser, minsize=minsize, maxsize=maxsize, timeout=timeout, loop=loop) await pool.discover() return pool class SentinelPool: """Sentinel connections pool. Holds connection pools to known and discovered (TBD) Sentinels as well as services' connections. """ def __init__(self, sentinels, *, db=None, password=None, ssl=None, encoding=None, parser=None, minsize, maxsize, timeout, loop=None): # TODO: deprecation note # if loop is None: # loop = asyncio.get_event_loop() # TODO: add connection/discover timeouts; # and what to do if no master is found: # (raise error or try forever or try until timeout) # XXX: _sentinels is unordered self._sentinels = set(sentinels) self._timeout = timeout self._pools = [] # list of sentinel pools self._masters = {} self._slaves = {} self._parser_class = parser self._redis_db = db self._redis_password = password self._redis_ssl = ssl self._redis_encoding = encoding self._redis_minsize = minsize self._redis_maxsize = maxsize self._close_state = CloseEvent(self._do_close) self._close_waiter = None self._monitor = monitor = Receiver() async def echo_events(): try: while await monitor.wait_message(): _, (ev, data) = await monitor.get(encoding='utf-8') ev = ev.decode('utf-8') _logger.debug("%s: %s", ev, data) if ev in ('+odown',): typ, name, *tail = data.split(' ') if typ == 'master': self._need_rediscover(name) # TODO: parse messages; # watch +new-epoch which signals `failover in progres` # freeze reconnection # wait / discover new master (find proper way) # unfreeze reconnection # # discover master in default way # get-master-addr... # connnect # role # etc... except asyncio.CancelledError: pass self._monitor_task = asyncio.ensure_future(echo_events()) @property def discover_timeout(self): """Timeout (seconds) for Redis/Sentinel command calls during master/slave address discovery. """ return self._timeout def master_for(self, service): """Returns wrapper to master's pool for requested service.""" # TODO: make it coroutine and connect minsize connections if service not in self._masters: self._masters[service] = ManagedPool( self, service, is_master=True, db=self._redis_db, password=self._redis_password, encoding=self._redis_encoding, minsize=self._redis_minsize, maxsize=self._redis_maxsize, ssl=self._redis_ssl, parser=self._parser_class, ) return self._masters[service] def slave_for(self, service): """Returns wrapper to slave's pool for requested service.""" # TODO: make it coroutine and connect minsize connections if service not in self._slaves: self._slaves[service] = ManagedPool( self, service, is_master=False, db=self._redis_db, password=self._redis_password, encoding=self._redis_encoding, minsize=self._redis_minsize, maxsize=self._redis_maxsize, ssl=self._redis_ssl, parser=self._parser_class, ) return self._slaves[service] def execute(self, command, *args, **kwargs): """Execute sentinel command.""" # TODO: choose pool # kwargs can be used to control which sentinel to use if self.closed: raise PoolClosedError("Sentinel pool is closed") for pool in self._pools: return pool.execute(command, *args, **kwargs) # how to handle errors and pick other pool? # is the only way to make it coroutine? @property def closed(self): """True if pool is closed or closing.""" return self._close_state.is_set() def close(self): """Close all controlled connections (both sentinel and redis).""" if not self._close_state.is_set(): self._close_state.set() async def _do_close(self): # TODO: lock tasks = [] task, self._monitor_task = self._monitor_task, None task.cancel() tasks.append(task) while self._pools: pool = self._pools.pop(0) pool.close() tasks.append(pool.wait_closed()) while self._masters: _, pool = self._masters.popitem() pool.close() tasks.append(pool.wait_closed()) while self._slaves: _, pool = self._slaves.popitem() pool.close() tasks.append(pool.wait_closed()) await asyncio.gather(*tasks) async def wait_closed(self): """Wait until pool gets closed.""" await self._close_state.wait() async def discover(self, timeout=None): # TODO: better name? """Discover sentinels and all monitored services within given timeout. If no sentinels discovered within timeout: TimeoutError is raised. If some sentinels were discovered but not all — it is ok. If not all monitored services (masters/slaves) discovered (or connections established) — it is ok. TBD: what if some sentinels/services unreachable; """ # TODO: check not closed # TODO: discovery must be done with some customizable timeout. if timeout is None: timeout = self.discover_timeout tasks = [] pools = [] for addr in self._sentinels: # iterate over unordered set tasks.append(self._connect_sentinel(addr, timeout, pools)) done, pending = await asyncio.wait(tasks, return_when=ALL_COMPLETED) assert not pending, ("Expected all tasks to complete", done, pending) for task in done: result = task.result() if isinstance(result, Exception): continue # FIXME if not pools: raise Exception("Could not connect to any sentinel") pools, self._pools[:] = self._pools[:], pools # TODO: close current connections for pool in pools: pool.close() await pool.wait_closed() # TODO: discover peer sentinels for pool in self._pools: await pool.execute_pubsub( b'psubscribe', self._monitor.pattern('*')) async def _connect_sentinel(self, address, timeout, pools): """Try to connect to specified Sentinel returning either connections pool or exception. """ try: with async_timeout(timeout): pool = await create_pool( address, minsize=1, maxsize=2, parser=self._parser_class, ) pools.append(pool) return pool except asyncio.TimeoutError as err: sentinel_logger.debug( "Failed to connect to Sentinel(%r) within %ss timeout", address, timeout) return err except Exception as err: sentinel_logger.debug( "Error connecting to Sentinel(%r): %r", address, err) return err async def discover_master(self, service, timeout): """Perform Master discovery for specified service.""" # TODO: get lock idle_timeout = timeout # FIXME: single timeout used 4 times; # meaning discovery can take up to: # 3 * timeout * (sentinels count) # # having one global timeout also can leed to # a problem when not all sentinels are checked. # use a copy, cause pools can change pools = self._pools[:] for sentinel in pools: try: with async_timeout(timeout): address = await self._get_masters_address( sentinel, service) pool = self._masters[service] with async_timeout(timeout), \ contextlib.ExitStack() as stack: conn = await pool._create_new_connection(address) stack.callback(conn.close) await self._verify_service_role(conn, 'master') stack.pop_all() return conn except asyncio.CancelledError: # we must correctly handle CancelledError(s): # application may be stopped or function can be cancelled # by outer timeout, so we must stop the look up. raise except asyncio.TimeoutError: continue except DiscoverError as err: sentinel_logger.debug("DiscoverError(%r, %s): %r", sentinel, service, err) await asyncio.sleep(idle_timeout) continue except RedisError as err: raise MasterReplyError("Service {} error".format(service), err) except Exception: # TODO: clear (drop) connections to schedule reconnect await asyncio.sleep(idle_timeout) continue # Otherwise raise MasterNotFoundError("No master found for {}".format(service)) async def discover_slave(self, service, timeout, **kwargs): """Perform Slave discovery for specified service.""" # TODO: use kwargs to change how slaves are picked up # (eg: round-robin, priority, random, etc) idle_timeout = timeout pools = self._pools[:] for sentinel in pools: try: with async_timeout(timeout): address = await self._get_slave_address( sentinel, service) # add **kwargs pool = self._slaves[service] with async_timeout(timeout), \ contextlib.ExitStack() as stack: conn = await pool._create_new_connection(address) stack.callback(conn.close) await self._verify_service_role(conn, 'slave') stack.pop_all() return conn except asyncio.CancelledError: raise except asyncio.TimeoutError: continue except DiscoverError: await asyncio.sleep(idle_timeout) continue except RedisError as err: raise SlaveReplyError("Service {} error".format(service), err) except Exception: await asyncio.sleep(idle_timeout) continue raise SlaveNotFoundError("No slave found for {}".format(service)) async def _get_masters_address(self, sentinel, service): # NOTE: we don't use `get-master-addr-by-name` # as it can provide stale data so we repeat # after redis-py and check service flags. state = await sentinel.execute(b'sentinel', b'master', service, encoding='utf-8') if not state: raise UnknownService() state = make_dict(state) address = state['ip'], int(state['port']) flags = set(state['flags'].split(',')) if {'s_down', 'o_down', 'disconnected'} & flags: raise BadState(state) return address async def _get_slave_address(self, sentinel, service): # Find and return single slave address slaves = await sentinel.execute(b'sentinel', b'slaves', service, encoding='utf-8') if not slaves: raise UnknownService() for state in map(make_dict, slaves): address = state['ip'], int(state['port']) flags = set(state['flags'].split(',')) if {'s_down', 'o_down', 'disconnected'} & flags: continue return address raise BadState() # XXX: only last state async def _verify_service_role(self, conn, role): res = await conn.execute(b'role', encoding='utf-8') if res[0] != role: raise RoleMismatch(res) def _need_rediscover(self, service): sentinel_logger.debug("Must redisover service %s", service) pool = self._masters.get(service) if pool: pool.need_rediscover() pool = self._slaves.get(service) if pool: pool.need_rediscover() class ManagedPool(ConnectionsPool): def __init__(self, sentinel, service, is_master, db=None, password=None, encoding=None, parser=None, *, minsize, maxsize, ssl=None, loop=None): super().__init__(_NON_DISCOVERED, db=db, password=password, encoding=encoding, minsize=minsize, maxsize=maxsize, ssl=ssl, parser=parser, loop=loop) assert self._address is _NON_DISCOVERED self._sentinel = sentinel self._service = service self._is_master = is_master # self._discover_timeout = .2 @property def address(self): if self._address is _NON_DISCOVERED: return return self._address def get_connection(self, command, args=()): if self._address is _NON_DISCOVERED: return None, _NON_DISCOVERED return super().get_connection(command, args) async def _create_new_connection(self, address): if address is _NON_DISCOVERED: # Perform service discovery. # Returns Connection or raises error if no service can be found. await self._do_clear() # make `clear` blocking if self._is_master: conn = await self._sentinel.discover_master( self._service, timeout=self._sentinel.discover_timeout) else: conn = await self._sentinel.discover_slave( self._service, timeout=self._sentinel.discover_timeout) self._address = conn.address sentinel_logger.debug("Discoverred new address %r for %s", conn.address, self._service) return conn return await super()._create_new_connection(address) def _drop_closed(self): diff = len(self._pool) super()._drop_closed() diff -= len(self._pool) if diff: # closed connections were in pool: # * reset address; # * notify sentinel pool sentinel_logger.debug( "Dropped %d closed connnection(s); must rediscover", diff) self._sentinel._need_rediscover(self._service) async def acquire(self, command=None, args=()): if self._address is _NON_DISCOVERED: await self.clear() return await super().acquire(command, args) def release(self, conn): was_closed = conn.closed super().release(conn) # if connection was closed while used and not by release() if was_closed: sentinel_logger.debug( "Released closed connection; must rediscover") self._sentinel._need_rediscover(self._service) def need_rediscover(self): self._address = _NON_DISCOVERED def make_dict(plain_list): it = iter(plain_list) return dict(zip(it, it)) class DiscoverError(Exception): """Internal errors for masters/slaves discovery.""" class BadState(DiscoverError): """Bad master's / slave's state read from sentinel.""" class UnknownService(DiscoverError): """Service is not monitored by specific sentinel.""" class RoleMismatch(DiscoverError): """Service reported to have other Role."""