ÿØÿà JFIF    ÿÛ „ !.%+&8&+/1555$;@;4?.451 4,$,44444444444414444444444444444444444444444444444444ÿÀ  á á" ÿÄ     ÿÄ ?    !1AQaq"2‘¡±ÁðBRbrÑá#‚’¢²3S CñÿÄ   ÿÄ !    !1QAa‘2ÿÚ   ? 5˜Z¯V¦cø)›t/? z¨±>Õ5€¶‹Á¤·¼z¼Ü¬+ñ®v¤¨_ˆR­BFn©—˜ý®ç̝P8gýt·ÉSTŦˆìät?þé¼íìN/Þa)ì–í6ô… Ï¿øÃj´¿KÇü]ÿ ªô¹-eKànëÕHTx}ýSÜ›ÿ ”7Ø×&µ<¦  ¥ÑO¶[Ù¯ä¨ÞÃÿ PZ-¬;#õ|•oaÿ ©CìÞz3˜öː/¤­ñTûIØ}š^ mÓ%ªxˆ¥ÉŸu=Z+ISe¿45™¼u;ú&WØ÷€æßQ™®{|íx*TC“#ZŠìZ§²‹ 6pv…³¿¡äª*áZÐ%ÒOáˆo"x«OHk w±æ+¬V(kMúŸ5Vö«$ ÁrÏbàb57/luR ¸ÑÛj Òµì`Ðœq­û žICÀÊ•©4€Âcà¨Ï€O´<èÐ:›ù(Ë^L8þ‘ÍÌ#¸Ð_Ì©ÙK(Öz 4¬û+¸;ü’V’84‘¬ÃŽ:[â‡ÔÌáõp¢~§ªlæ£ö{®G>J¼"°‡7¯ÆÉèßû ‹É‹§ÁòÃýâßî ^ƾÙõ‹×óH#«LP½ïX=xÑÍ$|W?•~• îëÔ©ª‹ {ÝT…Kÿ ”hûâá)J*ö˜–ÔU;iÇ€/ ÆþjóZ\ýwØ=Ìm ºèËL9 ýèÆð/¨’¥öo=nË.%Îì ŽÕ¯È|{Oj²ƒE6e/ßdÄõ²Ìâ1O®ò×TsəԸhOMýíMˆ¿¼H˜l²,7Â¥#MF/Úf°Ö½± ¸–dr‹NýÊ íjqx{œÉ ä-È ¦ øÄër¨q°ð †nцýÑÄÆ’mä…n<0È™;ÁÝá¯ÁZƒ7FÀmì­ É&9ˆîéi¶ùN§Y• ÃZãAâ?•‡©‰ , ó¾IŸŠc1 4â&y­&pŠ­6;M À 0¹qç»p.á …ŸÅáK@%6·y6ƒ‰3?”úºŽ‰éX5ªPT §µ!=Mž«Ú½‹ÅgÂSâÉaþÓoö–¯ÁÔìR>5éÿ üs¶ÆUcÌ kÇR ]ÿ ù¬¼«VŽ;Â|‡~¢¦”ÏÅ°æ {L™Õ°Óv¹ò¸írŽ¡×¢CÃ!íVÕ {¶»sŒNPg/ "uÕbkm²“$ďå¿é¹§°½æz¯6 †s¿!s–wÚÝ“™Œ °.ûj>·+™Òa…©Œ&rÝÎtÛë긪Ît’LAVp%c Úý[ÄzJ¾ÇàXXç@˜ó<êL]·T˜¾¥1Ó©V‡g´æ½¦Ý@¹óø!_@´ÞâSÁ —S3™•& ]@JHÚý©ZŽ €×æÔr»Áf!‡yÞ4Mv*èÓã_{‘åóUuљØ«Oïé*®EvÑ Œ÷‡U \"㪒ÍK+À 4“M¡ï:0¥5í!'<@î´”>Ç»&Z–ïCCV˜Ì5Šo&îhè.žû |ÓK©h$s6KìŒëã)¹hI¦GïOåóI;ììü#É$Š0…Ææ¥TØ.5­¾gn´ “ÂÖ\:hœ89G)J@„}œ:’Ò{/Š"¦_Æ×7Æ3VÇŠÊa]ÚŒÙ€Ä–=®uÁßâACZƒ§§£ Qnâ:«,×{tyø¬iÛcœÜÄ€H½ÄÍCk´÷šß .W'b¤Íåh]÷€=,Žv×cÚEÚHXJX¶îo¨FÒtèöŸ>ªª6[J®Fµ£sGÁeqõfe\íjÒÐïÄÐGˆe1Ø‹.Ø”‘Ëuø Y­ˆÜ ŽG|zùªüMpDnQWÄ”%JŠ™)â*p@Örš«ÕT2Ð%ˆG#ª„ ·¤!°ŸOTÂT¸aÚ%4&h™LµšØüÐ.F¿²ÐÞ_Ç‚¾ÅÃaÜ÷09Æ q€öy˜v‡85õN÷]¬äѼóS{°_MŽ¬úÔ#°Ç¸0åÞè2ëôPcvÆw9®ií1Ä8F™˜à‰´+‰Ik1òÝ7“Ñ×ÒsÝ\x‚h`ÞÑ`ó"|µEcý£n˜h`}GÞ !±ù²Ápü²ß6 0ïi󜵩SÈÇ7˜-ÕURO˜¦´f$ªž-Í6(œ}<„ éc øs]ŽŽ„*—¾ ìdŽ„)méª\¿êÎIg¾ØÞ~I#C/¼¼´EÁÈŽi8“©õådô·>euä ƒ'Ê×लR1ÉJE1ÐAát`t;ÇР%Ý<‡¥„ÍÆ`×Oyó)õiI€ñQaŸ4Ûù\áàaÃÔ¹HÃu¹*k€¦<„e S‡&õÏ B!ŽhüÞ`yj}mªf×\¿ Ç~æ­9‡û\՞Ǖg²1Žû5V7 !àöšm° c`ܬøÇìµÒ'P"?…´Ö,"§^•õŽ¨sÔ)6˜sæéÍR¼ ò|Sl”‹7 nPW Gòú÷½§O¯‡„l¡kSÞŒr½PÊ@æ¢pŽ-mÿ #Ÿ˜Àº¶Áä¦;ïÔæ$1££`“Õ>„—·ž)ßð³ñ#Ï Ô$¶œ‰ÊE‹À;÷º ¯«P:Ñ”8–IÊtpÞ3ª“>ê“þës4ò2OÏÕ­±zô†Õ§‰.÷ä¸;¿˜“'œ›žª}«Œ{ª±Ì 9ÔóÞÕ‡0 $íWV3Üì¬ —@kÝ4@¿r¼±½¬™›?øØæ´'Áé®CË3-g$˜ö‡×auÚi´Žp/êÛ æF›Ú2v‹ã¿¿,nB1̨ƃqÞa5͝@&Æû“él÷ \C²½UÍc ¯k×¢U ÖéQå™—-r wô ÞÏ<Ò=&=ÿ Ôê Òêˈt,i—;LîÜ á¸*ÚÃ1$êL•LÍ <É)ýÐà’ ;F™{ƒ™˜€&'}‚ãÄK`¡ÞT@I;®žZóè‚s’7®°›+§O­Åq©é»²9<Ô J ¼9O’HL»Ùïì¸rk¼Ž_ý‘TŸu[²ßÚŒ·ü÷B%¯E ŸÔX5êO´ Ç•€’I0 ÉJX` ñ¹õ%;µŸD‘«´€àwÒ™U ûئžÖö\×®×´8 ½‡ºÐÆÓ§?Àkmœ=;d5*@-ì0F Rªýš[Ü6âö̃ڸr*KA9· u*µæ£?U¸Âêí†8@¦X4 e-ò„0s{ HâUpU?¼mñRa°®a%Ð'tÉ×’\¾ÊÉ]t›h>·(Ë@R¼¡Ãt h}’O÷au<+nT…Ö…MӐ??Óe95 q>í/;&JSû °¯ÊéÞ øƒ*Ã2½Ài&:nôUl=¾¿5eˆ3”ñc|Ú2V”>„»&eE;«ÚäC p¢Û úy 9š[ŒÌx¼擼A&DåÒ¯ˆ¤ÀÌ;"˜ ÏQä¸åhÊ}Ûq«Û0WžÒ|»€ø®öCm5•\ÇÀ§Pe3£]0ÃàLDÉ‰1øªxjgwT‚÷¿LΨK‹›ùs—xˆÜ±µ kæ¸f‰‰ÜGk/LÛØ6d9ò¶ùA{ƒA3š/¬D¬khÓk‰`˜"㯒r¿±Óã jx‡°e}<Ñø\3y:'À•/h½Í€Ç4~g ?Û(¼]v‘ªlKÎâ~?O‚W%{Ì:“'©úNq¾›úo(X’¥¯ˆ nFê{Ç€ü?º'ë ø‹ì Þ09ŒÌç9Æ —ËC`j@ÓÄ(+a‹un¸#ÂꟋ{K`‘ÑÍÍ'à´»/Û,KW;Þ4²þð ï Nm|~fGÏ(…³Ã)«1ö­Õ ¥‡¨©ƒÃ™ü-s=à=U66Ï«Ýc蓦W¹íž®›nÔ%êÇìŒ<#Ü×84ån®Ð ÒåOC` ñânÑs‡¢ç 1õ%Îhì½Ã½® e:ݼUZo™`  ÅZŸŒÊ«ê1ÏÄo$q¹Þ€©ˆhÐÉä¯ñ[!…Ú˜àJ:x2$Íß&PåT£6ç— ‡Í*4Ýšçjÿ ‰É nófÐ ó(L5C•åÆ\rMÒ@ò }y-W}™üýVù—ú¢=Ù”c®‘< M ž ´Phr ¦©TD ‘ù.$´÷O‡‘V2Æò.=IUŒ=ž‡â¬i™aþÓåÙ?òUø'ØÖ•.~* šTŒ!•-×áºTâ®ä#õü'´ eýlYÅÓeÕKÂrT"CÚ@u!Óxƒ{š3€}1¿(r}%«nËamjÑ%ÑNEò v ˜à  σöK³,*º.àzù¨™Ó ÚçâU¦*¿ 9{%Ö¹ njûdaXöb) kÛƱûÓ\°M7ˆÂ=û›ç¿Ã‚­V»Cg–8ÙêE- j)k$º`Ã-ùEýeBÆÇ]c¡°ñty&Òd0nõ'¡W+ƒ*|–øµFa\GQªEAÔp5\Ǽ·¼Ç8·õ -â§Ú[ ‡ uZeÖ 3}×d'+¹:ð+K†Û®s!Ï$úe€<Û”x)1»a­¡LC]¸µík…ÚàA»AYº{†ªS[¦5HÒ7ù --,ísòDØ€èk ÞÀîÜ ò@â( ËNˆë›4ô½•/¦o‡€Û7 ê•ÆêòðÜy'Án½µ á˜ݦ ndeo…[ì¶Ê,¥R³Ä=À±—–ß;£™´ñSâ*g§”ïaið‘Jå~™ÓÞ ß³Õ¢»8x埒²52>AÊb&-÷\7´éÄù€T˜,w;3{ï˜k…à¹ÄqÀ«œ{€\ ˆ¾[´¨јr &Úé„Ívˆ±8†¿]|¬ņ4I×pÞS1ÈÖz‰#Ìv‡G!YNògñ:màTz¢Ý1ô©^O=~ë|5Bã™ç•¼µõ•bÆ@úÕS¬ÈŒ#¬zünrŸ û” Z²•ï›èðV"ÁHÚý©wÝ €7¼Ìu1hÑa3Éä û f$o¿É ™Ú›ÝçnpÒ3äÌ3†Í§,Äï]$‰/pê †«À¼¸e9­Æê_C]žƒ·ý·frÁN«, E=›Çq -‰öŒ:aÏ¿±í&£Í:-} 84‘ÿ eƒQÑeëSsuiA ³g㟥ú£?ÿ ʼn*”“÷aühe:ÊWa@ÒÞk±eØ] F Ô—r.åä˜ @ö¥ªZoÐýYL·¥S²G/‡ñ <~*ZÆ´è>JlòàÛƽÿ 窘ìGN¢:I®KšJp/`íIÁÀõ#Ä-€ö­šµŒoF4|ÆQØÆ@Ì|£Ô…¢À{9˜è½Üó›€ôYÒÎYsið;ís¤€à²ˆ‚4qÉVŒI$ ‰"° æµ8cXGjœˏ¡Aâý•ËÜ¢ûï e·çLx']á"oÅÎê3¯Ç—¹”ó0nå‚âg{Œñ> S´˜îè°g238‚ãköÝfÚd´6Ò€;ò÷±¢™¼›º ¢Æ'¥Ðx'e¬ç ]bÈÆV¢ó‹kýBO ðÊâ$Ÿ!×T 3Mýמ žìٍàÌü‘8÷€àæØ8æ©6‰©L´«…oãpð„~Çk‰!ñ;‹”ÛžÍ àž±z Ÿôû øŸÝužÏ;ÿ #|u6™Þ¬ÚˆÐõA4¶â|ôl|Ê2ŽÇ¤ÝÅÇY.<#Aí.k§hóF‚”Y; M½Ö4hŸ4&›­¿tès´%FìL¥£Ãk‰ÇT¤haÁ¤ÚxfÉ`ÑìË›>i 3t‚:,–+^÷´–{Û–Nxi"x‘Ûg î¨>¥Õ܁ùZH,2Û“:8xÊ¢Çí9.É-Ìâã-=çjwµS˜dütžçwýGòú®®ûº_ˆýx$–¡ãøO EÚÛÏ÷R„×w+3£Á£öUMyR²¹âŒ°š›¸Ñãò9§Ó_Dl+Ùßc›úšGÅÌc†Ž!Ko=¶.‘Îÿ c²(2®V mª.ÿ ¹B›¹å ù„öŸSV>™ü¯$y:G¢Z×àøúdî¹û­·ýÇ´:•c LÍõi_‹ö+ÎæGÊè>OŠ•äž´§Þ{X}¨1ÚTc›»Qþ•êô°t¿OP?eæ~É{5]•ÙR£r5†nZ\ã@ &îJõ ¾àC°þV>fé¥/ü5ñÊIº_é5 ;e­h<@ Ä&æÃëE%;X,ÒãÆÞ`Oò¦kŸm#˜!ÀyÄ¢| óLšò¥Ä` ¶R=|ÈCâh5ò3DˆïF†ðÒ#ÅìÛœ?¸yhBãœí ZxßÎÄhºRK„`Þödvײ™ÀÈÑÒgŒuY w³%†ƒÓzõ ÖÏp‚dH®¦A´ù§»ÓÇMæ~)ˆð‡û:ù&Ä •vGD´À n ݇¼Ö8Fö óáà£~Ë¥x`oK|Ä?fxiØü%pìR>éò+Û±éÎ>núlFŤ'tq8LZÏvÃ?„¡ß±È⽆¯³íü@x|PöUäèØã¡ð‚ŒAìÏ"vÍwóŸÍ{ ý0.z È•Ö{,N¡£¡ŸKÕÙž>Ýœþ ÍÀ°<×EA!Å‚D™IúOÍ¡>ôG}Â` ÍßkÜL™Ž Þð™ {IøF²¹òQ3&!ÃÂÞz.d&Ï-sH¸,Ôõ˜ŽP€ 77ˆÝ¼ÊëÜw =cÕ Ú,ØÐ5ÎYÐ)ì´öœgŒ[¤ßv㙑8心>h]§µháYš£²ºÑ.{Ï7Sð•?´~×SÃKýJÛ˜ ™Íäiúu<µX¶1õ^kâçIÑ£sZ4h>j*ÔšD:4­¿_ ÷¸ Õxæÿ ¸?Mù _•­ÊÐ ä ÷ý ÑwL œ­ïnTkÛUÍN©ë:¦fV ¶ÜÔÜMªÅâA½–¿R×TXš-%iTÊT•‡Ù‚JôϐZxWÑè‰f‰òG º ×Õû2aZ7OU3[“×AT–ÞŒ…-‘¤”Ì ì&(ˆ¿­•ƒkï’:ðY¦W‘ Å)“†‘˜³Åtcø˜ñTÂwÚÇ4|üLÇªí–v- qˆèU qPE.†â‘˜µ Æ,ÐÅs]8¾„oúÑ i>ÜxxÈó)ƒ ´æÁâØ$À‰vžŸf$Ž |ãw;ÀÁIJ»b` {¦Ó¤Ú$©YÀ‘n@Óïž«9J¼êG m¤ ܯ¹ÌW4€ÐÒÅÛ‡#褕Ÿn-?í|с¥÷Ú¹¬'´ÞÜ9ÓK `hê£SÄSà?7—Wí_´…óB›»:=Ãïq`<8ñÓŒÑlú2d¬ê³£hÖ[l|$vÝro~'R®‰§°ñmY ͧäP |PUª¹·:3Œ[Û{Xÿ ºâ@‚W–Äé u‚ ¯´*=íή.pûÒdt @G‰¬ s¸ ëÉücr ÞæѨÊ@>¤¢Ö±. Þ'¯°ÌME[YéïĵÂCå½ Ué©Áû'Ê9%eÔðNU”ë‘ÌsD3/®+UI˜9h.WC”빓$#:pz:YÓ ¿xž* ³$Í +$kñAŠ‹†¢ Uê>¸)_š¬÷©ßAÂÔb9ÇU ¯¾á•9¯ÏÏ÷O÷¼¼Fähal1‰3Ì[Ïr•´UCksNÐ] R‘¸¥H+§Šé†c©vÖÞ0iÓ76s†î!§=ß ¼~Ô'°Ãmäoäš³ªøi1úÉ)³yV8 CLÄØÁ‘WYïi€H6ÖÑiámø^ÈY´°Ñ7¥Û*—Ñ©L«Qƒï—Ùrÿ ›£Ð*š¸ˆL©ˆ$ˆ ÷¾D§9È®«qbqC)–ˆïv´çñsÑVT­Ø, <àïºÀO«Jý·õ àfPìð .wFšir´þ’2_Y *Æ€x\« ì€9š@ Ž|F⇥ˆkZ@hÖÄ0t¿-<“‹qµ¾*ZL¤Ú)&BJpÓF5=$„at*Zš$’ÑtdûÝRI1 2Ž‰$€$I$#‰SÞ’Hë¬ï;Á$¡t$’`<(ñÇt)$‡Ð.Êf¢X’Kt=Éé$‚ˆªè¢oÝëòI%Rgcª÷ŠyI%¡‰ÿ !ñ)´õ $¤ Ô’IIGÿÙimport logging import os import sys try: import psutil except: psutil = None import math import re import agent_util DOCKER_SOCKET = "/var/run/docker.sock" NANOSECONDS = 1000000000 CLOCK_TICKS = 100 class DockerPlugin(agent_util.Plugin): textkey = "docker" label = "Docker" ######################################################### # Metadata # ######################################################### @classmethod def is_cgroups_v2(self): return os.path.isfile("/sys/fs/cgroup/cgroup.controllers") @classmethod def map_cgroup_v2_io_textkey_to_metric(self, textkey): textkey_to_metric = { "io.bytes_read": "rbytes", "io.bytes_written": "wbytes", "io.read_ops": "rios", "io.write_ops": "wios", } return textkey_to_metric.get(textkey, None) @classmethod def get_metadata(self, config): status = agent_util.SUPPORTED if not agent_util.which("docker"): self.log.info("docker not present") status = agent_util.UNSUPPORTED msg = "Docker binary not found on instance" return {} return { "containers.num_running": { "label": "Number of containers running", "options": None, "status": status, "error_message": "", "unit": "count", }, "containers.num_running_img": { "label": "Number of containers running image", "options": None, "option_string": True, "status": status, "error_message": "", "unit": "count", }, "containers.num_running_name": { "label": "Number of containers running by name", "options": None, "option_string": True, "status": status, "error_message": "", "unit": "count", }, } @classmethod def get_metadata_docker(self, container, config): status = agent_util.SUPPORTED msg = None metadata = {} metadata.update(self.get_cpu_metadata(container, config)) metadata.update(self.get_memory_metadata(container, config)) metadata.update(self.get_network_metadata(container, config)) metadata.update(self.get_io_metadata(container, config)) metadata.update( { # Container is running "status.running": { "label": "Container is Running", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "boolean", }, # Disk metrics are always available "disk.size_rw": { "label": "Size RW", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "disk.size_root_fs": { "label": "Size Root FS", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, } ) return metadata @classmethod def get_cpu_metadata(self, container, config): container_id = container["Id"] cpu_metadata = { "cpu.usage_percentage": { "label": "CPU Usage Percentage", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "%", }, "cpu.user_usage": { "label": "CPU Percent Used [User]", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "%", }, "cpu.sys_usage": { "label": "CPU Percent Used [System]", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "%", }, } if self.is_cgroups_v2(): stat_file = "/sys/fs/cgroup/system.slice/docker-{}.scope/cpu.stat".format( container_id ) stats = DockerPlugin.read_stats_from_file(stat_file) textkey_map = { "cpu.usage_percentage": "usage_usec", "cpu.user_usage": "user_usec", "cpu.sys_usage": "system_usec", } for key in cpu_metadata.keys(): if stats.get(textkey_map[key], None) is None: cpu_metadata[key]["error_message"] = ( "Cannot access docker stats file" ) cpu_metadata[key]["status"] = agent_util.UNSUPPORTED else: # map textkey to docker interface file used for metrics textkey_map = { "cpu.usage_percentage": "cpuacct.usage", "cpu.user_usage": "cpuacct.usage_user", "cpu.sys_usage": "cpuacct.usage_sys", } for textkey in cpu_metadata.keys(): file_name = textkey_map.get(textkey, None) if file_name is None: self.log.warning( "Docker CPU metadata: missing map key {}".format(textkey) ) continue fpath = "/sys/fs/cgroup/cpuacct/docker/{}/{}".format( container_id, file_name ) metric = DockerPlugin.read_single_stat_file(fpath) if metric is None: cpu_metadata[textkey]["status"] = agent_util.UNSUPPORTED cpu_metadata[textkey]["error_msg"] = "Can't access '{}'".format( file_name ) return cpu_metadata @classmethod def read_single_stat_file(self, file_name): try: with open(file_name, "r") as f: metric = f.read() return float(metric) except Exception: self.log.exception("Read stat file {} failure".format(file_name)) return None @classmethod def read_stats_from_file(self, file_name): try: with open(file_name, "r") as f: output = f.readlines() stats = {} for line in output: stat_type, num = line.split(" ") stats[stat_type] = float(num) return stats except Exception: self.log.exception("Read stats from {} failure".format(file_name)) return {} @classmethod def read_io_stats_v2(self, container): result = {} try: successes = 0 identifier = None container_stats = ( "/sys/fs/cgroup/system.slice/docker-{}.scope/io.stat".format(container) ) with open(container_stats, "r") as cf: line = cf.read() identifier = line.split()[0] metric_line_split = None with open("/sys/fs/cgroup/io.stat", "r") as sf: lines = sf.readlines() for line in lines: items = line.split(" ") if items[0] == identifier: result["identifier"] = items[0] metric_line_split = items break if metric_line_split is None: raise Exception("No identifier for container") else: for item in metric_line_split[1:]: metric_and_value = item.strip("\n").split("=") if 2 == len(metric_and_value): try: result[metric_and_value[0]] = int(metric_and_value[1]) except: pass except Exception: _, e = sys.exc_info()[:2] result["error_msg"] = str(e) return result @classmethod def get_memory_metadata(self, container, config): container_id = container["Id"] if self.is_cgroups_v2(): memory_metadata = { "memory.usage": { "label": "Memory Used", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "memory.mapped_file": { "label": "Memory Mapped File", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, } stats_file = ( "/sys/fs/cgroup/system.slice/docker-{}.scope/memory.stat".format( container_id ) ) metrics = DockerPlugin.read_stats_from_file(stats_file) if metrics.get("file_mapped", None) is None: memory_metadata["memory.mapped_file"]["error_message"] = ( "Cannot read {}".format(stats_file) ) memory_metadata["memory.mapped_file"]["status"] = agent_util.UNSUPPORTED memory_current = ( "/sys/fs/cgroup/system.slice/docker-{}.scope/memory.current".format( container_id ) ) metric = DockerPlugin.read_single_stat_file(memory_current) if metric is None: memory_metadata["memory.usage"]["error_message"] = ( "Cannot read {}".format(stats_file) ) memory_metadata["memory.usage"]["status"] = agent_util.UNSUPPORTEDev return memory_metadata memory_metadata = { "memory.usage": { "label": "Memory Used", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "memory.cache": { "label": "Memory Cached", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "memory.rss": { "label": "Memory RSS", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "memory.mapped_file": { "label": "Memory Mapped File", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, "memory.swap": { "label": "Swap Used", "options": None, "status": agent_util.SUPPORTED, "error_message": "", "unit": "bytes", }, } total_metric = self.read_single_stat_file( "/sys/fs/cgroup/memory/docker/%s/memory.usage_in_bytes" % container_id ) if total_metric is None: memory_metadata["memory.usage"]["status"] = agent_util.UNSUPPORTED memory_metadata["memory.usage"]["error_message"] = ( "Can't access 'memory.usage_in_bytes'" ) stats = self.read_stats_from_file( "/sys/fs/cgroup/memory/docker/%s/memory.stat" % container_id ) for key in memory_metadata.keys(): if "memory.usage" == key: continue metric_key = key.split(".")[1] if metric_key not in stats: memory_metadata[key]["status"] = agent_util.UNSUPPORTED memory_metadata[key]["error_msg"] = "Unable to read stats file" return memory_metadata @classmethod def find_physical_ethernet_interface(self): try: cmd = """ find /sys/class/net -type l -not -lname '*virtual*' -printf '%f\n' """ rc, out = agent_util.execute_command(cmd) if 0 != rc: raise Exception("Non-zero rc") return out.strip("\n") except: return "eth0" @classmethod def get_network_metadata(self, container, config): container_id = container["Id"] status = agent_util.UNSUPPORTED msg = "" # Get the PID try: conn = agent_util.UnixHTTPConnection(DOCKER_SOCKET) conn.request( "GET", "/containers/%s/json" % container_id, headers={"Host": "localhost"}, ) r = conn.getresponse().read() j = agent_util.json_loads(r) container_pid = j["State"]["Pid"] except Exception: container_pid = None msg = "Can't get container's PID" if container_pid: phys_eth = "{}:".format(self.find_physical_ethernet_interface()) try: with open("/proc/%s/net/dev" % container_pid, "r") as f: output = f.readlines() eth0 = False for line in output: if line.lstrip().startswith(phys_eth): eth0 = True split = line.split() if len(split) == 17: status = agent_util.SUPPORTED else: msg = "Unexpected # of columns in /proc//net/dev" break if not eth0: msg = "Can't find {} device on container".format(phys_eth) except Exception: msg = "Can't access /proc//net/dev" return { "net.rx_bytes": { "label": "Bytes In Per Second", "options": None, "status": status, "error_message": msg, "unit": "bytes/sec", }, "net.rx_packets": { "label": "Packets In Per Second", "options": None, "status": status, "error_message": msg, "unit": "packets/sec", }, "net.rx_errs": { "label": "RX Errors Per Second", "options": None, "status": status, "error_message": msg, "unit": "errors/sec", }, "net.tx_bytes": { "label": "Bytes Out Per Second", "options": None, "status": status, "error_message": msg, "unit": "bytes/sec", }, "net.tx_packets": { "label": "Packets Out Per Second", "options": None, "status": status, "error_message": msg, "unit": "packets/sec", }, "net.tx_errs": { "label": "TX Errors Per Second", "options": None, "status": status, "error_message": msg, "unit": "errors/sec", }, } @classmethod def get_io_metadata(self, container, config): io_metadata = { "io.bytes_written": { "label": "Bytes Written Per Second", "options": None, "status": agent_util.UNSUPPORTED, "error_message": None, "unit": "bytes/s", }, "io.bytes_read": { "label": "Bytes Read Per Second", "options": None, "status": agent_util.UNSUPPORTED, "error_message": None, "unit": "bytes/s", }, "io.write_ops": { "label": "Writes Per Second", "options": None, "status": agent_util.UNSUPPORTED, "error_message": None, "unit": "w/s", }, "io.read_ops": { "label": "Reads Per Second", "options": None, "status": agent_util.UNSUPPORTED, "error_message": None, "unit": "r/s", }, } container_id = container["Id"] if self.is_cgroups_v2(): cgv2_status = agent_util.UNSUPPORTED cgv2_msg = "" io_stats_data = self.read_io_stats_v2(container_id) cgv2_msg = io_stats_data.get("eror_msg", None) if cgv2_msg is not None: for v in io_metadata.values(): v["status"] = cgv2_status v["error_message"] = cgv2_msg return io_metadata for textkey in io_metadata.keys(): metric_key = self.map_cgroup_v2_io_textkey_to_metric(textkey) if metric_key is None: io_metadata[textkey]["error_message"] = ( "Unknown metric from {}".format(textkey) ) continue if io_stats_data.get(metric_key, None) is not None: io_metadata[textkey]["status"] = agent_util.SUPPORTED else: io_metadata[textkey]["error_message"] = "Unable to read i/o stats" return io_metadata service_bytes = agent_util.UNSUPPORTED service_bytes_message = "" try: fpath = ( "/sys/fs/cgroup/blkio/docker/%s/blkio.throttle.io_service_bytes" % container_id ) with open(fpath, "r") as f: f.read() service_bytes = agent_util.SUPPORTED except Exception: service_bytes_message = "Can't access 'blkio.throttle.io_service_bytes'" for tk in ["io.bytes_written", "io.bytes_read"]: io_metadata[tk]["status"] = service_bytes io_metadata[tk]["error_message"] = service_bytes_message operations = agent_util.UNSUPPORTED operations_message = "" try: fpath = ( "/sys/fs/cgroup/blkio/docker/%s/blkio.throttle.io_serviced" % container_id ) with open(fpath, "r") as f: f.read() operations = agent_util.SUPPORTED except Exception: operations_message = "Can't access 'blkio.throttle.io_serviced'" for tk in ["io.write_ops", "io.read_ops"]: io_metadata[tk]["status"] = operations io_metadata[tk]["error_message"] = operations_message return io_metadata ######################################################### # Checks # ######################################################### def check(self, textkey, data, config): if textkey.startswith("containers."): return self.get_containers_metric(textkey, data, config) def get_containers_metric(self, textkey, data, config): def get_running_containers(): try: conn = agent_util.UnixHTTPConnection(DOCKER_SOCKET) conn.request("GET", "/containers/json", headers={"Host": "localhost"}) r = conn.getresponse().read() j = agent_util.json_loads(r) return j except Exception: self.log.exception("Get running containers error") return None if textkey == "containers.num_running": running = get_running_containers() return len(running) elif textkey == "containers.num_running_img": running = get_running_containers() search = data or "*" search = search.replace("*", ".*") search = search.replace('""', ".*") count = 0 for container in running: image = container.get("Image", "") if re.search(search, image): count += 1 return count elif textkey == "containers.num_running_name": running = get_running_containers() search = data or "*" search = search.replace("*", ".*") search = search.replace('""', ".*") count = 0 for container in running: names = container.get("Names", []) for name in names: if re.search(search, name): count += 1 return count def check_docker(self, container, textkey, data, config): if textkey.startswith("cpu."): return self.get_cpu_metric(container, textkey, data, config) elif textkey.startswith("memory."): return self.get_memory_metric(container, textkey, data, config) elif textkey.startswith("net."): return self.get_network_metric(container, textkey, data, config) elif textkey.startswith("disk."): return self.get_disk_metric(container, textkey, data, config) elif textkey.startswith("io."): return self.get_io_metric(container, textkey, data, config) elif textkey.startswith("status."): return self.get_status_metric(container, textkey, data, config) return None def _read_cpu_metric(self, textkey, container_id): if self.is_cgroups_v2(): stat_file = "/sys/fs/cgroup/system.slice/docker-{}.scope/cpu.stat".format( container_id ) stats = DockerPlugin.read_stats_from_file(stat_file) if "cpu.usage_percentage" == textkey: return stats.get("usage_usec", None) elif "cpu.user_usage" == textkey: return stats.get("user_usec", None) elif "cpu.sys_usage" == textkey: return stats.get("system_usec", None) self.log.warning( "Unrecognized textkey {} in _read_cpu_metric".format(textkey) ) return None stat_file = None base_path = "/sys/fs/cgroup/cpuacct/docker/{}".format(container_id) if "cpu.usage_percentage" == textkey: stat_file = os.path.join(base_path, "cpuacct.usage") elif "cpu.user_usage" == textkey: stat_file = os.path.join(base_path, "cpuacct.usage_user") elif "cpu.sys_usage" == textkey: stat_file = os.path.join(base_path, "cpuacct.usage_sys") if stat_file is None: self.log.error( "Unrecognized textkey {} in _read_cpu_metric".format(textkey) ) return None return DockerPlugin.read_single_stat_file(stat_file) def get_cpu_metric(self, container, textkey, data, config): container_id = container["Id"] def get_total_system(): cpu_times = psutil.cpu_times() total_system = 0 for key in ["user", "nice", "system", "idle", "iowait", "irq", "softirq"]: total_system += getattr(cpu_times, key) * 100 total_system = (total_system * NANOSECONDS) / CLOCK_TICKS if self.is_cgroups_v2(): total_system = total_system / 1000 return total_system if textkey == "cpu.usage_percentage": last_system = self.get_cache_results( "docker.cpu.usage_percentage", "total_system" ) if last_system: last_system = last_system[0][1] else: last_system = None last_container = self.get_cache_results( "docker.cpu.usage_percentage", container_id ) if last_container: last_container = last_container[0][1] else: last_container = None total_system = get_total_system() self.cache_result( "docker.cpu.usage_percentage", "total_system", total_system, replace=True, ) total_container = self._read_cpu_metric(textkey, container_id) if total_container is None: return None self.cache_result( "docker.cpu.usage_percentage", container_id, total_container, replace=True, ) if last_system is None or last_container is None: return None container_delta = total_container - last_container system_delta = total_system - last_system num_cpus = psutil.cpu_count() return (float(container_delta) / system_delta) * num_cpus * 100.0 elif textkey == "cpu.user_usage": last_system = self.get_cache_results( "docker.cpu.user_usage", "total_system" ) if last_system: last_system = last_system[0][1] else: last_system = None last_container = self.get_cache_results( "docker.cpu.user_usage", container_id ) if last_container: last_container = last_container[0][1] else: last_container = None total_system = get_total_system() self.cache_result( "docker.cpu.user_usage", "total_system", total_system, replace=True ) container_val = self._read_cpu_metric(textkey, container_id) if container_val is None: return None self.cache_result( "docker.cpu.user_usage", container_id, container_val, replace=True ) if last_system is None or last_container is None: return None container_delta = container_val - last_container system_delta = total_system - last_system num_cpus = psutil.cpu_count() return (float(container_delta) / system_delta) * num_cpus * 100.0 elif textkey == "cpu.sys_usage": last_system = self.get_cache_results("docker.cpu.sys_usage", "total_system") if last_system: last_system = last_system[0][1] else: last_system = None last_container = self.get_cache_results( "docker.cpu.sys_usage", container_id ) if last_container: last_container = last_container[0][1] else: last_container = None total_system = get_total_system() self.cache_result( "docker.cpu.sys_usage", "total_system", total_system, replace=True ) container_val = self._read_cpu_metric(textkey, container_id) if container_val is None: return None self.cache_result( "docker.cpu.sys_usage", container_id, container_val, replace=True ) if last_system is None or last_container is None: return None container_delta = container_val - last_container system_delta = total_system - last_system num_cpus = psutil.cpu_count() return (float(container_delta) / system_delta) * num_cpus * 100.0 def get_memory_metric(self, container, textkey, data, config): container_id = container["Id"] def get_total_bytes(): fname = ( "/sys/fs/cgroup/memory/docker/%s/memory.usage_in_bytes" % container_id ) if self.is_cgroups_v2(): fname = ( "/sys/fs/cgroup/system.slice/docker-{}.scope/memory.current".format( container_id ) ) return DockerPlugin.read_single_stat_file(fname) def get_memory_stats(): fname = "/sys/fs/cgroup/memory/docker/%s/memory.stat" % container_id if self.is_cgroups_v2(): fname = ( "/sys/fs/cgroup/system.slice/docker-{}.scope/memory.stat".format( container_id ) ) return DockerPlugin.read_stats_from_file(fname) if textkey == "memory.usage": try: total_bytes = get_total_bytes() if self.is_cgroups_v2(): return total_bytes memory_stats = get_memory_stats() return total_bytes - memory_stats["cache"] except Exception: self.log.exception("Docker get memory.usage error") return None elif textkey in [ "memory.cache", "memory.rss", "memory.mapped_file", "memory.swap", ]: try: memory_stats = get_memory_stats() if not self.is_cgroups_v2(): stat_type = textkey.split(".")[1] return memory_stats[stat_type] if "memory.mapped_file" == textkey: return memory_stats["file_mapped"] raise Exception("Unrecognized textkey {}".format(textkey)) except Exception: self.log.exception("Docker get {} error".format(textkey)) return None def get_container_pid(self, container): conn = None try: container_id = container["Id"] conn = agent_util.UnixHTTPConnection(DOCKER_SOCKET) conn.request( "GET", "/containers/%s/json" % container_id, headers={"Host": "localhost"}, ) r = conn.getresponse().read() j = agent_util.json_loads(r) return j["State"]["Pid"] except Exception: self.log.exception("Get container pid error") return None finally: try: conn.close() except: pass def get_network_metric(self, container, textkey, data, config): container_id = container["Id"] # Find the container's PID container_pid = self.get_container_pid(container) if container_pid is None: return None phys_eth = "{}:".format(DockerPlugin.find_physical_ethernet_interface()) def get_proc_stats(pid): proc_file = "/proc/%s/net/dev" % pid with open(proc_file, "r") as f: content = f.readlines() eth0_line = None for line in content: if line.lstrip().startswith(phys_eth): eth0_line = line break if not eth0_line: raise Exception("No line for {} in {}".format(phys_eth, proc_file)) eth0_line = eth0_line.split() keys = [ "", "rx_bytes", "rx_packets", "rx_errs", "", "", "", "", "", "", "tx_bytes", "tx_packets", "tx_errs", "", "", "", "", "", ] stats = {} for col, text in enumerate(eth0_line): key = keys[col] if key: stats[key] = int(text) return stats if textkey in [ "net.rx_bytes", "net.rx_packets", "net.rx_errs", "net.tx_bytes", "net.tx_packets", "net.tx_errs", ]: key = textkey.split(".")[1] last = self.get_cache_results("docker.net", key) if last: last_val = last[0][1] seconds = last[0][0] else: last_val = None seconds = None try: stats = get_proc_stats(container_pid) stat = stats[key] except Exception: self.log.exception( "Error accessing /proc/%s/net/dev: %s", container_pid, e ) return None self.cache_result("docker.net", key, stat, replace=True) if last_val is None: return None return (stat - last_val) / seconds def get_disk_metric(self, container, textkey, data, config): container_id = container["Id"] try: conn = agent_util.UnixHTTPConnection(DOCKER_SOCKET) conn.request( "GET", "/containers/%s/json?size=true" % container_id, headers={"Host": "localhost"}, ) r = conn.getresponse().read() j = agent_util.json_loads(r) except Exception: self.log.exception("Docker get disk metric error") return None if textkey == "disk.size_rw": return j.get("SizeRw", None) elif textkey == "disk.size_root_fs": return j.get("SizeRootFs", None) def get_metric_as_bytes(self, metric_string): try: index = 0 while True: if metric_string[index].isdigit() or "." == metric_string[index]: index += 1 continue break metric_value = float(metric_string[0:index]) units = metric_string[index:].lower() self.log.info( "metric_string {} -> {} {}".format(metric_string, metric_value, units) ) conversion = 1 if "kib" == units: conversion = 1000 elif "mib" == units: conversion = math.pow(1024, 2) elif "gib" == units: conversion = math.pow(1024, 3) elif "kb" == units: conversion = 1000 elif "mb" == units: conversion = math.pow(1000, 2) elif "gb" == units: conversion = math.pow(1000, 3) return metric_value * conversion except Exception: self.log.exception("get_metric_as_bytes error") return None def _get_docker_block_stats(self, container, textkey): """ Read the I/O metrics from docker stats, because the /proc io file is read-only to root. """ import json def parse_multi_metric_entry(metric_line): items = metric_line.split("/") items = metric_line.split("/") metrics = [item.strip() for item in items] rv = [] for metric in metrics: rv.append(self.get_metric_as_bytes(metric)) return rv try: container_id = container["Id"] rc, output = agent_util.execute_command( "docker stats {} --no-stream --format json".format(container_id), cache_timeout=agent_util.DEFAULT_CACHE_TIMEOUT, ) if 0 != rc: self.log.error("Docker stats failure: {}".format(rc)) return None data = agent_util.json_loads(output) self.log.debug("Docker Stats result: {}".format(json.dumps(data, indent=1))) mld = parse_multi_metric_entry(data["BlockIO"]) if 2 != len(mld): self.log.error("get_docker_block_stats error: Unexpected metric count") self.log.info(output) return None if "io.bytes_out" == textkey: return mld[1] elif "io.bytes_in" == textkey: return mld[0] else: return None except Exception: self.log.exception("get_docker_block_stats error") return None def get_io_metric(self, container, textkey, data, config): container_id = container["Id"] def get_total(fname, operation_type): with open(fname, "r") as f: lines = f.readlines() total = 0 for line in lines: if line.startswith("Total"): continue device, op_type, num = line.split(" ") if op_type == operation_type: total += int(num) return total key = textkey.split(".")[1] last = self.get_cache_results("docker.io", key) if last: last_val = last[0][1] seconds = last[0][0] else: last_val = None seconds = None new_val = None if self.is_cgroups_v2(): io_metrics = DockerPlugin.read_io_stats_v2(container_id) if io_metrics.get("error_msg", None) is not None: self.log.error("I/O stats error {}".format(io_metrics["error_msg"])) return None mtk = DockerPlugin.map_cgroup_v2_io_textkey_to_metric(textkey) if io_metrics.get(mtk, None) is None: self.log.error("No metric found for {}".format(textkey)) return None new_val = io_metrics[mtk] else: if textkey in ["io.bytes_written", "io.bytes_read"]: try: fname = ( "/sys/fs/cgroup/blkio/docker/%s/blkio.throttle.io_service_bytes" % container_id ) if "written" in textkey: new_val = get_total(fname, "Write") elif "read" in textkey: new_val = get_total(fname, "Read") except Exception: self.log.error("Error accessing %s", fname) return None elif textkey in ["io.write_os", "io.read_ops"]: try: fname = ( "/sys/fs/cgroup/blkio/docker/%s/blkio.throttle.io_serviced" % container_id ) if "write" in textkey: new_val = get_total(fname, "Write") elif "read" in textkey: new_val = get_total(fname, "Read") except Exception: self.log.error("Error accessing %s", fname) return None if new_val is None: return None self.cache_result("docker.io", key, new_val, replace=True) if last_val is None: return None if new_val < last_val: return None return (new_val - last_val) / seconds def get_status_metric(self, container, textkey, data, config): if textkey == "status.running": if container.get("State") != "running": return 0 return 1