linux_gpu_monitoring_and_diagnostic_tools.html 55 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235
  1. <!DOCTYPE html>
  2. <html dir="ltr" lang="en-US" amp="" i-amphtml-layout="" i-amphtml-no-boilerplate="" transformed="self;v=1">
  3. <head>
  4. <meta charset="UTF-8"><style amp-runtime="" i-amphtml-version="012103122145004">html{overflow-x:hidden!important}html.i-amphtml-fie{height:100%!important;width:100%!important}html:not([amp4ads]),html:not([amp4ads]) body{height:auto!important}html:not([amp4ads]) body{margin:0!important}body{-webkit-text-size-adjust:100%;-moz-text-size-adjust:100%;-ms-text-size-adjust:100%;text-size-adjust:100%}html.i-amphtml-singledoc.i-amphtml-embedded{-ms-touch-action:pan-y pinch-zoom;touch-action:pan-y pinch-zoom}html.i-amphtml-fie>body,html.i-amphtml-singledoc>body{overflow:visible!important}html.i-amphtml-fie:not(.i-amphtml-inabox)>body,html.i-amphtml-singledoc:not(.i-amphtml-inabox)>body{position:relative!important}html.i-amphtml-ios-embed-legacy>body{overflow-x:hidden!important;overflow-y:auto!important;position:absolute!important}html.i-amphtml-ios-embed{overflow-y:auto!important;position:static}#i-amphtml-wrapper{overflow-x:hidden!important;overflow-y:auto!important;position:absolute!important;top:0!important;left:0!important;right:0!important;bottom:0!important;margin:0!important;display:block!important}html.i-amphtml-ios-embed.i-amphtml-ios-overscroll,html.i-amphtml-ios-embed.i-amphtml-ios-overscroll>#i-amphtml-wrapper{-webkit-overflow-scrolling:touch!important}#i-amphtml-wrapper>body{position:relative!important;border-top:1px solid transparent!important}#i-amphtml-wrapper+body{visibility:visible}#i-amphtml-wrapper+body .i-amphtml-lightbox-element,#i-amphtml-wrapper+body[i-amphtml-lightbox]{visibility:hidden}#i-amphtml-wrapper+body[i-amphtml-lightbox] .i-amphtml-lightbox-element{visibility:visible}#i-amphtml-wrapper.i-amphtml-scroll-disabled,.i-amphtml-scroll-disabled{overflow-x:hidden!important;overflow-y:hidden!important}amp-instagram{padding:54px 0px 0px!important;background-color:#fff}amp-iframe iframe{box-sizing:border-box!important}[amp-access][amp-access-hide]{display:none}[subscriptions-dialog],body:not(.i-amphtml-subs-ready) [subscriptions-action],body:not(.i-amphtml-subs-ready) [subscriptions-section]{display:none!important}amp-experiment,amp-live-list>[update]{display:none}.i-amphtml-jank-meter{position:fixed;background-color:rgba(232,72,95,0.5);bottom:0;right:0;color:#fff;font-size:16px;z-index:1000;padding:5px}amp-list[resizable-children]>.i-amphtml-loading-container.amp-hidden{display:none!important}amp-list [fetch-error],amp-list[load-more] [load-more-button],amp-list[load-more] [load-more-end],amp-list[load-more] [load-more-failed],amp-list[load-more] [load-more-loading]{display:none}amp-list[diffable] div[role=list]{display:block}amp-story-page,amp-story[standalone]{min-height:1px!important;display:block!important;height:100%!important;margin:0!important;padding:0!important;overflow:hidden!important;width:100%!important}amp-story[standalone]{background-color:#202125!important;position:relative!important}amp-story-page{background-color:#757575}amp-story .amp-active>div,amp-story .i-amphtml-loader-background{display:none!important}amp-story-page:not(:first-of-type):not([distance]):not([active]){transform:translateY(1000vh)!important}amp-autocomplete{position:relative!important;display:inline-block!important}amp-autocomplete>input,amp-autocomplete>textarea{padding:0.5rem;border:1px solid rgba(0,0,0,0.33)}.i-amphtml-autocomplete-results,amp-autocomplete>input,amp-autocomplete>textarea{font-size:1rem;line-height:1.5rem}[amp-fx^=fly-in]{visibility:hidden}amp-script[nodom]{position:fixed!important;top:0!important;width:1px!important;height:1px!important;overflow:hidden!important;visibility:hidden}
  5. /*# sourceURL=/css/ampdoc.css*/[hidden]{display:none!important}.i-amphtml-element{display:inline-block}.i-amphtml-blurry-placeholder{transition:opacity 0.3s cubic-bezier(0.0,0.0,0.2,1)!important;pointer-events:none}[layout=nodisplay]:not(.i-amphtml-element){display:none!important}.i-amphtml-layout-fixed,[layout=fixed][width][height]:not(.i-amphtml-layout-fixed){display:inline-block;position:relative}.i-amphtml-layout-responsive,[layout=responsive][width][height]:not(.i-amphtml-layout-responsive),[width][height][heights]:not([layout]):not(.i-amphtml-layout-responsive),[width][height][sizes]:not([layout]):not(.i-amphtml-layout-responsive){display:block;position:relative}.i-amphtml-layout-intrinsic,[layout=intrinsic][width][height]:not(.i-amphtml-layout-intrinsic){display:inline-block;position:relative;max-width:100%}.i-amphtml-layout-intrinsic .i-amphtml-sizer{max-width:100%}.i-amphtml-intrinsic-sizer{max-width:100%;display:block!important}.i-amphtml-layout-container,.i-amphtml-layout-fixed-height,[layout=container],[layout=fixed-height][height]:not(.i-amphtml-layout-fixed-height){display:block;position:relative}.i-amphtml-layout-fill,.i-amphtml-layout-fill.i-amphtml-notbuilt,[layout=fill]:not(.i-amphtml-layout-fill){display:block;overflow:hidden!important;position:absolute;top:0;left:0;bottom:0;right:0}.i-amphtml-layout-flex-item,[layout=flex-item]:not(.i-amphtml-layout-flex-item){display:block;position:relative;-ms-flex:1 1 auto;flex:1 1 auto}.i-amphtml-layout-fluid{position:relative}.i-amphtml-layout-size-defined{overflow:hidden!important}.i-amphtml-layout-awaiting-size{position:absolute!important;top:auto!important;bottom:auto!important}i-amphtml-sizer{display:block!important}@supports (aspect-ratio:1/1){i-amphtml-sizer.i-amphtml-disable-ar{display:none!important}}.i-amphtml-blurry-placeholder,.i-amphtml-fill-content{display:block;height:0;max-height:100%;max-width:100%;min-height:100%;min-width:100%;width:0;margin:auto}.i-amphtml-layout-size-defined .i-amphtml-fill-content{position:absolute;top:0;left:0;bottom:0;right:0}.i-amphtml-replaced-content,.i-amphtml-screen-reader{padding:0!important;border:none!important}.i-amphtml-screen-reader{position:fixed!important;top:0px!important;left:0px!important;width:4px!important;height:4px!important;opacity:0!important;overflow:hidden!important;margin:0!important;display:block!important;visibility:visible!important}.i-amphtml-screen-reader~.i-amphtml-screen-reader{left:8px!important}.i-amphtml-screen-reader~.i-amphtml-screen-reader~.i-amphtml-screen-reader{left:12px!important}.i-amphtml-screen-reader~.i-amphtml-screen-reader~.i-amphtml-screen-reader~.i-amphtml-screen-reader{left:16px!important}.i-amphtml-unresolved{position:relative;overflow:hidden!important}.i-amphtml-select-disabled{-webkit-user-select:none!important;-ms-user-select:none!important;user-select:none!important}.i-amphtml-notbuilt,[layout]:not(.i-amphtml-element),[width][height][heights]:not([layout]):not(.i-amphtml-element),[width][height][sizes]:not([layout]):not(.i-amphtml-element){position:relative;overflow:hidden!important;color:transparent!important}.i-amphtml-notbuilt:not(.i-amphtml-layout-container)>*,[layout]:not([layout=container]):not(.i-amphtml-element)>*,[width][height][heights]:not([layout]):not(.i-amphtml-element)>*,[width][height][sizes]:not([layout]):not(.i-amphtml-element)>*{display:none}amp-img:not(.i-amphtml-element)[i-amphtml-ssr]>img.i-amphtml-fill-content{display:block}.i-amphtml-notbuilt:not(.i-amphtml-layout-container),[layout]:not([layout=container]):not(.i-amphtml-element),[width][height][heights]:not([layout]):not(.i-amphtml-element),[width][height][sizes]:not([layout]):not(.i-amphtml-element){color:transparent!important;line-height:0!important}.i-amphtml-ghost{visibility:hidden!important}.i-amphtml-element>[placeholder],[layout]:not(.i-amphtml-element)>[placeholder],[width][height][heights]:not([layout]):not(.i-amphtml-element)>[placeholder],[width][height][sizes]:not([layout]):not(.i-amphtml-element)>[placeholder]{display:block;line-height:normal}.i-amphtml-element>[placeholder].amp-hidden,.i-amphtml-element>[placeholder].hidden{visibility:hidden}.i-amphtml-element:not(.amp-notsupported)>[fallback],.i-amphtml-layout-container>[placeholder].amp-hidden,.i-amphtml-layout-container>[placeholder].hidden{display:none}.i-amphtml-layout-size-defined>[fallback],.i-amphtml-layout-size-defined>[placeholder]{position:absolute!important;top:0!important;left:0!important;right:0!important;bottom:0!important;z-index:1}amp-img.i-amphtml-ssr:not(.i-amphtml-element)>[placeholder]{z-index:auto}.i-amphtml-notbuilt>[placeholder]{display:block!important}.i-amphtml-hidden-by-media-query{display:none!important}.i-amphtml-element-error{background:red!important;color:#fff!important;position:relative!important}.i-amphtml-element-error:before{content:attr(error-message)}i-amp-scroll-container,i-amphtml-scroll-container{position:absolute;top:0;left:0;right:0;bottom:0;display:block}i-amp-scroll-container.amp-active,i-amphtml-scroll-container.amp-active{overflow:auto;-webkit-overflow-scrolling:touch}.i-amphtml-loading-container{display:block!important;pointer-events:none;z-index:1}.i-amphtml-notbuilt>.i-amphtml-loading-container{display:block!important}.i-amphtml-loading-container.amp-hidden{visibility:hidden}.i-amphtml-element>[overflow]{cursor:pointer;position:relative;z-index:2;visibility:hidden;display:initial;line-height:normal}.i-amphtml-layout-size-defined>[overflow]{position:absolute}.i-amphtml-element>[overflow].amp-visible{visibility:visible}template{display:none!important}.amp-border-box,.amp-border-box *,.amp-border-box :after,.amp-border-box :before{box-sizing:border-box}amp-pixel{display:none!important}amp-analytics,amp-auto-ads,amp-story-auto-ads{position:fixed!important;top:0!important;width:1px!important;height:1px!important;overflow:hidden!important;visibility:hidden}html.i-amphtml-fie>amp-analytics{position:initial!important}[visible-when-invalid]:not(.visible),form [submit-error],form [submit-success],form [submitting]{display:none}amp-accordion{display:block!important}amp-accordion>section{float:none!important}amp-accordion>section>*{float:none!important;display:block!important;overflow:hidden!important;position:relative!important}amp-accordion,amp-accordion>section{margin:0}amp-accordion:not(.i-amphtml-built)>section>:last-child{display:none!important}amp-accordion:not(.i-amphtml-built)>section[expanded]>:last-child{display:block!important}
  6. /*# sourceURL=/css/ampshared.css*/</style><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1"><meta name="description" content="Here are the top 7 Linux and BSD GPU monitoring, diagnostic, benchmarking command line tools to get the most out of your GPU."><meta property="og:locale" content="en_US"><meta property="og:type" content="article"><meta property="og:title" content="Linux GPU Monitoring and Diagnostic Commands Line Tools - nixCraft"><meta property="og:description" content="Here are the top 7 Linux and BSD GPU monitoring, diagnostic, benchmarking command line tools to get the most out of your GPU."><meta property="og:url" content="https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/"><meta property="og:site_name" content="nixCraft"><meta property="article:published_time" content="2020-11-24T10:49:11+00:00"><meta property="article:modified_time" content="2020-11-24T13:19:49+00:00"><meta property="og:image" content="https://www.cyberciti.biz/media/new/cms/2020/11/nvtop.png"><meta property="og:image:width" content="599"><meta property="og:image:height" content="579"><meta name="twitter:card" content="summary_large_image"><meta name="twitter:creator" content="@https://twitter.com/nixcraft"><meta name="twitter:site" content="@nixcraft"><meta name="twitter:label1" content="Written by"><meta name="twitter:data1" content="Vivek Gite"><meta name="twitter:label2" content="Est. reading time"><meta name="twitter:data2" content="6 minutes"><meta name="generator" content="AMP Plugin v2.0.11; mode=standard"><meta name="msapplication-TileImage" content="https://www.cyberciti.biz/media/new/cms/2018/11/new_icon_amp.png"><title>Linux GPU Monitoring and Diagnostic Commands Line Tools - nixCraft</title><link rel="preconnect" href="https://cdn.ampproject.org"><link rel="preload" as="script" href="https://cdn.ampproject.org/v0.js"><script async="" src="https://cdn.ampproject.org/v0.js"></script><script src="https://cdn.ampproject.org/v0/amp-ad-0.1.js" async="" custom-element="amp-ad"></script><script src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js" async="" custom-element="amp-analytics"></script><script src="https://cdn.ampproject.org/v0/amp-anim-0.1.js" async="" custom-element="amp-anim"></script><script src="https://cdn.ampproject.org/v0/amp-auto-ads-0.1.js" async="" custom-element="amp-auto-ads"></script><script src="https://cdn.ampproject.org/v0/amp-bind-0.1.js" async="" custom-element="amp-bind"></script><script src="https://cdn.ampproject.org/v0/amp-form-0.1.js" async="" custom-element="amp-form"></script><script src="https://cdn.ampproject.org/v0/amp-mustache-0.2.js" async="" custom-template="amp-mustache"></script><script src="https://cdn.ampproject.org/v0/amp-sticky-ad-1.0.js" async="" custom-element="amp-sticky-ad"></script><link href="https://www.cyberciti.biz/cms/wp-content/themes/thesis/lib/images/favicon.ico" rel="shortcut icon"><link rel="icon" href="https://www.cyberciti.biz/media/new/cms/2018/11/new_icon_amp.png" sizes="32x32"><link rel="icon" href="https://www.cyberciti.biz/media/new/cms/2018/11/new_icon_amp.png" sizes="192x192"><style amp-custom="">@font-face{font-family:icon-social;src:url("data:application/font-woff;charset=utf-8;base64,d09GRgABAAAAABAAAA0AAAAAGCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAAP5AAAABoAAAAciBdo/UdERUYAAA/IAAAAHAAAAB4AJwAaT1MvMgAAAZQAAAA+AAAAYA8TDzFjbWFwAAACBAAAAIsAAAHOr7N3G2dhc3AAAA/AAAAACAAAAAgAAAAQZ2x5ZgAAArwAAAuuAAARaCnnGf9oZWFkAAABMAAAAC8AAAA2FZKGYGhoZWEAAAFgAAAAHAAAACQHwgPUaG10eAAAAdQAAAAwAAAATjkrAEtsb2NhAAACkAAAACoAAAAqJSYg7G1heHAAAAF8AAAAGAAAACAAIAD5bmFtZQAADmwAAADcAAABm/pYTdhwb3N0AAAPSAAAAHUAAADRy6/tinjaY2BkYGAAYm9TxX/x/DZfGbhZGEDgpuWCXwj6/3YWBuYdQC4HAxNIFAAv9AsnAHjaY2BkYGA+8P8AAwMLAwgASUYGVCAMAExlAqB42mNgZGBgEGH4zsDNAAJMDGgAABuZARp42mNgZm5lnMDAysDANJPpDAMDQz+EZnzNYMzIyYAKGAXQBBgcGBg/NTIf+H+AwYEZiEFqkGQVGBgBWlwLWAAAeNpjYYAAxlAIzQTELEDMnAfE2xkkwXQeRIzpB4MCkM0PEwPKg2hmmDwDAwDdcAaqeNpjYGBgZoBgGQZGBhA4AeQxgvksDGuAtBqDApDFBhRTeMnwoefDzA+zP1z68OBj1Mf0j7kfp3yc8fHtJ4NPVp8a///9/x9sAlHq/j8RZ+Wv4s/jz+U359fi28i3lG8BXxVfOZ8y70PeG7yTwK5QYyAGMILcz8gGxMxQASYgwYShCuiN4Q0Auvo/xgAAAAAAAAgACAAQABgATgCoARICGAK2AyQDpAT0BfoGZAbwB34HtggSCLQAAHjanVdrjBtXFZ5773genpfH8/Lb6xmvvXZ27bHHM2Z37d3ubt6k6ba7SZpudpuU0JaKRpQCbdWSCNGAgFZElYCiSKBKgPjBD1oKAfUH9E8lhASIl3iJiof4CUi8BCQuZ2bszbZFqoRlX59777l3zj33O985QyFq8sEUNY+pt+pD+9r3aIo8R+WpMkUhm5URyxi6VUKW6XX7S6gf+L16C9VryFMdfOXY2tr5InzOr60d2yPfqL74Inmu9MbhWC5db5KfURQdPpZcIBdASlMtaomi+rCnqvVqjg3PNL1ON/CR50d91vMd1yQd6AZghtc1iwipOuNU7Jqv9gKv0jU/+uSTT8/YdmPGdmbQ8SdPrfi+hJ5E8CWsna+WkHhu/41fHzh79gCu7j/3SfTU6H2YLdbrxeLMTPHGv35/5lRbn780+utvOJVVy92Zn+CFm+rnQnMRlaWOkM+RaeoAdMA3pm7okY8cu1YvgVlBH5wzEYYI/NZCoclm5MK4A8Zb4ElYBi35nNbOI8e+fM89l267MhRM8Zb3u8OdNK/yUka8ZTPdtKbnJNFuNs8FwQLB7761fjA7tZgXmsVCkMXfL7g6IzJbO9ufcd3ZOkKNGjrcx4gQ6Z1rOCmi7O1utm00bacrKyRpCMcenMozVmrKV4kihGfi4KdF97BF3Uc9TF2iPkE9S32RegFO2I2sHh+RZWpgPFxHfQnFbb3GdFoovB23xjKmJSO4NmhBHTDSsc1OCXXMoFOAcaZei3Dj9yIUeSiowfJwlRlYQTgXyvC4oBfJEeZeNx7pAyzMoG/1amyEkT5ABBztyMiCDtiFrm2trz9iO5a53u+vqDJKi0cvZQySTOe38/1C4W257Xw6SYzMpSNJhWjrxXRnqmjmtQ9J6kq/v26ZtvPI+vrW6M8IE0yjqNUwwphBYZvQMDQEY5LIgUTYsCGMgxI0YREdNlWGphOJBGEqiCF0AkGXJhccO9z2yOrqgzMNk6Q4YXW2PJPk7rhUXbD45Tk+YDmODfi5Zd5aqF7c4LiSwTFeWc4ng0U23+K5FDEbMw+urh6Jz3fjY6F1kX0IpREDduCozWCGwFjUWpjFJLSdECSSRKTNhAYTGkUNDV2C4DwEIZpQMR9EWLgAnZvIhhtzg34I670U4EzoIbzDaCK8rskFdwE34eJ+hJYwpN0QGx24qtcjYQyyiY8qSyUpywq6MKj2ZmeX0qquH/F6y8cXFo/r+myLleJzMnJKUeZn6t3TO7cu2fu4ZLFUcjpEqBoMRgsd94CuxVcZ+2uCC57XO9m5Y2bFtG4HgLituYOGkc2ud9z2/BmNMBxjMTzPphrFZq22oKpnD9/98Y7T3Czl6ysd1/jsyaxvZRuN+5eG++MNY6NjBg1JDbihSh0C7+2SUxT442YRATHEvgobX4cIiINLRvsQeDp2TT3mCeA0g6aC6evfmA6kYsqpVg85Vad6qoVbp+DfOVStVpWiNKhV7MqaXakUNm1sbxYqFXsNRkr7W3Pk6HSAK6OF6SCYRq/IWSGTyXSy2VzO93O5TLYLXSErf9m9I5exrBb8tMVFzbIyLfjpJ+rNpaAWQYKaeu0a/ingYok6Tm3D6UyGLaGJ7YbjN1E9avehPgQ8HNL3msjxITSBGnxrAc7qGd3AgxZcQeIYd4xuSJSBH/Wd3T6+mmjWlg+yCcM40Gq97c6Uk74zVU69N+0qmoR4nMDYzCvuY1JBuRMyxp0nHmMfL5VubzQsq9G4vVRqZLNrjq0BANayWXxk/pCmbGz4kBmSPHnmk+jKM+9CSOf/+AU5i9AOQleeRk9d4ZaPj77bNHQFoZcVQ2+gVD0ND/uqnNYoSqCU166RHXKNkqkMVaEG1Cbw5NPU16iXqV9R/6Co6RjrdT9KTYYFvoh43piGvGV4qqdqkNsMGYVKvdoQOWHUxHnC6IdsGvOo76ieHw+H8HB0Jl4zjPLgWElGRbS7OoRSzx+HWDgOg0iPLiZ+WivkYr8Tpc04c4ZgLCEDeeHFRdN1rxxS9O6KIZpkWVDsq45/c8IzgM3jR/se3icIrMQwEisIL3uiJEk3fj58YTD/xW6XT3EJBqEuxrQqysXU34EyDUXOi54n5WTFANJ5uYewyEvSvRwnlDS9JHDwEcq6XhZoFqHRFwRJEmFTAV2lBa5bue3zt8G34nECnUhy3m6fFcmNVxYOzNOqzGv8cObfLJ8s6dp4v3DnJM9+p5cqymKaxriHmASX4nt/eUzTGPYrDKNpKJfTNU3/YBCMXkIH4Jf7uSgyDEryP3xVMRghqabLmw/pBV5RU8oTo8v5lJJS+GKWS6lYSOp6Z9GqJBX4JCvWO0w7qciKwio8nBwnHw13flTX9dF93Xtdu0V4At+W7d7b7dzn2nN02KfnbPe+Dv7FH65eNQYzeT//6Znzli3ANrJgW+fDLcPd7YOj35fTalJgDeW3P+KTiGFE8ZcUldjlbYGqUz3qMvV1iNHdQgruasLl/4N5+xF0J3i1ndqYk8a0ZIT5P6T4GDbRvxeunTDZmMzqPb/nB2MGG5MYC8Sr725mO000mez5AzRevAeT8MVHTg6GJ04MBydvS6vDRqPdbjSGaVVNT2Q1vVQ2zVLJNMuj32ac6lHbNpWMICoMT9OQDDFN84wiChnFtO2jVScz0bHESAdyISS/SEe0JjroGuRGKAXP+HNbc6275vxtKOcgUeJSvbbtz93VgmH/DJSKYbq9NjgZ2/hm0yKTX4qtK5sX3/yg/8/g0bNvZdOu9XEeirDQo/YDCuK6KXLyG0hhfC/WnguIGWX5jZU1/h1pdVZPW6IlQi0FVAEhJVkCkZJcazrVLlstq2hmSaosSzlJaxXmHlxZXkFP7S238dGlZYyP3ppgCvmUW54qu65yeDHwaFzI6029riie195pe97m5vql1qXXF97xma4B/56hzoZ1aVhbKBGLReBtwyn2yNEZAdDLqL5XnhTfZdTfK8eUHe0Xyol+nSXnHbeQfccUQW07P8uXny9j4jr5fdzUKVZhnVY5d3pwmpVZuw1SKZ7jSy+AVqcC+qVtKMntjpO5a7DDyMxo4fnB8+g/LSdf4+27WYm1O+XcznB7LG3DspaTg2VfAwkeCFttMXI0tzXYGmudmUJ41sk1ufKLUxh/avDiMPILgbz0APilH/mlY07qCSgmGIXERcXNOFtAnhnsdvtBdOv13WzRjNLLOKnrTLRWg2WVCB4GOc9A2VUqbvr+IKuYGE8hYspXRBlzCsulWCwl1apZPuC0NpzyPplNEsB/Jp+4RxHNEyvO4emclyvYCcKCD1mIjC9LKr6Mcyo6mT5cHK7gg7nFqUNuZ1GSNVl4hVM0VXTPuu7driCgBJesV4qHZrSaBpW42i4856xVVrYyopgvpBrNndlZk2BNJEdSpiJYOyt7a9lajBg1PJHq9aPU3PejXFpEYXbeRToZ6sWi/upD6Q9nGgf5+erb73744dEZo1AwiEguFPXrf4Pst3l5s+igDyD6rhPnLo++Es7pxfh9loH6737KhLdEmwrGNSBE2zSgTY3f+xJhdYCmxzNtNC6yWaLFURjOoxOSkuATo3+KKqRXkUFVBCs6ox/gj6RkVmauI6/Vftjr8q9+K5HiaZaG1x6EX7lxj2RgfJnJS/ippMqQZzCaH6nPomM3viSnaXIRL6Bjx166Y0NCD6A5jsPoKkZRyUq99hew+5vUMvV26jz1nrDyD6s74w05I3yjvfn2CvaW0SR0whCrE8BSPJCIck5ckiTgfa5mj2kHhOjFLqSZ+kQDMhRN2QVCV6tPbGzs7GxsPF6t0qRYXWtn9qXmP3NyFcvcj4RGptrtVp3utxdPPruoNDPttet/7jrxGPoBwzAqkztbr5/NgcAwjYuWdbGBnW6kgn5cCWQ5L+9fHJyG4jd7erC4X85Lcr/irmrqTJegBsfjlpGLtJ3m6K/EbajpVfeheP/unxiVZZi8rudh7zTTUNXG+6MJ0Keo/wKOw49TAAB42nXOsWrCUBjF8X80WrQgnUrpdEenoOADdCp1cOkgHRvjJQT0XogRdO8jdOwz9GF8Ik/CtyZww+87N+cjwIx/Eton4YEn80Cem4fymzmVv8wjHrmYx8p/zVNeuamVpBMls25D64H8Yh7KC3Mqf5hHPPNtHiv/MU9Z8UdFQeSoEwlQFfEYo/CJp+TMgZxaoy/Ph1zo+74v32pPzUn3be5Ykukv2fr6VMXgltmiv/vezY1apbaEblOu2bNXtuOq97rrbqybqRRD40offJ03fu92V7cu4kb7Mu7l2z5DeNp9yz0OwjAUA+DnlLZAofyIg7wESpORknAVQEKIhYHbI3DmevkkyxYj49mJwIhBIQUmKFGhxhQzzNFggSVarLDGBtvq+vi8brZ+P++q+jcF1Z8X9WcaQnag0dHEne1O9NhnIw2HrKeJvdtrlj/n7ReuoybYAAAAAAEAAf//AA942mNgZGBg4AFiMSBmYmAEQmEgZgHzGAAEbwBCeNpjYGBgZACCq0vUOUD0TcsFv2A0AEI9BwIAAA==") format("woff");font-weight:normal;font-style:normal}.icon-social{font-family:icon-social;font-size:1.5em;vertical-align:middle}.icon-social + .icon-social{margin-left:.25em}*{margin:0;padding:0;word-wrap:break-word}html{-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}h1,h2,h3{font-weight:normal}table{border-collapse:collapse;border-spacing:0}amp-img,amp-anim{border:0}abbr,acronym{text-decoration:none}code{line-height:1em}pre{overflow:auto;word-wrap:normal;-moz-tab-size:4;tab-size:4}amp-img,amp-anim,.wp-caption{max-width:100%;height:auto}amp-img,amp-anim{display:block}.aligncenter{display:block;float:none;text-align:center;margin-right:auto;margin-left:auto}input[type=submit]{cursor:pointer;overflow:visible;-webkit-appearance:none}body{font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:18px;line-height:30px;color:#111}a{color:#2692f0;text-decoration:none}p a,.post_content a{text-decoration:underline}code,pre,kbd{font-family:Menlo,Consolas,Monaco,"Andale Mono",Courier,Verdana,sans-serif}.grt h1{font-size:37px;line-height:58px;color:#111;margin-bottom:30px}.grt h2{font-size:29px;line-height:46px;margin-top:49px;margin-bottom:19px}.grt h3{font-size:23px;line-height:38px;margin-top:30px;margin-bottom:12px}.grt h1 + h2,.grt h2 + h3,.grt h2:first-child,.grt h3:first-child{margin-top:0}.grt ul{margin-left:30px}.grt li{margin-bottom:12px}.grt li:last-child{margin-bottom:0}.grt code,.grt pre,.grt kbd{font-size:16px}.grt code{background-color:rgba(0,0,0,.08);padding:4px;border-radius:7px;margin:0 1px}.grt pre{background-color:#eee;padding:19px}.grt kbd{font-size:14px;color:#111;text-shadow:0 1px 0 white;background-color:#f7f7f7;padding:5px;border:1px solid #ccc;border-radius:7px;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 2px white inset;margin:0 1px}.grt p,.grt ul,.grt pre,.grt amp-img,.grt amp-anim,.grt .aligncenter{margin-bottom:30px}.grt ul ul{margin-top:12px;margin-bottom:0}@media all and (min-width: 682px){body:not(.has-sidebar) .grt h1{font-size:47px;line-height:72px}body:not(.has-sidebar) .grt h2{font-size:37px;line-height:58px}body:not(.has-sidebar) .grt h3{font-size:29px;line-height:46px}}.grt .headline_area .headline{margin-top:0;margin-bottom:0}.grt .headline_area .headline a{color:#111}.grt .headline_area .headline a:hover{color:#2692f0}.grt .byline{font-size:14px;line-height:25px;color:#828282;margin-top:7px}.grt .byline .post_author:after,.grt .byline .post_date:after,.grt .byline .num_comments_link:after{display:block;height:0;content:"\A";white-space:pre}.grt .drop_cap{font-size:60px;line-height:1em;margin-right:7px;float:left}.grt .box{padding:19px}.grt .box{background-color:#cce8cc}.grt .highlight{background-color:#fffb7a;padding:4px 0}.grt .pop{box-shadow:0 0 12px rgba(0,0,0,.4)}.grt .wp-caption{box-sizing:border-box;color:#828282;background-color:#eee;padding:19px}.grt .wp-caption amp-img,.grt .wp-caption amp-anim{margin-bottom:19px}.grt .wp-caption.aligncenter amp-img,.grt .wp-caption.aligncenter amp-anim{margin-right:auto;margin-left:auto}.grt .wp-caption .wp-caption-text{margin-bottom:0}.grt .wp-caption-text{font-size:14px;line-height:25px}.grt .wp-caption-text code{font-size:13px}.grt .post_box + .post_box{margin-top:49px}.grt .box,.grt .wp-caption,.grt .headline_area{margin-bottom:30px}.grt .box > :last-child{margin-bottom:0}.grt .post_box:after,.grt .headline_area:after,.grt .byline:after,.grt .post_content:after{display:table;clear:both;content:""}@media all and (min-width: 682px){.grt .headline_area{position:relative}.grt .byline{margin-top:12px}.grt .byline .post_author:after,.grt .byline .post_date:after,.grt .byline .num_comments_link:after{display:none}.grt .byline .post_author ~ .post_date_intro:before,.grt .byline .post_date_intro ~ .post_date_intro:before,.grt .byline .post_author ~ .num_comments_link:before,.grt .byline .post_date ~ .num_comments_link:before{color:#828282;content:"• "}}label{display:block}input,textarea{display:block;width:100%;box-sizing:border-box;font-family:inherit;font-size:inherit;line-height:1em;font-weight:inherit;padding:7px;border:1px solid rgba(0,0,0,.2)}input[type=checkbox],input[type=radio],input[type=image],input[type=submit]{width:auto}input[type=checkbox],input[type=radio]{display:inline-block}input[type=checkbox] + label,input[type=radio] + label{display:inline;font-weight:normal;margin-left:12px}textarea{line-height:inherit}input[type=submit]{display:inline-block;width:auto;font-family:inherit;font-size:inherit;font-weight:normal;line-height:1em;vertical-align:top;color:#111;text-decoration:none;background-color:#eee;padding:7px 12px;border:1px solid rgba(0,0,0,.1);border-bottom-width:4px;border-bottom-color:rgba(0,0,0,.25);border-radius:12px;cursor:pointer;-webkit-appearance:none}input[type=submit]:hover,input[type=submit]:active{text-decoration:none;background-color:white;transition:background-color .3s ease}input[type=submit]:active{border-bottom-width:2px;margin-top:2px}#header{padding-top:19px;padding-bottom:19px;border-bottom:1px solid rgba(0,0,0,.1)}#nav{background-color:#f8f9fa;border-bottom:1px solid rgba(0,0,0,.1)}#content{padding-top:30px;padding-bottom:30px}#footer{color:#828282;background-color:#f8f9fa;padding-top:19px;padding-bottom:19px;border-top:1px solid rgba(0,0,0,.1)}.container{box-sizing:border-box;padding-left:19px;padding-right:19px}@media all and (min-width: 682px){.container{max-width:644px;padding-left:0;padding-right:0;margin-left:auto;margin-right:auto}}@media all and (min-width: 704px){.container{max-width:100%;padding-left:30px;padding-right:30px;margin-left:0;margin-right:0}}@media all and (min-width: 1100px){.container{max-width:1040px;padding-left:0;padding-right:0;margin-left:auto;margin-right:auto}}.prev_next{border-top:1px dotted rgba(0,0,0,.15);padding-top:19px;padding-bottom:19px;margin-top:30px;margin-bottom:-30px}.prev_next .next_post,.prev_next .previous_post{margin-bottom:0}.prev_next .next_post + .previous_post{margin-top:19px}#comments .comments_intro{margin-top:49px;margin-bottom:30px}#comments .comments_intro:after{content:" ↓"}#comments .num_comments{font-weight:bold}#comments #respond{margin-top:49px}#comments #respond p{margin-bottom:19px}#comments #respond .comment-reply-title{font-weight:bold}#comments #respond #cancel-comment-reply-link{font-weight:normal;margin-left:12px}#comments #respond .submit{padding:12px}.content,.sidebar{box-sizing:border-box}.sidebar{margin-top:30px}@media all and (min-width: 682px){.has-sidebar .content,.has-sidebar .sidebar{width:100%;max-width:644px}}@media all and (min-width: 1100px){.has-sidebar .columns{display:flex;justify-content:space-between}.has-sidebar .sidebar{margin-top:0;max-width:347px}}.sidebar{border-top:1px solid rgba(0,0,0,.1);padding-top:30px}.sidebar > div{margin-bottom:49px}.sidebar > div > :last-child{margin-bottom:0}.sidebar .widget_title{font-size:23px;line-height:38px;margin-bottom:12px}.sidebar p,.sidebar ul{margin-bottom:30px}.sidebar ul{margin-left:30px}.sidebar li{margin-bottom:12px}.sidebar li:last-child{margin-bottom:0}.sidebar > :last-child,.sidebar .textwidget > :last-child{margin-bottom:0}@media all and (min-width: 1100px){.sidebar{font-size:14px;line-height:25px;border-top:0;padding-top:0}.sidebar > div{margin-bottom:30px}.sidebar .widget_title{font-size:18px;line-height:30px;margin-bottom:7px}.sidebar p,.sidebar ul{margin-bottom:19px}.sidebar ul{margin-left:19px}.sidebar li{margin-bottom:7px}.sidebar input[type=submit]{padding:7px}}.wp_rp_wrap{clear:both}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) div.wp_rp_content{display:block}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post{margin:0 0 18px 0;padding:0}ul.related_post li{min-height:80px}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li{position:relative;list-style:none;padding:10px 0 10px 0;margin:0;border-bottom:1px solid rgba(0,0,0,.1);background:none}ul.related_post li a{text-decoration:none}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a{position:relative;text-indent:0}ul.related_post li a amp-img,ul.related_post li a amp-anim{width:60px;height:60px}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a amp-img,:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a amp-anim{float:left;float:none;margin:0 10px 0 0;padding:0;max-width:100%}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a.wp_rp_thumbnail{float:left;margin-right:10px}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a.wp_rp_thumbnail amp-img,:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) ul.related_post li a.wp_rp_thumbnail amp-anim{float:none;margin-right:0;display:block}table,th,td{border:1px solid #d1d1d1}table{border-collapse:separate;border-spacing:0;border-width:1px 0 0 1px;margin:0 0 1.75em;table-layout:auto;width:100%}code{overflow:auto;clear:both;word-wrap:normal;display:block;line-height:1.5em}.grt code{padding:19px}.grt .wp-caption{background-color:white}.nixcraftsearchbox{padding:7px;border:1px solid rgba(0,0,0,.2);box-sizing:border-box;width:100%;background-color:#f7f4d7;border-radius:4px}.crumbs{color:#828282;padding-top:5px;padding-bottom:5px}#header{border-bottom:0px;padding-top:0px;padding-bottom:0px}amp-img.amp-wp-enforced-sizes[layout="intrinsic"] > img,amp-anim.amp-wp-enforced-sizes[layout="intrinsic"] > img{object-fit:contain}.amp-wp-default-form-message > p{margin:1em 0;padding:.5em}.amp-wp-default-form-message[submitting] > p,.amp-wp-default-form-message[submit-success] > p.amp-wp-form-redirecting{font-style:italic}.amp-wp-default-form-message[submit-success] > p:not(.amp-wp-form-redirecting){border:solid 1px #008000;background-color:#90ee90;color:#000}.amp-wp-default-form-message[submit-error] > p{border:solid 1px #f00;background-color:#ffb6c1;color:#000}.amp-wp-default-form-message[submit-success] > p:empty{display:none}table.nixcraftcmdtabls{width:100%;height:100%;border-collapse:collapse}table.nixcraftcmdtabls td,table.nixcraftcmdtabls th{padding:5px;font-size:90%}table.nixcraftcmdtabls tr:nth-child(even){background:#d0e4f5}table.nixcraftcmdtabls thead{background:#1d3557;background:-moz-linear-gradient(top,#556781 0%,#334967 66%,#1d3557 100%);background:-webkit-linear-gradient(top,#556781 0%,#334967 66%,#1d3557 100%);background:linear-gradient(to bottom,#556781 0%,#334967 66%,#1d3557 100%)}table.nixcraftcmdtabls thead th{font-weight:bold;color:#a8dadc;text-align:center}.topnum{color:#ccc;font-size:40px;float:left;margin-top:-1px;width:15%}.morenum{color:#ccc;font-size:40px;float:left;width:15%}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-cb4b39f{width:609px}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-905953b{font-weight:bold;font-size:20px;margin-bottom:10px}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-434118d{width:20%}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-210797b{width:80%}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-cada9ac{text-align:right;width:20%}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-487f3fd{text-align:left;width:80%}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-1f50463{color:#828282}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-224b51a{display:none}:root:not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-3e2e7c3{display:none}:root:not(#_):not(#_):not(#_):not(#_):not(#_) .amp-wp-89f4d16{font-size:.85em}
  7. /*# sourceURL=amp-custom.css */</style><link href="https://www.cyberciti.biz/feed/" rel="alternate" type="application/rss+xml" title="nixCraft feed"><!-- This site is optimized with the Yoast SEO Premium plugin v16.0.2 - https://yoast.com/wordpress/plugins/seo/ --><link rel="canonical" href="https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/"><script type="application/ld+json" class="yoast-schema-graph">{"@context":"https://schema.org","@graph":[{"@type":"WebSite","@id":"https://www.cyberciti.biz/#website","url":"https://www.cyberciti.biz/","name":"nixCraft","description":"Linux Tips, Hacks, Tutorials, And Ideas In Blog","publisher":{"@id":"https://www.cyberciti.biz/#/schema/person/8d22b4a449eb131d4fed273b514cda82"},"potentialAction":[{"@type":"SearchAction","target":"https://www.cyberciti.biz/?s={search_term_string}","query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#primaryimage","inLanguage":"en-US","url":"https://www.cyberciti.biz/media/new/cms/2020/11/Linux-glmark2.gif"},{"@type":"WebPage","@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#webpage","url":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/","name":"Linux GPU Monitoring and Diagnostic Commands Line Tools - nixCraft","isPartOf":{"@id":"https://www.cyberciti.biz/#website"},"primaryImageOfPage":{"@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#primaryimage"},"datePublished":"2020-11-24T10:49:11+00:00","dateModified":"2020-11-24T13:19:49+00:00","description":"Here are the top 7 Linux and BSD GPU monitoring, diagnostic, benchmarking command line tools to get the most out of your GPU.","breadcrumb":{"@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/"]}]},{"@type":"BreadcrumbList","@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"item":{"@type":"WebPage","@id":"https://www.cyberciti.biz/","url":"https://www.cyberciti.biz/","name":"Home"}},{"@type":"ListItem","position":2,"item":{"@type":"WebPage","@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/","url":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/","name":"Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools"}}]},{"@type":"Article","@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#article","isPartOf":{"@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#webpage"},"author":{"@id":"https://www.cyberciti.biz/#/schema/person/8d22b4a449eb131d4fed273b514cda82"},"headline":"Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools","datePublished":"2020-11-24T10:49:11+00:00","dateModified":"2020-11-24T13:19:49+00:00","mainEntityOfPage":{"@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#webpage"},"commentCount":0,"publisher":{"@id":"https://www.cyberciti.biz/#/schema/person/8d22b4a449eb131d4fed273b514cda82"},"image":{"@id":"https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#primaryimage"},"keywords":"Debian,Linux,Sysadmin,Ubuntu","articleSection":"Command Line Hacks,Hardware,Linux Games","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#respond"]}]},{"@type":["Person","Organization"],"@id":"https://www.cyberciti.biz/#/schema/person/8d22b4a449eb131d4fed273b514cda82","name":"Vivek Gite","logo":{"@id":"https://www.cyberciti.biz/#personlogo"},"description":"The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.","sameAs":["http://www.cyberciti.biz/tips/about-us","https://twitter.com/https://twitter.com/nixcraft"]}]}</script><!-- / Yoast SEO Premium plugin. --><link rel="alternate" type="application/rss+xml" title="nixCraft » Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools Comments Feed" href="https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/feed/"><link rel="apple-touch-icon" href="https://www.cyberciti.biz/media/new/cms/2018/11/new_icon_amp.png"></head>
  8. <body class="template-single has-sidebar">
  9. <amp-auto-ads type="adsense" data-ad-client="ca-pub-7825705102693166" class="i-amphtml-layout-container" i-amphtml-layout="container">
  10. </amp-auto-ads>
  11. <div id="header">
  12. <div class="container">
  13. </div>
  14. </div>
  15. <nav id="nav">
  16. <div class="container">
  17. <div class="crumbs" itemscope itemtype="https://schema.org/BreadcrumbList">
  18. <span class="crumbs-crumb crumbs-item" itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem">
  19. <a href="https://www.cyberciti.biz/" class="crumbs-home" itemprop="item"><span itemprop="name">🐧 nixCraft</span></a>
  20. <meta itemprop="position" content="1">
  21. </span>
  22. <span class="crumbs-sep crumbs-item">→</span>
  23. <span class="crumbs-crumb crumbs-item" itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem">
  24. <a href="https://www.cyberciti.biz/topics/open-source/" itemprop="item"><span itemprop="name">Open Source</span></a>
  25. <meta itemprop="position" content="2">
  26. </span>
  27. <span class="crumbs-sep crumbs-item">→</span>
  28. <span class="crumbs-crumb crumbs-item" itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem">
  29. <a href="https://www.cyberciti.biz/topics/open-source/command-line-hacks/" itemprop="item"><span itemprop="name">Command Line Hacks</span></a>
  30. <meta itemprop="position" content="3">
  31. </span>
  32. <span class="crumbs-sep crumbs-item">→</span>
  33. <span class="crumbs-crumb crumbs-item" itemprop="itemListElement" itemscope itemtype="https://schema.org/ListItem">
  34. <span itemprop="name">Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools</span>
  35. <meta itemprop="position" content="4">
  36. </span>
  37. </div>
  38. </div>
  39. </nav>
  40. <div id="content">
  41. <div class="container columns">
  42. <div class="content grt">
  43. <article id="post-1569" class="post_box text top">
  44. <div class="headline_area">
  45. <h1 class="headline"><a href="https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/" rel="bookmark">Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools</a></h1>
  46. <div class="byline">
  47. <span class="post_author_intro">Author:</span> <span class="post_author">Vivek Gite</span>
  48. <span class="post_date_intro">Last updated:</span> <span class="post_date date_modified" title="2020-11-24">November 24, 2020</span>
  49. <a class="num_comments_link" href="https://www.cyberciti.biz/open-source/command-line-hacks/linux-gpu-monitoring-and-diagnostic-commands/#respond" rel="nofollow"><span class="num_comments">0</span> comments</a>
  50. </div>
  51. </div>
  52. <div class="post_content">
  53. <p><span class="drop_cap">A</span> video card is a special circuit board that controls what is displayed on a computer monitor. It is also called a graphics processing unit (GPU), which calculates 3D images and graphics for Linux gaming and other usages. Let us see the top 7 Linux GPU monitoring and diagnostic command-line tools to solve issues.<br>
  54. <span id="more-1569"></span><br>
  55. The following tools work on Linux for GPU monitoring and diagnostic purposes and other operating systems such as FreeBSD. The majority of Linux and FreeBSD users these days use Nvidia, Intel, and AMD GPUs.</p><center>
  56. <amp-ad width="100vw" height="320" type="adsense" data-ad-client="ca-pub-7825705102693166" data-ad-slot="4157409961" data-auto-format="rspv" data-full-width="" class="i-amphtml-layout-fixed i-amphtml-layout-size-defined" style="width:100vw;height:320px;" i-amphtml-layout="fixed">
  57. <div overflow=""></div>
  58. </amp-ad>
  59. </center>
  60. <h2>Linux GPU Monitoring and Diagnostic Commands Line Tools</h2>
  61. <p>We can use the following tools to monitor, diagnostic, and inspect our Linux or *BSD based systems. </p>
  62. <h3><a href="https://www.cyberciti.biz/faq/linux-tell-which-graphics-vga-card-installed/">Finding information about GPU on Linux</a></h3>
  63. <p>To get the GPU info simply run:<br>
  64. <code>sudo lshw -C display -short<br>
  65. lspci -v | more</code><br>
  66. Which is output something as follows:</p>
  67. <pre>H/W path Device Class Description
  68. ===============================================================
  69. /0/100/1/0 display TU117M [GeForce GTX 1650 Mobile / Max-Q]
  70. /0/100/2 /dev/fb0 display UHD Graphics 630 (Mobile)</pre>
  71. <h2>1. glmark2 – Stress-testing GPU performance on Linux</h2>
  72. <p>glmark2 is an OpenGL 2.0 and ES 2.0 benchmark command-line utility. We can install it as follows:<br>
  73. <code>$ sudo apt install glmark2</code><br>
  74. Now run it as follows:<br>
  75. <code>$ glmark2</code><br>
  76. Then it will begin the test as follows and would stress test your GPU on Linux:<br>
  77. </p><div id="attachment_1571" class="wp-caption aligncenter amp-wp-cb4b39f" data-amp-original-style="width: 609px"><amp-anim aria-describedby="caption-attachment-1571" src="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-glmark2.gif" alt="Linux glmark2" width="599" height="613" class="size-full wp-image-1571 amp-wp-enforced-sizes i-amphtml-layout-intrinsic i-amphtml-layout-size-defined" layout="intrinsic" i-amphtml-layout="intrinsic"><i-amphtml-sizer class="i-amphtml-sizer"><img alt="" aria-hidden="true" class="i-amphtml-intrinsic-sizer" role="presentation" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzYxMycgd2lkdGg9JzU5OScgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJyB2ZXJzaW9uPScxLjEnLz4="></i-amphtml-sizer><noscript><img aria-describedby="caption-attachment-1571" loading="lazy" src="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-glmark2.gif" alt="Linux glmark2" width="599" height="613" class="size-full wp-image-1571"></noscript></amp-anim><p id="caption-attachment-1571" class="wp-caption-text">Linux glmark2 test screen</p></div>
  78. My benchmark result for Nvidia GeForce GTX 1650 running on Ubuntu Linux 20.04 LTS:
  79. <pre>=======================================================
  80. glmark2 2014.03+git20150611.fa71af2d
  81. =======================================================
  82. OpenGL Information
  83. GL_VENDOR: NVIDIA Corporation
  84. GL_RENDERER: GeForce GTX 1650 with Max-Q Design/PCIe/SSE2
  85. GL_VERSION: 4.6.0 NVIDIA 450.80.02
  86. =======================================================
  87. [build] use-vbo=false: FPS: 4980 FrameTime: 0.201 ms
  88. [build] use-vbo=true: FPS: 6927 FrameTime: 0.144 ms
  89. [texture] texture-filter=nearest: FPS: 5144 FrameTime: 0.194 ms
  90. [texture] texture-filter=linear: FPS: 4979 FrameTime: 0.201 ms
  91. [texture] texture-filter=mipmap: FPS: 4030 FrameTime: 0.248 ms
  92. [shading] shading=gouraud: FPS: 6358 FrameTime: 0.157 ms
  93. [shading] shading=blinn-phong-inf: FPS: 5810 FrameTime: 0.172 ms
  94. [shading] shading=phong: FPS: 6425 FrameTime: 0.156 ms
  95. [shading] shading=cel: FPS: 5720 FrameTime: 0.175 ms
  96. [bump] bump-render=high-poly: FPS: 4772 FrameTime: 0.210 ms
  97. [bump] bump-render=normals: FPS: 7187 FrameTime: 0.139 ms
  98. [bump] bump-render=height: FPS: 6724 FrameTime: 0.149 ms
  99. [effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 5278 FrameTime: 0.189 ms
  100. [effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 3649 FrameTime: 0.274 ms
  101. [pulsar] light=false:quads=5:texture=false: FPS: 5793 FrameTime: 0.173 ms
  102. [desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 2776 FrameTime: 0.360 ms
  103. [desktop] effect=shadow:windows=4: FPS: 3913 FrameTime: 0.256 ms
  104. [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1555 FrameTime: 0.643 ms
  105. [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 1703 FrameTime: 0.587 ms
  106. [buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1800 FrameTime: 0.556 ms
  107. [ideas] speed=duration: FPS: 5480 FrameTime: 0.182 ms
  108. [jellyfish] : FPS: 4283 FrameTime: 0.233 ms
  109. [terrain] : FPS: 746 FrameTime: 1.340 ms
  110. [shadow] : FPS: 4878 FrameTime: 0.205 ms
  111. [refract] : FPS: 1580 FrameTime: 0.633 ms
  112. [conditionals] fragment-steps=0:vertex-steps=0: FPS: 5081 FrameTime: 0.197 ms
  113. [conditionals] fragment-steps=5:vertex-steps=0: FPS: 4556 FrameTime: 0.219 ms
  114. [conditionals] fragment-steps=0:vertex-steps=5: FPS: 5293 FrameTime: 0.189 ms
  115. [function] fragment-complexity=low:fragment-steps=5: FPS: 5048 FrameTime: 0.198 ms
  116. [function] fragment-complexity=medium:fragment-steps=5: FPS: 4602 FrameTime: 0.217 ms
  117. [loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 4744 FrameTime: 0.211 ms
  118. [loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 4515 FrameTime: 0.221 ms
  119. [loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 4948 FrameTime: 0.202 ms
  120. =======================================================
  121. glmark2 Score: 4584
  122. =======================================================
  123. </pre>
  124. <h2>2. glxgears – Simple Linux GPU performance testing tool</h2>
  125. <p>It will displays a set of rotating gears and prints out the frame rate at regular intervals. It has become quite popular as basic benchmarking tool for Linux and Unix-like system such as FreeBSD. Install and run it as follows:<br>
  126. <code>$ apt install mesa-utils<br>
  127. $ glxgears</code><br>
  128. <amp-anim src="https://www.cyberciti.biz/media/new/cms/2020/11/glxgears-mesa-utils.gif" alt="glxgears" width="298" height="294" class="aligncenter size-full wp-image-1572 amp-wp-enforced-sizes i-amphtml-layout-intrinsic i-amphtml-layout-size-defined" layout="intrinsic" i-amphtml-layout="intrinsic"><i-amphtml-sizer class="i-amphtml-sizer"><img alt="" aria-hidden="true" class="i-amphtml-intrinsic-sizer" role="presentation" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzI5NCcgd2lkdGg9JzI5OCcgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJyB2ZXJzaW9uPScxLjEnLz4="></i-amphtml-sizer><noscript><img loading="lazy" src="https://www.cyberciti.biz/media/new/cms/2020/11/glxgears-mesa-utils.gif" alt="glxgears" width="298" height="294" class="aligncenter size-full wp-image-1572"></noscript></amp-anim><br>
  129. The GPU frame rate is measured and published out on the screen every five seconds. The final result will look as follows:</p>
  130. <pre>Running synchronized to the vertical refresh. The framerate should be
  131. approximately the same as the monitor refresh rate.
  132. 299 frames in 5.0 seconds = 59.416 FPS
  133. 299 frames in 5.0 seconds = 59.731 FPS
  134. 300 frames in 5.0 seconds = 59.940 FPS
  135. 300 frames in 5.0 seconds = 59.968 FPS
  136. 300 frames in 5.0 seconds = 59.943 FPS
  137. 300 frames in 5.0 seconds = 59.967 FPS
  138. 299 frames in 5.0 seconds = 59.742 FPS
  139. 300 frames in 5.0 seconds = 59.951 FPS
  140. .....
  141. ...
  142. ...</pre>
  143. <h2>3. gpustat – A simple tool to get Nvidia GPU stats on Linux and FreeBSD Unix</h2>
  144. <p>It is written in Python and the perfect tool for CLI users, especially ML/AI developers. One can install it as follows using <a href="https://pypi.python.org/pypi/gpustat" rel="noopener noreferrer" target="_blank">PIP</a><br>
  145. <code>$ pip install gpustat<br>
  146. $ pip3 install gpustat</code><br>
  147. Run it as follows:<br>
  148. <code>$ gpustat<br>
  149. $ gpustat -cp</code><br>
  150. Here we see name of running process and their PIDs running on Nvidia GPU:</p>
  151. <pre>
  152. nixcraft-wks01 Tue Nov 24 15:46:37 2020 450.80.02
  153. [0] GeForce GTX 1650 with Max-Q Design | 39'C, ?? %, 2 % | 962 / 3911 MB | Xorg/2454(100M) Xorg/3504(325M) gnome-shell/3689(181M) firefox/4614(1M) firefox/5036(1M) firefox/5143(1M)
  154. </pre>
  155. <p>See help:<br>
  156. <code>$ gpustat -h</code></p>
  157. <h2>4. intel_gpu_top – Displying a top-like summary of Intel GPU usage on Linux</h2>
  158. <p>First install the tool, run:<br>
  159. <code>$ sudo apt install intel-gpu-tools<br>
  160. ## CentOS/RHEL/Fedora Linux user try the dnf command ##<br>
  161. $ sudo dnf install intel-gpu-tools</code><br>
  162. Fedora, RHEL and CentOS Linux user can use the podman command as follows to install the same:<br>
  163. <code>$ podman run --rm --priviledged registry.freedesktop.org/drm/igt-gpu-tools/igt:master</code><br>
  164. The tool gathers data using perf performance counters (PMU) exposed by i915 and other platform drivers like RAPL (power) and Uncore IMC (memory bandwidth). Run it as follows on Linux system:<br>
  165. <code>$ sudo intel_gpu_top</code><br>
  166. <amp-img src="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel.png" alt="Linux GPU Monitoring and Diagnostic Commands Line Tools for Intel" width="599" height="466" class="aligncenter size-full wp-image-1573 amp-wp-enforced-sizes i-amphtml-layout-intrinsic i-amphtml-layout-size-defined" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel-300x233.png 300w" sizes="(max-width: 599px) 100vw, 599px" layout="intrinsic" disable-inline-width="" i-amphtml-layout="intrinsic"><i-amphtml-sizer class="i-amphtml-sizer"><img alt="" aria-hidden="true" class="i-amphtml-intrinsic-sizer" role="presentation" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzQ2Nicgd2lkdGg9JzU5OScgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJyB2ZXJzaW9uPScxLjEnLz4="></i-amphtml-sizer><noscript><img loading="lazy" src="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel.png" alt="Linux GPU Monitoring and Diagnostic Commands Line Tools for Intel" width="599" height="466" class="aligncenter size-full wp-image-1573" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/Linux-GPU-Monitoring-and-Diagnostic-Commands-Line-Tools-for-Intel-300x233.png 300w" sizes="(max-width: 599px) 100vw, 599px"></noscript></amp-img></p>
  167. <h2>5. nvidia-smi – NVIDIA System Management Interface program</h2>
  168. <p>The nvidia-smi provides monitoring and management capabilities for each of NVIDIA’s Tesla, Quadro, GRID and GeForce devices from Fermi and higher architecture families. GeForce Titan series devices are supported for most functions with very limited information provided for the remainder of the Geforce brand. NVSMI is a cross platform tool that supports all standard NVIDIA driver-supported Linux and FreeBSD. Install it as follows once Nvidia driver installed on Ubuntu Linux:<br>
  169. <code>$ apt install nvidia-smi</code><br>
  170. Open the terminal and then run:<br>
  171. <code>$ nvidia-smi -q -g 0 -d UTILIZATION -l 1<br>
  172. $ sudo nvidia-smi<br>
  173. $ nvidia-smi --help</code><br>
  174. Here is what we see:</p>
  175. <pre>
  176. Tue Nov 24 15:57:43 2020
  177. +-----------------------------------------------------------------------------+
  178. | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
  179. |-------------------------------+----------------------+----------------------+
  180. | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
  181. | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
  182. | | | MIG M. |
  183. |===============================+======================+======================|
  184. | 0 GeForce GTX 165... Off | 00000000:01:00.0 On | N/A |
  185. | N/A 40C P8 3W / N/A | 1011MiB / 3911MiB | 2% Default |
  186. | | | N/A |
  187. +-------------------------------+----------------------+----------------------+
  188. +-----------------------------------------------------------------------------+
  189. | Processes: |
  190. | GPU GI CI PID Type Process name GPU Memory |
  191. | ID ID Usage |
  192. |=============================================================================|
  193. | 0 N/A N/A 2454 G /usr/lib/xorg/Xorg 100MiB |
  194. | 0 N/A N/A 3504 G /usr/lib/xorg/Xorg 357MiB |
  195. | 0 N/A N/A 3689 G /usr/bin/gnome-shell 179MiB |
  196. | 0 N/A N/A 4614 G /usr/lib/firefox/firefox 1MiB |
  197. | 0 N/A N/A 5036 G /usr/lib/firefox/firefox 1MiB |
  198. | 0 N/A N/A 5143 G /usr/lib/firefox/firefox 1MiB |
  199. | 0 N/A N/A 6406 G ...AAAAAAAA== --shared-files 225MiB |
  200. | 0 N/A N/A 14462 G ...AAAAAAAA== --shared-files 131MiB |
  201. +-----------------------------------------------------------------------------+
  202. </pre>
  203. <h2>6. nvtop – NVIDIA GPU top</h2>
  204. <p>Another fancy but very useful tool for NVIDIA GPU. It is a ncurses-based GPU status viewer for NVIDIA GPUs similarly to the htop command or top command. We can install it as follows:<br>
  205. <code>$ apt install nvtop<br>
  206. ## RUN the tool ##<br>
  207. $ nvtop</code><br>
  208. <amp-img src="https://www.cyberciti.biz/media/new/cms/2020/11/nvtop.png" alt="nvtop" width="599" height="579" class="aligncenter size-full wp-image-1574 amp-wp-enforced-sizes i-amphtml-layout-intrinsic i-amphtml-layout-size-defined" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/nvtop.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/nvtop-300x290.png 300w" sizes="(max-width: 599px) 100vw, 599px" layout="intrinsic" disable-inline-width="" i-amphtml-layout="intrinsic"><i-amphtml-sizer class="i-amphtml-sizer"><img alt="" aria-hidden="true" class="i-amphtml-intrinsic-sizer" role="presentation" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzU3OScgd2lkdGg9JzU5OScgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJyB2ZXJzaW9uPScxLjEnLz4="></i-amphtml-sizer><noscript><img loading="lazy" src="https://www.cyberciti.biz/media/new/cms/2020/11/nvtop.png" alt="nvtop" width="599" height="579" class="aligncenter size-full wp-image-1574" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/nvtop.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/nvtop-300x290.png 300w" sizes="(max-width: 599px) 100vw, 599px"></noscript></amp-img><br>
  209. The following commands are available while in nvtop is on screen:</p>
  210. <ul>
  211. <li><kbd>Up</kbd> – Select (highlight) the previous process.</li>
  212. <li><kbd>Down</kbd> – Select (highlight) the next process.</li>
  213. <li><kbd>Left</kbd> / <kbd>Right</kbd> – Scroll in the process row.</li>
  214. <li><kbd>+</kbd> – Sort increasingly.</li>
  215. <li><kbd>-</kbd> – Sort decreasingly.</li>
  216. <li><kbd>F1</kbd> – Select a signal to send to the highlighted process.</li>
  217. <li><kbd>F2</kbd> – Select the field for sorting. The current sort field is highlighted inside the header bar.</li>
  218. <li><kbd>F3</kbd>, <kbd>q</kbd>, <kbd>Esc</kbd> – Exit nvtop and return to your shell</li>
  219. </ul>
  220. <h2>7. radeontop – Tool to show AMD GPU utilization on Linux</h2>
  221. <p>View your <a href="https://github.com/clbr/radeontop" rel="noopener noreferrer" target="_blank">AMD</a> GPU utilization, both for the total activity percent and individual blocks on Linux. Install it as follows:<br>
  222. <code>$ sudo apt install radeontop<br>
  223. $ sudo radeontop</code><br>
  224. It works with R600 and up GPUs, even Southern Islands should work fine. Works with both the open source AMD drivers and AMD Catalyst cloused-source drivers:<br>
  225. <amp-img src="https://www.cyberciti.biz/media/new/cms/2020/11/radontop.png" alt="radontop" width="599" height="383" class="aligncenter size-full wp-image-1575 amp-wp-enforced-sizes i-amphtml-layout-intrinsic i-amphtml-layout-size-defined" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/radontop.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/radontop-300x192.png 300w" sizes="(max-width: 599px) 100vw, 599px" layout="intrinsic" disable-inline-width="" i-amphtml-layout="intrinsic"><i-amphtml-sizer class="i-amphtml-sizer"><img alt="" aria-hidden="true" class="i-amphtml-intrinsic-sizer" role="presentation" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9JzM4Mycgd2lkdGg9JzU5OScgeG1sbnM9J2h0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnJyB2ZXJzaW9uPScxLjEnLz4="></i-amphtml-sizer><noscript><img loading="lazy" src="https://www.cyberciti.biz/media/new/cms/2020/11/radontop.png" alt="radontop" width="599" height="383" class="aligncenter size-full wp-image-1575" srcset="https://www.cyberciti.biz/media/new/cms/2020/11/radontop.png 599w, https://www.cyberciti.biz/media/new/cms/2020/11/radontop-300x192.png 300w" sizes="(max-width: 599px) 100vw, 599px"></noscript></amp-img></p>
  226. <h2>Conclusion</h2>
  227. <p>You learned about the various Linux GPU commands and tools for monitoring and diagnostic purposes on Linux and BSD-based systems. Let me know if I missed your favorite tool in the comment section below.</p>
  228. <br>
  229. </body></html>