File manager - Edit - /usr/share/doc/python3-pygments/html/docs/unicode.html
Back
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Unicode and Encodings — Pygments 2.2.0 documentation</title> <link rel="stylesheet" href="../_static/pygments14.css" type="text/css" /> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script> <script type="text/javascript" src="../_static/jquery.js"></script> <script type="text/javascript" src="../_static/underscore.js"></script> <script type="text/javascript" src="../_static/doctools.js"></script> <link rel="shortcut icon" href="../_static/favicon.ico"/> <link rel="index" title="Index" href="../genindex.html" /> <link rel="search" title="Search" href="../search.html" /> <link rel="next" title="Builtin Tokens" href="tokens.html" /> <link rel="prev" title="Styles" href="styles.html" /> <link href='http://fonts.googleapis.com/css?family=PT+Sans:300,400,700' rel='stylesheet' type='text/css'> <style type="text/css"> table.right { float: right; margin-left: 20px; } table.right td { border: 1px solid #ccc; } </style> <script type="text/javascript"> // intelligent scrolling of the sidebar content $(window).scroll(function() { var sb = $('.sphinxsidebarwrapper'); var win = $(window); var sbh = sb.height(); var offset = $('.sphinxsidebar').position()['top']; var wintop = win.scrollTop(); var winbot = wintop + win.innerHeight(); var curtop = sb.position()['top']; var curbot = curtop + sbh; // does sidebar fit in window? if (sbh < win.innerHeight()) { // yes: easy case -- always keep at the top sb.css('top', $u.min([$u.max([0, wintop - offset - 10]), $(document).height() - sbh - 200])); } else { // no: only scroll if top/bottom edge of sidebar is at // top/bottom edge of window if (curtop > wintop && curbot > winbot) { sb.css('top', $u.max([wintop - offset - 10, 0])); } else if (curtop < wintop && curbot < winbot) { sb.css('top', $u.min([winbot - sbh - offset - 20, $(document).height() - sbh - 200])); } } }); </script> </head><body> <div class="outerwrapper"> <div class="pageheader"> <ul> <li><a href="../index.html">Home</a></li> <li><a href="../languages.html">Languages</a></li> <li><a href="../faq.html">FAQ</a></li> <li><a href="../download.html">Get it</a></li> <li><a href="index.html">Docs</a></li> </ul> <div> <a href="../index.html"> <img src="../_static/logo.png" alt="Pygments logo" /> </a> </div> </div> <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> <div class="sphinxsidebarwrapper"> <strong>« <a href="index.html">Back to docs index</a></strong> <div id="searchbox" style="display: none" role="search"> <h3>Quick search</h3> <div class="searchformwrapper"> <form class="search" action="../search.html" method="get"> <input type="text" name="q" /> <input type="submit" value="Go" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> </div> <script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> <div class="document"> <div class="documentwrapper"> <div class="bodywrapper"> <div class="body" role="main"> <div class="section" id="unicode-and-encodings"> <h1>Unicode and Encodings<a class="headerlink" href="#unicode-and-encodings" title="Permalink to this headline">¶</a></h1> <p>Since Pygments 0.6, all lexers use unicode strings internally. Because of that you might encounter the occasional <code class="xref py py-exc docutils literal notranslate"><span class="pre">UnicodeDecodeError</span></code> if you pass strings with the wrong encoding.</p> <p>Per default all lexers have their input encoding set to <cite>guess</cite>. This means that the following encodings are tried:</p> <ul class="simple"> <li>UTF-8 (including BOM handling)</li> <li>The locale encoding (i.e. the result of <cite>locale.getpreferredencoding()</cite>)</li> <li>As a last resort, <cite>latin1</cite></li> </ul> <p>If you pass a lexer a byte string object (not unicode), it tries to decode the data using this encoding.</p> <p>You can override the encoding using the <cite>encoding</cite> or <cite>inencoding</cite> lexer options. If you have the <a class="reference external" href="https://chardet.github.io/">chardet</a> library installed and set the encoding to <code class="docutils literal notranslate"><span class="pre">chardet</span></code> if will analyse the text and use the encoding it thinks is the right one automatically:</p> <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pygments.lexers</span> <span class="kn">import</span> <span class="n">PythonLexer</span> <span class="n">lexer</span> <span class="o">=</span> <span class="n">PythonLexer</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'chardet'</span><span class="p">)</span> </pre></div> </div> <p>The best way is to pass Pygments unicode objects. In that case you can’t get unexpected output.</p> <p>The formatters now send Unicode objects to the stream if you don’t set the output encoding. You can do so by passing the formatters an <cite>encoding</cite> option:</p> <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="n">HtmlFormatter</span> <span class="n">f</span> <span class="o">=</span> <span class="n">HtmlFormatter</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span> </pre></div> </div> <p><strong>You will have to set this option if you have non-ASCII characters in the source and the output stream does not accept Unicode written to it!</strong> This is the case for all regular files and for terminals.</p> <p>Note: The Terminal formatter tries to be smart: if its output stream has an <cite>encoding</cite> attribute, and you haven’t set the option, it will encode any Unicode string with this encoding before writing it. This is the case for <cite>sys.stdout</cite>, for example. The other formatters don’t have that behavior.</p> <p>Another note: If you call Pygments via the command line (<cite>pygmentize</cite>), encoding is handled differently, see <a class="reference internal" href="cmdline.html"><span class="doc">the command line docs</span></a>.</p> <div class="versionadded"> <p><span class="versionmodified">New in version 0.7: </span>The formatters now also accept an <cite>outencoding</cite> option which will override the <cite>encoding</cite> option if given. This makes it possible to use a single options dict with lexers and formatters, and still have different input and output encodings.</p> </div> </div> </div> </div> </div> <div class="clearer"></div> </div> <div class="footer" role="contentinfo"> © Copyright 2006-2017, Georg Brandl and Pygments contributors. Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.6. <br/> Pygments logo created by <a href="http://joelunger.com">Joel Unger</a>. Backgrounds from <a href="http://subtlepatterns.com">subtlepatterns.com</a>. </div> </div> </body> </html>
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings