File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/gitdb/__pycache__/stream.cpython-310.pyc
Back
o ;j�k � @ s. d dl mZ d dlZd dlZd dlZd dlZd dlmZmZm Z m Z m Z d dl m Z mZmZmZmZ d dlmZmZ d dlmZ dZz d dlmZ d ZW n eyU Y nw d Z G d d � d e�Z G d d� de�Z G dd� d�Z G dd� de �Z G dd� de �Z!G dd� de �Z"G dd� d�Z#G dd� d�Z$dS )� )�BytesION)�msb_size� stream_copy�apply_delta_data�connect_deltas� delta_types)�allocate_memory� LazyMixin�make_sha�write�close)� NULL_BYTE� BYTE_SPACE)� force_bytesF)� apply_deltaT) �DecompressMemMapReader�FDCompressedSha1Writer�DeltaApplyReader� Sha1Writer�FlexibleSha1Writer�ZippedStoreShaWriterr �FDStream� NullStreamc @ s~ e Zd ZdZdZdZd dd�Zdd� Zd d � Zd d � Z e d dd��Z dd� Z dd� Z dd� Zeedd�fdd�Zd dd �ZdS ) r a� Reads data in chunks from a memory map and decompresses it. The client sees only the uncompressed data, respective file-like read calls are handling on-demand buffered decompression accordingly A constraint on the total size of bytes is activated, simulating a logical file within a possibly larger physical memory area To read efficiently, you clearly don't want to read individual bytes, instead, read a few kilobytes at least. **Note:** The chunk-size should be carefully selected as it will involve quite a bit of string copying due to the way the zlib is implemented. Its very wasteful, hence we try to find a good tradeoff between allocation time and number of times we actually allocate. An own zlib implementation would be good here to better support streamed reading - it would only need to keep the mmap and decompress it into chunks, that's all ... ) �_m�_zip�_buf�_buflen�_br�_cws�_cwe�_s�_close�_cbr�_phii Nc C sR || _ t�� | _d| _d| _|dur|| _d| _d| _d| _ d| _ d| _ || _ dS )z|Initialize with mmap for stream reading :param m: must be content data - use new if you have object data and no sizeNr F) r �zlib� decompressobjr r r r r r r r" r# r! )�self�m�close_on_deletion�size� r* �@/opt/saltstack/salt/lib/python3.10/site-packages/gitdb/stream.py�__init__E s z DecompressMemMapReader.__init__c C s |dksJ �| � � d S )Nr )�_parse_header_info�r& �attrr* r* r+ � _set_cache_U s z"DecompressMemMapReader._set_cache_c C s | � � d S �N)r �r& r* r* r+ �__del__[ � z DecompressMemMapReader.__del__c C s~ d}|| _ | �|�}|�t�}|d|� �t�\}}t|�}|| _ d| _|d7 }t||d� �| _ t |�| | _ d| _ ||fS )z�If this stream contains object data, parse the header info and skip the stream to a point where each read will yield object content :return: parsed type_string, sizei Nr � T) r �read�findr �splitr �intr r r �lenr r# )r& Zmaxb�hdrZhdrend�typr) r* r* r+ r- ^ s z)DecompressMemMapReader._parse_header_infoFc C s" t ||d�}|�� \}}|||fS )a� Create a new DecompressMemMapReader instance for acting as a read-only stream This method parses the object header from m and returns the parsed type and size, as well as the created stream instance. :param m: memory map on which to operate. It must be object data ( header + contents ) :param close_on_deletion: if True, the memory map will be closed once we are being deletedr )r r- )r& r' r( �instr< r) r* r* r+ �new{ s zDecompressMemMapReader.newc C � | j S )z8:return: random access compatible data we are working on)r r2 r* r* r+ �data� s zDecompressMemMapReader.datac C s* | j rt| jd�r| j�� d| _ dS dS )z�Close our underlying stream of compressed bytes if this was allowed during initialization :return: True if we closed the underlying stream :note: can be called safely r FN)r! �hasattrr r r2 r* r* r+ r � s �z DecompressMemMapReader.closec C s� | j | jkrJ| jjsJd| _ t| jd�r(| jjtjkr'| �t j � | jjtjksn | jjsF| j t | j �krF| �t j � | jjsF| j t | j �ks4| j| _ | j S )z� :return: number of compressed bytes read. This includes the bytes it took to decompress the header ( if there was one )r �status)r r r � unused_datarA rB r$ ZZ_OKr6 �mmap�PAGESIZEr" r: r r2 r* r* r+ �compressed_bytes_read� s �� �z,DecompressMemMapReader.compressed_bytes_read�SEEK_SETr c C sZ |dks |t tdd�krtd��t�� | _d | _ | _ | _| _ | j r+d| _ | ` dS dS )zgAllows to reset the stream to restart reading :raise ValueError: If offset and whence are not 0r rG �Can only seek to position 0FN) �getattr�os� ValueErrorr$ r% r r r r r" r# r �r& �offset�whencer* r* r+ �seek� s �zDecompressMemMapReader.seek���c C s� |dk r | j | j }n t|| j | j �}|dkrdS d}| jrR| j|kr:| j�|�}| j|8 _| j|7 _|S | j�� }|| j8 }| j| j7 _d| _d | _| jj}|rg| jt |� | _ | j | | _n | j }| j| _ || | _| j| j dk r�| j d | _| j | j | j� }| j t |� | _| j� ||�}t tdtj�dv r�tjdks�t | jj�}n t | jj�t | jj� }| jt |�| 7 _| jt |�7 _|r�|| }|r�t |�t |� |k r�| j| j k r�|| �|t |� �7 }|S )Nr5 r � � �ZLIB_RUNTIME_VERSION)z1.2.7z1.2.5�darwin)r r �minr r r6 r �unconsumed_tailr r: r r � decompressrI r$ � ZLIB_VERSION�sys�platformrC r" )r& r) Zdat�tailZcwsZindataZdcompdatZunused_datalenr* r* r+ r6 � sN $ zDecompressMemMapReader.readr1 �F)rP )�__name__� __module__� __qualname__�__doc__� __slots__Z max_read_sizer, r0 r3 r- � classmethodr> r@ r rF rI rJ rO r6 r* r* r* r+ r . s 1r c @ s� e Zd ZdZdZdZdd� Zdd� Zdd � Ze seZ neZ dd d �Z e e d d �fdd�Zedd� �Zedd� �Zedd� �Zedd� �ZdS )r a� A reader which dynamically applies pack deltas to a base object, keeping the memory demands to a minimum. The size of the final object is only obtainable once all deltas have been applied, unless it is retrieved from a pack index. The uncompressed Delta has the following layout (MSB being a most significant bit encoded dynamic size): * MSB Source Size - the size of the base against which the delta was created * MSB Target Size - the size of the resulting data after the delta was applied * A list of one byte commands (cmd) which are followed by a specific protocol: * cmd & 0x80 - copy delta_data[offset:offset+size] * Followed by an encoded offset into the delta data * Followed by an encoded size of the chunk to copy * cmd & 0x7f - insert * insert cmd bytes from the delta buffer into the output stream * cmd == 0 - invalid operation ( or error in delta stream ) )�_bstream� _dstreams� _mm_target�_sizer i���c C s: t |�dks J d��|d | _t|dd� �| _d| _dS )z�Initialize this instance with a list of streams, the first stream being the delta to apply on top of all following deltas, the last stream being the base object onto which to apply the deltasr5 z+Need at least one delta and one base streamrP Nr )r: rc �tuplerd r )r& � stream_listr* r* r+ r, h s zDeltaApplyReader.__init__c C s� t | j�dkr | �|�S t| j�}|�� dkr!d| _td�| _d S |�� | _t| j�| _t| jj �}t | jj |j | jj dt j � | jj }|�||� | j�d� d S )Nr5 r � )r: rd �_set_cache_brute_r Zrboundrf r re rc r) r r6 r rD rE ZapplyrO )r& r/ Zdcl�bbufr r* r* r+ � _set_cache_too_slow_without_cr s z.DeltaApplyReader._set_cache_too_slow_without_cc C sl t � }d}| jD ]&}|�d�}t|�\}}t||�\}}|�||d� |||f� t||�}q| jj} |}t| j�dkrCt| |� } }t | �} t | jj| j | dt j � t |�} d} tt|�t| j��D ]I\\} }}}}t |j| �}|� | � t |j|j |jdt j � dt� v r�t| || � n t| ||t|�| j � | | } } | �d� | �d� |} qd| | _| | _dS )z*If we are here, we apply the actual deltasr i Nr5 ri � c_apply_delta)�listrd r6 r �append�maxrc r) r: r r r rD rE �zip�reversed�globalsrm r rO re rf )r& r/ Zbuffer_info_listZmax_target_sizeZdstream�bufrM Zsrc_sizeZ target_sizeZ base_sizerk ZtbufZfinal_target_size�dbufZddatar* r* r+ rj � s: $ z"DeltaApplyReader._set_cache_brute_r c C sB | j | j }|dk s||kr|}| j�|�}| jt|�7 _|S )Nr5 )rf r re r6 r: )r& �countZblr@ r* r* r+ r6 � s zDeltaApplyReader.readrG c C s6 |dks |t tdd�krtd��d| _| j�d� dS )zhAllows to reset the stream to restart reading :raise ValueError: If offset and whence are not 0r rG rH N)rI rJ rK r re rO rL r* r* r+ rO � s zDeltaApplyReader.seekc C s< t |�dk r td��|d jtv rtd|d j ��| |�S )a Convert the given list of streams into a stream which resolves deltas when reading from it. :param stream_list: two or more stream objects, first stream is a Delta to the object that you want to resolve, followed by N additional delta streams. The list's last stream must be a non-delta stream. :return: Non-Delta OPackStream object whose stream can be used to obtain the decompressed resolved data :raise ValueError: if the stream list cannot be handled� zNeed at least two streamsrP zNCannot resolve deltas if there is no base object stream, last one was type: %s)r: rK �type_idr �type)�clsrh r* r* r+ r> � s �zDeltaApplyReader.newc C � | j jS r1 )rc ry r2 r* r* r+ ry � zDeltaApplyReader.typec C r{ r1 )rc rx r2 r* r* r+ rx r| zDeltaApplyReader.type_idc C r? )z3:return: number of uncompressed bytes in the stream)rf r2 r* r* r+ r) s zDeltaApplyReader.sizeN�r )r] r^ r_ r` ra Zk_max_memory_mover, rl rj � has_perf_modr0 r6 rI rJ rO rb r> �propertyry rx r) r* r* r* r+ r B s( "K r c @ s. e Zd ZdZdZdd� Zdd� Zd dd �Zd S ) r zpSimple stream writer which produces a sha whenever you like as it degests everything it is supposed to write�sha1c C s t � | _d S r1 )r r� r2 r* r* r+ r, 2 r4 zSha1Writer.__init__c C s | j �|� t|�S )z{:raise IOError: If not all bytes could be written :param data: byte object :return: length of incoming data)r� �updater: �r& r@ r* r* r+ r 7 s zSha1Writer.writeFc C s |r| j �� S | j �� S )z]:return: sha so far :param as_hex: if True, sha will be hex-encoded, binary otherwise)r� � hexdigest�digest)r& Zas_hexr* r* r+ �shaD s zSha1Writer.shaNr\ )r] r^ r_ r` ra r, r r� r* r* r* r+ r , s r c @ s$ e Zd ZdZdZdd� Zdd� ZdS )r zZWriter producing a sha1 while passing on the written bytes to the given write function�writerc C s t �| � || _d S r1 )r r, r� )r& r� r* r* r+ r, T s zFlexibleSha1Writer.__init__c C s t �| |� | �|� d S r1 )r r r� r� r* r* r+ r X s zFlexibleSha1Writer.writeN)r] r^ r_ r` ra r, r r* r* r* r+ r N s r c @ sP e Zd ZdZdZdd� Zdd� Zdd� Zd d � Ze e d d �fd d�Z dd� Z dS )r z=Remembers everything someone writes to it and generates a sha)rt rq c C s$ t �| � t� | _t�tj�| _d S r1 )r r, r rt r$ � compressobj� Z_BEST_SPEEDrq r2 r* r* r+ r, b s z ZippedStoreShaWriter.__init__c C s t | j|�S r1 )rI rt r. r* r* r+ � __getattr__g r4 z ZippedStoreShaWriter.__getattr__c C s$ t �| |�}| j�| j�|�� |S r1 )r r rt rq �compress)r& r@ �alenr* r* r+ r j s zZippedStoreShaWriter.writec C s | j �| j�� � d S r1 )rt r rq �flushr2 r* r* r+ r p s zZippedStoreShaWriter.closerG r c C s0 |dks |t tdd�krtd��| j�d� dS )z`Seeking currently only supports to rewind written data Multiple writes are not supportedr rG rH N)rI rJ rK rt rO rL r* r* r+ rO s s zZippedStoreShaWriter.seekc C s | j �� S )zA:return: string value from the current stream position to the end)rt �getvaluer2 r* r* r+ r� { s z ZippedStoreShaWriter.getvalueN) r] r^ r_ r` ra r, r� r r rI rJ rO r� r* r* r* r+ r ] s r c s< e Zd ZdZdZed�Z� fdd�Zdd� Zdd � Z � Z S ) r z�Digests data written to it, making the sha available, then compress the data and write it to the file descriptor **Note:** operates on raw file descriptors **Note:** for this to work, you have to use the close-method of this instance)�fdr� rq z+Failed to write all bytes to filedescriptorc s" t � �� || _t�tj�| _d S r1 )�superr, r� r$ r� r� rq �r& r� �� __class__r* r+ r, � s z FDCompressedSha1Writer.__init__c C s> | j �|� | j�|�}t| j|�}|t|�kr| j�t|�S )zZ:raise IOError: If not all bytes could be written :return: length of incoming data)r� r� rq r� r r� r: �exc)r& r@ ZcdataZ bytes_writtenr* r* r+ r � s z FDCompressedSha1Writer.writec C s. | j �� }t| j|�t|�kr| j�t| j�S r1 )rq r� r r� r: r� r )r& � remainderr* r* r+ r � s z FDCompressedSha1Writer.close) r] r^ r_ r` ra �IOErrorr� r, r r � __classcell__r* r* r� r+ r � s r c @ sF e Zd ZdZdZdd� Zdd� Zddd �Zd d � Zd d � Z dd� Z dS )r z�A simple wrapper providing the most basic functions on a file descriptor with the fileobject interface. Cannot use os.fdopen as the resulting stream takes ownership��_fd�_posc C s || _ d| _d S �Nr r� r� r* r* r+ r, � s zFDStream.__init__c C s$ | j t|�7 _ t�| j|� d S r1 )r� r: rJ r r� r� r* r* r+ r � s zFDStream.writer c C s: |dkr t j�| j�}t �| j|�}| jt|�7 _|S r� )rJ �path�getsizeZ _filepathr6 r� r� r: )r& rv �bytesr* r* r+ r6 � s z FDStream.readc C r? r1 )r� r2 r* r* r+ �fileno� � zFDStream.filenoc C r? r1 )r� r2 r* r* r+ �tell� r� z FDStream.tellc C s t | j� d S r1 )r r� r2 r* r* r+ r � s zFDStream.closeNr} ) r] r^ r_ r` ra r, r r6 r� r� r r* r* r* r+ r � s r c @ s0 e Zd ZdZe� Zd dd�Zdd� Zdd� Zd S ) r zVA stream that does nothing but providing a stream interface. Use it like /dev/nullr c C s dS )N� r* )r& r) r* r* r+ r6 � � zNullStream.readc C s d S r1 r* r2 r* r* r+ r � r� zNullStream.closec C s t |�S r1 )r: r� r* r* r+ r � s zNullStream.writeNr} ) r] r^ r_ r` rg ra r6 r r r* r* r* r+ r � s r )%�ior rD rJ rY r$ Z gitdb.funr r r r r Z gitdb.utilr r r r r Z gitdb.constr r Zgitdb.utils.encodingr r~ Zgitdb_speedups._perfr rm � ImportError�__all__r r r r r r r r r* r* r* r+ �<module> s8 � k"#("
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings