File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/__pycache__/s3fs.cpython-310.pyc
Back
o ;j�j � @ s> d Z ddlZddlZddlZddlZddlZddlZddlm Z ddl Z ddl Z ddlZ ddlZ ddlZ e�e�ZdZdd� Zdd� Zd6d d �Zd d � Zd d� Zdd� Zdd� Zdd� Z dd� Z dd� Z dd� Z dd � Z d d � Z!d d � Z"d!d"� Z#d#d$� Z$d%d&� Z%d'd(� Z&d)d*� Z'd+d,� Z(d-d.� Z)d/d0� Z*d7d2d3�Z+d4d5� Z,dS )8aj Amazon S3 Fileserver Backend .. versionadded:: 0.16.0 This backend exposes directories in S3 buckets as Salt environments. To enable this backend, add ``s3fs`` to the :conf_master:`fileserver_backend` option in the Master config file. .. code-block:: yaml fileserver_backend: - s3fs S3 credentials must also be set in the master config file: .. code-block:: yaml s3.keyid: GKTADJGHEIQSXMKKRBJ08H s3.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs Alternatively, if on EC2 these credentials can be automatically loaded from instance metadata. This fileserver supports two modes of operation for the buckets: 1. :strong:`A single bucket per environment` .. code-block:: yaml s3.buckets: production: - bucket1 - bucket2 staging: - bucket3 - bucket4 2. :strong:`Multiple environments per bucket` .. code-block:: yaml s3.buckets: - bucket1 - bucket2 - bucket3 - bucket4 Note that bucket names must be all lowercase both in the AWS console and in Salt, otherwise you may encounter ``SignatureDoesNotMatch`` errors. A multiple-environment bucket must adhere to the following root directory structure:: s3://<bucket name>/<environment>/<files> .. note:: This fileserver back-end requires the use of the MD5 hashing algorithm. MD5 may not be compliant with all security policies. .. note:: This fileserver back-end is only compatible with MD5 ETag hashes in the S3 metadata. This means that you must use SSE-S3 or plaintext for bucket encryption, and that you must not use multipart upload when uploading to your bucket. More information here: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html Objects without an MD5 ETag will be fetched on every fileserver update. If you deal with objects greater than 8MB, then you should use the following AWS CLI config to avoid mutipart upload: .. code-block:: text s3 = multipart_threshold = 1024MB More info here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html .. note:: This fileserver back-end will by default sync all buckets on every fileserver update. If you want files to be only populated in the cache when requested, you can disable this in the master config: .. code-block:: yaml s3.s3_sync_on_update: False � N�md5c C s t � } t| �� �S )z^ Return a list of directories within the bucket that can be used as environments. )�_init�list�keys)�metadata� r �H/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py�envsm s r c C s� t � } t�dd�rIt�d� | �� D ]/\}}t|�D ]&}|�� D ] \}}|D ]}t|||�}t�d|||� t | ||||� q&q qqt�d� dS dS )z/ Update the cache file for the bucket. zs3.s3_sync_on_updateTz Syncing local cache from S3...z %s - %s : %sz#Sync local cache from S3 completed.N) r �__opts__�get�log�info�items� _find_files�_get_cached_file_name�debug�_get_file_from_s3)r �saltenv�env_meta� bucket_files�bucket�filesZ file_path�cached_file_pathr r r �updatex s& � �����r �basec K s� d|v r |� d� ddd�}t� }|r||vr|S t|| �}t� s)tj�|| �} |D ] }|�� D ]\}}| |v rIt� t | �sI||d< | |d< nq1q+ |d rT|d sV|S t |d || �} t |||d | | � |S )z� Look through the buckets cache file for a match. If the field is found, it is retrieved from S3 only if its cached version is missing, or if the MD5 does not match. �envN)r �pathr r ) �popr r �_is_env_per_bucket�osr �joinr �fs�is_file_ignoredr r r ) r r �kwargs�fndr � env_filesr � bucket_namer r r r r � find_file� s. �r' c C s� d| v r | � d� i }d| vr|S d|vs d|vs |d s |S t|d | d |d �}tj�|�r?t|d< tjj� |t�|d< |S )z4 Return the hash of an object's cached copy r r r r Z hash_typeZhsum) r r r r �isfile� S3_HASH_TYPE�salt�utils� hashutils�get_hash)�loadr$ �retr r r r � file_hash� s � r0 c C s" d| v r | � d� ddd�}d| vsd| vsd| vr |S d|vs$d|vr&|S | �dd �}t|d | d |d �}t|d g| d �d |d < tjj�|d ��9}|�| d � |� t d �}|rktjj� |�sk|� t �}|r{|r{tjj�||�}||d< ||d< W d � |S 1 s�w Y |S )z? Return a chunk from a file based on the data received r � )�data�destr �locr r �gzipNr r3 �rbZfile_buffer_sizer2 )r r r �_trim_env_off_pathr* r+ r �fopen�seek�readr Z is_binary�decodeZ__salt_system_encoding__Z gzip_util�compress)r. r$ r/ r5 r �fp_r2 r r r � serve_file� s2 � � � r> c C s� d| v r | � d� g }d| vr|S | d }t� }|r ||vr |S t|| �D ]}|�� D ]}dd� |D �}|t||�7 }q,q&|S )zR Return a list of all files on the file server in a specified environment r r c S s g | ] }t �t|�s|�qS r )r! r" r )�.0�fr r r � <listcomp> � z file_list.<locals>.<listcomp>)r r r �valuesr7 )r. r/ r r r �bucketsr r r r � file_list s �rE c C s t � g S )z> Return a list of all empty directories on the master )r )r. r r r �file_list_emptydirs s rF c C s� d| v r | � d� g }d| vr|S | d }t� }|r ||vr |S t|| �D ]}|�� D ]}t||dd�}|dd� |D �7 }q,q&|S )z8 Return a list of all directories on the master r r T)� trim_slashc S s g | ]}|r|�qS r r )r? Z_fr r r rA = � z dir_list.<locals>.<listcomp>)r r � _find_dirsrC r7 )r. r/ r r r �dirsr r r �dir_list$ s �rK c C s� dt v rt d nd} dt v rt d nd}dt v r t d nd}dt v r&t d nd}dt v r0t d nd}dt v r:t d nd}d t v rDt d nd}d t v rNt d nd}| |||||||fS ) z, Get AWS keys from pillar or config zs3.keyNzs3.keyidzs3.service_urlz s3.verify_sslz aws.kms.keyidz aws.kmw.keyidz s3.locationz s3.path_stylezs3.https_enable�r )�key�keyid� service_url� verify_ssl� kms_keyid�location� path_style� https_enabler r r � _get_s3_keyB s$ ��rU c C sj t � } t�� t�dd� }d}ztj�| �|kr t| �}W n ty& Y nw |du r/t | �}t |� |S )zx Connect to S3 and download the metadata for each file in all buckets specified and cache the data to disk. zs3.s3_cache_expire� N) �_get_buckets_cache_filename�timer r r r �getmtime�_read_buckets_cache_file�OSError�_refresh_buckets_cache_file�_prune_deleted_files)� cache_file�expr r r r r ^ s � �r c C s t j�td d�S )z, Return the path to the s3cache dir ZcachedirZs3cache)r r r r r r r r �_get_cache_dirx s r` c C s t j�t� || |�S )z< Return the cached file name for a bucket path file �r r r r` )r& r r r r r r � � r c C s t j�t� d�S )z? Return the filename of the cache for bucket contents. zbuckets_files.cachera r r r r rW � s rW c sz t �d� t� \}}��}��� i }||f� ����fdd� }t� r�t� �� D ]m\�}g }|D ]`}i } ||�} | s:q/dd� | D �| |< |�| � | | s�i } | D ]} d| v sZd| v r_| �| � qPz t �d| d |� W q/ t y� d| v r�t �d| d |� Y q/t �d � i Y S w q/||�< q'n�t� D ]�}||�} | s�q�d d� | D �} | s�i } | D ]} d| v s�d| v r�| �| � q�z t �d| d |� W q� t y� d| v r�t �d| d |� Y q�t �d � i Y S w d d� | D �}t |�}|D ];��fd d�| D �}�|v�r g |�< d }|� D ]} || v �r&| | |7 < d} n�q|�s4|� �||i� q�q�t || � |S )zb Retrieve the content of all buckets and cache the metadata to the buckets cache file z Refreshing buckets cache filec s� g d}} t d |||| ���d�� d|id� }g }|D ] }d|v r% n|�|� q |�|� tdd � |D ��r< |S |d d }q) Nr1 T�s3.queryF�marker) rM rN rQ r rO rP rR Z return_binrS rT �params�Keyc S s g | ] }|� d d�dk�qS )Z IsTruncated�false)r )r? �headerr r r rA � rB zF_refresh_buckets_cache_file.<locals>.__get_s3_meta.<locals>.<listcomp>���)� __utils__�append�extend�all)r rM rN r/ rd �tmp�headersrh )rT rR rS rO rP r r � __get_s3_meta� s8 � � ��z2_refresh_buckets_cache_file.<locals>.__get_s3_metac S � g | ]}d |v r|�qS �rf r �r? �kr r r rA � � z/_refresh_buckets_cache_file.<locals>.<listcomp>ZCode�Messagez '%s' response for bucket '%s'z2S3 Error! Do you have any files in your S3 bucket?c S rq rr r rs r r r rA � ru c S s( g | ]}t j�|d ��dd�d �qS )rf �/� r )r r �dirname�splitrs r r r rA s ( c s g | ] }|d � � �r|�qS rr )� startswithrs )r r r rA # s FT) r r rU r � _get_bucketsr rk r �warning�KeyError�set�_write_buckets_cache_file)r^ rM rN rQ r rp rD Zbucket_files_listr& r Zs3_metaZ meta_responsert r Z environmentsr% �foundr )rT rR rS r rO rP r r\ � s� � �� ���� � 0 �� �� � ��� r\ c C s� t � }t� }t� }t� rE| �� D ]3\}}|D ],}|�� D ]%\}}tj�|||�} tj�| �r3|�| � |D ] } | d } |�| � q5q qqn-| �� D ](\}}t � D ]}tj�||�} tj�| �rd|�| � qP|D ] } |�| d � qgqIt � t j �r�dd l} t �d| �|�� |D ]f} t�| �D ]^\} }}|D ]V}tj�| |�} tj�| | �}||vr�t �d| � t �d|� t�| � tj�| �}||kr�tt�|��dkr�t �d|� t�|� tj�|�}||kr�tt�|��dks�q�q�q�d S )Nrf r zcached file list: %sz'File '%s' not found in cached file listz6File '%s' was deleted from bucket, deleting local copyz%Directory '%s' is now empty, removing)r` r r r r r r �exists�addr| r � isEnabledFor�logging�DEBUG�pprintr Zpformat�walk�relpathr �unlinkry �len�listdir�rmdir)r � cache_dirZ cached_files�rootsr Zenv_data� bucket_metar Z bucket_data�root�metar r� r rJ r � file_namer� ry r r r r] 7 sf ���� �� � �����r] c C s~ t � }tj�|�st�|� tj�|�rt�|� t�d� t j j � |d��}t �| |� W d � dS 1 s8w Y dS )z6 Write the contents of the buckets cache file zWriting buckets cache file�wbN)r` r r r� �makedirsr( �remover r r* r+ r r8 �pickle�dump)r r^ r� r= r r r r� p s "�r� c C s� t �d� tj�| �st �d� dS tjj�| d��;}zt � |�}W n" t j t t ttttfyD } zt �d|� d}W Y d}~n d}~ww W d � |S W d � |S 1 sXw Y |S )z7 Return the contents of the buckets cache file zReading buckets cache filezCache file does not existNr6 z*Exception reading buckets cache file: '%s')r r r r r� r* r+ r r8 r� r. ZUnpicklingError�AttributeError�EOFError� ImportError� IndexErrorr~ � ValueError)r^ r= r2 �excr r r rZ � s8 � ��� �� ��rZ c C s� g }i }| D ]<}|� � D ]5\}}dd� |D �}dd� |D �}||vr.d||< |�||i� q |D ]}||v r@|| |7 < nq0q q|S )zA Looks for all the files in the S3 bucket cache metadata c S � g | ]}|d �qS rr r rs r r r rA � rH z _find_files.<locals>.<listcomp>c S s g | ] }|� d �s|�qS )rw )�endswithrs r r r rA � s T)r rk )r r/ r� � bucket_dictr& r2 Z filepathsr r r r r � s" ��� r c C s� g }i }| D ]d}|� � D ]]\}}t� }dd� |D �D ] }d}|�d�dd� D ]} || d } |�| � | }q'q||vrJd||< |�|t|�i� q |D ] } || v rh| | t|�7 < tt| | ��| |< nqLq q|S )z� Looks for all the directories in the S3 bucket cache metadata. Supports trailing '/' keys (as created by S3 console) as well as directories discovered in the path of file keys. c S r� rr r rs r r r rA � rH z _find_dirs.<locals>.<listcomp>r1 rw Nri T)r r rz r� rk r ) r r/ r� r� r& r2 Zdirpathsr �prefix�part� directoryr r r r rI � s0 ����rI c C s� || v r| | ni }i }|D ] }||v r|| }qt t tdd� |���}|D ]'}d|v rM|d |krMz|d �d�|d< W | S tyL Y | S w q&dS )zA Looks for a file's metadata in the S3 bucket cache file c S s d| v S )Nrf r )rt r r r �<lambda>� s z!_find_file_meta.<locals>.<lambda>rf �ETag�"N)r �filter�stripr~ ) r r& r r r r� r Z files_metaZ item_metar r r �_find_file_meta� s$ � ����r� c C s dt v rt d S i S )z* Return the configuration buckets z s3.bucketsrL r r r r r| � rb r| c C s� t j�|�}t j�|�st �|� t� \}}}} } } } } t j�|�r�t| |||�}|r�|d }|�d�dkrM|}t j j � |t �}||krFdS t�d|� n�t �|�}|j}tj�|j�}tj�|d d�}|t|d �kr�||kr�t�d � td |||d ||| | tj�|�|d | | d � }|dur�d }}|d � � D ](\}}|� � }|� � } t |�� � dkr�tj�| d�}q�t |�� � dkr�t| �}q�|r�||kr�|r�||kr�t�d|||� dS td |||||| | tj�|�|| | d� dS )zw Checks the local cache for the file, if it's old or missing go grab the file from S3 and update the cache r� �-ri Nz-found different hash for file %s, updating...Z LastModifiedz%Y-%m-%dT%H:%M:%S.%fZZSizezicached file size equal to metadata size and cached file mtime later than metadata last modification time.rc �HEADT) rM rN rQ �methodr rO rP rR r � local_fileZ full_headersrS rT ro z last-modifiedz%a, %d %b %Y %H:%M:%S %Zzcontent-lengthzW%s - %s : %s skipped download since cached file size equal to and mtime after s3 values) rM rN rQ r rO rP rR r r� rS rT ) r r ry r� r� rU r( r� �findr* r+ r, r- r) r r �stat�st_size�datetime� fromtimestamp�st_mtime�strptime�intr rj �urllib�parse�quoter r� �str�lower) r r r& r r � target_dirrM rN rO rP rQ rR rS rT Z file_metaZ file_etagZfile_md5Z cached_md5Zcached_file_statZcached_file_sizeZcached_file_mtimeZcached_file_lastmodr/ Z s3_file_mtimeZ s3_file_size� header_nameZ header_value�name�valuer r r r � s� � � ��� ��� �� �r Fc s6 t � rdnt|�d � |rdnd�� �fdd�| D �S )zH Return a list of file paths with the saltenv directory removed Nrx ri c s g | ]}|� �� �qS r r )r? �d�Zenv_lenZ slash_lenr r rA k ru z&_trim_env_off_path.<locals>.<listcomp>)r r� )�pathsr rG r r� r r7 d s r7 c C s* t � } t| t�r dS t| t�rdS td��)z� Return the configuration mode, either buckets per environment or a list of buckets that have environment dirs in their root TFz)Incorrect s3.buckets type given in config)r| � isinstance�dictr r� )rD r r r r n s r )r )F)-�__doc__r� r� r r� rX � urllib.parser� Zsalt.fileserverZ fileserverr! Z salt.modulesr* Zsalt.utils.filesZsalt.utils.gzip_utilZsalt.utils.hashutilsZsalt.utils.versions� getLogger�__name__r r) r r r' r0 r> rE rF rK rU r r` r rW r\ r] r� rZ r rI r� r| r r7 r r r r r �<module> sP Y +% (9 q
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings