File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__pycache__/s3.cpython-310.pyc
Back
o ;j�9 � @ s� d Z ddlZddlZddlZddlZddlZddlmZ ddl Z ddl Z ddl m Z e�e�ZG dd� d�Z d d d �Zdd� Zdd� Zdd� Zdd� Zdd� Zdd� Zdd� Zd d � Zd d � ZdS )!a� Copy pillar data from a bucket in Amazon S3 The S3 pillar can be configured in the master config file with the following options .. code-block:: yaml ext_pillar: - s3: bucket: my.fancy.pillar.bucket keyid: KASKFJWAKJASJKDAJKSD key: ksladfDLKDALSFKSD93q032sdDasdfasdflsadkf multiple_env: False environment: base prefix: somewhere/overthere verify_ssl: True service_url: s3.amazonaws.com kms_keyid: 01234567-89ab-cdef-0123-4567890abcde s3_cache_expire: 30 s3_sync_on_update: True path_style: False https_enable: True The ``bucket`` parameter specifies the target S3 bucket. It is required. The ``keyid`` parameter specifies the key id to use when access the S3 bucket. If it is not provided, an attempt to fetch it from EC2 instance meta-data will be made. The ``key`` parameter specifies the key to use when access the S3 bucket. If it is not provided, an attempt to fetch it from EC2 instance meta-data will be made. The ``multiple_env`` defaults to False. It specifies whether the pillar should interpret top level folders as pillar environments (see mode section below). The ``environment`` defaults to 'base'. It specifies which environment the bucket represents when in single environments mode (see mode section below). It is ignored if multiple_env is True. The ``prefix`` defaults to ''. It specifies a key prefix to use when searching for data in the bucket for the pillar. It works when multiple_env is True or False. Essentially it tells ext_pillar to look for your pillar data in a 'subdirectory' of your S3 bucket The ``verify_ssl`` parameter defaults to True. It specifies whether to check for valid S3 SSL certificates. *NOTE* If you use bucket names with periods, this must be set to False else an invalid certificate error will be thrown (issue #12200). The ``service_url`` parameter defaults to 's3.amazonaws.com'. It specifies the base url to use for accessing S3. The ``kms_keyid`` parameter is optional. It specifies the ID of the Key Management Service (KMS) master key that was used to encrypt the object. The ``s3_cache_expire`` parameter defaults to 30s. It specifies expiration time of S3 metadata cache file. The ``s3_sync_on_update`` parameter defaults to True. It specifies if cache is synced on update rather than jit. The ``path_style`` parameter defaults to False. It specifies whether to use path style requests or dns style requests The ``https_enable`` parameter defaults to True. It specifies whether to use https protocol or http protocol This pillar can operate in two modes, single environment per bucket or multiple environments per bucket. Single environment mode must have this bucket structure: .. code-block:: text s3://<bucket name>/<prefix>/<files> Multiple environment mode must have this bucket structure: .. code-block:: text s3://<bucket name>/<prefix>/<environment>/<files> If you wish to define your pillar data entirely within S3 it's recommended that you use the `prefix=` parameter and specify one entry in ext_pillar for each environment rather than specifying multiple_env. This is due to issue #22471 (https://github.com/saltstack/salt/issues/22471) � N)�deepcopy)�Pillarc @ s e Zd Z ddd�ZdS )� S3CredentialsTNFc C s: || _ || _|| _|| _|| _|| _|| _|| _| | _d S �N) �key�keyid� kms_keyid�bucket� service_url� verify_ssl�location� path_style� https_enable) �selfr r r r r r r r r � r �B/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/s3.py�__init__i s zS3Credentials.__init__)TNNFT)�__name__� __module__� __qualname__r r r r r r h s �r TF�base� � c C sN t |||| || |||� }tj�tj�t� ||��}| r&tj�tj�|| ��}td �|g �|gkr3i S t||||| | �}| rxt � d� |� � D ]+\}}t |�� � D ] \}}|D ]}t |||�}t � d|||� t||||||� qWqQqGt � d� tt�}|r�tj�||�gn|g|d |< dd� |d D �|d< t|t| |�}|jdd �}|S ) z7 Execute a command and read the output as YAML Z pillar_rootsz%Syncing local pillar cache from S3...z %s - %s : %sz*Sync local pillar cache from S3 completed.c S s g | ]}d |vr|�qS )Zs3r )�.0�xr r r � <listcomp>� � z ext_pillar.<locals>.<listcomp>� ext_pillarF)�ext)r �os�path�normpath�join�_get_cache_dir�__opts__�get�_init�log�info�items� _find_files�_get_cached_file_name�_get_file_from_s3r r Z __grains__Zcompile_pillar)Z minion_idZpillarr r r r r � multiple_env� environment�prefixr r �s3_cache_expireZs3_sync_on_updater r Zs3_credsZ pillar_dir�metadata�saltenv�env_meta�files� file_path�cached_file_path�optsZpilZcompiled_pillarr r r r � sN � � ��� � r c C s� t ||�}t�� | }tj�|�rtj�|�}nd}||k} t�d|| r&dnd|| |� | r8t| ||||�} nt |�} t�d| � | S )zx Connect to S3 and download the metadata for each file in all buckets specified and cache the data to disk. r zDS3 bucket cache file %s is %sexpired, mtime_diff=%ss, expiration=%ssr znot z S3 bucket retrieved pillars %s) �_get_buckets_cache_filename�timer r �isfile�getmtimer' �debug�_refresh_buckets_cache_file�_read_buckets_cache_file) �credsr r- r. r/ r0 � cache_file�expZcache_file_mtimeZexpiredZpillarsr r r r& � s( � � r& c C s6 t j�td d�} t j�| �st�d� t �| � | S )zI Get pillar cache directory. Initialize it if it does not exist. ZcachedirZ pillar_s3fsz Initializing S3 Pillar Cache)r r r" r$ �isdirr' r< �makedirs)� cache_dirr r r r# � s r# c C s> t j�t� || |�}t j�t j�|��s t �t j�|�� |S )z< Return the cached file name for a bucket path file )r r r" r# �exists�dirnamerC )r r2 r r5 r r r r+ � s r+ c C s6 t � }tj�|�st�|� tj�|| � d|� d��S )zi Return the filename of the cache for bucket contents. Create the path if it does not exist. �-z -files.cache)r# r r rE rC r" )r r/ rD r r r r8 s r8 c s� d� �fdd� }dd� }dd� }dd � }t �d � i } � j} |sNt �d � i } |� } | rM|| �| | < || �} | s:n|| �} | | || �7 < q3| | |< nQt �d � |� } | r�|| �} || �} | sdn || �} ||| �7 }q]||�}|D ])��fdd�|D �}�| vr�i | �< | | � vr�g | � | < | � | |7 < qutj�|�r�t�|� t �d� tjj � |d��}t � | |� W d � | S 1 s�w Y | S )zb Retrieve the content of all buckets and cache the metadata to the buckets cache file Nc sJ �dd�}| r | |d< t d � j� j� j� j� j� j� jd|� j� j d� S )N� )r/ z list-typezcontinuation-token�s3.queryF) r r r r r r r Z return_bin�paramsr r ) � __utils__r r r r r r r r r )�continuation_tokenrJ )r? r/ r r � __get_s3_meta! s �z2_refresh_buckets_cache_file.<locals>.__get_s3_metac S s dd� | D �S )Nc S s g | ]}d |v r|�qS ��Keyr �r �kr r r r 8 r zX_refresh_buckets_cache_file.<locals>.__get_pillar_files_from_s3_meta.<locals>.<listcomp>r ��s3_metar r r � __get_pillar_files_from_s3_meta7 s zD_refresh_buckets_cache_file.<locals>.__get_pillar_files_from_s3_metac S s dd� | D �}t |�S )Nc S s( g | ]}t j�|d ��dd�d �qS )rO �/� r )r r rF �splitrP r r r r < s ( zR_refresh_buckets_cache_file.<locals>.__get_pillar_environments.<locals>.<listcomp>)�set)r4 � environmentsr r r �__get_pillar_environments; s z>_refresh_buckets_cache_file.<locals>.__get_pillar_environmentsc S s t dd� | D �d �S )Nc s s$ � | ] }|� d �r|� d �V qdS )ZNextContinuationTokenN)r% )r �itemr r r � <genexpr>A s � � �zP_refresh_buckets_cache_file.<locals>.__get_continuation_token.<locals>.<genexpr>)�nextrR r r r �__get_continuation_token? s ��z=_refresh_buckets_cache_file.<locals>.__get_continuation_tokenz'Refreshing S3 buckets pillar cache filez"Single environment per bucket modeTz$Multiple environment per bucket modec s g | ] }|d � � �r|�qS rN )� startswithrP )r2 r r r y s z/_refresh_buckets_cache_file.<locals>.<listcomp>z$Writing S3 buckets pillar cache file�wbr ) r' r< r r r r: �remove�salt�utilsr4 �fopen�pickle�dump)r? r@ r- r. r/ rM rT rZ r^ r1 r Z bucket_filesrS rL r4 rY Z env_files�fp_r )r? r/ r2 r r= sd �� � ��r= c C sJ t �d� tjj�| d��}t�|�}W d � |S 1 s w Y |S )z7 Return the contents of the buckets cache file zReading buckets cache file�rbN)r' r< rb rc r4 rd re �load)r@ rg �datar r r r> � s ��r> c C sR i }| � � D ] \}}||vrg ||< dd� |D �}|| dd� |D �7 < q|S )zA Looks for all the files in the S3 bucket cache metadata c S s g | ]}|d �qS rN r rP r r r r � s z _find_files.<locals>.<listcomp>c S s g | ] }|� d �s|�qS )rU )�endswithrP r r r r � s )r) )r1 �retr rj Z filePathsr r r r* � s r* c C sh || v r| | ni }||v r|| ni }t t tdd� |���}|D ]}d|v r1|d |kr1| S q!dS )zA Looks for a file's metadata in the S3 bucket cache file c S s d| v S )NrO r )rQ r r r �<lambda>� s z!_find_file_meta.<locals>.<lambda>rO N)�list�filter)r1 r r2 r r3 Z bucket_metaZ files_metaZ item_metar r r �_find_file_meta� s ��rp c C s� t j�|�r4t||||�}|r d�tttj|d ���nd}t j j � |d�}t �d|||� ||kr4dS td | j| j| j|| jtj�|�|| j| j| j| jd� dS )zw Checks the local cache for the file, if it's old or missing go grab the file from S3 and update the cache r ZETagN�md5z%Cached file: path=%s, md5=%s, etag=%srI ) r r r r r r Z local_filer r r r )r r r: rp r" rn ro �str�isalnumrb rc Z hashutilsZget_hashr' r< rK r r r r �urllib�parse�quoter r r r ) r? r1 r2 r r r6 Z file_metaZfile_md5Z cached_md5r r r r, � s4 �� �r, ) NNTNFr r NNr TFT) �__doc__�loggingr re r9 � urllib.parsert �copyr Zsalt.utils.filesrb Zsalt.utils.hashutilsZ salt.pillarr � getLoggerr r' r r r& r# r+ r8 r= r> r* rp r, r r r r �<module> sF Y �L% u
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings