File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/states/__pycache__/boto_dynamodb.cpython-310.pyc
Back
o ;jyt � @ s� d Z ddlZddlZddlZddlZddlZddlm mZ ej ej dej d� e� � Z G dd� de�Zdd� Z d#d d �Zd d� Zdd� Zdd� Zdd� Zdd� Zdd� Zdd� Zdd � Zd d � Zd d � Zd$d!d"�Z dS )%a� Manage DynamoDB Tables ====================== .. versionadded:: 2015.5.0 Create and destroy DynamoDB tables. Be aware that this interacts with Amazon's services, and so may incur charges. This module uses ``boto``, which can be installed via package, or pip. This module accepts explicit DynamoDB credentials but can also utilize IAM roles assigned to the instance through Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More information available `here <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html>`_. If IAM roles are not used you need to specify them either in a pillar file or in the minion's config file: .. code-block:: yaml keyid: GKTADJGHEIQSXMKKRBJ08H key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs region: us-east-1 It's also possible to specify ``key``, ``keyid`` and ``region`` via a profile, either passed in as a dict, or as a string to pull from pillars or minion config: .. code-block:: yaml myprofile: keyid: GKTADJGHEIQSXMKKRBJ08H key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs region: us-east-1 .. code-block:: yaml Ensure DynamoDB table does not exist: boto_dynamodb.absent: - table_name: new_table - keyid: GKTADJGHEIQSXMKKRBJ08H - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs - region: us-east-1 Ensure DynamoDB table exists: boto_dynamodb.present: - table_name: new_table - read_capacity_units: 1 - write_capacity_units: 2 - hash_key: primary_id - hash_key_data_type: N - range_key: start_timestamp - range_key_data_type: N - keyid: GKTADJGHEIQSXMKKRBJ08H - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs - region: us-east-1 - local_indexes: - index: - name: "primary_id_end_timestamp_index" - hash_key: primary_id - hash_key_data_type: N - range_key: end_timestamp - range_key_data_type: N - global_indexes: - index: - name: "name_end_timestamp_index" - hash_key: name - hash_key_data_type: S - range_key: end_timestamp - range_key_data_type: N - read_capacity_units: 3 - write_capacity_units: 4 It's possible to specify cloudwatch alarms that will be setup along with the DynamoDB table. Note the alarm name will be defined by the name attribute provided, plus the DynamoDB resource name. .. code-block:: yaml Ensure DynamoDB table exists: boto_dynamodb.present: - name: new_table - read_capacity_units: 1 - write_capacity_units: 2 - hash_key: primary_id - hash_key_data_type: N - range_key: start_timestamp - range_key_data_type: N - alarms: ConsumedWriteCapacityUnits: name: 'DynamoDB ConsumedWriteCapacityUnits **MANAGED BY SALT**' attributes: metric: ConsumedWriteCapacityUnits namespace: AWS/DynamoDB statistic: Sum comparison: '>=' # threshold_percent is used to calculate the actual threshold # based on the provisioned capacity for the table. threshold_percent: 0.75 period: 300 evaluation_periods: 2 unit: Count description: 'DynamoDB ConsumedWriteCapacityUnits' alarm_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ] insufficient_data_actions: [] ok_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ] - keyid: GKTADJGHEIQSXMKKRBJ08H - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs - region: us-east-1 You can also use alarms from pillars, and override values from the pillar alarms by setting overrides on the resource. Note that 'boto_dynamodb_alarms' will be used as a default value for all resources, if defined and can be used to ensure alarms are always set for a resource. Setting the alarms in a pillar: .. code-block:: yaml boto_dynamodb_alarms: ConsumedWriteCapacityUnits: name: 'DynamoDB ConsumedWriteCapacityUnits **MANAGED BY SALT**' attributes: metric: ConsumedWriteCapacityUnits namespace: AWS/DynamoDB statistic: Sum comparison: '>=' # threshold_percent is used to calculate the actual threshold # based on the provisioned capacity for the table. threshold_percent: 0.75 period: 300 evaluation_periods: 2 unit: Count description: 'DynamoDB ConsumedWriteCapacityUnits' alarm_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ] insufficient_data_actions: [] ok_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ] Ensure DynamoDB table exists: boto_dynamodb.present: - name: new_table - read_capacity_units: 1 - write_capacity_units: 2 - hash_key: primary_id - hash_key_data_type: N - range_key: start_timestamp - range_key_data_type: N - alarms: ConsumedWriteCapacityUnits: attributes: threshold_percent: 0.90 period: 900 � Nz.%(asctime)s %(name)s %(levelname)s %(message)s)�level�format�streamc @ s e Zd ZdZdS )�GsiNotUpdatableErrorz7Raised when a global secondary index cannot be updated.N)�__name__� __module__� __qualname__�__doc__� r r �M/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/boto_dynamodb.pyr � s r c C s dt v rdS dS )z2 Only load if boto_dynamodb is available. �boto_dynamodb.existsZ boto_dynamodb)Fz(boto_dynamodb module could not be loaded)�__salt__r r r r � __virtual__� s r �boto_dynamodb_alarms� boto_dynamodb_backup_configsc " C s | ddi d�}|rdg|d< ||d< |} g }i }i }t d | ||||�}|s�td r8d |d < d | � d �|d < |S t d | ||||||| | | | ||� }|s_d|d < d| � �|d < t|||� |S |�d | � d�� | |d< ||d< ||d< | |d< | |d< | |d< | |d< ||d< ||d< n |�d | � d�� t d | ||||�}|�d i ��d i �}|�d �}|�d �}||ko�||k}|�std r�d |d < |�d | � d!�� nKt d" | ||d#�||||d$�} | s�d|d < d%| � �|d < t|||� |S |�d | � d&�� |f|d< |f|d< |f|d< |f|d< n |�d | � d'�� |�d i ��d(g �} t| ||||| ||||� } | d �s[|�| d � | d |d < |d du �r[d)�|�|d < t|||� |S t| || ||||||� } t� |d* | d* �|d*< |�| d � | d �s�| d |d < |d du �r�d)�|�|d < t|||� |S t � t d+ |g ��} | D ]T} t | | d | d, | d- | d. d/�}!|!d d0v �r�|!d |d < |�|!d � |!�d*��r�|!�d*�f|d* d1� | d �< �q�d)�|d |!d g�|d < t|||� | S d)�|�|d < t|||� |S )2a� Ensure the DynamoDB table exists. Table throughput can be updated after table creation. Global secondary indexes (GSIs) are managed with some exceptions: - If a GSI deletion is detected, a failure will occur (deletes should be done manually in the AWS console). - If multiple GSIs are added in a single Salt call, a failure will occur (boto supports one creation at a time). Note that this only applies after table creation; multiple GSIs can be created during table creation. - Updates to existing GSIs are limited to read/write capacity only (DynamoDB limitation). name Name of the DynamoDB table table_name Name of the DynamoDB table (deprecated) region Region to connect to. key Secret key to be used. keyid Access key to be used. profile A dict with region, key and keyid, or a pillar key (string) that contains a dict with region, key and keyid. read_capacity_units The read throughput for this table write_capacity_units The write throughput for this table hash_key The name of the attribute that will be used as the hash key for this table hash_key_data_type The DynamoDB datatype of the hash key range_key The name of the attribute that will be used as the range key for this table range_key_data_type The DynamoDB datatype of the range key local_indexes The local indexes you would like to create global_indexes The global indexes you would like to create backup_configs_from_pillars Pillars to use to configure DataPipeline backups T� ��name�result�comment�changeszMboto_dynamodb.present: `table_name` is deprecated. Please use `name` instead.�warningsr r �testNr �DynamoDB table z would be created.r zboto_dynamodb.create_tableFzFailed to create table z was successfully created�table�read_capacity_units�write_capacity_units�hash_key�hash_key_data_type� range_key�range_key_data_type� local_indexes�global_indexes� existszboto_dynamodb.describeZTable�ProvisionedThroughput�WriteCapacityUnits�ReadCapacityUnitsz is set to be updated.zboto_dynamodb.update��read�write)Z throughput�region�key�keyid�profilezFailed to update table z was successfully updatedz throughput matchesZGlobalSecondaryIndexesz, r z pillar.get�period�utc_hour�s3_base_location)r � schedule_namer. r/ r0 )TNzbackup_datapipeline_{})r �__opts__� _add_changes�append�get�_global_indexes_present�join�_alarms_present� dictupdate�update�copy�deepcopy�#_ensure_backup_datapipeline_presentr )"r � table_namer* r+ r, r- r r �alarms�alarms_from_pillarr r r r r! r" Zbackup_configs_from_pillars�ret�comments� changes_old� changes_newZ table_existsZ is_created� descriptionZprovisioned_throughputZ current_write_capacity_unitsZcurrent_read_capacity_unitsZthroughput_matchesZ is_updated�provisioned_indexesZ_retZdatapipeline_configsZconfigZdatapipeline_retr r r �present� s S�� � ��� � � �� �� rG c C s( |r|| d d< |r|| d d< d S d S )Nr �old�newr )rA rC rD r r r r3 � s �r3 c C s� ddi} | rdd� | D �} ni } t | �� �} i } |rR|D ]5}tt|�� ��}d}|D ]}|�� dgkr;tt|�� ��}q*|sMd| d< d|� �| d < | S || |< q t| t | �� ��\}}}|rpd| d< d �d �|��| d < | S t|�d kr�d| d< d �|�| d < | S |r�tt|��}t | |||||| |||| � | d s�| S |r�t | ||||| | ||||| � | d s�| S d|vr�d|vr�|� d� | S )z;Handles global secondary index for the table present state.r Tc S s i | ]}|d |�qS )Z IndexNamer )�.0�indexr r r � <dictcomp>� s �z+_global_indexes_present.<locals>.<dictcomp>Nr Fz Index name not found for table r zSDeletion of GSIs ({}) is not supported! Please do this manually in the AWS console.�, � ziCreation of multiple GSIs ({}) is not supported due to API limitations. Please create them one at a time.r" z"All global secondary indexes match) �set�keys�next�iter�values�_partition_index_namesr r7 �len�_add_global_secondary_index� _update_global_secondary_indexesr4 )rF r" rC rD rB r r* r+ r, r- rA �provisioned_gsi_config�provisioned_index_names� gsi_configrK Z index_config� index_name�entry�existing_index_names�new_index_names�index_names_to_be_deletedr r r r6 � s� � � � � � � � r6 c C sF t � }t � }|D ]}|| v r|�|� q|�|� q| | }|||fS )zOReturns 3 disjoint sets of indexes: existing, to be created, and to be deleted.)rO �add)rY Z index_namesr] r^ r r_ r r r rT $ s rT c C s� t d rd| d< d�||�| d< dS |�di � td |td || d d �||| | d �} | r@|�d |� �� || |d |< dS d | d< d|� �| d< dS )�4Updates ret iff there was a failure or in test mode.r Nr z)Dynamo table {} will have a GSI added: {}r r" z+boto_dynamodb.create_global_secondary_index�boto_dynamodb.extract_indexT�Z global_index�r* r+ r, r- z Created GSI FzFailed to create GSI )r2 r � setdefaultr r4 ) rA r r[ rC rD rB rZ r* r+ r, r- �successr r r rV 1 s, � �� rV c C s z t |||�\} } W n ty' } zd| d< t|�| d< W Y d}~dS d}~ww | r�td rAd| d< d�|d�| �� ��| d< dS |�di � |�di � td || || | | d �}|rz|� d | � �� | D ]}| | |d |< | | |d |< qedS d| d< d | � �| d< dS dS ) ra Fr r Nr z*Dynamo table {} will have GSIs updated: {}rM r" z+boto_dynamodb.update_global_secondary_indexrd z"Updated GSIs with new throughputs z!Failed to update GSI throughputs ) �_determine_gsi_updatesr �strr2 r r7 rP re r r4 )rA rC rD rB r] rX rZ r r* r+ r, r- �provisioned_throughputs� index_updates�erf r[ r r r rW Y sL � ��� � ���rW c C s. i }i }| D ]�}|| }t d || dd��� }|D ]C}||v r]|dkr]|| } || } |dkrQ| d | d kr<td��t| �dg ��t| �dg ��krPtd ��q| | kr]td �||���q|�d�} | �d �} | �d �} | | d �||< |�d�}|�d �}|�d �}| |ks�| |kr�||d �||< q||fS )Nrb Trc r$ Z ProjectionZProjectionTypez!GSI projection types do not matchZNonKeyAttributesz0NonKeyAttributes do not match for GSI projectionz.GSI property {} cannot be updated for index {}r& r% r' )r Zschemar rO r5 r )r] rX rZ ri rj r[ Zcurrent_configZ new_configr+ Z new_valueZ current_valueZcurrent_throughputZ current_readZ current_writeZnew_throughputZnew_readZ new_writer r r rg � sV �� ������ � �rg c C s� t �td |i ��} |rt�| |�} | ddi d�} | �� D ]�\} } | d | d | d< | d | d d | d d< d | gi| d d < | d d d krrd | d vrrt�|| d d �| d d < | d d= | d d | d d 9 < | d d dkr�d | d vr�t�|| d d �| d d < | d d= | d d | d d 9 < | d | d ||||d�} td di | ��}|d s�|d | d< |� di �i kr�|d | d | d < d|v r�| d |d 7 < q | S )zAhelper method for present. ensure that cloudwatch_alarms are setz config.optionTr r � r � attributesrE Z TableNameZ dimensionsZmetricZConsumedWriteCapacityUnitsZ thresholdZthreshold_percentr. ZConsumedReadCapacityUnits)r rm r* r+ r, r- z boto_cloudwatch_alarm.presentr r r Nr ) r; r< r r9 r: �items�math�ceil� __states__r5 )r r? r@ r r r* r+ r, r- �tmpZmerged_return_value�_�info�kwargs�resultsr r r r8 � sN � � � � �r8 c C sX | � d|� d�d||dt | |��� d�d�i| |� d| � d�d�d �}td d i |��S ) N�-z-backupZDefaultScheduleZSchedule)r. �typeZ startDateTime)r �fields�/)ZmyDDBTableNameZ myOutputS3Loc)r Zpipeline_objectsZparameter_valueszboto_datapipeline.presentr )� _next_datetime_with_utc_hour� isoformatrq )r r1 r. r/ r0 ru r r r r= s" ���� ��r= c C s t | �| S )zV For a given table_name, returns hash of the table_name limited by max_value. )�hash)r> Z max_valuer r r �'_get_deterministic_value_for_table_name! s r~ c C sX t j�� }t j |j|j|j|t| d�t| d�d�}|t j �� k r*t jdd�}||7 }|S )ax Datapipeline API is throttling us, as all the pipelines are started at the same time. We would like to uniformly distribute the startTime over a 60 minute window. Return the next future utc datetime where hour == utc_hour minute = A value between 0-59 (depending on table name) second = A value between 0-59 (depending on table name) �< )�year�month�day�hour�minute�secondrN )�days) �datetime�date�todayr� r� r� r~ �utcnow� timedelta)r> r/ r� Zstart_date_timeZone_dayr r r r{ ( s � r{ c C s� | ddi d�}t d | ||||�}|s d| � d�|d< |S td r/d| � d �|d< d |d < |S t d | ||||�}|r\d | � �|d< |d �dd| � d�� |d �dd| � d�� |S d| � �|d< d|d < |S )ag Ensure the DynamoDB table does not exist. name Name of the DynamoDB table. region Region to connect to. key Secret key to be used. keyid Access key to be used. profile A dict with region, key and keyid, or a pillar key (string) that contains a dict with region, key and keyid. Tr r r r z does not existr r z is set to be deletedNr zboto_dynamodb.deletezDeleted DynamoDB table r rH zTable r# rI z deletedz Failed to delete DynamoDB table F)r r2 re )r r* r+ r, r- rA �existsZ is_deletedr r r �absentF s$ �r� )NNNNNNNNNr NNNNNNr )NNNN) r r; r� �loggingro �sysZsalt.utils.dictupdate�utilsr9 � basicConfig�INFO�stdout� getLogger�log� Exceptionr r rG r3 r6 rT rV rW rg r8 r= r~ r{ r� r r r r �<module> s\ � � }g (62F
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings