File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/__pycache__/boto_kinesis.cpython-310.pyc
Back
o ;jwN � @ s4 d Z ddlZddlZddlZddlZzddlZddlZe�d�� ej � dZ W n e y2 dZ Y nw e�e �ZdZdd� Zd d � Zd+d d �Z d+d d�Zd+dd�Z d+dd�Zd+dd�Z d+dd�Z d+dd�Z d+dd�Z d+dd �Zd d � Zd d � Z d,d!d"�Z d+d#d$�Z d%d&� Z d'd(� Z d)d*� Z!dS )-a� Connection module for Amazon Kinesis .. versionadded:: 2017.7.0 :configuration: This module accepts explicit Kinesis credentials but can also utilize IAM roles assigned to the instance trough Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More Information available at: .. code-block:: text http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html If IAM roles are not used you need to specify them either in a pillar or in the minion's config file: .. code-block:: yaml kinesis.keyid: GKTADJGHEIQSXMKKRBJ08H kinesis.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs A region may also be specified in the configuration: .. code-block:: yaml kinesis.region: us-east-1 If a region is not specified, the default is us-east-1. It's also possible to specify key, keyid and region via a profile, either as a passed in dict, or as a string to pull from pillars or minion config: .. code-block:: yaml myprofile: keyid: GKTADJGHEIQSXMKKRBJ08H key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs region: us-east-1 :depends: boto3 � N�boto3TFZ boto_kinesisc C s* t jj�� } | du rtd td� tS | S )z- Only load if boto3 libraries exist. Tzboto3.assign_funcsZkinesis)�salt�utilsZversionsZcheck_boto_reqsZ __utils__�__name__�__virtualname__)Z has_boto_reqs� r �M/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/boto_kinesis.py� __virtual__G s r c C s t |d| d�S )a Stream info from AWS, via describe_stream Only returns the first "page" of shards (up to 100); use _get_full_stream() for all shards. CLI Example: .. code-block:: bash salt myminion boto_kinesis._get_basic_stream my_stream existing_conn �describe_stream�� StreamName)�_execute_with_retries)� stream_name�connr r r �_get_basic_streamR s r c C s� t ||||d�}i }t| |�d }|}|d d rAt|d| |d d d d d �}|d }|d d |d d 7 < |d d s||d< |S ) z� Get complete stream info from AWS, via describe_stream, including all shards. CLI Example: .. code-block:: bash salt myminion boto_kinesis._get_full_stream my_stream region=us-east-1 ��region�key�keyid�profile�result�StreamDescription� HasMoreShardsr �Shards����ShardId)r ZExclusiveStartShardId)� _get_connr r ) r r r r r r �r�streamZ full_streamr r r �_get_full_stream` s$ �� � r c C s� t ||||d�}d}d}d}|dkr5t�t||�� |d7 }t| |�} d| v r)| S | d d d }|dks| d d d rEt| ||||�} | S ) aw Get complete stream info from AWS, returning only when the stream is in the ACTIVE state. Continues to retry when stream is updating or creating. If the stream is deleted during retries, the loop will catch the error and break. CLI Example: .. code-block:: bash salt myminion boto_kinesis.get_stream_when_active my_stream region=us-east-1 r Nr � ZACTIVE� �errorr r Z StreamStatusr )r �time�sleep�_jittered_backoffr r ) r r r r r r Z stream_status�attempt�max_retry_delay�stream_responser r r �get_stream_when_active� s � r) c C sJ t ||||d�}i }t| |�}d|v r d|d< |d |d< |S d|d< |S )z� Check if the stream exists. Returns False and the error if it does not. CLI Example: .. code-block:: bash salt myminion boto_kinesis.exists my_stream region=us-east-1 r r" Fr T)r r )r r r r r r r r r r r �exists� s �r* c C s4 t ||||d�}t|d|| d�}d|vrd|d< |S )z� Create a stream with name stream_name and initial number of shards num_shards. CLI Example: .. code-block:: bash salt myminion boto_kinesis.create_stream my_stream N region=us-east-1 r � create_stream)Z ShardCountr r" Tr �r r )r Z num_shardsr r r r r r r r r r+ � s �r+ c C s2 t ||||d�}t|d| d�}d|vrd|d< |S )z� Delete the stream with name stream_name. This cannot be undone! All data will be lost!! CLI Example: .. code-block:: bash salt myminion boto_kinesis.delete_stream my_stream region=us-east-1 r � delete_streamr r" Tr r, )r r r r r r r r r r r- � s r- c C �4 t ||||d�}t|d| |d�}d|vrd|d< |S )z� Increase stream retention period to retention_hours CLI Example: .. code-block:: bash salt myminion boto_kinesis.increase_stream_retention_period my_stream N region=us-east-1 r � increase_stream_retention_period�r ZRetentionPeriodHoursr" Tr r, �r Zretention_hoursr r r r r r r r r r/ � � �r/ c C r. )z� Decrease stream retention period to retention_hours CLI Example: .. code-block:: bash salt myminion boto_kinesis.decrease_stream_retention_period my_stream N region=us-east-1 r � decrease_stream_retention_periodr0 r" Tr r, r1 r r r r3 � r2 r3 c C r. )a Enable enhanced monitoring for the specified shard-level metrics on stream stream_name CLI Example: .. code-block:: bash salt myminion boto_kinesis.enable_enhanced_monitoring my_stream ["metrics", "to", "enable"] region=us-east-1 r �enable_enhanced_monitoring�r ZShardLevelMetricsr" Tr r, �r Zmetricsr r r r r r r r r r4 � �r4 c C r. )a Disable enhanced monitoring for the specified shard-level metrics on stream stream_name CLI Example: .. code-block:: bash salt myminion boto_kinesis.disable_enhanced_monitoring my_stream ["metrics", "to", "disable"] region=us-east-1 r �disable_enhanced_monitoringr5 r" Tr r, r6 r r r r8 * r7 r8 c C s� d}d}g | d< | d D ]N}|d }d|d v r t �d|� q | d �|� t|d d �|d d < t|d d �|d d < |d d |k rL|d d }|d d |krZ|d d }q | d jd d � d � ||| fS )a9 Collect some data: number of open shards, key range, etc. Modifies stream_details to add a sorted list of OpenShards. Returns (min_hash_key, max_hash_key, stream_details) CLI Example: .. code-block:: bash salt myminion boto_kinesis.get_info_for_reshard existing_stream_details r � OpenShardsr r �EndingSequenceNumber�SequenceNumberRangezskipping closed shard %s� HashKeyRange�StartingHashKey� EndingHashKeyc S s t | d d �S )Nr< r= )�long_int)�shardr r r �<lambda>d s z&get_info_for_reshard.<locals>.<lambda>)r )�log�debug�appendr? �sort)�stream_details� min_hash_key� max_hash_keyr@ �shard_idr r r �get_info_for_reshardC s0 � � �� rJ c C s t | �S )al The hash key is a 128-bit int, sent as a string. It's necessary to convert to int/long for comparison operations. This helper method handles python 2/3 incompatibility CLI Example: .. code-block:: bash salt myminion boto_kinesis.long_int some_MD5_hash_as_string :return: long object if python 2.X, int object if python 3.X )�int)Zhash_keyr r r r? i s r? c C s& t ||||d�}i }t| ||||�} d| v r| S | d d } t| �\} } } t�dt| d �| | � t| d �D ]�\} }|d }d|d v rLt�d |� q7|d d }|d d }| | | | | }| | | | d | }|| krt| }t�d| ||||k� t�d| ||||k� ||kr�d|d< | S ||kr�q7||d kr�|r�t�d|||� t|d| |t|d �d�}nt�d|||� d|vr�d|d< | S t | |�}|s�d|� �|d< | S |r�t�d|||� t|d| ||d�}nt�d|||� d|v�rd|d< | S t�d � d |d< |S ) a8 Reshard a kinesis stream. Each call to this function will wait until the stream is ACTIVE, then make a single split or merge operation. This function decides where to split or merge with the assumption that the ultimate goal is a balanced partition space. For safety, user must past in force=True; otherwise, the function will dry run. CLI Example: .. code-block:: bash salt myminion boto_kinesis.reshard my_stream N True region=us-east-1 :return: True if a split or merge was found/performed, False if nothing is needed r r" r r z5found %s open shards, min_hash_key %s max_hash_key %sr9 r r: r; z)this should never happen! closed shard %sr< r= r> r! z$Shard %s (%s) should start at %s: %sz"Shard %s (%s) should end at %s: %sz3starting hash keys mismatch, don't know what to do!z)%s should end at %s, actual %s, splittingZ split_shard)r Z ShardToSplitZNewStartingHashKeyz*%s should end at %s, actual %s would splitTz failed to find next shard after z+%s should continue past %s, merging with %sZ merge_shards)r Z ShardToMergeZAdjacentShardToMergez/%s should continue past %s, would merge with %sz"No split or merge action necessaryF) r r) rJ rB rC �len� enumerater �str�_get_next_open_shard)r Z desired_size�forcer r r r r r r( rF rG rH Z shard_numr@ rI Zstarting_hash_keyZending_hash_keyZexpected_starting_hash_keyZexpected_ending_hash_keyZ next_shard_idr r r �reshardz s� � ��� ���� � �� ��� rQ c C s� t | |||d�}g }d}|durN|rd|ini }t|dfi |��}d|v r'|S |r2|�d�r2|d ni }||�dg �7 }|�d d �d v rH|d nd}|dusd|iS ) z� Return a list of all streams visible to the current account CLI Example: .. code-block:: bash salt myminion boto_kinesis.list_streams r � NZExclusiveStartStreamName� list_streamsr" r Z StreamNamesZHasMoreStreamsF)T�truer )r r �get) r r r r r ZstreamsZexclusive_start_stream_name�args�retr r r rS s" ����rS c C s8 d}| d D ]}|d }||krd}q|r| S qdS )z� Return the next open shard after shard_id CLI Example: .. code-block:: bash salt myminion boto_kinesis._get_next_open_shard existing_stream_details shard_id Fr9 r TNr )rF rI �foundr@ Zcurrent_shard_idr r r rO . s ��rO c K s i }d}d}t |�D ]k}t�d||� zt| |�}|di |��|d< |W S tjjyu } zA|jd d } d| v s?d| v rOtjd d d � t � t ||�� n |jd |d < t� |d � d |d< |W Y d }~ S W Y d }~q d }~ww d� ||�|d < t� |d � |S )a� Retry if we're rate limited by AWS or blocked by another call. Give up and return error message if resource not found or argument is invalid. conn The connection established by the calling method via _get_conn() function The function to call on conn. i.e. create_stream **kwargs Any kwargs required by the above function, with their keywords i.e. StreamName=stream_name Returns: The result dict with the HTTP response and JSON data if applicable as 'result', or an error as 'error' CLI Example: .. code-block:: bash salt myminion boto_kinesis._execute_with_retries existing_conn function_name function_kwargs � r zattempt: %s function: %sr �ErrorZCodeZLimitExceededExceptionZResourceInUseExceptionz Retrying due to AWS exceptionT)�exc_infor" Nz5Tried to execute function {} {} times, but was unabler )�rangerB �info�getattr�botocore� exceptionsZ ClientError�responserC r# r$ r% r" �format) r �function�kwargsr Z max_attemptsr' r& �fn�eZ error_coder r r r B s6 ����r c C s t t�� d| |�S )z� Basic exponential backoff CLI Example: .. code-block:: bash salt myminion boto_kinesis._jittered_backoff current_attempt_number max_delay_in_seconds � )�min�random)r& r' r r r r% } s r% )NNNN)FNNNN)"�__doc__�loggingri r# Zsalt.utils.versionsr r r_ � getLogger�setLevel�CRITICALZHAS_BOTO� ImportErrorr rB r r r r r) r* r+ r- r/ r3 r4 r8 rJ r? rQ rS rO r r% r r r r �<module> sZ 0 � ! � $ � � � � �& � ;
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings