File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/__pycache__/redis_cache.cpython-310.pyc
Back
o ;j(U � @ s\ d Z ddlZddlZddlZddlZddlmZ zddlZddl m Z ddl m Z dZW n ey7 dZY nw z ddlmZ dZW n eyM dZY nw d Zd d iZe�e�Zd Zd ZdZdZdZ da dd� Z dd� Z dd� Z d3dd�Z!dd� Z"dd � Z#d d � Z$d d � Z%d!d"� Z&d#d$� Z'd4d&d'�Z(d(d)� Z)d*d+� Z*d3d,d-�Z+d.d � Z,d/d0� Z-d1d2� Z.dS )5a Redis ===== Redis plugin for the Salt caching subsystem. .. versionadded:: 2017.7.0 .. versionchanged:: 3005 To enable this cache plugin, the master will need the python client for redis installed. This can be easily installed with pip: .. code-block:: bash salt \* pip.install redis As Redis provides a simple mechanism for very fast key-value store, in order to provide the necessary features for the Salt caching subsystem, the following conventions are used: - A Redis key consists of the bank name and the cache key separated by ``/``, e.g.: ``$KEY_minions/alpha/stuff`` where ``minions/alpha`` is the bank name and ``stuff`` is the key name. - As the caching subsystem is organised as a tree, we need to store the caching path and identify the bank and its offspring. At the same time, Redis is linear and we need to avoid doing ``keys <pattern>`` which is very inefficient as it goes through all the keys on the remote Redis server. Instead, each bank hierarchy has a Redis SET associated which stores the list of sub-banks. By default, these keys begin with ``$BANK_``. - In addition, each key name is stored in a separate SET of all the keys within a bank. By default, these SETs begin with ``$BANKEYS_``. For example, to store the key ``my-key`` under the bank ``root-bank/sub-bank/leaf-bank``, the following hierarchy will be built: .. code-block:: text 127.0.0.1:6379> SMEMBERS $BANK_root-bank 1) "sub-bank" 127.0.0.1:6379> SMEMBERS $BANK_root-bank/sub-bank 1) "leaf-bank" 127.0.0.1:6379> SMEMBERS $BANKEYS_root-bank/sub-bank/leaf-bank 1) "my-key" 127.0.0.1:6379> GET $KEY_root-bank/sub-bank/leaf-bank/my-key "my-value" There are four types of keys stored: - ``$BANK_*`` is a Redis SET containing the list of banks under the current bank. - ``$BANKEYS_*`` is a Redis SET containing the list of keys under the current bank. - ``$KEY_*`` keeps the value of the key. - ``$TSTAMP_*`` stores the last updated timestamp of the key. These prefixes and the separator can be adjusted using the configuration options: bank_prefix: ``$BANK`` The prefix used for the name of the Redis key storing the list of sub-banks. bank_keys_prefix: ``$BANKEYS`` The prefix used for the name of the Redis key storing the list of keys under a certain bank. key_prefix: ``$KEY`` The prefix of the Redis keys having the value of the keys to be cached under a certain bank. timestamp_prefix: ``$TSTAMP`` The prefix for the last modified timestamp for keys. .. versionadded:: 3005 separator: ``_`` The separator between the prefix and the key body. The connection details can be specified using: host: ``localhost`` The hostname of the Redis server. port: ``6379`` The Redis server port. cluster_mode: ``False`` Whether cluster_mode is enabled or not cluster.startup_nodes: A list of host, port dictionaries pointing to cluster members. At least one is required but multiple nodes are better .. code-block:: yaml cache.redis.cluster.startup_nodes - host: redis-member-1 port: 6379 - host: redis-member-2 port: 6379 cluster.skip_full_coverage_check: ``False`` Some cluster providers restrict certain redis commands such as CONFIG for enhanced security. Set this option to true to skip checks that required advanced privileges. .. note:: Most cloud hosted redis clusters will require this to be set to ``True`` db: ``'0'`` The database index. .. note:: The database index must be specified as string not as integer value! password: Redis connection password. unix_socket_path: .. versionadded:: 2018.3.1 Path to a UNIX socket for access. Overrides `host` / `port`. Configuration Example: .. code-block:: yaml cache.redis.host: localhost cache.redis.port: 6379 cache.redis.db: '0' cache.redis.password: my pass cache.redis.bank_prefix: #BANK cache.redis.bank_keys_prefix: #BANKEYS cache.redis.key_prefix: #KEY cache.redis.timestamp_prefix: #TICKS cache.redis.separator: '@' Cluster Configuration Example: .. code-block:: yaml cache.redis.cluster_mode: true cache.redis.cluster.skip_full_coverage_check: true cache.redis.cluster.startup_nodes: - host: redis-member-1 port: 6379 - host: redis-member-2 port: 6379 cache.redis.db: '0' cache.redis.password: my pass cache.redis.bank_prefix: #BANK cache.redis.bank_keys_prefix: #BANKEYS cache.redis.key_prefix: #KEY cache.redis.separator: '@' � N)�SaltCacheError)�ConnectionError)� ResponseErrorTF)� RedisCluster�redis�list_�listz$BANKz$KEYz$TSTAMPz$BANKEYS�_c C s t sdS ts t� d r dS tS )z� The redis library must be installed for this module to work. The redis redis cluster library must be installed if cluster_mode is True )Fz(Please install the python-redis package.� cluster_mode)Fz,Please install the redis-py-cluster package.)� HAS_REDIS�HAS_REDIS_CLUSTER�_get_redis_cache_opts�__virtualname__� r r �J/opt/saltstack/salt/lib/python3.10/site-packages/salt/cache/redis_cache.py� __virtual__� s r c C s i S )z9 Effectively a noop. Return an empty dictionary. r )�kwargsr r r � init_kwargs� s r c C sV t �dd�t �dd�t �dd�t �dd�t �d d �t �d d �t �d i �t �dd �d�S )zG Return the Redis server connection details from the __opts__. zcache.redis.hostZ localhostzcache.redis.porti� z cache.redis.unix_socket_pathNzcache.redis.db�0zcache.redis.password� zcache.redis.cluster_modeFz!cache.redis.cluster.startup_nodesz,cache.redis.cluster.skip_full_coverage_check)�host�port�unix_socket_path�db�passwordr � startup_nodes�skip_full_coverage_check)�__opts__�getr r r r r � s ��r c C s^ t rt S | s t� } | d rt| d | d d�a t S tj| d | d | d | d | d d �a t S ) zL Return the Redis server instance. Caching the object instance. r r r )r r r r r r r )r r r )� REDIS_SERVERr r r Z StrictRedis)�optsr r r �_get_redis_server� s$ � ��r! c C s8 t �dt�t �dt�t �dt�t �dt�t �dt�d�S )z7 Build the key opts based on the user options. zcache.redis.bank_prefixz cache.redis.bank_keys_prefixzcache.redis.key_prefixzcache.redis.separatorz cache.redis.timestamp_prefix)� bank_prefix�bank_keys_prefix� key_prefix� separator�timestamp_prefix)r r � _BANK_PREFIX�_BANK_KEYS_PREFIX� _KEY_PREFIX� _SEPARATOR�_TIMESTAMP_PREFIXr r r r �_get_redis_keys_opts s � ��r, c C � t � }dj|d |d | d�S )z; Return the Redis key for the bank given the name. �{prefix}{separator}{bank}r" r% ��prefixr% �bank�r, �format�r1 r r r r �_get_bank_redis_key! � �r5 c C s" t � }d�|d |d | h|h�S )Nz {}{}{}/{}r& r% r2 �r1 �keyr r r r �_get_timestamp_key+ s �r9 c C s* t � }dj|d |d | tjj�|�d�S )zD Return the Redis key given the bank name and the key name. z {prefix}{separator}{bank}/{key}r$ r% )r0 r% r1 r8 )r, r3 �salt�utils� stringutils�to_strr7 r r r �_get_key_redis_key4 s �r> c C r- )z] Return the Redis key for the SET of keys under a certain bank, given the bank name. r. r# r% r/ r2 r4 r r r �_get_bank_keys_redis_keyA r6 r? c C s� | � d�}t|�D ]=\}}d�|d|d � �}t|�}t�d| |� |�|d� |dkrFd�|d|� �}|�t|�|� |�t|�|� q dS )aP Build the bank hierarchy from the root of the tree. For each level in the bank path: - ensure a ``.`` placeholder exists in ``$BANK_<level>`` so that an empty bank still has a recognisable record; - for every non-root level, register this segment as a child of its parent in both ``$BANK_<parent>`` (consumed by the flush tree-traversal in ``_get_banks_to_remove``) and ``$BANKEYS_<parent>`` (consumed by ``list_()``). Without this, ``list_("minions")`` reads an empty set even though the data is stored correctly under ``minions/<id>``, breaking ``CkMinions.connected_ids()`` and therefore ``salt-run manage.present`` / ``manage.up`` for any deployment that uses the redis cache backend. Uses the Redis pipeline so there is only one round-trip with the server. �/N� �Adding %s to %s�.r )�split� enumerate�joinr5 �log�debug�saddr? )r1 � redis_pipe�parts�index�segment� bank_pathZbank_set� parent_pathr r r �_build_bank_hierK s ��rP r c C st |s|n|� d|� �}|g}t |�}| �|�}|s|S |D ]}t|t�r(|�� }|dkr-q |�t| ||d�� q |S )z� A simple tree traversal algorithm that builds the list of banks to remove, starting from an arbitrary node in the tree. r@ rC )�path)r5 �smembers� isinstance�bytes�decode�extend�_get_banks_to_remove)� redis_serverr1 rQ Z current_path�bank_paths_to_remove�bank_keyZ child_banksZ child_bankr r r rW r s �rW c C s� t � }|�� }t| |�}t| �}z?t| |� tj�|�}|�||� t � d|| |� |� ||� |�t | |d�tj�t t�� ��� t � d||� |�� W dS ttfym } zdj||d�} t �| � t| ��d}~ww )z( Store the data in a Redis key. z&Setting the value for %s under %s (%s)�r1 r8 rB z-Cannot set the Redis cache key {rkey}: {rerr}�Zrkey�rerrN)r! �pipeliner> r? rP r: �payload�dumps�setrG rH rI r9 �int�time�execute�RedisConnectionError�RedisResponseErrorr3 �errorr ) r1 r8 �datarX rJ � redis_keyZredis_bank_keys�valuer] �mesgr r r �store� s0 �� ��rl c C sx t � }t| |�}d}z|�|�}W n ttfy/ } zdj||d�}t�|� t|��d}~ww |du r6i S t j � |�S )z* Fetch data from the Redis cache. Nz/Cannot fetch the Redis cache key {rkey}: {rerr}r\ ) r! r> r re rf r3 rG rg r r: r_ �loads)r1 r8 rX ri Z redis_valuer] rk r r r �fetch� s � �� rn c C s8 t � }|�� }|du r�t|| �}|D ]}t|�}|�|� t�d||� qz t�d� |�� }W n! tt fyQ } zdj d� |�|d�} t� | � t | ��d}~ww t|�} t| �D ]N} || } || } | D ]!}t| |�}|�|� t| |�� d�}|�|� t�d|| |� qft| �}|�|� t�d | |� t| �}|�|� t�d | |� qZd | v r�| �d d �\}}|�t|�|� |�t|�|� n/t| |�}|�|� t| |d�}|�|� t�d|| |� t| �}|�||� t�d || |� z|�� W dS tt f�y } zdj | |d�} t� | � t | ��d}~ww )ad Remove the key from the cache bank with all the key content. If no key is specified, remove the entire bank with all keys and sub-banks inside. This function is using the Redis pipelining for best performance. However, when removing a whole bank, in order to re-create the tree, there are a couple of requests made. In total: - one for node in the hierarchy sub-tree, starting from the bank node - one pipelined request to get the keys under all banks in the sub-tree - one pipeline request to remove the corresponding keys This is not quite optimal, as if we need to flush a bank having a very long list of sub-banks, the number of requests to build the sub-tree may grow quite big. An improvement for this would be loading a custom Lua script in the Redis instance of the user (using the ``register_script`` feature) and call it whenever we flush. This script would only need to build this sub-tree causing problems. It can be added later and the behaviour should not change as the user needs to explicitly allow Salt inject scripts in their Redis instance. Nz%Fetching the keys of the %s bank (%s)zExecuting the pipe...zBCannot retrieve the keys under these cache banks: {rbanks}: {rerr}z, )Zrbanksr] r[ z*Removing the key %s under the %s bank (%s)z/Removing the bank-keys key for the %s bank (%s)zRemoving the %s bank (%s)r@ rA z@De-referencing the key %s from the bank-keys of the %s bank (%s)z1Cannot flush the Redis cache bank {rbank}: {rerr})Zrbankr] T)r! r^ rW r? rR rG rH rd re rf r3 rF rg r �len�ranger> �deleter9 rU r5 �rsplitZsrem)r1 r8 rX rJ rY Zbank_to_removeZbank_keys_redis_keyZ subtree_keysr] rk Z total_banksrL Z bank_keysrN ri � timestamp_keyrZ rO rM r r r �flush� s� � � � �� � � � � �� ��rt c C sp t � }t| �}z|�|�}W n ttfy, } zdj||d�}t�|� t|��d}~ww |s1g S dd� |D �S )z5 Lists entries stored in the specified bank. z.Cannot list the Redis cache key {rkey}: {rerr}r\ Nc S s g | ] }|d kr|� � �qS )� .)rU )�.0r1 r r r � <listcomp>I s zlist_.<locals>.<listcomp>) r! r? rR re rf r3 rG rg r )r1 rX �bank_redis_keyZbanksr] rk r r r r 9 s � ��c C s| t � }t| �}z|du rtjj�|�|��dkW S |�||�W S tt fy= } zdj ||d�}t � |� t |��d}~ww )zB Checks if the specified bank contains the specified key. N�nonez2Cannot retrieve the Redis cache key {rkey}: {rerr}r\ )r! r? r: r; r< r= �typeZ sismemberre rf r3 rG rg r )r1 r8 rX rx r] rk r r r �containsL s" ��� ��r{ c C s4 t � }t| |d�}|�|�}|durtj�|�}|S )zp Return the Unix Epoch timestamp of when the key was last updated. Return None if key is not found. r[ N)r! r9 r r: r_ rm )r1 r8 rX rs rj r r r �updatedb s r| )N)r )/�__doc__�loggingrc Z salt.payloadr: Zsalt.utils.stringutilsZsalt.exceptionsr r Zredis.exceptionsr re r rf r � ImportErrorZ redisclusterr r r Z__func_alias__� getLogger�__file__rG r' r) r+ r( r* r r r r r! r, r5 r9 r> r? rP rW rl rn rt r r{ r| r r r r �<module> s^ � � '% q
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings