File manager - Edit - /opt/saltstack/salt/lib/python3.10/site-packages/salt/__pycache__/state.cpython-310.pyc
Back
o ;j\J � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlmZ ddlZddlZddlZddlZddlZddlZddl Zddl Zddl m m Z ddl!Zddl"Zddl#Zddl$Zddl%Zddl&m m'Z' ddl(m)Z)m*Z*m+Z+ ddl,m-Z. ddl,m/Z0 ddl1m2Z2m3Z3 ddl4m5Z5m6Z6 e�7e8�Z9e:g d��Z;e:g d ��Z<e:g d ��Z=e;�>e<��>e=�Z?d d � Z@d d� ZAdd� ZBdd� ZCdd� ZDdd� ZEdd� ZFdd� ZGd:d d �ZHd d � ZId d!� ZJd"d#� ZKd$d%� ZLd&d'� ZMG d(d)� d)eN�ZOG d*d+� d+�ZPG d,d-� d-�ZQG d.d/� d/�ZRG d0d1� d1�ZSG d2d3� d3eS�ZTG d4d5� d5eQ�ZUG d6d7� d7eT�ZVG d8d9� d9�ZWdS );a� The State Compiler is used to execute states in Salt. A state is unlike an execution module in that instead of just executing a command, it ensures that a certain state is present on the system. The data sent to the state calls is as follows: { 'state': '<state module name>', 'fun': '<state function name>', 'name': '<the name argument passed to all states>' 'argn': '<arbitrary argument, can have many of these>' } � N)�CommandExecutionError�SaltRenderError�SaltReqTimeoutError)� deserialize)� serialize)�compile_template�compile_template_str)�DefaultOrderedDict�HashableOrderedDict) � onchanges� onchanges_any�onfail� onfail_any� onfail_all� onfail_stop�prereq� prerequired�watch� watch_any�require� require_any�listen)� onchanges_in� onfail_in� prereq_in�watch_in� require_in� listen_in)'�fun�state� check_cmd�cmd_opts_exclude�failhard�onlyif�unless�creates�retry�order�parallelr r r �reload_modules� reload_grains� reload_pillar�runas�runas_password� fire_event�saltenv�umask�use�use_in�__env__�__sls__�__id__�__orchestration_jid__Z __pub_userZ __pub_argZ __pub_jidZ __pub_funZ __pub_tgtZ __pub_retZ __pub_pidZ__pub_tgt_type� __prereq__�__prerequired__� __umask__c C s | � d�\}}}}||||d�S )zN Take a low tag and split it back into the low dict that it came from z_|-)r r5 �namer )�split)�tagr �id_r: r � r> �>/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py� split_low_tag� s r@ c C s d� | �S )zJ Generate the running dict tag string from the low data structure z/{0[state]}_|-{0[__id__]}_|-{0[name]}_|-{0[fun]}��format��lowr> r> r? �_gen_tag� � rE c C s t jj�| �S )z* Make tag name safe for filenames )�salt�utils�filesZsafe_filename_leaf)r< r> r> r? � _clean_tag� s rJ c C s" d| � �d|� �ddd�}t |�S )N�listen_Z Listen_Error)r: r5 r r �rE )r: r= rD r> r> r? �_l_tag� s �rM c C s\ t j �� } | t j �� t j �� }t j �� }|�� �� }|| }|jd |j d }||fS )zk Generate a NULL duration for when states do not run but we want the results to be consistent. �@B � @�@)�datetime�utcnow�now�time� isoformat�seconds� microseconds)�utc_start_time�local_start_time�utc_finish_time� start_time�delta�durationr> r> r? �_calculate_fake_duration� s � r] c C s( t j�| d�}t j�|�st �|� |S )zg Return the directory that accumulator data is stored in, creating it if it doesn't exist. Z accumulator)�os�path�join�isdir�makedirs)�cachedir�fn_r> r> r? �get_accumulator_dir� s re c C s. t t| ��}d|v r|�d�d | | iS | S )z. Trim any function off of a requisite �.r )�next�iterr; )�req�reqfirstr> r> r? �trim_req� s rk c C sd t � }| |vr |S ||| vr|S || | D ]}t|t�s qt|�dkr&q|�tt|��� q|S )zA Return a set of the arguments passed to the named state � )�set� isinstance�dict�len�addrg rh )r= r �high�args�itemr> r> r? � state_args� s ru Fc C s� g }|du r d}n||� | i �v }| |v r |r |�| |f� |S |dkr@|�� D ]\}}|d | kr=|�|tt|��f� q(|S |D ]8}||| v rzt|| | t�rz|| | D ] }t|t�saqYt|�dkrhqY|tt|�� | kry|�||f� qYqB|S )z� Scan high data for the id referencing the given name and return a list of (IDs, state) tuples that match Note: if `state` is sls, then we are looking for all IDs that match the given SLS FT�slsr4 rl ) �get�append�itemsrg rh rn �listro rp ) r: r rr �strict�ext_idZcheck2�nidrt �argr> r> r? � find_name� s2 � �� �r c C s| g }|� � D ]5\}}z|d }W n ty$ |dkr"t�d||� Y qw || kr;|D ]}|�d�r3q+|�||f� q+q|S )zT Scan for all ids in the given sls and return them in a dict; {name: state} r4 � __exclude__z2Invalid non-dict item '%s' in high data. Value: %r�__)ry � TypeError�log�error� startswithrx )rv rr �retr} rt Zsls_tgt�st_r> r> r? � find_sls_ids s$ �� �r� c C sL d}t | t�r�d| v r�| d }|s | d r| d }n_d�| �}nYt |t�ryd|v r7t |d t�r7d�|d �}tdd� |�� D ��rytd d� |�� D ��ryd }|D ]'}|| d }|sa|d vrad }|| d }|so|d vrod }|d�|||�7 }qQ|s�t| d �}| d du s�| d du r�t�|� dS t�|� dS dS t�t| �� dS )z- Format the state into a log message � �changes�commentz No changes made for {0[name]}�diffzFile changed: {}c S s g | ]}t |t��qS r> )rn ro ��.0�xr> r> r? � <listcomp>* s z format_log.<locals>.<listcomp>c S s g | ] }d |v o d|v �qS )�old�newr> r� r> r> r? r� + � z Made the following changes: r� �FNZabsentr� z '{}' changed from '{}' to '{}' �resultTN) rn ro rB �str�all�valuesr� �infor� )r� �msgZchg�pkgr� r� r> r> r? � format_log s@ � �#r� c C s t | ||||�}|�� S )zQ Compile the master side low state data, and build the hidden state file )�MasterHighState�compile_highstate)� master_opts� minion_opts�grainsr= r/ r� r> r> r? �master_compileD s r� c C s$ zt | � W dS ty Y dS w )NFT)�hashr� )�objr> r> r? � ishashableL s ��r� c C s0 | d r | d d }n| d d }|di dd�S )zo Returns a mocked return dict with information about the run, without executing the state function rs r �kwargsr: zNot called, mockedT�r: r� r� r� r> )�cdatar: r> r> r? �mock_retT s �r� c @ s e Zd ZdZdS )� StateErrorz! Custom exception class. N)�__name__� __module__� __qualname__�__doc__r> r> r> r? r� g s r� c @ sH e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dS )�CompilerzB Class used to compile and manage the High Data structure c C s || _ || _d S �N)�opts�rend)�selfr� � renderersr> r> r? �__init__r s zCompiler.__init__c K s> t || j| jd | jd | jd fi |��}|s|S | �|�S �z2 Enforce the states in a template �renderer�renderer_blacklist�renderer_whitelist)r r� r� � pad_funcs)r� �templater� rr r> r> r? �render_templatev s �� zCompiler.render_templatec C sP |D ]�}t || t�s?t || t�r?d|| v r>|| �d�}t|�dkr2d�|dt|�� �|d< |d |d gi||< qqt� }t|| �D ]\}|�d�rPqHt || | t �sZqHd|v r�|�d�}t|�dkrvd�|dt|�� �|d< |d |v r}qH|| � |�|| |d < || |d � |d � |� |d � qH|� |� qHq|S )zI Turns dot delimited function refs into function strings rf � rl r �_) rn ro r� r; rp r` rm �sortedr� rz �poprx rq )r� rr r: �comps�skeys�keyr> r> r? r� � s< � �zCompiler.pad_funcsc C s� g }t |t�s |�d� t� }|�s�|�� D �]�\}}z |�d�r#W qW n ttfy/ Y nw t |t�sD|�d� ||d t |�j �� t |t�sUd� ||�}|�|� q|D �]i}|�d�r`qWt || t �ss|�d� ||d �� qWd}d|v r}|d 7 }|| D ]�} t | t�r�|d 7 }d | � � v r�|�d | � d |� d |d � d�� q�t | t��rntt| ��} | dv �rnt | | t �s�|�d� | ||d �� n�d|i||< | | D ]�} t | t�r�d| i} t | t�s�|�d� | |d �� q�tt| ��} | | } d| v �r|�d� | ||d | d| �d�� �� t| ��s%|�d� t| �|d �� q�| || | < | |v �rY||| v �rY|| | |k�rY|| d || | k�rY|�d� |d || �� q�tt | ��d k�rn|�d� ||d �� q�|�s�|dk�s||dk�r}qW|�d|� d |d � d �� qW|d k�r�d|v �r�|jdd d �d gng }|�d d � || D �� |�d!|� d |d � d"�d#�|� � qWq|S )$�T Verify that the high data is viable and follows the data structure �,High data is not a dictionary and is invalidr� zTID '{}' in SLS '{}' is not formed as a string, but is a {}. It may need to be quotedr4 �2The type {} in {} is not formatted as a dictionary�.State '{}' in SLS '{}' is not formed as a listr rf rl � �The function "� " in state "� " in SLS "�p" has whitespace, a function with whitespace is not supported, perhaps this is an argument that is missing a ":"�r r r r �GThe {} statement in state '{}' in SLS '{}' needs to be formed as a listr �id�KRequisite declaration {} in SLS {} is not formed as a single key dictionary�qInvalid requisite type '{}' in state '{}', in SLS '{}'. Requisite types must not contain dots, did you mean '{}'?Nz"Illegal requisite "{}", is SLS {} �9A recursive requisite was found, SLS "{}" ID "{}" ID "{}"�CMultiple dictionaries defined in argument of state '{}' in SLS '{}'r r � No function declared in state '� ' in SLS '�'��maxsplitc s � � | ] }t |t�r|V qd S r� �rn r� �r� r~ r> r> r? � <genexpr>V s � � �z'Compiler.verify_high.<locals>.<genexpr>�&Too many functions declared in state '�''. Please choose one of the following: �, )rn ro rx r ry r� �AttributeErrorr� r� rB �typer� rz �striprg rh �findr� rp r; �extendr` �r� rr �errors�reqsr: �body�errr r r~ �argfirstri �req_key�req_valZfunsr> r> r? � verify_high� s �� �� � �� ���� �� �� �� �� ���� ����� �� ����� �� zCompiler.verify_highc C s d}|D ] }d|v r$t |d t�sq|d }||d kr$|dkr$|d }q|D ]\}d|vr2||d< q't |d ttf�sW|d dkrH|d |d< n|d dkrSd|d< n||d< d|v rh|d |�d�d |d< |d dk rx|d |d |d< tjj�|d �|d < q'|jd d � d � |S )�x Sort the chunk list verifying that the chunks follow the order specified in the order options. rl r' r �d �lastrN �first� name_order� ��@r: c S � | d d� | �fS �Nr' z{0[state]}{0[name]}{0[fun]}rA ��chunkr> r> r? �<lambda>� � �z'Compiler.order_chunks.<locals>.<lambda>�r� ) rn �int�floatr� rG rH �data�decode�sort�r� �chunksZcapr� Z chunk_orderr> r> r? � order_chunks` s8 � �zCompiler.order_chunksc C s� g }|� � D ]�\}}|�d�rq|� � D ]�\}}t� }g }|�d�r#q||d�} d|v r2|d | d< d|v r<|d | d< || d< |D ]3} t| t�rO|�| � qBt| t�ru| � � D ] \} } | dkro| D ] } | |vrm|�| � qbqX| �| � qXqB|r�d}|D ]=}t � | �}t|t�r�t t |� � ��}||d< tt|j|| �� n||d< ||d < |d }|D ] }||d < |�|� q�q|qt � | �}|D ] }||d < |�|� q�qq| �|�}|S ) �� "Compile" the high data as it is retrieved from the CLI or YAML into the individual state executor structures r� �r r: r4 r3 r5 �namesrl r: r� r )ry r� rm rn r� rq ro rx �update�copy�deepcopyrg rh �keysrz �mapr )r� rr r r: r� r �run�funcsr r� r~ r� �val�_namer� �entry�live�low_namer r> r> r? �compile_high_data� sj � � �� �� ,zCompiler.compile_high_datac C s� d|vr|S t � }t � }|�d�}|D ]7}t|t�r |�|� t|t�rJt|�dkr+qtt|� � ��}|dkr?|�|d � q|dkrJ|�|d � q|ri|� � D ]\}}|� d�r[qQ|� dd�|v rh|�|� qQ|D ] } | |v rv|�| � qk|S �zi Read in the __exclude__ list and remove all excluded objects from the high data r� rl rv r� r� r4 r� ) rm r� rn r� rq ro rp rg rh r ry r� rw ) r� rr �ex_sls�ex_id�exclude�excr� r: r� r= r> r> r? � apply_exclude� s: � � �zCompiler.apply_excludeN) r� r� r� r� r� r� r� r� r r r r> r> r> r? r� m s 1 *' 8r� c @ s� e Zd ZdZ dgdd�Zdd� Zd d � Zd d � Zd d� Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd � Zd d � Zd d � Zdhd!d"�Zd#d$� Zd%d&� Zd'd(� Zd)d*� Zd+d,� Zd-d.� Zdid/d0�Zdjd1d2�Zd3d4� Zd5d6� Z e d7d8� �Z d9d:� Z e j!j"j#�$d;d<�dkd>d?��Z%d@dA� Z&dBdC� Z'dDdE� Z(dFdG� Z)dHdI� Z*dJdK� Z+dLdM� Z,djdNdO�Z-djdPdQ�Z.dldSdT�Z/dUdV� Z0dWdX� Z1didYdZ�Z2d[d\� Z3d]d^� Z4d_d`� Z5dadb� Z6dcdd� Z7dedf� Z8dS )m�Statez+ Class used to execute salt states NF�statesc C s� |dur|} | du rt jj�|�} ||||||||| | d� | _|| _d|vr.t j�|�|d< || _| r:| | _ d| _ n t j � | j�| _ d| _ || _ || _|durez|�� }W n tyd t|��� }Y nw || _t�d� | rx| jsx| | jd< n+| �� | jd< | jr�t jj�| jd | j| j�dd �| j�d d �| j�d d��| jd< t�d � |du r�i | _n|| _| j | jd< | �� t� | _ t� | _ i | _ d| _ || _| | _ tt!| ��| _"i | _#|| _$d| _%t� | _&dS )a When instantiating an object of this class, do not pass ``_invocation_id``. It is an internal field for tracking parallel executions where no jid is available (Salt-SSH) and only exposed as an init argument to work on spawning platforms. N) r� �pillar_override�jid� pillar_enc�proxy�context�mocked�loader�initial_pillar�_invocation_idr� TFz#Gathering pillar data for state run�pillarZ pillar_source_merging_strategyZsmartr� ZyamlZpillar_merge_listsz,Finished gathering pillar data for state run� fileclientr )'rG rH r Zgen_jid� _init_kwargs� states_loaderr! r� r� � file_client�preserve_file_clientr% �get_file_clientr �_pillar_override�lowerr� r� � _pillar_encr� �debug�_gather_pillarZ dictupdate�mergerw � state_con� load_modulesrm �active�mod_init�pre�_State__run_num� invocation_idr� � instance_id�inject_globalsr �global_state_conditions�_processed_sls_files) r� r� r r r r r r r! r" r( r# r> r> r? r� � s| � � � zState.__init__c s� g }d}|di dd�}t � jt�s� jd d�pi � _� j�� D ]$\}}|d||fv rDt |t�r3|g}t |t�rD|�� fdd�|D �� q |rMt|�sM|}|S ) a+ Return ``None`` if global state conditions are met. Otherwise, pass a return dictionary which effectively creates a no-op outcome. This operation is "explicit allow", in that ANY state and condition combination which matches will allow the state to be run. Nz9Failed to meet global state conditions. State not called.r� � config.optionr: �*c 3 s � | ] }� j d |�V qdS )zmatch.compoundN)� functions)r� Z condition�r� r> r? r� ` s � � �z7State._match_global_state_conditions.<locals>.<genexpr>) rn r: ro r> ry r� rz r� �any) r� �fullr r: �matchesr� Zret_dictZ state_matchZ conditionsr> r? r? � _match_global_state_conditionsE s, � � �� z$State._match_global_state_conditionsc C s* | j rv| jr9ztjjj| j | jdt| dd�| j| jd d�| _ W n ty8 } z t � d|� W Y d}~nd}~ww t | j t �rhz t j| j t jd�| _ W n tyg } zt � d� t �|� W Y d}~nd}~ww t | j t�svt � d � d| _ tjj| j| jd | jd | jd | j | j�d �d�}|�� S )zK Whenever a state run starts, gather the pillar data fresh Tr� NZdecrypt_pillar_renderers)Ztranslate_newlinesr� r� Z valid_rendz%Failed to decrypt pillar override: %s)�Loaderz"Failed to load CLI pillar overridez.Pillar override was not passed as a dictionaryr� r� r/ � pillarenv)r rE )r+ r- rG rH ZcryptZdecrypt�getattrr� � Exceptionr� r� rn r� � yamlloader�loadZSaltYamlSafeLoader� exceptionro r$ Z get_pillarrw Zcompile_pillar)r� r r$ r> r> r? r/ j sJ ��� � �� �zState._gather_pillarc C s� z| j d�|d |d � W n ty Y dS w d�|d �}|d | jvrC|| j jv rE| j | |�}|s9dS | j�|d � dS dS dS )z� Check the module initialization function, if this is the first run of a state package that has a mod_init function, then execute the mod_init function in the state module. �{}.{}r r Nz {}.mod_init)r rB �KeyErrorr4 �_dictrq )r� rD ZminitZmretr> r> r? � _mod_init� s � � �zState._mod_initc C s� i }|D ]9}|d |d kr=|� d�r=|d |d kr=tjttg� D ]}||v r<||v r6|| �|| � q"|| ||< q"q|�|� |S )z* Aggregate the requisites r �__agg__r: )rw � frozenset�union�STATE_REQUISITE_KEYWORDS�STATE_REQUISITE_IN_KEYWORDSr� r )r� rD r � requisitesr� ri r> r> r? �_aggregate_requisites� s �� � zState._aggregate_requisitesc C s | j d d�}d|v r|d }|du r|d g}nt|t�s |S |d |v r�|�d�s�d�|d �}d| jvr<i | jd< || jd v rL| jd | sL|S || jv r�d| jd |< zd|d< | j| |||�}| �||�}W |S ty� t � d |d � Y |S w d | jd |< |S ) zQ Execute the aggregation systems to runtime modify the low chunk r< �state_aggregateZ aggregateTr rO z{}.mod_aggregateZ loader_cachez(Failed to execute aggregate for state %sF) r> rn rz rw rB r1 r rU r� r� r� )r� rD �runningr Zagg_optZagg_funr> r> r? �_mod_aggregate� s8 � ��zState._mod_aggregatec C s� dg d�}dD ] }|� |�||< qi }d}d|v r+t|d t�s&|d g}n|d }ng }|D ] }||vr<|� |�||< q/d|v rLd|vrL|d |d< nd| jd v r]| jd � d�|d< d|v r�| �||�}|d |d < |d �|d � d |v r�|d |d < d |v r�| �||�}|d p�|d |d < |d �|d � d |v r�|d p�|d |d < d |v r�| �|�}|d p�|d |d < |d �|d � d |v r�|d p�|d |d < |S )zR Check that unless doesn't return 0, and that onlyif returns a 0. F)r� r� �r4 r5 r: )�cwd�rootr, �envZ prepend_pathr0 �timeoutZsuccess_retcodesr! �shellr� r# r� r� � skip_watchr$ r% )rw rn rz r� �_run_check_onlyifrx �_run_check_unless�_run_check_creates) r� �low_datar� r� �cmd_optsZPOSSIBLE_CMD_ARGSr! Z run_cmd_argZ_retr> r> r? � _run_check� sJ � zState._run_checkc C sP |� d�}d|v r|� d�ng }||d�}| �|� | j| |d i |d ��S )z0Format slot args and run unless/onlyif function.r rs )rs r� r� )r� � format_slotsr> )r� r r rs r� r> r> r? �_run_check_function% s zState._run_check_functionc s� ddi� dD ] }|� |�� |< qt|d t�s |d g}n|d }� fdd�}|D ]�}t|t�r[z| jd |fddd �|��}W n tyK d }Y nw t�d |� ||�sZ� S q)t|t�r�d |vrvd |� �� d< t� � d � � S |� dd�}| � |�} |r�t j j�| |�} | j� dd�r�|| jd �s�� S q)| s�� �dddd�� � S � �ddd�� q)� �ddd�� � S � S )z� Make sure that all commands return True for the state to run. If any command returns False (non 0), the state will not run r� FrY r# c sF | dkr� d du r� � dddd�� dS | dkr!� � ddd�� dS ) Nr r� F�onlyif condition is falseT�r� r_ r� �onlyif condition is true�r� r� )r ��cmd�r� r> r? � _check_cmd< s ��z+State._run_check_onlyif.<locals>._check_cmd� cmd.retcodeT�Zignore_retcode� python_shellr� � Last command return code: %sr z no `fun` argument in onlyif: r� � get_returnN�retcoder rh ri rj rk z(onlyif execution failed, bad type passed)rw rn rz r� r> r r� r. ro �warningr� rg rG rH r� �traverse_dict_and_listr1 r ) r� rc rd r� Zlow_data_onlyifro r rm rt r� r> rn r? r` - sj �� � � � �����zState._run_check_onlyifc s� ddi� dD ] }|� |�� |< qt|d t�s |d g}n|d }� fdd�}|D ]�}t|t�r[z| jd |fddd �|��}t�d |� W n tyQ d }Y nw ||�rZ� S q)t|t�r�d |vrvd |� �� d< t� � d � � S |� dd�}| � |�} |r�t j j�| |�} | j� dd �r�|| jd �r�� S q)| r�� �dddd�� q)� �ddd�� � S � �ddd�� q)� S )zq Check if any of the commands return False (non 0). If any are False the state will run. r� FrY r$ c s> | dkr� � dddd�� dS � �dd � � � ddd�� dS ) Nr �unless condition is trueTri Fr_ �unless condition is falserk )r r� rl rn r> r? ro � s �� z+State._run_check_unless.<locals>._check_cmdrp Trq rs r r z no `fun` argument in unless: r� rt Nru rx ri ry rk z*unless condition is false, bad type passed)rw rn rz r� r> r� r. r ro rv r� rg rG rH r� rw r1 r ) r� rc rd r� Zlow_data_unlessro r rm rt r� r> rn r? ra z sl ��� �� ������zState._run_check_unlessc C s� ddi}dD ] }|� |�||< qi }d| jd v r#| jd � d�|d< t|d t�r1|d g|d< |d D ]9}| jd |fddd �|��}t�d |� |d kr^|d du r^|�d dd �� q5|d krn|�ddd �� | S q5|S )zD Alter the way a successful state run is determined r� FrY r^ r� r rp Trq rs r z(check_cmd determined the state succeededrk z%check_cmd determined the state failed)rw r� rn r� r> r� r. r )r� rc r� r� rd r rm r> r> r? �_run_check_cmd� s@ ��� �����zState._run_check_cmdc C s� ddi}dD ] }|� |�||< qt|d t�r2tj�|d �r2d�|d �|d< d|d< d|d< |S t|d t�rRtd d � |d D ��rRd |d< d|d< d|d< |S d |d< d|d< |S ) z/ Check that listed files exist r� FrY r% z {} existsr� Tr_ c S s g | ]}t j�|��qS r> )r^ r_ �exists)r� r_ r> r> r? r� � � z,State._run_check_creates.<locals>.<listcomp>zAll files in creates existzCreates files not found) rw rn r� r^ r_ r{ rB rz r� )r� rc r� r� r> r> r? rb � s$ ���zState._run_check_createsc C s d| _ dS )z- Rest the run_num value to 0 r N)r6 r? r> r> r? � reset_run_num rF zState.reset_run_numc C sZ | j dkrtj�| j| ji �| _dS tjj| j| j| j| j| j | j tj � | j �d�| _dS )zU Read the state loader value and loadup the correct states subsystem �thorium)r r r( N)r' rG r! r~ r� r> r rH � serializersr1 r r% �ContextlessFileClientr( r? r> r> r? � _load_states s � �zState._load_statesc C sB t �d� tjj| j| jd�| _tjj| j| j| j| j tj � | j�d�| _ t |t�r�|�dd�r�t |d t�rA|d |d ig}nt |d t�rM|d }ni }|D ].}|D ])}tj�| j|| | j �}|r~|D ]}d�|||�d�d � �}|| | j |< qgqUqQtj�| j�| _| �� tjj| j| j | j| j | j| jd �| _d S ) �1 Load the modules into the state �(Loading fresh modules for state activity)r( )rH r r( �providerFr z{}{}rf N)r r r( r )r� r� rG r! rH r� r( Z minion_modsr1 r r% r� r> rn ro rw r� rz Zraw_modrB �rindexr r� �renderr r� ) r� r� r Z providersr� �modr �funcZf_keyr> r> r? r2 sJ � ��� �zState.load_modulesc C s� t �d� | jd �d�dkr2zt�t� W n ty$ t �d� Y n t y1 t �d� Y nw | � � | j�dd�sL| j�dd �rN| j d � d S d S d S ) z) Refresh all the modules zRefreshing modules...r� r^ ZMacOSzBError encountered during module reload. Modules were not reloaded.�localF�multiprocessingTzsaltutil.refresh_modulesN) r� r. r� rw � importlib�reload�site� RuntimeErrorr� r� r2 r> r? r> r> r? �module_refreshF s" � �� �zState.module_refreshc C s, d}|� dd�rt�d� tj�| j�| jd< d}|� dd�r-t�d� | �� | jd< d}|d s=|� d d�r;| �� d S |� d d�sE|rK| �� d S |d dkr�|d dkrf|d � d�rd| �� d S d S |d dkrr| �� d S |d dkr�d|d v r�| �� d S d S d S |d dv r�| �� d S d S )a� Check to see if the modules for this state instance need to be updated, only update if the state is a file or a package and if it changed something. If the file function is managed check to see if the file is a possible module type, e.g. a python, pyx, or .so. Always refresh if the function is recurse, since that can lay down anything. Fr* zRefreshing grains...r� Tr+ zRefreshing pillar...r$ r� Zforce_reload_modulesNr) r �filer Zmanagedr: )z.pyz.pyxz.pyoz.pycz.soZrecurse�symlink�bin)r� ZportsZpip) rw r� r. rG r! r� r� r/ r� �endswith)r� r� r� Z_reload_modulesr> r> r? � check_refresh] s> � � �zState.check_refreshc C s� g }d|vr |� d� d|vr|� d� d|vr |� d� |d rDt|d t�sD|� d�|d d|v r:d �|d �nd t|d �j�� d |v rl|�d �}z tt|�d �|d < W n tt fyk |� d|� �� Y nw |rp|S |d d |d }|| j vr�d|v r�|� d�||d �� | j � |�}|r�|� d|� �� nJ|� d|� d�� n@t j j�| j | �}d}d}t|jt�r�t|j�}t|jt�r�t|j�}t|| �D ]} |j| |vr�|� d�|j| |�� q�d } d|v r�d} d|v �r d�|d �| j v�rd|v �r|d �|�d�� n|�d�|d< d} nd} | �rY|| D ]4} tt| ��} |d | k�rWt�|d | | ��sIt�|d | | ��rW|� d�|d | |d �� �q$|S )zR Verify the data, return an error statement if something is wrong r zMissing "state" datar zMissing "fun" datar: zMissing "name" dataz0ID '{}' {}is not formed as a string, but is a {}r4 z in SLS '{}' r� r0 � r9 zInvalid umask: rf z$State '{}' was not found in SLS '{}'zReason: zSpecified state 'z' was not foundr z!Missing parameter {} for state {}r r � {}.mod_watchr5 z<Recursive require detected in SLS {} for require {} in ID {})rx rn r� rB r� r� r� r� r� � ValueErrorr Zmissing_fun_stringrG rH rs Zget_function_argspecrz rp �defaults�tuple�ranger� rg rh �fnmatch) r� r� r� r0 rA �reasonZaspecZarglenZdeflen�indZreqdecri rj r> r> r? � verify_data� s� �� � � � ��� ����zState.verify_datac C s� g }t |t�s |�d� t� }|�� D �]�\}}z |�d�r W qW n ty* Y nw t |t�s?|�d�||d t |�j �� t |t�sPd�||�}|�|� q|D �]�}|�d�r[qR|| du rn|�d�||d |�� qRt || t �s�|�d�||d �� qRd }d |v r�|d 7 }|| D �]} t | t�r�|d 7 }d | � � v r�|�d | � d|� d|d � d�� q�t | t��r�t t| ��} | dkr�t | | t �s�|�d�||d �� | dv �r�t | | t �s�|�d�| ||d �� n�t|d�||< | | D ]�} t | t��rd| i} t | t��s|�d�| |d �� q�t t| ��} | | } d | v �r:|�d�| ||d | d| �d �� �� t| ��sH|�d�| �� q�| || | < | |v �r|||| v �r||| | |k�r||| d || | k�r||�d�|d || �� q�tt | ��d k�r�|�d �||d �� q�|�s�|d k�s�|d k�r�qR|�d |� d |d � d!�� qR|d k�r�d |v �r�|jd d d"�d gng }|�d#d$� || D �� |�d%|� d |d � d&�d'�|� � qRq|S )(r� r� r� zUID '{}' in SLS '{}' is not formed as a string, but is a {}. It may need to be quoted.r4 r� Nz�ID '{}' in SLS '{}' contains a short declaration ({}) with a trailing colon. When not passing any arguments to a state, the colon must be omitted.r� r rf rl r� r� r� r� r� r zKThe 'names' argument in state '{}' in SLS '{}' needs to be formed as a listr� r� )r r� r� r� z2Illegal requisite "{}", please check your syntax. r r� r� r r r� r� r� r� c s r� r� r� r� r> r> r? r� ~ � � z$State.verify_high.<locals>.<genexpr>r� r� r� )rn ro rx r ry r� r� r� rB r� r� rz r� rg rh r� r� rp r; r� r` r� r> r> r? r� � s( � � �� � ���� ���� � � �� �� � ���� ����� "���� �� zState.verify_highc C s" g }|D ] }|� | �|�� q|S )zD Verify the chunks in a list of low data structures )r� r� )r� r r� r� r> r> r? � verify_chunks� s zState.verify_chunksc C s d}|D ] }d|v r$t |d t�sq|d }||d kr$|dkr$|d }q|D ]Q}d|vr2||d< q't |d ttf�sW|d dkrH|d |d< n|d dkrSd|d< n||d< d|v rh|d |�d�d |d< |d dk rx|d |d |d< q'|jd d � d � |S ) r� rl r' r r� r� rN r� r� r� c S r� r� rA r� r> r> r? r� � r� z$State.order_chunks.<locals>.<lambda>r� )rn r� r� r� r� r r> r> r? r � s8 � ��zState.order_chunksc C s� g }|� � D ]�\}}|�d�rqd|v r | j�|d � |� � D ]�\}}t� }g } |�d�r/q ||d�} |dur<|| d< d|v rF|d | d< d|v rP|d | d< || d< |D ]E} t| t�rc|�| � qVt| t�r�| � � D ].\} } | dkr�| D ] }|| vr�| �|� qvql| d kr�ql| d kr�t| t�s�|| | < ql| | | < qlqV| r�d }| D ]=}t � | �}t|t�r�t t |� � ��}||d < tt|j|| �� n||d < ||d < |d 7 }|D ] }||d < |�|� q�q�q t � | �}|D ] }||d < |�|� q�q q| �|�}|S )r r� r4 r Nr6 r3 r5 r r r: rl r� r )ry r� r; rq rm rn r� ro rx r r rg rh r rz r r r )r� rr �orchestration_jidr r: r� r r r r r� r~ r� r r r� r r r r r> r> r? r � sz �� � �� �� 3zState.compile_high_datac C s� g }d|vr ||fS |� d�}|D �]B}|�� D �]:\}}tdd� |D ��}||vs0||| vrWt||||d�} t| �dkrQ|�d�||�dd�|�d d��� q| d d }g } |�d g �D ]} t| t �rl| �| � q_| �d �rx| �| d � q_|�d �| v r�t � d ||�dd�|�d d�|�dd�|�d d�� q|�� D ]�\} } | � d�r�q�| || vr�| || | < q�| D ]�}d}t || | �D ]�\}}t|t �r�t|t �r�|| | � |� || | �||� d}q�t|t��rDt|t��rDtt|��}|tt|| | | ��k�r(|tv �r || | | | �|| � n||| | |< d}|dk�rDtt|| | | ��dk�rD||| | |< q�|�sQ|| | �|� q�q�qq||fS )zM Pull the extend data and add it to the respective high data � __extend__c s s � | ] }|� d �s|V qdS )r� N�r� r� r> r> r? r� r� z)State.reconcile_extend.<locals>.<genexpr>�r{ rl z�Cannot extend ID '{0}' in '{1}:{2}'. It is not part of the high state. This is likely due to a missing include statement or an incorrectly typed ID. Ensure that a state with an ID of '{0}' is available in environment '{1}' and to SLS '{2}'r3 �baser4 r r� rv z=Cannot extend ID '%s' in '%s:%s' because '%s:%s' is excluded.r� FTr: r )r� ry rg r rp rx rB rw rn r� r� r. r� � enumerate�insertro rh rR r� )r� rr r{ r� �extZ ext_chunkr: r� Z state_typeZidsZ sls_excludesr r r r~ r Zhindr r� r> r> r? �reconcile_extend� s� �� � � � �����OzState.reconcile_extendc C s d|vr|S t � }t � }|�d�}|D ]7}t|t�r |�|� t|t�rJt|�dkr+qtt|� � ��}|dkr?|�|d � q|dkrJ|�|d � q|ru|� � D ]#\}}|� d�r[qQ|� dd�} | sdqQ|D ] } t � | | �rs|�|� qfqQ|D ] } | |v r�|�| � qw|S r )rm r� rn r� rq ro rp rg rh r ry r� rw r� ) r� rr r r r r r� r: r� rv Zex_r= r> r> r? r S sD � �� �zState.apply_excludec $ C s� h d�}|� h d��}i }g }| j�dg �}t|t�s |g}|�� D �]|\}}t|t�s-q"|�� D �]l\} } | �d�r<q1| D �]^} t| t��r�t| �dk rNq>t t | ��} | |vrYq>| |v rdt � d| � q>| � d�d } | | }t|t��r|�� D ]�\}}d }||vr�t� ||< d |v r�|�d �| |||d |d |�d �� �� |� d �d }||| vr�g || |< |d || d< |d || d < tt|| | ��D ]!}t t || | | ��| kr�|| | | | �| |i� d}q�|r�qy|| | �| | |igi� qyt|t��r�g }|D �]�}t|t��sl||v �r*dd� || D �}|d |i}nBd }t |�D ]6}dd� t || �D �D ]'}t || | �D ]}t|t��rad|v �ra|d |k�ra||i}d}�qG�q=�q0|�sl�q t|�dk �ru�q t t |��}|| }|dk�r�t||�}n|�||f� d |v �r�|�d �| |||d |d |�d �� �� |� d �d }|D �]�}|\}}| dk�r�||v�r�t� ||< | || v�r�g || | < || | �d||igi� | dk�r!t|||dd�}|D ]*\} } | |v�rt� || < | || v�rg || | < || | �d| |igi� �q�q�| dk�r�t|||dd�}|D ]u\} } | �s9�q0t| ||�} | |v�rIt� || < | || v�rVg || | < |� | �} || | D ]B} t| t��sk�qat| �dk�rt�qat t | ��| v �r�qat t | �� ��dk�r��qat t | �� ��dk�r��qa|| | �| � �qa�q0�q�| dk�r/t|||dd�}|D ]u\} } | �s��q�t|| |�} ||v�r�t� ||< | || v�r�g || | < |� | �} || | D ]B} t| t��s�q�t| �dk�r��q�t t | ��| v �r�q�t t | �� ��dk�r�q�t t | �� ��dk�r �q�|| | �| � �q�q��q�d }||v�r;t� ||< ||| v�rHg || |< |d || d< |d || d < tt|| | ��D ]#}t t || | | ��| k�r�|| | | | �| |i� d}�qb|�r��q�|| | �| | |igi� �q��q q>q1q"g |d < |�� D ]\} }!|d �| |!i� �q�| j|dd�\}"}#|�|#� |"|fS ) zG Extend the data reference with requisite_in arguments > r r1 r r r2 r r r > r r r r r �disabled_requisitesr� rl �-The %s requisite has been disabled, Ignoring.r� r Frf zeInvalid requisite in {}: {} for {}, in SLS '{}'. Requisites must not contain dots, did you mean '{}'?r4 Nr3 Tc S � g | ] }|� d �s|�qS �r� r� r� r> r> r? r� � � ��z&State.requisite_in.<locals>.<listcomp>c S r� r� r� )r� Z_st8r> r> r? r� � r� r: rv r r r r� r2 r r1 r� )rQ r� rw rn rz ry ro r� rp rg rh r� rv r; r rx rB r� r� r� r ru r r� r� )$r� rr Zreq_inZ req_in_allr� r� � disabled_reqsr= r� r r r~ r� Zrkeyry �_stater: �foundr� ZhingesZ _ind_high�_idZst8�jZpstateZpname�tupZext_idsr| Z _req_stateZext_argsZ ignore_argsZloc_argsr Z req_in_high� req_in_errorsr> r> r? � requisite_in{ s� � � �� �� � �� ��� �� � � � � � � � ��� ���� Y zState.requisite_inc C s� |du r| d#i |��}||j _tj�� }|�|� t|�}z|j |d |d i |d ��} W n) tyX } z tjd| t j d� t � � } d|i d| � �d �} W Y d} ~ nd} ~ ww tj�� } tj�� tj� � } | | }|| }|�� �� | d < | | }|jd |j d }|| d < d| d< d|v �rnd}|�|d �|d< |j jd �sR|d d |k�rQ|d d | d kr�n�|d d }|d d dkr�|t�d|d d � }t�d|� t�|� |j |d |d i |d ��}tj�� }tj�� } | | }|jd |j d }||d < | }|} | d �sd}n d �|| d | d �}d �|d |g�| d< | d |d |d | d < |dk�rE|d | d < |d }|d d |ks�n d �| d �s[dnt| d �d jd#i |d ��g�| d< tj �|jd! |j �}tj �|t j j �!|��}tj �"|��s�zt�#|� W n t$�y� Y nw t j j%�&|d"��}|�'t(| �� W d � dS 1 �s�w Y dS )$zZ The target function to call that will create the parallel thread/process NrA rs r� �'An exception occurred in this state: %s�Zexc_info_on_loglevelF�%An exception occurred in this state: �r� r: r� r� rZ rN rO r\ TZ __parallel__r& rl �test�attempts�untilr� �interval�splayr �QState result does not match retry until value, state will be re-run in %s secondsr� r� �GAttempt {}: Returned a result of "{}", with the following comment: "{}"� �� � ��The state would be retried every {interval} seconds (with a splay of up to {splay} seconds) a maximum of {attempts} times or until a result of {until} is returnedrc zwb+r> ))r r9 rP rQ rf rE rG r� r. �logging�DEBUG� traceback� format_excrR rS rT rU rV �verify_retry_datar� �random�randintr� �sleeprB r` r� r^ r_ r7 rG rH � hashutils� sha1_digestra rb �OSErrorrI �fopen�write�msgpack_serialize)�cls�instance� init_kwargsr: r� rD r9 rW r< r� r �trbrY �timezone_delta�local_finish_timerX r[ r\ �retriesr� � retry_ret�orig_retZ_commentZtrootZtfile�fp_r> r> r? �_call_parallel_targets s� $���� � �� ��� ��-���� �$�zState._call_parallel_targetc C s\ |� d�pdgd p|d � d�}|s |� d|� d��}tjj�� r%d}n| }d}tjjj| j|| j||||fd|� d�d �}z|� � W n^ t y� } zRd t |�vrT� i }| jd � � D ]\} } zt �| � W n t yq Y q]w | || < q]| j�� } || d < tjjj| j|| ||||fd|� d�d �}|� � W Y d}~nd}~ww |di d |d �} | S )zG Call the state defined in the given cdata in parallel rs Nr r� r: r5 zParallelState(�))�targetrs r: z cannot pickler z Started in a separate process)r: r� r� r� �proc)rw rG rH �platformZspawning_platform�processZProcessr� r& �startr� r� ry �pickle�dumpsr ) r� r� rD r9 r: r� r� r� Z clean_context�varr r� r� r> r> r? � call_parallel� sT " � � ����zState.call_parallelZ content_checkZunifyrl c s� t j �� }|t j �� t j �� }t�dt� d t�r!� d �� n� d |�� � � � | � � �}|rcd� d i dd�}|D ] } |d | � d�7 < q=| j |d< | j d 7 _ t |� | � � |� |S d� d i d �}� �d d �| jd < � d dkr�d� v r�� d | jd< n � �dd �| jd< � �d�s�t�d� d � d t� d t�r�� d �� n� d � d� v r�| �� � d�� �} tjjj| j| � d| itd�} t�� �|r�t�|�ni | j|r�t�|�ni | j�dd�d�} d� v r�t� d �| d< | j �r| � | j � � �d��r| jjd } d | jjd < �zv�zd }t � fd d �|D ���r7d!�� �| jv�r7|� | � � �� | j�d"d��scd#� v �rMt� d# �| d< nt| d$ �d%d �t��rct| d$ d% �| d< d| v�rld&| d< d'� v �rw� d' | d'< d(|v�s�|d( du �r�| | j_ | j �r�t!| �}n]| �"| d � d � d �}|�r�t�d)� d � nD� �d��s�� �d*��r�| �#| � | �}n0| �$| � tjj%�&� �d+��� | j| d | d, i | d$ ��}W d � n 1 �s�w Y i | j_ d-� v �r%d.�� �}d�� �}t'�(| j| �}|| jv�r|� | �)� �� nd-|j*v�r%|� | �)� �� W nI t+�yo } z<tj,d/|t-j.d0� t/�0� }| �d,��pDd gd1 �pN| d$ �d�}|�s[� �d� �d2��}d|i d3|� �d�}W Y d }~nd }~ww W � �d��r}| | jjd < | j�1d d � | j�1dd � n� �d��r�| | jjd < | j�1d d � | j�1dd � w t|t2��s�|S d4| v �r�|�3d4g ��4| d4 � d� v �r�| �� � �d��r�d� d< |S � �d5�|d5< | j |d< | j d 7 _ t |� | � � |� t j �� }t j �� t j �� }|| }|| }|�� � � |d6< || }|j5d7 |j6 d8 }||d9< � d2 |d2< t�d:t� d t��r9� d �� n� d |�� � � |� d;� v �r�d*� v�r�| �7� d; �� d;< | jjd �s߈ d; d< |d( k�r݈ d; d= |k�r݈ d; d> }� d; d? d1k�r�|t8�9d1� d; d? � }t�d@|� | j:dA |� | j;� |||d dB�}|} |}d�<dC�|| d( | d �|d �s�dn|d g�|d< |d9 | d9 |dD |d9< |d k�r�| d6 |d6< |S dE�<|d �s�dnt|d �dFjdGi � d; ��g�|d< |S )Hzk Call a state directly with the low data structure, verify data before processing. z Running state [%s] at time %sr: Fr� r� r� r� � __run_num__rl )r� r: r� r, Nr rm �passwordr- r7 z Executing state %s.%s for [%s]r r� z{0[state]}.{0[fun]}rA )Z initial_retZexpected_extra_kws�user�UNKNOWN)Z__low__Z __running__Z__instance_id__Z __lowstate__Z__user__r3 r� T)r$ r# r% c 3 s � | ]}|� v V qd S r� r> )r� ri rC r> r? r� y � � z State.call.<locals>.<genexpr>z{0[state]}.mod_run_checkZ lock_saltenvr/ r� r\ r� r6 r� z>Failed to meet global state conditions. State '%s' not called.r( r9 rs r z {0[state]}.mod_run_check_cmdr� r� r r5 r� �warningsr4 rZ rN rO r\ z3Completed state [%s] at time %s (duration_in_ms=%s)r& r� r� r� r� r� z test.sleep)r� r� r� r� r� r> )=rP rQ rR r� r� rn r� r� rS rT r� r6 r� r� rw r1 r2 rB rG rH rs Z format_callr �STATE_INTERNAL_KEYWORDS�immutabletypesZfreezer8 r� r9 r r@ re r r� rC r� rf rI � set_umask�inspectZ signaturerz � parametersrG r. r� r� r� r� r� ro � setdefaultr� rU rV r� r� r� r> �callr` ) r� rD r rW r� rW rX r� r� r� Zstate_func_namer� r9 r� Zreq_listZstate_check_cmdZ state_funcZstate_func_sigr r� r: rY r� r� r[ r\ r� r� r� r> rC r? r� s� � � � � � � � �� �� � ��&���� � " � ����� � ����� z State.callc C s� t �d|� |�dd�}t|�dkrt �d|� |S |d dkr-t �d|� t �d� |S tjj�|d �\}}}|r@|| j vrMt �d|� t �d � |S t �d |||� | j | |i |��}|d �d �d }|d �d d�dd � }t �d |� t �d|� d } z ||� d�d d � } W n t y� Y nw | r�| �dd�d � � } t �d|| � tjj j|| d dd�}|r�t|t�r�d�|�� � }t �d|� ||7 }|S t �d� |S )NzEvaluating slot: %s�:r� � zMalformed slot: %srl rG zkOnly execution modules are currently supported in slots. This means slot should start with "__slot__:salt:"z[Execution module should be specified in a function call format: test.arg('arg', kw='kwarg')zCalling slot: %s(%s, %s)�~r z slot_text: %szappend_data: %sr� rf z Searching slot result %s for %s)�default� delimiterr� z appending to slot result: %sz1Ignoring slot append, slot result is not a string)r� r. r; rp rv rG rH rs Zparse_functionr> r� r� r� r� rw rn r� r` r� ) r� Zslot�fmtr rs r� Z slot_returnZ slot_textZ append_dataZ return_getr> r> r? Z __eval_slot! sX � � �� �zState.__eval_slotc C s� d}dt |d �fd|d �� ff}|D ]�\}}|D ]�\}}tjjj|dd�}t|t�rW|�� D ]%\}} z| �|�rKt � d| � | � | �|| | |< W q0 t yU Y q0w qt|t �r�t |�D ]V\} } t � d| � t| t�r�| �� D ]'\}} z| �|�r�t � d| � | � | �|| | | |< W qs t y� Y qsw t| t�r�| �|�r�t � d | � | � | �|| | | < q`qt|t�r�|�|�r�t � d |� | � |�|| |< qqqd S ) a! Read in the arguments from the low level slot syntax to make a last minute runtime call to gather relevant data for the specific routine Will parse strings, first level of dictionary values, and strings and first level dict values inside of lists z __slot__:rs r� T)Zkeepz Slot processsing dict value %sz Slot processing list value: %sz%Slot processsing nested dict value %sz!Slot processsing nested string %szSlot processsing %sN)r� ry rG rH r� r� rn ro r� r� Ztrace�_State__eval_slotr� rz r� ) r� r� Z SLOT_TEXT�ctxZatypeZavaluesr� r~ r� �value�idxZ listvaluer> r> r? rf \ s` � �� ��� � ��� ��zState.format_slotsc C s� ddddd�}t tttd�}i }t|t�rH|�� D ],\}}||v r?t|| |�r/|| ||< qt�d||| � || ||< q|| ||< q|S t|t �ra|rS|}|S t�d� ddddd�}|S t�d � |}|S ) z3 verifies the specified retry data Tr� r � )r� r� r� r� )r� r� r� r� zFAn invalid value was passed for the retry %s, using default value '%s'z\State is set with explicit retry: False so using default retry configuration with 0 attemptszsState is set to retry, but retry: True or a valid dict for retry configuration was not found. Using retry defaults)�boolr� rn ro ry r� rv r. )r� Z retry_dataZretry_defaultsZ expected_dataZvalidated_retry_dataZ expected_keyZ value_typer> r> r? r� � sX �� � � ���� ��zState.verify_retry_datac C sT i }d| j d v rQ|dd� D ]A}d�|d |d �}| j d d D ]-}t�||�rOd�||�}t|�}i d|| j|d d �||< | jd 7 _|�|� nq"qi }|D ]7}d |v rd|�d � | S t|�} | |vr�| �|�} | d krw n| �|||�}| � ||�r�| S t � | _ qU | � |�r�nt �d� q�tt|�� �t|�� � �} | S )zU Iterate over a list of chunks and call them, checking for requires. Zstate_runs_disabledr� NrK r r z\The state function "{0}" is currently disabled by "{1}", to re-enable, run state.enable {1}.Fr4 )r� r� r� r� r4 rl � __FAILHARD__�killT�{�G�z�?)r� rB r� rE r6 �remover� � check_pause� call_chunk�check_failhardrm r3 �reconcile_procsrS r� ro rz ry ) r� r �disabledrD Zstate_�patr� Z_tagrW r< �actionr� r> r> r? � call_chunks� sZ �� � �� � zState.call_chunksc C sZ t |�}| j�dd�r dS |�d| jd �r+||v r+|| d du r$dS || d S dS )zK Check if the low data chunk should send a failhard signal r� Fr" r� N)rE r� rw )r� rD rW r< r> r> r? r s zState.check_failhardc C s� | j sdS tj�| jd d| j �}t�� }tj�|�r�z� d}tjj � |d���}zt |� � �}W n* tjj jjyZ |dkrIY W d � W dS |d7 }t�d� Y W d � q w |d }d }||v rh|}nd |v rnd }|r�d || v r�t�� } | | || d kr� W d � W d S d|| v r� W d � W dS n W d � W d S t�d� W d � n1 s�w Y q ty� } z t�d|� W Y d} ~ d S d} ~ ww d S )z@ Check to see if this low chunk has been paused Nrc � state_pauseTr �rb� rl r5 r� �__all__r\ r r z4Failed to read in pause data for file located at: %s)r r^ r_ r` r� rS �isfilerG rH rI r� �msgpack_deserialize�readZmsgpack� exceptionsZUnpackValueErrorr� rG r� r� ) r� rD � pause_pathr� �triesr� Zpdatr= r� rR r r> r> r? r sb � � � �� �� ���zState.check_pausec C s t � }|D ]z}|| �d�}|r|�� sztj�| jd | jtj j � |��}tj� |�s6dd|| d i d�}z tj j �|d��}t|�� �}W d � n1 sPw Y W n tyj dd || d i d�}Y nw || �|� || �d� q|�d� qd|vS ) zG Check the running dict for processes and resolve them r� rc Fz!Parallel process failed to returnr: )r� r� r: r� r NzParallel cache failure)rm rw �is_aliver^ r_ r` r� r7 rG rH r� r� r rI r� r r r� r r� rq )r� rW Zretsetr< r� Z ret_cacher� r� r> r> r? r < sB � � �� �� �zState.reconcile_procsc C s� | j �dg �}t|t�s|g}d}d|v r8d�|d �| jvr6d|v r.|d �|�d�� n |�d�|d< nd}d|v r_d�|d �| jvr]d |v rU|d �|�d�� n |�d�|d < nd}d|v red}d |v rkd}d |v rqd}d |v rwd}d |v r}d}d |v r�d}d|v r�d}d|v r�d}d|v r�d}|s�dS | �|� g g g g g g g g g g d� }|r�g |d < |D ]�}||v �r�|| du�r�||v r�t � d|� q�|| D ]�} t| t �r�d| i} t | �} d} t t| ��} | | } | du r�q�|D ]t} | dk�r t�| d | ��r d} || �| � q�z2t| t ��r;t�| d | ��s%t�| d | ��r:| dk�s1| d | k�r:d} || �| � nt�W q� t�yU } z td�| | d ���d}~w t�yf td�| | d ���w | �s�| dk�r�| jD ]}t�|| ��rd} n�qr| �s� dS q�q�t� }|�� D �]S\}}t� }|�d ��r�|�d ��s�| j}n|}i }|D ]} t| �}|�|�}|�r�|||< �q�|} | �|��r�nt�d � �q�|D ]�} t| �}||v�r�|� d � �q�|�d ��r�|| d du �r�|� d � �q�n|| d du �r|� d � �q�|�d��r)|| d �s"|� d� n|� d!� �q�|�d��r=|| d �r=|� d"� �q�|�d ��rW|| d du �rW|�d ��sW|� d#� |�d ��rs|| d du�rs|�d ��sq|� d$� �q�|| �d%d��r�|� d&� �q�|� d'��s�|d k�r�d&|v �s�d"|v �r�d |v �r�|� d � d!|v �r�d|v �r�|� d� d |v �r�|� d � d |v �r�d&|v �r�|� d � |� d(��r�d |v �r�d&|v �r�|� d&� |� |� �q�d |v �r�d }||fS d |v �r�d }||fS d$|v �rd#|v �r d&}||fS d$}||fS d |v �r"d!|v�r"d }||fS d|v �r2d!|v�r2d}||fS d"|v �r=d"}||fS d |v �rHd }||fS d&}||fS ))z` Look into the running data to check the status of all requisite states r� Fr r� r r Tr r r r r r r r r )�metr> ) r r r r r r r r r r Nr� r� rv r4 r: r5 zBCould not locate requisite of [{}] present in state with name [{}])�unmetr> r r r� �failr� Z onchangesmet�changeZpremetr5 � __state_ran__r Z_anyZ_all) r� rw rn rz rB r r� r� r r� rv r� rk rg rh r� rx rL r r� r; rm ry r� r5 rE rS r� rq r� r r )r� rD rW r r5 r� Zpresentr� Zr_stateri r� r� r� r� r � processed_slsZ fun_statsZ req_stats�run_dictZfiltered_run_dictr< Zrun_dict_chunk�statusr> r> r? �check_requisitea s� � � ���� �� �� ��� � � � � � � � � � � � ��zState.check_requisitec s� � j �d�s{� j �dd�s|r}� j �d�s d� fdd� }n� jd }d |i}|du r>tjj�� j� j d t|d �gd �}n/t |t�rUtjj�� j� j d t|�gd �}ntjj�� jd � j d t|d �gd�}||d< d� ji}||||d� dS dS dS )a Fire an event on the master bus If `fire_event` is set to True an event will be sent with the chunk name in the tag and the chunk result in the event data. If `fire_event` is set to a string such as `mystate/is/finished`, an event will be sent with the string added to the tag and the chunk result in the event data. If the `state_events` is set to True in the config, then after the chunk is evaluated an event will be set up to the master with the results. r� � state_eventsT� master_uriNc sN t jjj� j� jd dd��}|�| |� W d � d S 1 s w Y d S )NZsock_dirF)r )rG rH �eventZget_master_eventr� r. )r� r< �preloadZ_evtr? r> r? �ev_funcH s �"�z State.event.<locals>.ev_funczevent.fire_masterr� r� r: Z state_result�progr� Zjobrp r )r% r� ) r� rw r> rG rH r$ Ztagifyr r� rn )r� Z chunk_ret�lengthr. r& r� r< r% r> r? r? r$ 4 s6 �� � �� �z State.eventr c # C sd | � |||�}| �|� t|�}|�d�s| j�|� g d�}|�d�s5|�d� | j|||dd�\}}n | �|||�\}}|dk�r�i } g }|D ]�} g | | < | |vrTqI|| D ]�} t| t �rcd| i} t | �} d} t t | ��} | | }|D ]U}|d u r|qu| d kr�t � |d |�r�| d kr�d|d< |�|� d} qut � |d |�s�t � |d |�r�| dks�|d | kr�| d kr�d|d< n| dkr�d|d< |�|� d} qu| s�| d kr�| jD ] }t � ||�r�d} nq�| s�| | �| � qXqI| d �s | d �s | d �s | d �s | d �s | d �s | d �s | d �s | d �s | �d��r�d}| �� D ]+\} }|�s.�q%|d�d| �7 }|D ]}t t |��} || }|d �d | |�7 }�q8�q%|�d��r[| j}n|}t� \}}i d|||| jd �||< d D ] }|�|�|| |< �qp| jd 7 _| j|| t|�|�d!�d"� |S |D ]�}t|�}||v�r || jv �r|�d��r�|| jv�r�d|d< | �|||�| j|< | S | S ||v�r�t�d#� i dd#| jd$�||< d D ] }|�|�|| |< �q�| jd 7 _| j|| t|�|�d!�d"� | S | �|||�}| �||��r d|d%< | S �q�|�d��rV| �|||�\}}| �|||�| j|< | j| d& �sU|d'k�rUddi| j| d&< d | j| d(< n0|d 7 }|d)k�r~t�d#� i dd#| jd$�||< d D ] }|�|�|| |< �qpn| �||||�}| �||��r�d|d%< |S �n�|d*k�r�|�d��r�| �|||�| j|< �n�| �|||�||< �n�|d+k�ri|| jv �r�| j| �d&��r�i | j| d&< | j| ||< | j|| d,< d D ] }|�|�|| |< �q�nqt� }|� � D ]5}|D ]/}t|�} | j�| |�| ��} | d u �r�q�| d( du �r)d-j|d |d d.�}|�|� �q��q�d/�d0� d1d2� |D ���} t� \}}i d||| | jd �||< d D ] }|�|�|| |< �qM|| | j|< | jd 7 _n�|d'k�r�|�d��s�| �|||�} | d& �s�| �d3d��s�|� � }|d4 |d5< d6|d4< ||d7< | �|||�} | ||< n�|d8k�r�t� \}}i d||d9| jd �} d D ] }|�|�| |< �q�| ||< | | j|< | jd 7 _nv|dk�rt� \}}i d||d:d| jd;�||< d D ] }|�|�|| |< �q�| jd 7 _nH|dk�r3t� \}}i d||d<d| jd;�||< d D ] }|�|�|| |< �q | jd 7 _n|�d��rD| �|||�| j|< n | �|||�||< ||v �r�| j|| t|�|�d!�d"� || � d=d>�D ]F}!t� \}}| jd 7 _t|!d? �}"|!d? d |!d& |!d( |!�d@|�|!�dA|�|!�dBdC�d| jdD�||"< d D ] }|�|�||" |< �q��qi|S )Ezd Check if a chunk has any requires, execute the requires and then the chunk r ) r r r r r r r r r r7 T)r5 r r� FNrv r4 r r: r5 r r8 r r r r r r r r z)The following requisites were not found: z{}{}: z z {}{}: {} z )r� r� r\ rZ r� r� rY rl r. )r. zRecursive requisite found)r� r� r� r� r r� r r� � r r r� z {sls}.{_id})rv r� z One or more requisite failed: {}r� c s s � | ]}t |�V qd S r� )r� )r� �ir> r> r? r� 8 r� z#State.call_chunk.<locals>.<genexpr>r_ r �sfun� mod_watchZ__reqs__r5 zNo changes detectedz3State was not run because onfail req did not change)r� r� r\ rZ r� r r� z<State was not run because none of the onchanges reqs changedZ sub_state_runr> rD r\ rZ r� r� )r: r� r� r\ rZ r� r r� ) rX rN rE rw r3 rq rx r! rn r� rk rg rh r� r; ry rB r5 r] r6 r$ rp r� r� r� r r rm r� r` r r� )#r� rD rW r �depthr< rT r r� ZlostZ requisiteri r� r� r� r� r r� ZlreqsZlreqr rZ r\ r� ZctagZfailed_requisitesZreq_lowsZreq_lowZreq_tagZreq_retZ_cmtr� Zpre_retZsub_state_dataZsub_tagr> r> r? r e s � � ���'�������� � � � � ��� � ��� � � ��� � � � � � �zState.call_chunkc C s� g }|D ] }d|v r|� |� qg }i }|D ] }|�� }|d |d< d|d< d�|d �|d< |� |� q| �|�}|�|� |D ]} | j|| d< | jd7 _qA|�|� |S ) zY Find all of the beacon routines and call the associated mod_beacon runs Zbeaconr r+ Z mod_beaconz beacon_{}r5 r� rl )rx r rB r r r6 ) r� r rW Zbeaconsr� Z mod_beaconsr� rD r� r� r> r> r? � call_beacons� s( � zState.call_beaconsc s� g }i }|D ]J}|||d |d |d f< d|v r+|� |d |d |d f|d i� d|v rP|d D ] }|�� D ]\� }|� � |df|d |d igi� q9q3qg }i } |D ]�} | �� D ]�\� }|D ]�} t| t�s�d} |D ]}|d | ks||d | kr�|d |d i} d} qn| s�qc| �� D ]�\��t��fd d �|D ��s�t���d ����d �� d �� �di d�i} | �| � q���fdd�|�� D �}|D ]}}||vr�q�|| d �rEt� fdd �|D ���st� d � d �d �� d � d �d�� d � d �di d�i} | �| � qȇ fdd�|�� D �}|D ]/}|�� }|d |d< d|d< d�|d �|d< t D ] }||v �r<|� |� �q0|� |� �qq�q�qcq]qW| � |�}|�|� | D ]}| j | | d< | j d7 _ �qV|�| � |S ) zX Find all of the listen routines and call the associated mod_watch runs r r5 r: r r �lookupFTc 3 s$ � | ] }� |d ko �|v V qdS )r Nr> �r� �cref��lkey�lvalr> r? r� � s �" z$State.call_listen.<locals>.<genexpr>z&Referenced state {}: {} does not existrK r� )r� r: r� r� c s, g | ]\}}� |d kr�|v rt |��qS �r rL �r� r1 r� r2 r> r? r� � s �z%State.call_listen.<locals>.<listcomp>r� c 3 s, � | ]}� d |d ko� d |v V qdS )r rl Nr> r0 r� r> r? r� � s � � �r rl z listen_{}:{}c s0 g | ]\}}� d |d kr� d |v r|�qS �r rl r> r6 r� r> r? r� s �r r+ r, z listener_{}r� ) rx ry rn ro r@ rM rB r r rR r� r r6 )r� r rW Z listenersZcrefsr� Zl_inr Z mod_watchersr� Zl_dictZ listen_tor� ZrerrorZto_tagsZto_tagZ new_chunksrD ri r� r� r> )r� r3 r4 r? � call_listen� s� � ��� ���� � � � ��� � ������ I zState.call_listenc s� g }� � |�\}}|�|� |�� �|�� |r|S � �|�\}}|�|� � �|�}|r/|S � �||�}|r9|S � �|�}� �||�}� �||�}� fdd�}|� � j dur}t j � � j d d� j �} t j �| �r}zt �| � W |S ty| Y |S w |S )zI Process a high data call and ensure the defined states. c sX t j�t� jd �� j�} zt �| � t�d| � W d S t y+ t�d| � Y d S w )Nrc z Deleted accumulator data file %sz*File %s does not exist, no need to cleanup) r^ r_ r` re r� r8 r r� r. r� )Zaccum_data_pathr? r> r? �_cleanup_accumulator_data4 s � �z2State.call_high.<locals>._cleanup_accumulator_dataNrc r )r� r� r� r� r r r r8 r. r r^ r_ r` r� r r r� ) r� rr r� r� � ext_errorsr� r r� r9 r r> r? r? � call_high s< ��zState.call_highc C s� g }|s||fS t |t�s|�d|� d�� ||fS d}|D ]}||v r3|�d�||�� ||f S q |D ]�}t || t�sit || t�rid|| v r_|| �d�}|d |d gi||< q6|�d�||�� q6t� }t|| �D ]e} | �d �rzqr|| | d u r�|�d �||| �� qrt || | t �s�qrd| v r�| �d�}|d |v r�|�d �||�� qr|| � | �|| |d < || |d �|d � |� |d � qr|� | � qrq6||fS ) Nz Template � does not render to a dictionary)�includer ZextendszMThe '{}' declaration found on '{}' is invalid when rendering single templatesrf r rl z(ID {} in template {} is not a dictionaryr� z�ID '{}' in template {} contains a short declaration ({}) with a trailing colon. When not passing any arguments to a state, the colon must be omitted.zNID '{}' in template '{}' contains multiple state declarations of the same type) rn ro rx rB r� r; rm r� r� rz r� rq ) r� rr r� r� Z invalid_itemsrt r: r� r� r� r> r> r? r� J sp � � ��� � � �$zState.render_templatec C sN t || j| jd | jd | jd �}|s|S | �||�\}}|r"|S | �|�S r� )r r� r� r� r; �r� r� rr r� r> r> r? � call_template� � � zState.call_templatec C sN t || j| jd | jd | jd �}|s|S | �|d�\}}|r"|S | �|�S )zQ Enforce the states in a template, pass the template as a string r� r� r� z<template-str>)r r� r� r� r; r> r> r> r? �call_template_str� r@ zState.call_template_strc C � | j s | j�� d S d S r� )r) r( �closer? r> r> r? �destroy� � �z State.destroyc C � | S r� r> r? r> r> r? � __enter__� � zState.__enter__c G � | � � d S r� �rD �r� r� r> r> r? �__exit__� � zState.__exit__) NNNNNFr NNN�NNr� �F)NNrl r5 )9r� r� r� r� r� rC r/ rN rU rX re rg r` ra rz rb r} r� r2 r� r� r� r� r� r r r� r r� � classmethodr� r� rG rH Z decoratorsr Z OutputUnifierr� r� rf r� r r r r r! r$ r r. r8 r; r� r? rA rD rG rL r> r> r> r? r � s� �\%-!>MP" ))X ) & DZ( y q9 ;983 / % T 1 5 g2L r c @ s8 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd S ) �LazyAvailStatesz� The LazyAvailStates lazily loads the list of states of available environments. This is particularly usefull when top_file_merging_strategy=same and there are many environments. c C s || _ dd i| _d| _d S )Nr� F)�_hs�_avail�_filled)r� Zhsr> r> r? r� � s zLazyAvailStates.__init__c C s8 | j rd S | j�� D ] }|| jvrd | j|< q d| _ d S �NT)rT rR � _get_envsrS �r� r/ r> r> r? �_fill� s � zLazyAvailStates._fillc C s |dkrdS | � � || jv S )Nr� T)rX rS rW r> r> r? � __contains__� s z LazyAvailStates.__contains__c C sF |dkr| � � || jvs| j| d u r | jj�|�| j|< | j| S )Nr� )rX rS rR �clientZ list_statesrW r> r> r? � __getitem__� s zLazyAvailStates.__getitem__c C s. | � � g }| jD ] }|�|| | f� q |S r� )rX rS rx )r� r� r/ r> r> r? ry � s zLazyAvailStates.itemsN) r� r� r� r� r� rX rY r[ ry r> r> r> r? rQ � s rQ c @ s$ e Zd ZdZdd� Zdd� Zdd� Zdd � ZdCd d �Zd d� Z dd� Z dd� Z dd� Z dd� Z dCdd�Zdd� Zdd � Zd d � ZdDd d!�Zd"d#� Zd$d%� Zd&d'� Zd(d)� ZdCd*d+�Zd,d-� Zd.d/� ZdEd0d1�Zd2d3� Z 4 dFd5d6�Z dCd7d8�Z dCd9d:�Z d;d<� Z d=d>� Z d?d@� Z!dAdB� Z"d S )G� BaseHighStatea The BaseHighState is an abstract base class that is the foundation of running a highstate, extend it and add a self.state object of type State. When extending this class, please note that ``self.client`` and ``self.matcher`` should be instantiated and handled. c C s( | � |�| _d| _| �� | _t� | _d S )Ni' )�_BaseHighState__gen_optsr� �iorder� _BaseHighState__gather_avail�availr �building_highstate)r� r� r> r> r? r� � s zBaseHighState.__init__c C s t | �S )zO Lazily gather the lists of available sls data from the master )rQ r? r> r> r? Z__gather_avail� s z BaseHighState.__gather_availc C s� d|v r |d r |S | j �� }t|t�s3d|d< d|d< tjj�d�|d< i |d< d tj gi|d < |S |d |d< |� dd�|d< |d � d �rO|d |d< n"|d � d �rftjj�|d d d� �|d< n tjj�|d �|d< |� dd�|d< |� di �|d< |� d|d �|d< |d |d < |� d|� d��|d< |� d|� dg ��|d< |� d|� d��|d< |� d�p�|� d�p�d|d< |� d�p�|� d�p�d|d< |� di �|d< |� di �|d< |� dd�|d< |� dd�|d< |S )z� The options used by the High State object are derived from options on the minion and the master, or just the minion if the high state call is entirely local. Z local_statez jinja|yamlr� Fr" ztop.sls� state_top� nodegroupsr� � file_rootszsalt://�/rl N�state_top_saltenv�state_auto_order�top_file_merging_strategy� env_order� default_topr" rV Z jinja_envZ jinja_sls_envZjinja_lstrip_blocksZjinja_trim_blocks) rZ r� rn ro rG rH �url�create�syspathsZBASE_FILE_ROOTS_DIRrw r� )r� r� Zmoptsr> r> r? Z __gen_opts sL � � ���zBaseHighState.__gen_optsc s� dg�d| j v r���fdd�t| j d �D �� | j �dg �}t� ��fdd�|D �}| j�� � |r>� r>� fdd�|D �S |rB|S ���fdd�� D �� �S ) zM Pull the file server environments out of the master options r� rd c � g | ]}|� vr|�qS r> r> r� ��envsr> r? r� ? r| z+BaseHighState._get_envs.<locals>.<listcomp>ri c s" g | ] }|� v s� � |�s|�qS r> )rq �r� r\ )�membersr> r? r� C s c s g | ]}|� v r|�qS r> r> rq )� client_envsr> r? r� H r| c rn r> r> rq ro r> r? r� M r| )r� r� rz rw rm rZ rp )r� ri r> )rs rp rr r? rV 9 s " � zBaseHighState._get_envsNc C s t t�}t t�}t t�}d}| jd }|dkr%| jd s%| jd s%td��| jd rg| j�| jd | jd �}|r^d}t|| jj| jjd | jjd | jjd | jd |d �g|| jd < nzi g|| jd < nqd}| j� d d�}|r{t |t �s{t |�}|r�|gn| � � D ]>} | j�| jd | �}|r�|d }|| � t|| jj| jjd | jjd | jjd | |d �� q�|| � i � t�d| � q�|dkr�|dkr�| j� dd�s�t�d|� |dkr�t�dt| jjd �� |�� D ]!\} } | D ]} d| vr�q�| d D ] } || � | � q�| �d� q�q�|�r�g } |�� D ]Y\} }| � | � |�s%�q|D ]G}t�| j| |�D ]:} | || v �r=�q2|| � t| j�| | �� dd�| jj| jjd | jjd | jjd | |d�� || � | � �q2�q'�q| D ] } | |v �r|�| � �qs|�s|S )z& Gather the top files r rh Zsamer/ rj zXtop_file_merging_strategy set to 'same', but no default_top configuration option was setrb rl r� r� r� )r/ r rf Fz#No contents loaded for saltenv '%s'r0 ri Na top_file_merging_strategy is set to '%s' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.z�No contents found in top file. If this is not expected, verify that the 'file_roots' specified in 'etc/master' are accessible. The 'file_roots' configuration is: %srd r= �dest�r )r rz r� r rZ Z cache_filer r r� rw rn r� rV rx r� r. rv �reprry r� r� �filterr` � get_state)r� r �topsr= Zdoner� �merging_strategy�contentsrf r/ �ctops�ctoprv Zpopsr � sls_matchr> r> r? �get_topsP s� � � �� � �� �� � � �� �� ��zBaseHighState.get_topsc C s� | j d }z#d|� �}t| |�}t|d�s$d|� d�}t�|� t|��W ||�S ttfy? t�d|� | j}Y ||�S w )z- Cleanly merge the top files rh Z _merge_tops_�__call__r� z' is not callablez?Invalid top_file_merging_strategy '%s', falling back to 'merge') r� rF �hasattrr� r� r� r� rv �_merge_tops_merge)r� ry rz Z merge_attrZ merge_funcr� r> r> r? � merge_tops� s$ � ���zBaseHighState.merge_topsc C s t t�}|�dt t��}|D ],}|�� D ]%\}}|dkr qz|D ] }|| | || |< q W q ty9 td��w q|�� D ]K\}} | D ]D}|�� D ]=\}}|dkrTqK||kr`t�d||� qK||v rlt�d||� qKz|D ] }|| | || |< qoW qK ty� td��w qEq?|S )a� The default merging strategy. The base env is authoritative, so it is checked first, followed by the remaining environments. In top files from environments other than "base", only the section matching the environment from the top file will be considered, and it too will be ignored if that environment was defined in the "base" top file. r� r= �,Unable to render top file. No targets found.z�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'merge' and the saltenvs do not matchz�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as this saltenv was already defined in the 'base' top file)r r r� ry r� r r� r. ) r� ry �topZ base_topsr} r/ �targets�tgt�cenvr| r> r> r? r� � s^ � ��� ��� ���� z BaseHighState._merge_tops_mergec C sB t t�}|�� D ]�\}}tdd� |D ��rc| jd }|�|g �}tdd� |D ��r1t�d||� q|D ].}|�� D ] \}} ||krBq9t�d||� | D ] } || | || | < qK nt�d||� q3q|D ]8}|�� D ]1\}} |dkrtqk||kr�t�d ||� qkz| D ] } || | || | < q�W qk t y� t d ��w qeq|S ) z� For each saltenv, only consider the top file from that saltenv. All sections matching a given saltenv, which appear in a different saltenv's top file, will be ignored. c S � g | ]}|i k�qS r> r> r� r> r> r? r� ! � z2BaseHighState._merge_tops_same.<locals>.<listcomp>rj c S r� r> r> r� r> r> r? r� % r� zmThe '%s' saltenv has no top file, and the fallback saltenv specified by default_top (%s) also has no top filezDThe '%s' saltenv has no top file, using the default_top saltenv (%s)zlThe '%s' saltenv has no top file, and no matches were found in the top file for the default_top saltenv (%s)r= z�Section for saltenv '%s' in the '%s' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not matchr� ) r r ry r� r� rw r� r� r. r� r ) r� ry r� r� r| rj Z fallback_topsr} r/ r� r� r> r> r? �_merge_tops_same sl ������ ����z BaseHighState._merge_tops_samec s dd� }t t�}|�� D ]r}|D ]m}|�� D ]f\}}|dkr qzR|D ]M}||| vr5|| | || |< q"||| | �\} } ||| | �\} } g � | pN| } | durX� �| � � �| � � �� fdd�| D �� � || |< q"W q ty| td��w qq |S )z> Merge the top files into a single dictionary c S s< d }g }| D ]}t |t�r|}t |t�r|�|� q||fS r� )rn ro r� rx )r� � match_typer rt r> r> r? � _read_tgtf s �z6BaseHighState._merge_tops_merge_all.<locals>._read_tgtr= Nc rn r> r> r� �Zmergedr> r? r� � r| z7BaseHighState._merge_tops_merge_all.<locals>.<listcomp>r� )r r r� ry rx r� r� r )r� ry r� r� r| r} r/ r� r� Zm_type1Z m_states1Zm_type2Z m_states2r� r> r� r? �_merge_tops_merge_alla s< � ����z#BaseHighState._merge_tops_merge_allc C s g }t |t�s|�d� |S |�� D ]t\}}|dkrqt |t�s(|�d�|�� |dkr1|�d� t |t�s>|�d�|�� |�� D ]C\}}t |t�sY|�d�||t|�j�� qB|D ])}t |t�ru|� � D ] }|ss|�d�||�� qfq[t |t�r�|s�|�d �|�� q[qBq|S ) z: Verify the contents of the top file data z!Top data was not formed as a dictr= z4Environment {} in top file is not formed as a stringr� z#Empty saltenv statement in top filez?The top file matches for saltenv {} are not formatted as a dictzhMalformed topfile: state declarations for matcher {!r} in saltenv {!r} are not formed as a list (got {})z<Improperly formatted top file matcher in saltenv {}: {} filez*Environment {} contains an empty sls index) rn ro rx ry r� rB rz r� r� r� ) r� ry r� r/ rB �matchZslsmodsZslsmodr r> r> r? � verify_tops� s` �� � � ��� �����zBaseHighState.verify_topsc C sR z| j |d�}W n ty# } zt�d|j� i W Y d}~S d}~ww | �|�S )zA Returns the high data derived from the top file ru z Unable to render top file: %sN)r r r� r� r� )r� r ry r� r> r> r? �get_top� s �� zBaseHighState.get_topc s� t t��|�� D ],\�}�jd r��jd krq|�� D ]\}}� ���fdd�� � ||�jd � q q��� }|D ](����g �}�j�d�rP|� �|}n|�|� }��fdd�|D � ��< q;�S )z� Search through the top high data for matches and return the states that this minion needs to execute. Returns: {'saltenv': ['state1', 'state2', ...]} r/ c s� t |t�r|g}�jd | ||�rc��vrg ��< |D ]I}d|v r5|�d�}|�� D ] \}}� |||� q*t |t�rB�� �|� qt |t�rb|�� \}}|�jvrSq|�vr[g �|< �| �|� qd S d S )Nzconfirm_top.confirm_topZ subfilter) rn r� �matchersr� ry rx ro �popitemr` ) Z_match�_dataZ_optsrt Z_tmpdatar� r� �env_key�inc_sls)�_filter_matchesrB r/ r� r> r? r� � s, ��z2BaseHighState.top_matches.<locals>._filter_matchesrc Zmaster_tops_firstc rn r> r> r� )r� r> r? r� � r| z-BaseHighState.top_matches.<locals>.<listcomp>)r r ry r� � _master_topsrw )r� r� r� r� r� Z ext_matchesZtop_file_matches�secondr> )r� r� rB r/ r� r? � top_matches� s& � zBaseHighState.top_matchesc C s | j �� S )z� Get results from the master_tops system. Override this function if the execution of the master_tops needs customization. )rZ Z master_topsr? r> r> r? r� � s zBaseHighState._master_topsc C sb | j d sdS | jjd t|�dd�}|d r*tj�| j �| j d< | j�� | jj d< | j�� dS )zi If autoload_dynamic_modules is True then automatically load the dynamic modules Zautoload_dynamic_modulesNzsaltutil.sync_allF)Zrefreshr� r$ ) r� r r> rz rG r! r� r/ r� )r� rB Zsyncdr> r> r? � load_dynamic s zBaseHighState.load_dynamicFc s g }|s�j �||�}|�dd�} n|} tj�| �s#|�d|� d�� d} | s1|�d�||�� n�z t| �j j �j j d �j j d �j j d ||||d � } W nT t yu } zd |� d |� d | � �} t �| � |�| � W Y d} ~ n2d} ~ w ty� } z"d|� d| � �} t j| tjd� |�| � dt�� � �� W Y d} ~ nd} ~ ww z |�|� d |� �� W n ty� Y nw t�j d�rƈj j�|� | �r�t| t�s�|�d|� d�� | |fS g } d| v r�t| d t�s�d�|�}|�|� n| �d�} �� | |||� �� | |||� �� | |||� | D �]W� d}t� t��r$� � � \}� n|}|�j v�rFd�j v�rFd�|� ||�} t �!| � |�| � �q� �"d��r�t#�$d� �}|�r\|�%� \}} nd �� ||�} t �!| � |�| � �qt&|�}|�'d�}|�d d ��(d ��r�|�d � |t&|�k�r�d!�� ||�} t �!| � |�| � �qd�)|d| � | g �� ||k�r�|du �r�g }||v �s�t*�+�j | � ��r�|g}n g }n � �fd"d#�|D �}t&|�d$k�s�||v �r7t*�+�j | � ��p� g}|D ]?}t&|�d$k�r|d% n|}|� d |� �}||v�r3�j,|||||d&�\}}|�r+��-| ||� | �.|� |�r3|�/|� �q��qd } |�sNd'�|� ||k�rJd(�)|�n|�} nt&|�d$k�r_d)�|� d(�)|��} t �| � |�| � �qz ��0| � W | |fS t1�y� t �d*|� Y | |fS w i } | |fS )+zL Render a state file and retrieve all of the include states rt FzSpecified SLS z% on local filesystem cannot be found.NzeSpecified SLS {} in saltenv {} is not available on the salt master or through a configured fileserverr� r� r� )Z rendered_slsr zRendering SLS 'r� z ' failed: zRendering SLS z failed, render error: r� r� r; �SLS r< r= z5Include Declaration in SLS {} is not formed as a listZ_xenvr3 zDNonexistent saltenv '{}' found in include of '{}' within SLS '{}:{}'rf z ^(\.+)(.*)$z:Badly formatted include {} found in include in SLS '{}:{}'�sourcer� z /init.sls�initzTAttempted relative include of '{}' within SLS '{}:{}' goes beyond top level package c s" g | ] }t ��j| � �r|�qS r> )r� rw r` )r� Zaenv�r� r� r> r? r� � s ��z.BaseHighState.render_state.<locals>.<listcomp>rl r ru z\Unknown include: Specified SLS {}: {} is not available on the salt master in saltenv(s): {} r� zjAmbiguous include: Specified SLS {}: {} is available on the salt master in multiple available saltenvs: {}z/Could not render SLS %s. Syntax error detected.)2rZ rx rw r^ r_ r rx rB r r r� r� r r� �criticalrG r� r� r� r� rq r� r� r; rn ro rz r� �_handle_extend�_handle_exclude�_handle_state_declsr� r` r� r� �rer� �groupsrp r; r� r` r� rw � render_state�merge_included_statesr r� �_handle_iorderr� )r� rv r/ �modsrB r� r r� Z state_datard r r r� r= r� Zxenv_keyr� r� ZlevelsZ level_countZp_compsZ resolved_envsZ sls_targetsZ sls_target�r_envZmod_tgtZnstater> r� r? r� sJ �� � ��$�� � ��� �� � �� � ��� � �� ��� �� ��zBaseHighState.render_statec C s� | j d rp|D ]h}|| D ]a}t|t�sq t|| t�s q t|| | t�s'q d}|�d�r/q || | D ]}t|t�rNt|�dkrNtt|� � ��dkrNd}q5|snt|| | t�s[q || | � d| j i� | j d7 _ q q|S )z: Take a state and apply the iorder system rg Fr� r r' Trl ) r� rn r� ro rz r� rp rg rh r rx r^ )r� r r: Zs_decr� r~ r> r> r? r� � s4 ���z BaseHighState._handle_iorderc C sz |D ]�}t || t�sF|dkrq|dkrqt || t�r9d|| v r9|| �d�}d|d||d |d gi||< q|�d|� d |� d �� qt� }t|| �D ]R}|�d �rWqOt || | t�saqOd|v r�|�d�}|d |v rz|�d �||�� qO|| � |�|| |d < || |d �|d � |� |d � qO|� |� qOd|| vr�||| d< d|| vr�||| d< qd S )z= Add sls and saltenv components to the state r� r� rf r4 r3 r rl zID z in SLS z is not a dictionaryr� zIID '{}' in SLS '{}' contains multiple state declarations of the same typeN) rn ro r� r; rx rm rz r� rB r� rq ) r� r rv r/ r� r: r� r� r� r> r> r? r� sR � � ��z!BaseHighState._handle_state_declsc C s d|v r�|� d�}t|t�s|�d|� d�� dS |D ]g}t|| t�s.|�d�||�� qd|| vr:||| d< d|| vrF||| d< t|| �D ]5}|�d�rTqLt|| | t�s^qLd |v r�|�d �}|| � |�|| |d < || |d �|d � qLq|�d g ��|� dS dS ) z` Take the extend dec out of state and apply to the highstate global dec r� zExtension value in SLS 'z' is not a dictionaryNz3Extension name '{}' in SLS '{}' is not a dictionaryr4 r3 r� rf r rl r� ) r� rn ro rx rB rz r� r; r� ) r� r rv r/ r� r� r: r� r� r> r> r? r� > s< �� �� �z BaseHighState._handle_extendc C sJ d|v r#|� d�}t|t�sd�|�}|�|� |�dg ��|� dS dS )zh Take the exclude dec out of the state and apply it to the highstate global dec r z5Exclude Declaration in SLS {} is not formed as a listr� N)r� rn rz rB rx r� r� )r� r rv r/ r� r r� r> r> r? r� _ s � �z BaseHighState._handle_excludec C s4 | j }g }t� }g }|�� D ]�\}}|D ]{} || jv r%t�| j| | �}nd| jv r4t�| jd | �}n|�d�|�� |sA| g}|D ]K} |� d| � �} | |v rQqC| j| ||||d�\} } | rf| � || | � t | dd� �D ]\}}d|v r�d| � d�}||v r�d �| |�| |< qn|� | � qCqq| � |� ||fS ) zp Gather the state files and render them into a single unified salt high data structure. r3 z7No matching salt environment for environment '{}' foundr� ru Nzis not availabler� z in saltenvz*No matching sls found for '{}' in env '{}') ra rm ry r` r� rw rx rB r� r� r� r� �clean_duplicate_extends)r� rB r � highstateZ all_errorsr� Z statefilesr/ r r~ rv r� r r� r* r� Zthis_slsr> r> r? �render_highstatem sN � � ��� �� &z BaseHighState.render_highstatec C sZ d|v r+g }dd� |d D �D ]}|D ] }||vr |� |� qqdd� |D �|d< d S d S )Nr� c s s � | ]}|� � V qd S r� )ry )r� r� r> r> r? r� � r� z8BaseHighState.clean_duplicate_extends.<locals>.<genexpr>c S s g | ] }|d |d i�qS r7 r> )r� �tr> r> r? r� � r� z9BaseHighState.clean_duplicate_extends.<locals>.<listcomp>)rx )r� r� Zhighextry rt r> r> r? r� � s ���z%BaseHighState.clean_duplicate_extendsc C s� d|v r|� dg ��|�d�� d|v r |� dg ��|�d�� |D ]*}||v rL|| || krL|�d�||| d || d || d || d �� q"z|�|� W d S tyg |�d|� �� Y d S w )Nr� r� z�Detected conflicting IDs, SLS IDs need to be globally unique. The conflicting ID is '{}' and is found in SLS '{}:{}' and SLS '{}:{}'r3 r4 z*Error when rendering state with contents: )r� r� r� rx rB r r� )r� r� r r� r= r> r> r? r� � s. ��� �z#BaseHighState.merge_included_statesc C s |rdS d| j jd v rdS dS )z� Check the pillar for errors, refuse to run the state if there are errors in the pillar and return the pillar errors T�_errorsr$ F)r r� )r� �forcer> r> r? � _check_pillar� s zBaseHighState._check_pillarc C sj |s|S i }t |t�s|�d�}|D ] }|| D ]}||v r1||v r&|| ng ||< || �|� qq|S )zx Reads over the matches and returns a matches dict with just the ones that are in the whitelist �,)rn rz r; rx )r� rB � whitelistZ ret_matchesr\ rv r> r> r? �matches_whitelist� s ��z BaseHighState.matches_whitelistr� c C s� d}|dddi dd�i}t j�| jd |� d��} |rFt j�| �rFtjj�| d ��} tj � | �} | j � | |�W d � S 1 sAw Y g } z| � � } W n: tyu } zd || d < || d t| j�7 < |W Y d } ~ S d } ~ w ty� t�� }| �|� | Y S w | | �| �7 } | �| �}|s�d }||| d < |S | �||�}| �|� | �|�s�| dg7 } | | j jd d 7 } n'| �|�\} }|r�t|t�r�|� d�}d| v r�| d � |� n|| d< | |7 } | r�| S | s�|S tjj� d��] zDtjj � � �r| j j!d dd| gddd� tjj�| d�� } z tj �"| | � W n t#�y* Y nw W d � n 1 �s6w Y W n t$�yL t%�d| � Y nw W d � n 1 �sXw Y | j � | |�S )zP Run the sequence to execute the salt highstate for this minion zno_|-states_|-states_|-NoneFz No states found for this minionz No Statesr )r� r� r: r� r� rc z.cache.pr NzUnable to render top file: r� zQNo Top file or master_tops data matches found. Please see master log for details.z4Pillar failed to render with the following messages:r$ r� r� r� �? zcmd.runZattribz-R�quiet)rr Zoutput_loglevelzw+bz2Unable to write to "state.highstate" cache file %s)&r^ r_ r` r� r rG rH rI r� �payloadrI r r; r� r r� r� rG r� r� rx r� r� r� r� r� r� rn r; r� r� r� Z is_windowsr> �dumpr� r� r� )r� r �cacheZ cache_namer� r� r� Ztag_namer� Zcfnr� rr r� r� r� rB r� r� r> r> r? �call_highstate� s� �� � � � � ��� ���� �z BaseHighState.call_highstatec C sN g }| j |d�}|| �|�7 }| �|�}| j||d�\}}||7 }|r%|S |S )z9 Return just the highstate or the errors ru )r� r� r� r� )r� r r� r� rB rr r� r> r> r? r� 9 s z BaseHighState.compile_highstatec C s� | j |d�}| �|�}| j||d�\}}| j�|�\}}||7 }|| j�|�7 }| j�|�\}}||7 }| j�|�}|r>|S | j�|�}|S )z� Compile the highstate but don't run it, return the low chunks to see exactly what the highstate will execute ru ) r� r� r� r r� r� r� r r ) r� r r� rB rr r� r: r� r r> r> r? �compile_low_chunksI s z BaseHighState.compile_low_chunksc C s� g }| � � }|| �|�7 }|r|S | �|�}i }| j�� D ]F\}}g g dddd�}|�|g �}|D ]-} |d d7 < | |v rO|d d7 < |d �| � q1|d d7 < |d �| � q1|||< q |S ) z^ Return all used and unused states for the minion based on the top match data r )�used�unused� count_all� count_used� count_unusedr� rl r� r� r� r� )r� r� r� r` ry rw rx ) r� r� r� rB Z state_usager/ r Z env_usageZ env_matchesr r> r> r? �compile_state_usaged s0 � z!BaseHighState.compile_state_usagec C s t �r� )�NotImplementedErrorr? r> r> r? rD � rH zBaseHighState.destroyc C rF r� r> r? r> r> r? rG � rH zBaseHighState.__enter__c G rI r� rJ rK r> r> r? rL � rM zBaseHighState.__exit__r� r� rO )NNr� FNN)#r� r� r� r� r� r_ r] rV r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� rD rG rL r> r> r> r? r\ � sP 4 {:H) 5 4 Q#9! 3 � Z & r\ c @ st e Zd ZdZg Z ddd�Zdd� Zed d � �Zed d � �Z ed d� �Z dd� Z dd� Z dd� Z dS )� HighStatez� Generate and execute the salt "High State". The High State is the compound state derived from a group of template files stored on the salt master or in the local cache. NFr c C s� || _ | r | | _d| _n tj�| j �| _d| _t�| |� t| j |||||||| | jd� | _ tj � | j �| _ || _ i | _ g | _d S )NTF)r r r r! r" r( )r� rZ �preserve_clientrG r% r* r\ r� r r r! r� r Z_pydsl_all_declsZ_pydsl_render_stack) r� r� r r r r r r r! r" r( r> r> r? r� � s. � zHighState.__init__c C s | j �| � d S r� )�stackrx r? r> r> r? � push_active� � zHighState.push_activec C s g | _ d S r� )r� �r� r> r> r? � clear_active� s zHighState.clear_activec C s | j �� d S r� )r� r� r� r> r> r? � pop_active� s zHighState.pop_activec C s" z| j d W S ty Y d S w )N���)r� � IndexErrorr� r> r> r? � get_active� s �zHighState.get_activec C rB r� )r� rZ rD r? r> r> r? rD � rE zHighState.destroyc C rF r� r> r? r> r> r? rG � rH zHighState.__enter__c G rI r� rJ rK r> r> r? rL � rM zHighState.__exit__) NNNNNFr NN)r� r� r� r� r� r� r� rP r� r� r� rD rG rL r> r> r> r? r� � s. �* r� c @ s" e Zd ZdZdd� Zddd�ZdS )� MasterStatez9 Create a State object for master side compiling c C s t �| |� d S r� )r r� )r� r� Zminionr> r> r? r� � r� zMasterState.__init__Nc C s� t �d� tj�| j| jd �| _tj�| j�| _tj� | j�| _ tj� | j| j| j| j �| _ tjj | j| j| j | j d�| _ dS )r� r� r� )r r N)r� r� rG rZ ZFunctionWrapperr� r> r! rH r r r� r1 r� )r� r� r r> r> r? r2 � s � �zMasterState.load_modulesrN )r� r� r� r� r� r2 r> r> r> r? r� � s r� c @ s e Zd ZdZddd�ZdS )r� z7 Execute highstate compilation from the master Nc C sV t �|�}d|d< |d |d< |d |d< |d |d< ||d< ||d< t�| |� d S ) Nr� r( Z master_rootsrd r� rb r� r� )r r r� r� )r� r� r� r� r= r/ r� r> r> r? r� s zMasterHighState.__init__r� )r� r� r� r� r� r> r> r> r? r� s r� c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd S ) �RemoteHighStatez3 Manage gathering the data from the master c C s. || _ || _tjjj�| j d �| _d| _d S )Nr# F)r� r� rG �channelrZ Z ReqChannel�factory�_closing)r� r� r� r> r> r? r� s zRemoteHighState.__init__c C s< | j | jdd�}z | jj|ddd�W S ty i Y S w )z7 Return the state data from the master Z _master_state)r� r� rm r� i@ )r r] )r� r� r� �sendr )r� rI r> r> r? �compile_master% s �z RemoteHighState.compile_masterc C s | j rd S d| _ | j�� d S rU )r� r� rC r? r> r> r? rD / s zRemoteHighState.destroyc C rI r� rJ r? r> r> r? �__del__7 rM zRemoteHighState.__del__N)r� r� r� r� r� r� rD r� r> r> r> r? r� s r� rO )Xr� r rP r� r� r� r� r^ r� r� r� r� rS r� Zsalt.channel.clientrG Zsalt.fileclientZ salt.loaderZ salt.minionZ salt.pillarZ salt.syspathsrm Zsalt.utils.argsZsalt.utils.cryptZsalt.utils.dataZsalt.utils.decorators.stateZsalt.utils.dictupdateZsalt.utils.eventZsalt.utils.filesZsalt.utils.hashutilsZsalt.utils.immutabletypesrH r� Zsalt.utils.jidZsalt.utils.msgpackZsalt.utils.platformZsalt.utils.processZsalt.utils.urlZsalt.utils.yamlloaderrH Zsalt.exceptionsr r r Zsalt.serializers.msgpackr r r r� Z salt.templater r Zsalt.utils.datastructuresr r � getLoggerr� r� rP rR rS ZSTATE_RUNTIME_KEYWORDSrQ r� r@ rE rJ rM r] re rk ru r r� r� r� r� r� rG r� r� r rQ r\ r� r� r� r� r> r> r> r? �<module> s� ���,�� "- y x+ -V
| ver. 1.4 |
Github
|
.
| PHP 8.2.30 | Generation time: 0 |
proxy
|
phpinfo
|
Settings