Nightly Per-Antenna Quality Summary Notebook¶

Josh Dillon, Last Revised February 2021

This notebooks brings together as much information as possible from ant_metrics, auto_metrics and redcal to help figure out which antennas are working properly and summarizes it in a single giant table. It is meant to be lightweight and re-run as often as necessary over the night, so it can be run when any of those is done and then be updated when another one completes.

Contents:¶

  • Table 1: Overall Array Health
  • Table 2: RTP Per-Antenna Metrics Summary Table
  • Figure 1: Array Plot of Flags and A Priori Statuses
In [1]:
import os
os.environ['HDF5_USE_FILE_LOCKING'] = 'FALSE'
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
import pandas as pd
pd.set_option('display.max_rows', 1000)
from hera_qm.metrics_io import load_metric_file
from hera_cal import utils, io, redcal
import glob
import h5py
from copy import deepcopy
from IPython.display import display, HTML
from hera_notebook_templates.utils import status_colors
from hera_mc import mc
from pyuvdata import UVData

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the first few lines of this cell.

# JD = "2459122"
# data_path = '/lustre/aoc/projects/hera/H4C/2459122'
# ant_metrics_ext = ".ant_metrics.hdf5"
# redcal_ext = ".maybe_good.omni.calfits"
# nb_outdir = '/lustre/aoc/projects/hera/H4C/h4c_software/H4C_Notebooks/_rtp_summary_'
# good_statuses = "digital_ok,calibration_maintenance,calibration_triage,calibration_ok"
# os.environ["JULIANDATE"] = JD
# os.environ["DATA_PATH"] = data_path
# os.environ["ANT_METRICS_EXT"] = ant_metrics_ext
# os.environ["REDCAL_EXT"] = redcal_ext
# os.environ["NB_OUTDIR"] = nb_outdir
# os.environ["GOOD_STATUSES"] = good_statuses
In [3]:
# Use environment variables to figure out path to data
JD = os.environ['JULIANDATE']
data_path = os.environ['DATA_PATH']
ant_metrics_ext = os.environ['ANT_METRICS_EXT']
redcal_ext = os.environ['REDCAL_EXT']
nb_outdir = os.environ['NB_OUTDIR']
good_statuses = os.environ['GOOD_STATUSES']
print(f'JD = "{JD}"')
print(f'data_path = "{data_path}"')
print(f'ant_metrics_ext = "{ant_metrics_ext}"')
print(f'redcal_ext = "{redcal_ext}"')
print(f'nb_outdir = "{nb_outdir}"')
print(f'good_statuses = "{good_statuses}"')
JD = "2459885"
data_path = "/mnt/sn1/2459885"
ant_metrics_ext = ".ant_metrics.hdf5"
redcal_ext = ".known_good.omni.calfits"
nb_outdir = "/home/obs/src/H6C_Notebooks/_rtp_summary_"
good_statuses = "digital_ok,calibration_maintenance,calibration_triage,calibration_ok"
In [4]:
from astropy.time import Time, TimeDelta
utc = Time(JD, format='jd').datetime
print(f'Date: {utc.month}-{utc.day}-{utc.year}')
Date: 11-1-2022
In [5]:
# Per-season options
def ant_to_report_url(ant):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/antenna_report/antenna_{ant}_report.html'

Load Auto Metrics¶

In [6]:
use_auto_metrics = False

# find the auto_metrics file
glob_str = os.path.join(data_path, f'zen.{JD}*.auto_metrics.h5')
auto_metrics_file = sorted(glob.glob(glob_str))

# if it exists, load and extract relevant information
if len(auto_metrics_file) > 0:
    auto_metrics_file = auto_metrics_file[0]
    print(f'Found auto_metrics results file at {auto_metrics_file}.')
    
    auto_metrics = load_metric_file(auto_metrics_file)
    mean_round_modz_cut = auto_metrics['parameters']['mean_round_modz_cut']
    auto_ex_ants = auto_metrics['ex_ants']['r2_ex_ants']
    
    use_auto_metrics = True
else:
    print(f'No files found matching glob {glob_str}. Skipping auto_metrics.')
Found auto_metrics results file at /mnt/sn1/2459885/zen.2459885.25257.sum.auto_metrics.h5.

Load Ant Metrics¶

In [7]:
use_ant_metrics = False

# get a list of all ant_metrics files
glob_str = os.path.join(data_path, f'zen.{JD}.?????.sum{ant_metrics_ext}')
ant_metrics_files = sorted(glob.glob(glob_str))

# if they exist, load as many of them as possible
if len(ant_metrics_files) > 0:
    print(f'Found {len(ant_metrics_files)} ant_metrics files matching glob {glob_str}')
    ant_metrics_apriori_exants = {}
    ant_metrics_xants_dict = {}
    ant_metrics_dead_ants_dict = {}
    ant_metrics_crossed_ants_dict = {}
    ant_metrics_dead_metrics = {}
    ant_metrics_crossed_metrics = {}
    dead_cuts = {}
    crossed_cuts = {}
    for amf in ant_metrics_files:
        with h5py.File(amf, "r") as infile: # use h5py directly since it's much faster than load_metric_file
            # get out results for this file
            dead_cuts[amf] = infile['Metrics']['dead_ant_cut'][()]
            crossed_cuts[amf] = infile['Metrics']['cross_pol_cut'][()]
            xants = infile['Metrics']['xants'][:]
            dead_ants = infile['Metrics']['dead_ants'][:]
            crossed_ants = infile['Metrics']['crossed_ants'][:]        
            try:
                # look for ex_ants in history
                ex_ants_string = infile['Header']['history'][()].decode()
                ex_ants_string = ex_ants_string.split('--apriori_xants')[1]
                ex_ants_string = ex_ants_string.split('--')[0].strip()
            except:
                ex_ants_string = ''
                    
            # This only works for the new correlation-matrix-based ant_metrics
            if 'corr' in infile['Metrics']['final_metrics'] and 'corrXPol' in infile['Metrics']['final_metrics']:
                ant_metrics_dead_metrics[amf] = {eval(ant): infile['Metrics']['final_metrics']['corr'][ant][()]
                                                 for ant in infile['Metrics']['final_metrics']['corr']}
                ant_metrics_crossed_metrics[amf] = {eval(ant): infile['Metrics']['final_metrics']['corrXPol'][ant][()]
                                                    for ant in infile['Metrics']['final_metrics']['corrXPol']}                       
            else:
                raise(KeywordError)
        
        # organize results by file
        ant_metrics_xants_dict[amf] = [(int(ant[0]), ant[1].decode()) for ant in xants]
        ant_metrics_dead_ants_dict[amf] = [(int(ant[0]), ant[1].decode()) for ant in dead_ants]
        ant_metrics_crossed_ants_dict[amf] = [(int(ant[0]), ant[1].decode()) for ant in crossed_ants]
        ant_metrics_apriori_exants[amf] = [int(ant) for ant in ex_ants_string.split()]
    
    dead_cut = np.median(list(dead_cuts.values()))
    crossed_cut = np.median(list(crossed_cuts.values()))
        
    use_ant_metrics = True
else:
    print(f'No files found matching glob {glob_str}. Skipping ant_metrics.')
Found 1850 ant_metrics files matching glob /mnt/sn1/2459885/zen.2459885.?????.sum.ant_metrics.hdf5

Load chi^2 info from redcal¶

In [8]:
use_redcal = False
glob_str = os.path.join(data_path, f'zen.{JD}.?????.sum{redcal_ext}')

redcal_files = sorted(glob.glob(glob_str))
if len(redcal_files) > 0:
    print(f'Found {len(redcal_files)} ant_metrics files matching glob {glob_str}')
    post_redcal_ant_flags_dict = {}
    flagged_by_redcal_dict = {}
    cspa_med_dict = {}
    for cal in redcal_files:
        hc = io.HERACal(cal)
        _, flags, cspa, chisq = hc.read()
        cspa_med_dict[cal] = {ant: np.nanmedian(cspa[ant], axis=1) for ant in cspa}

        post_redcal_ant_flags_dict[cal] = {ant: np.all(flags[ant]) for ant in flags}
        # check history to distinguish antennas flagged going into redcal from ones flagged during redcal
        tossed_antenna_lines =  hc.history.replace('\n','').split('Throwing out antenna ')[1:]
        flagged_by_redcal_dict[cal] = sorted([int(line.split(' ')[0]) for line in tossed_antenna_lines])
        
    use_redcal = True
else:
    print(f'No files found matching glob {glob_str}. Skipping redcal chisq.')
No files found matching glob /mnt/sn1/2459885/zen.2459885.?????.sum.known_good.omni.calfits. Skipping redcal chisq.

Figure out some general properties¶

In [9]:
# Parse some general array properties, taking into account the fact that we might be missing some of the metrics
ants = []
pols = []
antpol_pairs = []

if use_auto_metrics:
    ants = sorted(set(bl[0] for bl in auto_metrics['modzs']['r2_shape_modzs']))
    pols = sorted(set(bl[2] for bl in auto_metrics['modzs']['r2_shape_modzs']))
if use_ant_metrics:
    antpol_pairs = sorted(set([antpol for dms in ant_metrics_dead_metrics.values() for antpol in dms.keys()]))
    antpols = sorted(set(antpol[1] for antpol in antpol_pairs))
    ants = sorted(set(antpol[0] for antpol in antpol_pairs) | set(ants))
    pols = sorted(set(utils.join_pol(ap, ap) for ap in antpols) | set(pols))
if use_redcal:
    antpol_pairs = sorted(set([ant for cspa in cspa_med_dict.values() for ant in cspa.keys()]) | set(antpol_pairs))
    antpols = sorted(set(antpol[1] for antpol in antpol_pairs))
    ants = sorted(set(antpol[0] for antpol in antpol_pairs) | set(ants))
    pols = sorted(set(utils.join_pol(ap, ap) for ap in antpols) | set(pols))

# Figure out remaining antennas not in data and also LST range
data_files = sorted(glob.glob(os.path.join(data_path, 'zen.*.sum.uvh5')))
hd = io.HERAData(data_files[0])
unused_ants = [ant for ant in hd.antpos if ant not in ants]    
hd_last = io.HERAData(data_files[-1])

Load a priori antenna statuses and node numbers¶

In [10]:
# try to load a priori antenna statusesm but fail gracefully if this doesn't work.
a_priori_statuses = {ant: 'Not Found' for ant in ants}
nodes = {ant: np.nan for ant in ants + unused_ants}
try:
    from hera_mc import cm_hookup

    # get node numbers
    hookup = cm_hookup.get_hookup('default')
    for ant_name in hookup:
        ant = int("".join(filter(str.isdigit, ant_name)))
        if ant in nodes:
            if hookup[ant_name].get_part_from_type('node')['E<ground'] is not None:
                nodes[ant] = int(hookup[ant_name].get_part_from_type('node')['E<ground'][1:])
    
    # get apriori antenna status
    for ant_name, data in hookup.items():
        ant = int("".join(filter(str.isdigit, ant_name)))
        if ant in a_priori_statuses:
            a_priori_statuses[ant] = data.apriori

except Exception as err:
    print(f'Could not load node numbers and a priori antenna statuses.\nEncountered {type(err)} with message: {err}')

Summarize auto metrics¶

In [11]:
if use_auto_metrics:
    # Parse modzs
    modzs_to_check = {'Shape': 'r2_shape_modzs', 'Power': 'r2_power_modzs', 
                      'Temporal Variability': 'r2_temp_var_modzs', 'Temporal Discontinuties': 'r2_temp_diff_modzs'}
    worst_metrics = []
    worst_zs = []
    all_modzs = {}
    binary_flags = {rationale: [] for rationale in modzs_to_check}

    for ant in ants:
        # parse modzs and figure out flag counts
        modzs = {f'{pol} {rationale}': auto_metrics['modzs'][dict_name][(ant, ant, pol)] 
                 for rationale, dict_name in modzs_to_check.items() for pol in pols}
        for pol in pols:
            for rationale, dict_name in modzs_to_check.items():
                binary_flags[rationale].append(auto_metrics['modzs'][dict_name][(ant, ant, pol)] > mean_round_modz_cut)

        # parse out all metrics for dataframe
        for k in modzs:
            col_label = k + ' Modified Z-Score'
            if col_label in all_modzs:
                all_modzs[col_label].append(modzs[k])
            else:
                all_modzs[col_label] = [modzs[k]]
                
    mean_round_modz_cut = auto_metrics['parameters']['mean_round_modz_cut']
else:
    mean_round_modz_cut = 0

Summarize ant metrics¶

In [12]:
if use_ant_metrics:
    a_priori_flag_frac = {ant: np.mean([ant in apxa for apxa in ant_metrics_apriori_exants.values()]) for ant in ants}
    dead_ant_frac = {ap: {ant: np.mean([(ant, ap) in das for das in ant_metrics_dead_ants_dict.values()])
                                 for ant in ants} for ap in antpols}
    crossed_ant_frac = {ant: np.mean([np.any([(ant, ap) in cas for ap in antpols])
                                      for cas in ant_metrics_crossed_ants_dict.values()]) for ant in ants}
    ant_metrics_xants_frac_by_antpol = {antpol: np.mean([antpol in amx for amx in ant_metrics_xants_dict.values()]) for antpol in antpol_pairs}
    ant_metrics_xants_frac_by_ant = {ant: np.mean([np.any([(ant, ap) in amx for ap in antpols])
                                     for amx in ant_metrics_xants_dict.values()]) for ant in ants}
    average_dead_metrics = {ap: {ant: np.nanmean([dm.get((ant, ap), np.nan) for dm in ant_metrics_dead_metrics.values()]) 
                                 for ant in ants} for ap in antpols}
    average_crossed_metrics = {ant: np.nanmean([cm.get((ant, ap), np.nan) for ap in antpols 
                                                for cm in ant_metrics_crossed_metrics.values()]) for ant in ants}
else:
    dead_cut = 0.4
    crossed_cut = 0.0

Summarize redcal chi^2 metrics¶

In [13]:
if use_redcal:
    cspa = {ant: np.nanmedian(np.hstack([cspa_med_dict[cal][ant] for cal in redcal_files])) for ant in antpol_pairs}
    redcal_prior_flag_frac = {ant: np.mean([np.any([afd[ant, ap] and not ant in flagged_by_redcal_dict[cal] for ap in antpols])
                                            for cal, afd in post_redcal_ant_flags_dict.items()]) for ant in ants}
    redcal_flagged_frac = {ant: np.mean([ant in fbr for fbr in flagged_by_redcal_dict.values()]) for ant in ants}

Get FEM switch states¶

In [14]:
HHautos = sorted(glob.glob(f"{data_path}/zen.{JD}.*.sum.autos.uvh5"))
diffautos = sorted(glob.glob(f"{data_path}/zen.{JD}.*.diff.autos.uvh5"))

try:
    db = mc.connect_to_mc_db(None)
    session = db.sessionmaker()
    startJD = float(HHautos[0].split('zen.')[1].split('.sum')[0])
    stopJD = float(HHautos[-1].split('zen.')[1].split('.sum')[0])
    start_time = Time(startJD,format='jd')
    stop_time = Time(stopJD,format='jd')

    # get initial state by looking for commands up to 3 hours before the starttime
    # this logic can be improved after an upcoming hera_mc PR
    # which will return the most recent command before a particular time.
    search_start_time = start_time - TimeDelta(3*3600, format="sec")
    initial_command_res = session.get_array_signal_source(starttime=search_start_time, stoptime=start_time)
    if len(initial_command_res) == 0:
        initial_source = "Unknown"
    elif len(command_res) == 1:
        initial_source = initial_command_res[0].source
    else:
        # multiple commands
        times = []
        sources = []
        for obj in command_res:
            times.append(obj.time)
            sources.append(obj.source)
        initial_source = sources[np.argmax(times)]
    
    # check for any changes during observing
    command_res = session.get_array_signal_source(starttime=start_time, stoptime=stop_time)
    if len(command_res) == 0:
        # still nothing, set it to None
        obs_source = None
    else:
        obs_source_times = []
        obs_source = []
        for obj in command_res:
            obs_source_times.append(obj.time)
            obs_source.append(obj.source)

    if obs_source is not None:
        command_source = [initial_source] + obs_source
    else:
        command_source = initial_source
    
    res = session.get_antenna_status(starttime=startTime, stoptime=stopTime)
    fem_switches = {}
    right_rep_ant = []
    if len(res) > 0:
        for obj in res:
            if obj.antenna_number not in fem_switches.keys():
                fem_switches[obj.antenna_number] = {}
            fem_switches[obj.antenna_number][obj.antenna_feed_pol] = obj.fem_switch
        for ant, pol_dict in fem_switches.items():
            if pol_dict['e'] == initial_source and pol_dict['n'] == initial_source:
                right_rep_ant.append(ant)
except Exception as e:
    print(e)
    initial_source = None
    command_source = None
    right_rep_ant = []
name 'command_res' is not defined

Find X-engine Failures¶

In [15]:
read_inds = [1, len(HHautos)//2, -2]
x_status = [1,1,1,1,1,1,1,1]
s = UVData()
s.read(HHautos[1])

nants = len(s.get_ants())
freqs = s.freq_array[0]*1e-6
nfreqs = len(freqs)

antCon = {a: None for a in ants}
rightAnts = []
for i in read_inds:
    s = UVData()
    d = UVData()
    s.read(HHautos[i])
    d.read(diffautos[i])
    for pol in [0,1]:
        sm = np.abs(s.data_array[:,0,:,pol])
        df = np.abs(d.data_array[:,0,:,pol])
        sm = np.r_[sm, np.nan + np.zeros((-len(sm) % nants,len(freqs)))]
        sm = np.nanmean(sm.reshape(-1,nants,nfreqs),axis=1)
        df = np.r_[df, np.nan + np.zeros((-len(df) % nants,len(freqs)))]
        df = np.nanmean(df.reshape(-1,nants,nfreqs),axis=1)

        evens = (sm + df)/2
        odds = (sm - df)/2
        rat = np.divide(evens,odds)
        rat = np.nan_to_num(rat)
        for xbox in range(0,8):
            xavg = np.nanmean(rat[:,xbox*192:(xbox+1)*192],axis=1)
            if np.nanmax(xavg)>1.5 or np.nanmin(xavg)<0.5:
                x_status[xbox] = 0
    for ant in ants:
        for pol in ["xx", "yy"]:
            if antCon[ant] is False:
                continue
            spectrum = s.get_data(ant, ant, pol)
            stdev = np.std(spectrum)
            med = np.median(np.abs(spectrum))
            if (initial_source == 'digital_noise_same' or initial_source == 'digital_noise_different') and med < 10:
                antCon[ant] = True
            elif (initial_source == "load" or initial_source == 'noise') and 80000 < stdev <= 4000000 and antCon[ant] is not False:
                antCon[ant] = True
            elif initial_source == "antenna" and stdev > 500000 and med > 950000 and antCon[ant] is not False:
                antCon[ant] = True
            else:
                antCon[ant] = False
            if np.min(np.abs(spectrum)) < 100000:
                antCon[ant] = False
for ant in ants:
    if antCon[ant] is True:
        rightAnts.append(ant)
            
x_status_str = ''
for i,x in enumerate(x_status):
    if x==0:
        x_status_str += '\u274C '
    else:
        x_status_str += '\u2705 '

Build Overall Health DataFrame¶

In [16]:
def comma_sep_paragraph(vals, chars_per_line=40):
    outstrs = []
    for val in vals:
        if (len(outstrs) == 0) or (len(outstrs[-1]) > chars_per_line):
            outstrs.append(str(val))
        else:
            outstrs[-1] += ', ' + str(val)
    return ',<br>'.join(outstrs)
In [17]:
# Time data
to_show = {'JD': [JD]}
to_show['Date'] = f'{utc.month}-{utc.day}-{utc.year}'
to_show['LST Range'] = f'{hd.lsts[0] * 12 / np.pi:.3f} -- {hd_last.lsts[-1] * 12 / np.pi:.3f} hours'

# X-engine status
to_show['X-Engine Status'] = x_status_str

# Files
to_show['Number of Files'] = len(data_files)

# Antenna Calculations
to_show['Total Number of Antennas'] = len(ants)

to_show[' '] = ''
to_show['OPERATIONAL STATUS SUMMARY'] = ''

status_count = {status: 0 for status in status_colors}
for ant, status in a_priori_statuses.items():
    if status in status_count:
        status_count[status] = status_count[status] + 1
    else:
        status_count[status] = 1
to_show['Antenna A Priori Status Count'] = '<br>'.join([f'{status}: {status_count[status]}' for status in status_colors if status in status_count and status_count[status] > 0])

to_show['Commanded Signal Source'] = ', '.join(command_source if hasattr(command_source, '__iter__') else [str(command_source)])
to_show['Antennas in Commanded State (reported)'] = f'{len(right_rep_ant)} / {len(ants)} ({len(right_rep_ant) / len(ants):.1%})'
to_show['Antennas in Commanded State (observed)'] = f'{len(rightAnts)} / {len(ants)} ({len(rightAnts) / len(ants):.1%})'

if use_ant_metrics:
    to_show['Cross-Polarized Antennas'] = ', '.join([str(ant) for ant in ants if (np.max([dead_ant_frac[ap][ant] for ap in antpols]) + crossed_ant_frac[ant] == 1) 
                                                                                 and (crossed_ant_frac[ant] > .5)])

# Node calculations
nodes_used = set([nodes[ant] for ant in ants if np.isfinite(nodes[ant])])
to_show['Total Number of Nodes'] = len(nodes_used)
if use_ant_metrics:
    node_off = {node: True for node in nodes_used}
    not_correlating = {node: True for node in nodes_used}
    for ant in ants:
        for ap in antpols:
            if np.isfinite(nodes[ant]):
                if np.isfinite(average_dead_metrics[ap][ant]):
                    node_off[nodes[ant]] = False
                if dead_ant_frac[ap][ant] < 1:
                    not_correlating[nodes[ant]] = False
    to_show['Nodes Registering 0s'] = ', '.join([f'N{n:02}' for n in sorted([node for node in node_off if node_off[node]])])
    to_show['Nodes Not Correlating'] = ', '.join([f'N{n:02}' for n in sorted([node for node in not_correlating if not_correlating[node] and not node_off[node]])])

# Pipeline calculations    
to_show['  '] = ''
to_show['NIGHTLY ANALYSIS SUMMARY'] = ''
    
all_flagged_ants = []
if use_ant_metrics:
    to_show['Ant Metrics Done?'] = '\u2705'
    ant_metrics_flagged_ants = [ant for ant in ants if ant_metrics_xants_frac_by_ant[ant] > 0]
    all_flagged_ants.extend(ant_metrics_flagged_ants)
    to_show['Ant Metrics Flagged Antennas'] = f'{len(ant_metrics_flagged_ants)} / {len(ants)} ({len(ant_metrics_flagged_ants) / len(ants):.1%})' 
else:
    to_show['Ant Metrics Done?'] = '\u274C'
if use_auto_metrics:
    to_show['Auto Metrics Done?'] = '\u2705'
    auto_metrics_flagged_ants = [ant for ant in ants if ant in auto_ex_ants]
    all_flagged_ants.extend(auto_metrics_flagged_ants)    
    to_show['Auto Metrics Flagged Antennas'] = f'{len(auto_metrics_flagged_ants)} / {len(ants)} ({len(auto_metrics_flagged_ants) / len(ants):.1%})' 
else:
    to_show['Auto Metrics Done?'] = '\u274C'
if use_redcal:
    to_show['Redcal Done?'] = '\u2705'    
    redcal_flagged_ants = [ant for ant in ants if redcal_flagged_frac[ant] > 0]
    all_flagged_ants.extend(redcal_flagged_ants)    
    to_show['Redcal Flagged Antennas'] = f'{len(redcal_flagged_ants)} / {len(ants)} ({len(redcal_flagged_ants) / len(ants):.1%})' 
else:
    to_show['Redcal Done?'] = '\u274C' 
to_show['Never Flagged Antennas'] = f'{len(ants) - len(set(all_flagged_ants))} / {len(ants)} ({(len(ants) - len(set(all_flagged_ants))) / len(ants):.1%})'

# Count bad antennas with good statuses and vice versa
n_apriori_good = len([ant for ant in ants if a_priori_statuses[ant] in good_statuses.split(',')])
apriori_good_flagged = []
aprior_bad_unflagged = []
for ant in ants:
    if ant in set(all_flagged_ants) and a_priori_statuses[ant] in good_statuses.split(','):
        apriori_good_flagged.append(ant)
    elif ant not in set(all_flagged_ants) and a_priori_statuses[ant] not in good_statuses.split(','):
        aprior_bad_unflagged.append(ant)
to_show['A Priori Good Antennas Flagged'] = f'{len(apriori_good_flagged)} / {n_apriori_good} total a priori good antennas:<br>' + \
                                            comma_sep_paragraph(apriori_good_flagged)
to_show['A Priori Bad Antennas Not Flagged'] = f'{len(aprior_bad_unflagged)} / {len(ants) - n_apriori_good} total a priori bad antennas:<br>' + \
                                            comma_sep_paragraph(aprior_bad_unflagged)

# Apply Styling
df = pd.DataFrame(to_show)
divider_cols = [df.columns.get_loc(col) for col in ['NIGHTLY ANALYSIS SUMMARY', 'OPERATIONAL STATUS SUMMARY']]
try:
    to_red_columns = [df.columns.get_loc(col) for col in ['Cross-Polarized Antennas', 'Nodes Registering 0s', 
                                                          'Nodes Not Correlating', 'A Priori Good Antennas Flagged']]
except:
    to_red_columns = []
def red_specific_cells(x):
    df1 = pd.DataFrame('', index=x.index, columns=x.columns)
    for col in to_red_columns:
        df1.iloc[col] = 'color: red'
    return df1

df = df.T
table = df.style.hide_columns().apply(red_specific_cells, axis=None)
for col in divider_cols:
    table = table.set_table_styles([{"selector":f"tr:nth-child({col+1})", "props": [("background-color", "black"), ("color", "white")]}], overwrite=False)

Table 1: Overall Array Health¶

In [18]:
HTML(table.render())
Out[18]:
JD 2459885
Date 11-1-2022
LST Range 22.223 -- 8.179 hours
X-Engine Status ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
Number of Files 1850
Total Number of Antennas 201
OPERATIONAL STATUS SUMMARY
Antenna A Priori Status Count dish_maintenance: 7
dish_ok: 1
RF_maintenance: 54
RF_ok: 19
digital_ok: 96
not_connected: 24
Commanded Signal Source None
Antennas in Commanded State (reported) 0 / 201 (0.0%)
Antennas in Commanded State (observed) 0 / 201 (0.0%)
Cross-Polarized Antennas
Total Number of Nodes 18
Nodes Registering 0s N20
Nodes Not Correlating N07
NIGHTLY ANALYSIS SUMMARY
Ant Metrics Done? ✅
Ant Metrics Flagged Antennas 74 / 201 (36.8%)
Auto Metrics Done? ✅
Auto Metrics Flagged Antennas 164 / 201 (81.6%)
Redcal Done? ❌
Never Flagged Antennas 32 / 201 (15.9%)
A Priori Good Antennas Flagged 70 / 96 total a priori good antennas:
3, 10, 17, 19, 20, 21, 29, 30, 44, 45, 51,
53, 54, 55, 56, 59, 66, 67, 68, 71, 72, 81,
83, 84, 86, 93, 94, 98, 99, 100, 101, 103,
106, 107, 108, 109, 111, 116, 117, 118, 121,
122, 123, 128, 136, 140, 141, 142, 143, 144,
146, 147, 158, 160, 161, 162, 164, 165, 167,
169, 170, 183, 184, 185, 186, 187, 189, 190,
191, 202
A Priori Bad Antennas Not Flagged 6 / 105 total a priori bad antennas:
4, 8, 89, 90, 125, 168
In [19]:
# write to csv
outpath = os.path.join(nb_outdir, f'array_health_table_{JD}.csv')
print(f'Now saving Table 2 to a csv at {outpath}')
df.replace({'\u2705': 'Y'}, regex=True).replace({'\u274C': 'N'}, regex=True).replace({'<br>': ' '}, regex=True).to_csv(outpath)
Now saving Table 2 to a csv at /home/obs/src/H6C_Notebooks/_rtp_summary_/array_health_table_2459885.csv

Build DataFrame¶

In [20]:
# build dataframe
to_show = {'Ant': [f'<a href="{ant_to_report_url(ant)}" target="_blank">{ant}</a>' for ant in ants],
           'Node': [f'N{nodes[ant]:02}' for ant in ants], 
           'A Priori Status': [a_priori_statuses[ant] for ant in ants]}
           #'Worst Metric': worst_metrics, 'Worst Modified Z-Score': worst_zs}
df = pd.DataFrame(to_show)

# create bar chart columns for flagging percentages:
bar_cols = {}
if use_auto_metrics:
    bar_cols['Auto Metrics Flags'] = [float(ant in auto_ex_ants) for ant in ants]
if use_ant_metrics:
    if np.sum(list(a_priori_flag_frac.values())) > 0:  # only include this col if there are any a priori flags
        bar_cols['A Priori Flag Fraction in Ant Metrics'] = [a_priori_flag_frac[ant] for ant in ants]
    for ap in antpols:
        bar_cols[f'Dead Fraction in Ant Metrics ({ap})'] = [dead_ant_frac[ap][ant] for ant in ants]
    bar_cols['Crossed Fraction in Ant Metrics'] = [crossed_ant_frac[ant] for ant in ants]
if use_redcal:
    bar_cols['Flag Fraction Before Redcal'] = [redcal_prior_flag_frac[ant] for ant in ants]
    bar_cols['Flagged By Redcal chi^2 Fraction'] = [redcal_flagged_frac[ant] for ant in ants]  
for col in bar_cols:
    df[col] = bar_cols[col]

# add auto_metrics
if use_auto_metrics:
    for label, modz in all_modzs.items():
        df[label] = modz
z_score_cols = [col for col in df.columns if 'Modified Z-Score' in col]        
        
# add ant_metrics
ant_metrics_cols = {}
if use_ant_metrics:
    for ap in antpols:
        ant_metrics_cols[f'Average Dead Ant Metric ({ap})'] = [average_dead_metrics[ap][ant] for ant in ants]
    ant_metrics_cols['Average Crossed Ant Metric'] = [average_crossed_metrics[ant] for ant in ants]
    for col in ant_metrics_cols:
        df[col] = ant_metrics_cols[col]   

# add redcal chisq
redcal_cols = []
if use_redcal:
    for ap in antpols:
        col_title = f'Median chi^2 Per Antenna ({ap})'
        df[col_title] = [cspa[ant, ap] for ant in ants]
        redcal_cols.append(col_title)

# sort by node number and then by antenna number within nodes
df.sort_values(['Node', 'Ant'], ascending=True)

# style dataframe
table = df.style.hide_index()\
          .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
          .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=z_score_cols) \
          .background_gradient(cmap='bwr_r', vmin=dead_cut-.25, vmax=dead_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .background_gradient(cmap='bwr_r', vmin=crossed_cut-.25, vmax=crossed_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .background_gradient(cmap='plasma', vmax=4, vmin=1, axis=None, subset=redcal_cols) \
          .applymap(lambda val: 'font-weight: bold' if val < dead_cut else '', subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val < crossed_cut else '', subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val > mean_round_modz_cut else '', subset=z_score_cols) \
          .applymap(lambda val: 'color: red' if val > mean_round_modz_cut else '', subset=z_score_cols) \
          .bar(subset=list(bar_cols.keys()), vmin=0, vmax=1) \
          .format({col: '{:,.4f}'.format for col in z_score_cols}) \
          .format({col: '{:,.4f}'.format for col in ant_metrics_cols}) \
          .format({col: '{:,.2%}'.format for col in bar_cols}) \
          .applymap(lambda val: 'font-weight: bold', subset=['Ant']) \
          .set_table_styles([dict(selector="th",props=[('max-width', f'70pt')])])

Table 2: RTP Per-Antenna Metrics Summary Table¶

This admittedly very busy table incorporates summary information about all antennas in the array. Its columns depend on what information is available when the notebook is run (i.e. whether auto_metrics, ant_metrics, and/or redcal is done). These can be divided into 5 sections:

Basic Antenna Info: antenna number, node, and its a priori status.

Flag Fractions: Fraction of the night that an antenna was flagged for various reasons. Note that auto_metrics flags antennas for the whole night, so it'll be 0% or 100%.

auto_metrics Details: If auto_metrics is included, this section shows the modified Z-score signifying how much of an outlier each antenna and polarization is in each of four categories: bandpass shape, overall power, temporal variability, and temporal discontinuities. Bold red text indicates that this is a reason for flagging the antenna. It is reproduced from the auto_metrics_inspect.ipynb nightly notebook, so check that out for more details on the precise metrics.

ant_metrics Details: If ant_metrics is included, this section shows the average correlation-based metrics for antennas over the whole night. Low "dead ant" metrics (nominally below 0.4) indicate antennas not correlating with the rest of the array. Negative "crossed ant" metrics indicate antennas that show stronger correlations in their cross-pols than their same-pols, indicating that the two polarizations are probably swapped. Bold text indicates that the average is below the threshold for flagging.

redcal chi^2 Details: If redcal is included, this shows the median chi^2 per antenna. This would be 1 in an ideal array. Antennas are thrown out when they they are outliers in their median chi^2, usually greater than 4-sigma outliers in modified Z-score.

In [21]:
HTML(table.render())
Out[21]:
Ant Node A Priori Status Auto Metrics Flags Dead Fraction in Ant Metrics (Jee) Dead Fraction in Ant Metrics (Jnn) Crossed Fraction in Ant Metrics ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score Average Dead Ant Metric (Jee) Average Dead Ant Metric (Jnn) Average Crossed Ant Metric
3 N01 digital_ok 100.00% 100.00% 0.00% 0.00% 18.770157 -0.978716 61.749978 -0.804382 14.083701 -1.122208 16.404485 0.424697 0.033452 0.716936 0.547385
4 N01 RF_maintenance 0.00% 0.00% 0.00% 0.00% 0.664685 3.670905 2.739994 -0.245964 0.964051 0.594955 3.471459 0.519720 0.731300 0.714431 0.359463
5 N01 digital_ok 0.00% 0.00% 0.00% 0.00% 0.497354 0.355478 -0.491095 2.418005 -0.525065 0.741378 0.843611 -0.665924 0.732991 0.716080 0.355844
7 N02 digital_ok 0.00% 0.00% 0.00% 0.00% -2.002913 -2.002301 0.244434 0.749028 -0.038214 0.559560 3.841649 3.402798 0.724278 0.712755 0.359129
8 N02 RF_maintenance 0.00% 0.00% 0.00% 0.00% 1.345781 -1.463466 1.551542 0.651830 -0.081069 -0.546536 1.720940 -0.190979 0.714513 0.702118 0.355048
9 N02 digital_ok 0.00% 0.00% 0.00% 0.00% 0.173757 -1.674492 0.095379 -0.052846 0.956047 -0.838281 0.384960 1.258313 0.714498 0.702760 0.364247
10 N02 digital_ok 100.00% 0.00% 0.00% 0.00% 5.425127 -1.122325 5.287752 7.826146 7.298706 4.650409 3.051925 1.063761 0.700610 0.696663 0.371281
15 N01 digital_ok 0.00% 0.00% 0.00% 0.00% 0.848977 0.448844 1.426532 -0.474846 -0.223192 -0.562843 2.207554 2.353269 0.733157 0.723346 0.355039
16 N01 digital_ok 0.00% 0.00% 0.00% 0.00% -1.028538 -0.501711 0.251240 -0.929729 0.041771 1.290041 2.351551 0.818367 0.734274 0.722226 0.351434
17 N01 digital_ok 100.00% 0.00% 0.00% 0.00% -0.104880 1.433266 -0.499461 0.623750 0.988581 0.667020 7.737781 2.805371 0.734236 0.723176 0.352774
18 N01 RF_maintenance 100.00% 0.00% 0.00% 0.00% 7.438934 21.003927 1.751114 1.606845 3.123070 9.613174 7.601187 18.021670 0.715102 0.515354 0.455435
19 N02 digital_ok 100.00% 0.00% 0.00% 0.00% 0.922832 -1.182002 0.846771 7.719810 22.450151 75.811567 2.116318 7.174845 0.726117 0.713702 0.355042
20 N02 digital_ok 100.00% 0.00% 0.00% 0.00% -1.093327 4.313802 0.148689 19.087543 -0.605422 3.028438 1.407777 2.911666 0.726305 0.706624 0.361330
21 N02 digital_ok 100.00% 0.00% 0.00% 0.00% 1.263977 0.761517 -0.252551 9.254093 0.480600 20.896401 0.857337 12.581623 0.712276 0.692045 0.360849
22 N06 not_connected 100.00% 0.00% 0.00% 0.00% 49.556534 14.157497 6.101564 19.163887 10.463789 6.362802 16.166270 9.915060 0.520841 0.644690 0.286901
27 N01 RF_maintenance 100.00% 100.00% 100.00% 0.00% 18.792130 20.038047 61.943453 62.568348 14.079860 19.304722 17.083091 11.784836 0.032809 0.037119 0.004483
28 N01 RF_maintenance 100.00% 0.00% 65.95% 0.00% 22.744129 43.796847 7.884937 5.072214 12.785999 22.096912 13.126635 17.075895 0.424193 0.197707 0.298985
29 N01 digital_ok 100.00% 0.00% 100.00% 0.00% -1.851718 20.753180 -0.453094 59.985964 -1.063184 19.260153 -0.890441 11.140178 0.737905 0.036137 0.581923
30 N01 digital_ok 100.00% 0.00% 0.00% 0.00% 0.706104 -1.059979 3.595919 -1.004995 12.680942 0.166376 6.479386 -0.237077 0.734234 0.726532 0.342191
31 N02 digital_ok 0.00% 0.00% 0.00% 0.00% 0.195999 1.868778 0.918401 -0.514856 0.197995 2.195647 1.241446 2.824968 0.744917 0.723803 0.352400
32 N02 RF_maintenance 100.00% 0.00% 0.00% 0.00% 13.685808 31.176155 3.134363 4.554277 13.937615 18.272781 35.672838 52.312110 0.688008 0.666581 0.305550
34 N06 not_connected 100.00% 100.00% 0.00% 0.00% 21.468270 4.321362 26.151523 22.785110 14.114221 4.496777 16.606797 4.591260 0.042328 0.690120 0.474455
35 N06 not_connected 100.00% 0.00% 0.00% 0.00% 1.336254 1.675601 26.891583 13.288938 26.531678 4.509871 11.093233 2.476900 0.684638 0.674451 0.367730
36 N03 RF_maintenance 100.00% 0.00% 0.00% 0.00% 13.845085 13.120256 0.864798 0.694327 2.168556 2.444370 2.256698 2.300297 0.721496 0.719606 0.351947
37 N03 digital_ok 0.00% 0.00% 0.00% 0.00% -0.250395 1.020789 -0.556454 0.164634 2.846890 0.016643 -0.562091 2.959754 0.731269 0.727797 0.355740
38 N03 digital_ok 0.00% 0.00% 0.00% 0.00% 0.043347 -0.000243 -0.858327 -0.657091 1.142173 0.253750 2.073167 -0.008885 0.736933 0.730272 0.354343
40 N04 digital_ok 0.00% 0.00% 0.00% 0.00% 0.217780 0.297910 -0.853060 -0.870753 0.685122 -0.468259 -0.572598 -0.498144 0.737000 0.727897 0.345016
41 N04 digital_ok 0.00% 0.00% 0.00% 0.00% 0.000243 0.684331 2.543742 1.628439 2.027999 -0.098109 0.737901 1.397799 0.741069 0.727494 0.336644
42 N04 digital_ok 0.00% 0.00% 0.00% 0.00% -0.440847 1.487059 -0.861897 1.438856 0.467639 -0.228199 -0.187157 -0.044059 0.748909 0.736856 0.348203
43 N05 RF_maintenance 100.00% 100.00% 0.00% 0.00% 17.660791 8.548833 61.261362 0.158543 14.232508 11.883479 17.473164 7.693207 0.037378 0.509368 0.300645
44 N05 digital_ok 100.00% 0.00% 0.00% 0.00% 3.944953 3.787264 1.771239 4.313904 3.573086 0.673699 27.693801 8.426073 0.726494 0.729262 0.336225
45 N05 digital_ok 100.00% 0.00% 0.00% 0.00% -0.920336 3.966953 -0.926376 -0.371514 1.404289 7.799080 0.343829 3.971319 0.737031 0.714538 0.343140
46 N05 RF_maintenance 100.00% 0.00% 100.00% 0.00% -0.742793 20.807736 -0.892565 62.812451 -0.143310 19.405274 1.934558 12.994773 0.728412 0.037612 0.489741
47 N06 not_connected 100.00% 100.00% 0.00% 0.00% 20.313189 4.064338 24.972435 14.226598 14.199475 3.630740 17.219371 5.168880 0.038251 0.691861 0.461942
48 N06 not_connected 100.00% 0.00% 0.00% 0.00% 3.694566 3.824045 27.026694 32.449842 3.505411 5.400949 9.537071 6.138842 0.693514 0.696809 0.369154
49 N06 not_connected 100.00% 0.00% 0.00% 0.00% 2.443800 3.299991 13.308801 31.069395 2.337540 5.977658 5.078656 6.052052 0.668566 0.683371 0.366451
50 N03 RF_maintenance 100.00% 0.00% 0.00% 0.00% 3.605268 4.918360 -0.474037 1.069040 4.173141 5.434265 22.352404 34.921792 0.707028 0.704385 0.344638
51 N03 digital_ok 100.00% 100.00% 0.00% 0.00% 40.424832 1.867507 79.665329 0.567933 14.296169 3.536145 20.681077 3.719170 0.041214 0.727506 0.531727
52 N03 RF_maintenance 100.00% 0.00% 0.00% 0.00% 12.102811 11.089060 0.998324 0.455090 1.303781 -0.052751 1.313648 0.267812 0.736507 0.733137 0.349624
53 N03 digital_ok 100.00% 0.00% 0.00% 0.00% 0.626202 4.396856 0.070571 1.169138 -0.405255 0.686889 1.115997 1.615779 0.743305 0.739137 0.346885
54 N04 digital_ok 100.00% 100.00% 100.00% 0.00% 18.800214 21.179852 61.978217 64.078653 14.281831 19.452024 17.951215 12.480633 0.044610 0.043945 0.001237
55 N04 digital_ok 100.00% 0.00% 100.00% 0.00% 4.349710 22.354674 1.705050 63.476435 10.864935 19.378450 8.280604 12.823962 0.740075 0.034662 0.504520
56 N04 digital_ok 100.00% 0.00% 100.00% 0.00% 0.636457 22.436369 0.511286 64.796319 0.265402 19.466344 1.152304 12.457789 0.747308 0.037483 0.523173
57 N04 RF_maintenance 100.00% 0.00% 0.00% 0.00% 52.878145 0.664016 24.631927 -0.682682 8.004571 1.617894 9.712701 1.314894 0.598322 0.737463 0.325781
58 N05 RF_maintenance 100.00% 100.00% 100.00% 0.00% 18.189794 20.952208 61.658237 63.769685 14.188846 19.418381 17.531797 12.346430 0.036556 0.034074 0.001483
59 N05 digital_ok 100.00% 0.00% 0.00% 0.00% 18.673605 5.687216 2.192555 3.243127 11.773032 2.101266 68.252611 13.089247 0.703134 0.724411 0.335005
60 N05 RF_maintenance 100.00% 100.00% 100.00% 0.00% 19.595955 20.520242 61.839552 63.624048 14.231491 19.434672 17.299686 12.655820 0.027337 0.027043 0.001409
61 N06 not_connected 100.00% 0.00% 0.00% 0.00% 6.433668 5.160788 5.569311 0.292036 2.891677 4.657733 3.365861 2.975466 0.689316 0.677915 0.344982
62 N06 not_connected 100.00% 0.00% 0.00% 0.00% 2.234866 3.869594 21.807644 30.474587 2.340094 7.179813 8.005411 5.997970 0.704477 0.705243 0.355018
63 N06 not_connected 100.00% 0.00% 100.00% 0.00% 13.735038 21.164070 21.602900 26.339747 3.227715 19.246477 7.981202 12.117684 0.652513 0.041879 0.427081
64 N06 not_connected 100.00% 0.00% 0.00% 0.00% 3.877360 2.337852 13.691799 25.217280 1.960207 4.562011 5.859834 4.531967 0.657607 0.672430 0.364063
65 N03 digital_ok 0.00% 0.00% 0.00% 0.00% 0.840348 1.352216 0.767916 1.594313 2.232361 0.714533 1.126493 0.840070 0.707684 0.715047 0.367811
66 N03 digital_ok 100.00% 0.00% 0.00% 0.00% -0.424411 2.342668 12.227390 7.989790 2.663205 0.142591 5.362433 1.958095 0.716352 0.722129 0.357743
67 N03 digital_ok 100.00% 0.00% 0.00% 0.00% -1.116630 -0.463656 11.400550 7.811476 0.760017 0.455504 5.152895 2.310935 0.725162 0.727097 0.347975
68 N03 digital_ok 100.00% 0.00% 100.00% 0.00% 3.608020 44.520904 1.429515 85.014722 0.085308 18.771071 1.716851 15.838617 0.733419 0.032236 0.530540
69 N04 digital_ok 0.00% 0.00% 0.00% 0.00% 0.134748 -1.182406 -0.188674 -0.390515 1.307660 2.127346 -0.183326 -0.620179 0.740947 0.739231 0.334493
70 N04 digital_ok 0.00% 0.00% 0.00% 0.00% 0.744118 -0.402673 1.523011 2.256663 1.121911 1.337993 1.184511 1.374235 0.747658 0.744038 0.330030
71 N04 digital_ok 100.00% 0.00% 0.00% 0.00% 10.902654 -0.322635 2.512872 2.244975 0.652709 0.431013 2.661665 0.622992 0.757241 0.744422 0.328183
72 N04 digital_ok 100.00% 0.00% 0.00% 0.00% 5.561226 -0.378408 2.121556 1.624443 1.798739 0.632835 4.559780 1.736087 0.742529 0.738307 0.322386
73 N05 RF_maintenance 100.00% 100.00% 100.00% 0.00% 17.396348 19.263610 60.858208 61.767862 14.327354 19.488757 18.495783 12.595709 0.027159 0.027244 0.001252
74 N05 RF_maintenance 100.00% 100.00% 85.95% 0.00% 19.452858 22.052552 63.707575 62.403756 14.296504 19.321803 17.981319 20.208763 0.030508 0.132051 0.057433
77 N06 not_connected 100.00% 0.00% 0.00% 0.00% 37.571263 40.691125 22.586458 17.969545 8.377965 12.993884 17.856437 8.430494 0.600424 0.563856 0.181758
78 N06 not_connected 100.00% 0.00% 0.00% 0.00% 52.727778 1.126734 16.939028 22.226406 8.492147 2.968253 10.698809 4.632404 0.532684 0.691607 0.335609
79 N11 not_connected 100.00% 0.00% 0.00% 0.00% 3.758250 1.475686 15.690326 16.726479 2.380642 3.322279 5.579992 2.698220 0.685907 0.694432 0.357394
80 N11 not_connected 100.00% 0.00% 100.00% 0.00% 16.704168 22.999301 36.994941 25.560499 13.019242 19.302957 18.041112 11.503767 0.349152 0.039438 0.217173
81 N07 digital_ok 100.00% 100.00% 100.00% 0.00% -0.155489 2.657015 0.134048 31.846696 -0.295988 22.628502 -0.539649 3.668588 0.061873 0.088740 0.016087
82 N07 RF_maintenance 100.00% 100.00% 100.00% 0.00% 1.372586 0.089127 -0.533425 7.940459 0.568408 1.558050 -0.757285 0.293432 0.071824 0.072838 0.009152
83 N07 digital_ok 0.00% 100.00% 100.00% 0.00% -0.828059 -0.641906 -0.472865 0.574988 -1.437036 -0.850137 -1.373926 -0.683057 0.080041 0.073329 0.009925
84 N08 digital_ok 100.00% 0.00% 100.00% 0.00% 3.545335 39.096026 28.502333 82.150057 3.959806 18.653871 16.212018 13.016000 0.711313 0.036488 0.423399
85 N08 digital_ok 0.00% 0.00% 0.00% 0.00% 0.361727 0.134370 -0.465412 -0.691891 -1.445026 0.119999 -1.586030 -1.363869 0.735650 0.731547 0.333392
86 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 2.749865 9.391840 5.149534 0.854108 7.249881 2.500499 0.399340 8.099601 0.731890 0.706155 0.319919
87 N08 RF_maintenance 100.00% 0.00% 0.00% 0.00% 13.841036 12.410116 4.380962 1.554403 30.811552 0.014968 16.032250 0.929180 0.695389 0.750517 0.303658
88 N09 digital_ok 0.00% 0.00% 0.00% 0.00% 1.113246 0.564339 0.137457 0.697043 -0.924114 1.299607 3.781113 0.008885 0.745224 0.739852 0.318173
89 N09 RF_maintenance 0.00% 0.00% 0.00% 0.00% 1.378942 -0.162414 -0.718986 0.222897 0.121159 -0.751825 -1.201009 -0.646635 0.749361 0.739196 0.325047
90 N09 RF_maintenance 0.00% 0.00% 0.00% 0.00% 0.089346 -0.612371 -0.155529 1.754396 -1.251935 -0.374181 -0.567714 1.291673 0.745893 0.736194 0.325032
91 N09 digital_ok 0.00% 0.00% 0.00% 0.00% -0.162276 -0.643461 -0.413353 0.207925 -0.529331 -1.077089 0.518903 0.114649 0.740986 0.741325 0.341338
92 N10 RF_maintenance 100.00% 0.00% 0.76% 0.00% 62.534812 71.412270 7.527394 9.074306 12.948448 19.312048 12.093894 9.688545 0.351998 0.299532 0.096168
93 N10 digital_ok 100.00% 0.00% 0.00% 0.00% 3.755046 0.249066 10.768340 -0.090540 2.344973 -0.328479 5.310050 -0.889672 0.727073 0.732235 0.346355
94 N10 digital_ok 100.00% 100.00% 0.00% 0.00% 20.457016 -1.734205 62.772328 1.339048 14.128716 3.624282 16.406383 2.010609 0.032455 0.723529 0.423419
95 N11 not_connected 100.00% 0.00% 0.00% 0.00% 2.695861 1.557713 16.171541 29.447813 3.278362 4.495643 5.386560 6.053138 0.694648 0.709046 0.363835
96 N11 not_connected 100.00% 100.00% 100.00% 0.00% 20.631144 22.392025 24.945492 26.703802 14.248975 19.496402 17.206890 12.013107 0.033422 0.037606 0.002335
97 N11 not_connected 100.00% 0.00% 0.00% 0.00% 0.610672 7.327288 23.434585 0.469969 2.402909 5.594481 7.088334 4.694931 0.679052 0.654359 0.360958
98 N07 digital_ok 100.00% 100.00% 100.00% 0.00% -0.497150 22.444952 0.043017 2.033399 1.148130 11.729432 1.635139 5.349287 0.066711 0.068245 0.008341
99 N07 digital_ok 0.00% 100.00% 100.00% 0.00% 2.476072 -1.340180 0.448633 -0.585698 -1.388500 3.511384 0.037814 -0.861540 0.068332 0.062953 0.006382
100 N07 digital_ok 0.00% 100.00% 100.00% 0.00% -0.492475 -1.454387 -0.576990 2.252435 0.663860 -0.980338 -0.528093 -0.409020 0.064081 0.063432 0.005909
101 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 11.901683 14.750334 2.711154 1.857310 -0.412824 -0.328347 0.228712 -0.492375 0.727897 0.728281 0.348651
102 N08 RF_maintenance 100.00% 0.00% 0.00% 0.00% 6.046228 1.423708 43.417245 15.654532 8.222617 0.551550 12.897296 3.495216 0.717215 0.733881 0.348767
103 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 6.170726 9.052989 43.049831 40.746063 10.229352 9.072268 13.636445 9.080359 0.720747 0.724814 0.333790
104 N08 RF_maintenance 100.00% 0.00% 0.00% 0.00% 4.160716 94.766169 40.844189 41.387004 3.427787 5.627578 13.041311 5.289648 0.702564 0.722740 0.325770
105 N09 digital_ok 0.00% 0.00% 0.00% 0.00% -0.510537 -0.684296 -0.942268 0.719720 -0.908634 -0.611943 -1.345090 -1.231974 0.749516 0.740760 0.317284
106 N09 digital_ok 100.00% 0.00% 0.00% 0.00% 0.184794 1.099373 4.823438 2.137779 4.257820 0.450025 0.457865 -0.673218 0.741679 0.738311 0.319844
107 N09 digital_ok 100.00% 0.00% 0.00% 0.00% 3.903109 8.167835 0.474071 1.374225 0.353827 2.236189 3.581804 4.988614 0.745133 0.737426 0.311623
108 N09 digital_ok 100.00% 41.14% 0.00% 0.00% 17.963726 3.494915 60.720488 0.855719 13.419128 2.455387 14.662846 1.013725 0.267268 0.740573 0.459986
109 N10 digital_ok 100.00% 0.00% 100.00% 0.00% -1.092770 20.699620 1.496510 61.789549 -0.452023 19.335578 0.948118 11.964826 0.743334 0.035363 0.443817
110 N10 RF_maintenance 100.00% 0.00% 100.00% 0.00% 0.390618 41.930808 0.003836 83.179769 -1.162350 18.763236 0.478446 12.580609 0.745781 0.032178 0.440141
111 N10 digital_ok 100.00% 0.00% 100.00% 0.00% 0.039682 20.410904 1.275456 62.455007 -1.269698 19.319292 1.057474 12.190713 0.732073 0.035405 0.441415
112 N10 digital_ok 0.00% 0.00% 0.00% 0.00% 0.066831 -0.469694 -0.880847 1.633064 -0.580164 1.280340 -0.540050 -0.787634 0.719822 0.727224 0.355498
113 N11 not_connected 100.00% 100.00% 100.00% 0.00% 21.950170 22.441739 23.551904 26.014454 14.145887 19.401089 16.917254 11.542472 0.035546 0.030803 0.002622
114 N11 not_connected 100.00% 0.00% 0.00% 0.00% 9.976955 2.621715 3.962356 20.777696 5.834468 3.655464 1.287150 3.421329 0.613676 0.690084 0.376085
115 N11 not_connected 100.00% 0.00% 0.00% 0.00% 4.114652 12.590414 37.053931 34.741758 5.727421 9.997169 11.480200 6.729366 0.673298 0.639983 0.363401
116 N07 digital_ok 100.00% 100.00% 100.00% 0.00% -1.108119 0.964878 5.296135 2.465688 0.458130 1.050993 0.651442 0.291809 0.083756 0.073053 0.011731
117 N07 digital_ok 100.00% 100.00% 100.00% 0.00% 20.337002 23.082248 62.229186 65.674176 14.090600 19.336940 16.519604 12.494786 0.026616 0.025408 0.000760
118 N07 digital_ok 0.00% 100.00% 100.00% 0.00% -1.046024 1.340033 0.000074 0.560655 2.874087 -0.278085 -0.438511 -0.426413 0.068641 0.061605 0.005228
119 N07 RF_maintenance 100.00% 100.00% 100.00% 0.00% 0.854011 3.826324 4.328483 23.202071 -1.318701 10.983388 0.649220 3.670985 0.072872 0.079014 0.011813
120 N08 RF_maintenance 100.00% 0.00% 0.00% 0.00% 4.456434 4.524928 14.964064 38.337779 0.215258 8.375474 6.519639 6.985081 0.718208 0.718272 0.341705
121 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 4.050438 8.053610 0.339071 1.003161 2.287329 -0.524079 11.052382 7.445016 0.738219 0.737666 0.333723
122 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 12.475062 11.699913 8.444890 1.081185 47.156605 -1.074223 3.302880 -1.191637 0.746940 0.744289 0.330174
123 N08 digital_ok 100.00% 0.00% 0.00% 0.00% 10.079817 14.635327 1.497606 1.901606 -1.700592 -1.177786 0.157658 0.372902 0.754424 0.749979 0.324470
124 N09 digital_ok 0.00% 0.00% 0.00% 0.00% -1.089696 1.591677 -0.099493 -0.620603 0.405871 -1.108909 -0.169249 0.167581 0.754167 0.745442 0.323351
125 N09 RF_maintenance 0.00% 0.00% 0.00% 0.00% 1.799644 2.631405 -0.000074 0.212549 0.185317 2.055603 -0.783350 -0.519748 0.744756 0.739886 0.320640
126 N09 RF_maintenance 100.00% 0.00% 0.00% 0.00% 24.380957 -0.853237 3.751230 0.493235 26.238927 0.055140 17.672884 0.479736 0.682689 0.739439 0.316715
127 N10 digital_ok 0.00% 0.00% 0.00% 0.00% -0.142248 -0.519395 -0.914862 -0.061084 1.176574 -0.016643 -1.053796 1.065398 0.747596 0.745782 0.338835
128 N10 digital_ok 100.00% 0.00% 0.00% 0.00% -0.886661 -0.139910 8.598554 2.345330 0.738961 1.746964 2.763717 0.540125 0.738795 0.740104 0.338061
129 N10 digital_ok 0.00% 0.00% 0.00% 0.00% 0.980583 -2.159107 2.442684 1.859245 -0.953227 -0.820722 0.012181 1.161477 0.734764 0.735854 0.346004
130 N10 digital_ok 0.00% 0.00% 0.00% 0.00% 2.502222 0.010065 -0.347242 -0.038360 -0.127491 -0.106437 -0.418610 0.418134 0.716980 0.727231 0.346632
131 N11 not_connected 100.00% 100.00% 100.00% 0.00% 20.368262 22.587132 25.174647 27.524445 14.132781 19.289267 17.564955 11.144407 0.034717 0.038678 0.001765
132 N11 not_connected 100.00% 0.00% 0.00% 0.00% 0.227784 1.545871 22.684759 8.859889 1.946690 2.482241 7.024781 0.905518 0.685932 0.685420 0.362883
133 N11 not_connected 100.00% 100.00% 62.81% 0.00% 21.185644 28.310981 23.581679 18.162028 14.176763 18.516754 17.002191 9.277280 0.041118 0.224685 0.123075
135 N12 RF_maintenance 100.00% 0.00% 100.00% 0.00% -1.960803 20.656532 -0.493905 63.963735 0.950131 19.542144 3.503593 12.926950 0.672566 0.038128 0.437583
136 N12 digital_ok 100.00% 0.00% 0.00% 0.00% 5.750963 0.997057 1.615547 2.597493 3.398182 5.050199 2.846321 1.400776 0.664318 0.681300 0.366979
137 N07 RF_maintenance 0.00% 100.00% 100.00% 0.00% 1.015740 -0.792751 0.379222 -0.978168 2.314304 -0.377248 0.325105 -0.473586 0.072014 0.070022 0.008174
138 N07 RF_maintenance 100.00% 100.00% 100.00% 0.00% 1.072396 -1.081565 3.904590 6.128044 -0.279589 -1.241331 3.450851 0.765681 0.075130 0.070225 0.009862
139 N13 RF_maintenance 100.00% 0.00% 0.00% 0.00% 1.452148 0.011983 30.563004 17.133711 4.054054 2.747433 9.691072 3.600817 0.705894 0.702731 0.353483
140 N13 digital_ok 100.00% 0.00% 100.00% 0.00% 1.218837 21.609651 22.499599 63.082570 1.249801 19.242807 7.775243 12.120359 0.724221 0.044763 0.506306
141 N13 digital_ok 100.00% 0.00% 0.00% 0.00% -2.004959 0.326510 2.705507 27.777703 1.416556 2.881562 0.446366 5.281561 0.734325 0.730936 0.336282
142 N13 digital_ok 100.00% 0.00% 100.00% 0.00% 1.997549 20.505788 0.707954 63.547107 2.795877 19.376439 1.214504 12.403661 0.735700 0.043530 0.511287
143 N14 digital_ok 100.00% 100.00% 0.00% 0.00% 19.132114 -1.780036 62.629788 -0.507755 14.172235 1.606766 16.328806 -0.619429 0.037761 0.744583 0.528187
144 N14 digital_ok 100.00% 0.00% 0.00% 0.00% -1.751429 -0.581688 0.333818 6.844844 -0.209220 3.842383 -0.354520 0.806784 0.748765 0.740629 0.328673
145 N14 RF_maintenance 100.00% 0.00% 0.00% 0.00% -2.125685 4.805676 -0.638863 41.357180 1.781412 25.333687 0.179535 4.686241 0.746869 0.670935 0.354108
146 N14 digital_ok 100.00% 0.00% 0.00% 0.00% 2.232008 3.150252 7.137951 30.100266 1.774593 5.432572 2.236833 5.182172 0.718914 0.736817 0.344145
147 N15 digital_ok 100.00% 0.00% 0.00% 0.00% -1.168310 -0.592792 5.411217 9.761718 1.427587 -0.824458 1.672820 0.811848 0.740446 0.735444 0.332079
148 N15 RF_maintenance 100.00% 0.00% 0.00% 0.00% -1.819544 -0.528696 18.683025 6.809639 2.242316 -0.677144 7.286846 -0.214463 0.726968 0.738044 0.345080
149 N15 RF_maintenance 100.00% 0.00% 0.00% 0.00% -1.478788 2.520451 13.089679 35.114287 -1.016275 7.070816 3.494075 6.436342 0.731914 0.728216 0.354663
150 N15 RF_maintenance 100.00% 100.00% 0.05% 0.00% 19.882689 0.052595 61.906918 20.841128 14.119374 7.065872 17.026458 5.777457 0.045152 0.385419 0.206838
155 N12 RF_maintenance 100.00% 0.00% 0.00% 0.00% -1.694639 -0.789317 0.108365 -0.964893 1.997905 5.182549 3.671310 2.463513 0.671437 0.680776 0.391103
156 N12 RF_maintenance 100.00% 89.84% 0.00% 0.00% 17.794057 -0.041653 61.607580 9.381434 13.955867 -0.470734 16.315746 2.999597 0.105567 0.679164 0.487502
157 N12 digital_ok 0.00% 0.00% 0.00% 0.00% -0.505952 -0.447134 -0.106215 -0.537035 -0.504353 0.566887 0.665244 0.218598 0.686865 0.696960 0.373879
158 N12 digital_ok 100.00% 0.00% 0.00% 0.00% 0.132380 -1.200056 1.777895 5.039082 1.523971 1.319672 1.520637 6.356436 0.704338 0.710523 0.374313
159 N13 RF_maintenance 100.00% 0.00% 0.00% 0.00% 2.275686 42.933182 10.777221 10.285617 1.307766 12.492708 4.483958 14.528052 0.682444 0.570250 0.343339
160 N13 digital_ok 100.00% 0.00% 0.00% 0.00% -1.049670 -1.726180 1.812707 5.905604 -0.885345 1.939518 0.043069 0.464476 0.721695 0.722224 0.346136
161 N13 digital_ok 100.00% 0.00% 0.00% 0.00% -0.687102 45.378981 0.205576 8.320910 -0.693808 7.234911 0.115692 5.163214 0.728318 0.612773 0.310797
162 N13 digital_ok 100.00% 0.00% 0.00% 0.00% 1.872693 0.452124 14.331150 10.797749 2.056933 -0.563777 5.092138 2.206865 0.740208 0.736838 0.337130
163 N14 digital_ok 0.00% 0.00% 0.00% 0.00% -0.637026 -0.529497 1.205290 -0.732415 -1.475136 0.624623 -0.509882 -0.530188 0.745500 0.737328 0.339040
164 N14 digital_ok 100.00% 0.00% 0.00% 0.00% -0.844863 -0.479108 4.947001 3.240294 5.942521 1.121282 1.610186 0.022042 0.743241 0.741899 0.333700
165 N14 digital_ok 100.00% 0.00% 0.00% 0.00% 29.890649 0.080599 45.691349 3.837277 9.567246 -0.876240 7.738801 0.040692 0.503403 0.740406 0.372387
166 N14 RF_maintenance 100.00% 0.00% 80.59% 0.00% 47.578154 19.180407 10.418960 60.607589 7.228620 19.123828 10.035172 11.464093 0.617757 0.131609 0.406038
167 N15 digital_ok 100.00% 0.00% 0.00% 0.00% -1.323772 -1.559253 14.630923 2.414750 2.204285 0.410161 3.620197 1.144944 0.750763 0.740466 0.339113
168 N15 RF_maintenance 0.00% 0.00% 0.00% 0.00% -0.787150 -1.431797 -0.971025 2.527891 -0.116569 -0.196996 -0.856426 -0.256594 0.742442 0.740626 0.341577
169 N15 digital_ok 100.00% 0.00% 0.00% 0.00% 0.651951 17.388746 5.965167 13.351646 7.809176 15.231087 4.574055 12.320961 0.730088 0.697531 0.348684
170 N15 digital_ok 100.00% 100.00% 0.00% 0.00% 19.949741 -1.029200 62.993066 13.038820 14.111529 31.755914 16.372497 1.139986 0.037902 0.732004 0.547950
179 N12 RF_maintenance 100.00% 99.89% 75.95% 0.00% 20.254400 21.994343 63.024278 66.386584 14.204998 19.460931 16.801512 11.494531 0.063592 0.149857 0.079062
180 N13 RF_maintenance 100.00% 100.00% 100.00% 0.00% 19.322096 21.779088 62.475535 64.386314 14.214735 19.454210 16.829175 12.554906 0.047240 0.049781 0.003679
181 N13 digital_ok 0.00% 0.00% 0.00% 0.00% -0.195275 -1.620765 0.847195 -0.966870 -0.086977 0.139250 -0.374200 1.297233 0.730780 0.723298 0.348997
182 N13 RF_maintenance 100.00% 0.00% 0.00% 0.00% 3.024743 6.901554 34.040132 44.477929 3.161382 11.800826 13.144565 8.995149 0.691793 0.711269 0.360479
183 N13 digital_ok 100.00% 100.00% 0.00% 0.00% 19.224474 -0.796795 57.539633 5.049302 14.161689 -0.018288 16.695605 0.250700 0.040814 0.727646 0.472913
184 N14 digital_ok 100.00% 83.89% 99.41% 0.00% 18.993611 21.150436 62.310392 63.583535 13.864013 19.164049 16.204618 11.680773 0.133610 0.062814 0.068491
185 N14 digital_ok 100.00% 100.00% 0.00% 0.00% 17.996744 -1.640922 62.081826 18.168631 14.096724 1.263880 16.522559 3.115962 0.036627 0.726010 0.486674
186 N14 digital_ok 100.00% 0.00% 0.00% 0.00% 1.305158 2.180845 18.363226 13.021466 8.669050 1.227785 7.626684 1.686084 0.729308 0.737009 0.341591
187 N14 digital_ok 100.00% 0.00% 0.00% 0.00% 4.799790 2.508711 1.434811 31.589074 18.849784 5.184908 1.886278 7.218506 0.735232 0.733538 0.343262
189 N15 digital_ok 100.00% 0.00% 0.00% 0.00% 1.891533 2.799733 6.839173 2.807666 0.446339 6.193626 2.353149 1.098567 0.723196 0.727167 0.345061
190 N15 digital_ok 100.00% 0.00% 100.00% 0.00% 67.517554 20.853105 7.958639 64.164238 10.160014 19.519776 37.462537 12.517655 0.571769 0.036373 0.406924
191 N15 digital_ok 100.00% 0.00% 0.00% 0.00% -0.222080 0.507974 22.156357 1.172721 1.354330 -0.939382 14.991021 -0.410501 0.699882 0.717670 0.369437
200 N18 RF_maintenance 100.00% 100.00% 25.73% 0.00% 21.474700 57.916167 24.786646 24.783386 14.083262 18.919603 16.833927 8.967722 0.047042 0.259837 0.169053
201 N18 RF_maintenance 100.00% 0.00% 0.00% 0.00% 11.296365 9.994208 53.045795 50.710908 12.203712 15.524985 14.834652 9.156740 0.672860 0.677023 0.362172
202 N18 digital_ok 100.00% 0.00% 0.00% 0.00% 2.948237 5.044939 25.382691 2.590964 3.060734 5.457879 8.612877 1.931348 0.713189 0.677710 0.353073
203 N18 RF_maintenance 100.00% 100.00% 100.00% 0.00% 22.303782 23.724979 23.272896 24.627927 14.053472 19.219501 17.079074 12.413105 0.033713 0.041528 0.001761
205 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 4.394940 8.137689 21.470471 7.216525 1.985968 7.598158 7.818033 3.240891 0.718626 0.651221 0.367310
206 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 2.551457 3.357928 23.094089 17.554922 17.749052 2.873890 9.674905 5.415368 0.717820 0.700797 0.340817
207 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 5.544696 4.848770 29.781756 23.732045 3.958590 5.952287 9.682489 4.050558 0.703909 0.695212 0.332652
208 N20 dish_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
209 N20 dish_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
210 N20 dish_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
211 N20 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
219 N18 RF_maintenance 100.00% 0.00% 0.00% 0.00% 10.858291 7.359713 53.525862 45.625668 13.380376 12.461667 14.373645 8.459822 0.643483 0.684252 0.385643
220 N18 RF_maintenance 100.00% 0.00% 0.00% 0.00% 0.340555 0.208690 20.761663 21.626366 1.750276 3.291439 7.628284 3.232855 0.706050 0.698102 0.362019
221 N18 RF_ok 100.00% 0.00% 0.00% 0.00% 5.149175 2.019462 2.790997 20.303946 2.759603 1.802071 2.158000 3.748535 0.682983 0.699418 0.361367
222 N18 RF_ok 100.00% 0.00% 0.00% 0.00% 3.449572 3.112126 24.721588 22.771035 3.240514 2.215111 9.783583 4.330699 0.713033 0.705661 0.358235
223 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 5.179063 3.364107 11.619251 21.525591 2.130391 47.033431 4.855287 7.057779 0.695485 0.702872 0.351677
224 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 11.420473 10.795532 54.391740 54.002656 13.134234 17.657586 14.731405 9.372972 0.689862 0.679034 0.362574
225 N19 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
226 N19 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
227 N20 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
228 N20 RF_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
229 N20 RF_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
237 N18 RF_ok 100.00% 0.00% 0.00% 0.00% 5.666500 3.952750 1.501317 15.255060 2.373749 3.556701 2.022193 2.939063 0.657765 0.671570 0.376842
238 N18 RF_ok 100.00% 0.00% 0.00% 0.00% 1.606812 0.664057 28.477064 27.805579 3.573798 5.594060 8.352222 4.306204 0.704384 0.692372 0.372250
239 N18 RF_ok 100.00% 0.00% 0.00% 0.00% 1.000528 4.609800 20.295315 37.202102 4.673985 40.195259 6.371259 11.227417 0.705768 0.692256 0.369334
240 N19 RF_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
241 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 3.690995 7.125497 16.274561 -0.054925 2.009793 3.998623 8.073022 8.179268 0.704327 0.657761 0.367353
242 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 44.110200 3.854386 21.294669 31.635749 7.901052 5.234374 9.164464 5.935277 0.583798 0.700309 0.352329
243 N19 RF_ok 100.00% 0.00% 0.00% 0.00% 87.631171 4.607163 21.100987 14.076697 17.839307 3.632187 10.233127 2.668044 0.419661 0.685695 0.454559
244 N20 RF_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
245 N20 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
246 N20 RF_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
261 N20 RF_ok 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
262 N20 dish_maintenance 100.00% 100.00% 100.00% 0.00% nan nan inf inf nan nan nan nan nan nan nan
320 N03 dish_maintenance 100.00% 0.00% 100.00% 0.00% 0.977387 21.614878 1.285402 40.050312 0.550250 19.414929 5.040027 13.310560 0.706461 0.047623 0.535855
324 N04 not_connected 100.00% 0.00% 0.00% 0.00% 2.814047 5.000235 28.448938 33.403214 3.534660 6.539627 9.331141 5.623053 0.610156 0.601870 0.362093
325 N09 dish_ok 100.00% 0.00% 0.00% 0.00% 1.539841 -0.472318 29.127330 15.130634 3.460332 2.332641 7.172909 2.269441 0.643647 0.618440 0.371782
329 N12 dish_maintenance 100.00% 0.00% 0.00% 0.00% 4.613474 -0.152804 5.625187 17.623039 39.735993 2.693874 8.648558 2.617897 0.582340 0.610445 0.368401
333 N12 dish_maintenance 100.00% 0.00% 0.00% 0.00% 5.751149 3.046334 5.022635 13.999413 1.857985 2.881514 3.682146 2.640259 0.554094 0.574647 0.380495
In [22]:
# print ex_ants for easy copy-pasting to YAML file
proposed_ex_ants = [ant for i, ant in enumerate(ants) if np.any([col[i] > 0 for col in bar_cols.values()])]
print('ex_ants: [' + ", ".join(str(ant) for ant in proposed_ex_ants) + ']')
print(f'\nunflagged_ants: [{", ".join([str(ant) for ant in ants if ant not in proposed_ex_ants])}]')
# "golden" means no flags and good a priori status
golden_ants = ", ".join([str(ant) for ant in ants if ((ant not in proposed_ex_ants) and (a_priori_statuses[ant] in good_statuses.split(',')))])
print(f'\ngolden_ants: [{golden_ants}]')
ex_ants: [3, 10, 17, 18, 19, 20, 21, 22, 27, 28, 29, 30, 32, 34, 35, 36, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 66, 67, 68, 71, 72, 73, 74, 77, 78, 79, 80, 81, 82, 83, 84, 86, 87, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 106, 107, 108, 109, 110, 111, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 126, 128, 131, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 155, 156, 158, 159, 160, 161, 162, 164, 165, 166, 167, 169, 170, 179, 180, 182, 183, 184, 185, 186, 187, 189, 190, 191, 200, 201, 202, 203, 205, 206, 207, 208, 209, 210, 211, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 261, 262, 320, 324, 325, 329, 333]

unflagged_ants: [4, 5, 7, 8, 9, 15, 16, 31, 37, 38, 40, 41, 42, 65, 69, 70, 85, 88, 89, 90, 91, 105, 112, 124, 125, 127, 129, 130, 157, 163, 168, 181]

golden_ants: [5, 7, 9, 15, 16, 31, 37, 38, 40, 41, 42, 65, 69, 70, 85, 88, 91, 105, 112, 124, 127, 129, 130, 157, 163, 181]
In [23]:
# write to csv
outpath = os.path.join(nb_outdir, f'rtp_summary_table_{JD}.csv')
print(f'Now saving Table 2 to a csv at {outpath}')
df.to_csv(outpath)
Now saving Table 2 to a csv at /home/obs/src/H6C_Notebooks/_rtp_summary_/rtp_summary_table_2459885.csv
In [24]:
# Load antenna positions
data_list = sorted(glob.glob(os.path.join(data_path, f'zen.{JD}.?????.sum.uvh5')))
hd = io.HERAData(data_list[len(data_list) // 2])

# Figure out where to draw the nodes
node_centers = {}
for node in sorted(set(list(nodes.values()))):
    if np.isfinite(node):
        this_node_ants = [ant for ant in ants + unused_ants if nodes[ant] == node]
        if len(this_node_ants) == 1:
            # put the node label just to the west of the lone antenna 
            node_centers[node] = hd.antpos[ant][node] + np.array([-14.6 / 2, 0, 0])
        else:
            # put the node label between the two antennas closest to the node center
            node_centers[node] = np.mean([hd.antpos[ant] for ant in this_node_ants], axis=0)
            closest_two_pos = sorted([hd.antpos[ant] for ant in this_node_ants], 
                                     key=lambda pos: np.linalg.norm(pos - node_centers[node]))[0:2]
            node_centers[node] = np.mean(closest_two_pos, axis=0)
In [25]:
def Plot_Array(ants, unused_ants, outriggers):
    plt.figure(figsize=(16,16))
    
    plt.scatter(np.array([hd.antpos[ant][0] for ant in hd.data_ants if ant in ants]), 
                np.array([hd.antpos[ant][1] for ant in hd.data_ants if ant in ants]), c='w', s=0)

    # connect every antenna to their node
    for ant in ants:
        if nodes[ant] in node_centers:
            plt.plot([hd.antpos[ant][0], node_centers[nodes[ant]][0]], 
                     [hd.antpos[ant][1], node_centers[nodes[ant]][1]], 'k', zorder=0)

    rc_color = '#0000ff'
    antm_color = '#ffa500'
    autom_color = '#ff1493'

    # Plot 
    unflagged_ants = []
    for i, ant in enumerate(ants):
        ant_has_flag = False
        # plot large blue annuli for redcal flags
        if use_redcal:
            if redcal_flagged_frac[ant] > 0:
                ant_has_flag = True
                plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=7 * (2 - 1 * float(not outriggers)), fill=True, lw=0,
                                                color=rc_color, alpha=redcal_flagged_frac[ant]))
                plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=6 * (2 - 1 * float(not outriggers)), fill=True, color='w'))
        
        # plot medium green annuli for ant_metrics flags
        if use_ant_metrics: 
            if ant_metrics_xants_frac_by_ant[ant] > 0:
                ant_has_flag = True
                plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=6 * (2 - 1 * float(not outriggers)), fill=True, lw=0,
                                                color=antm_color, alpha=ant_metrics_xants_frac_by_ant[ant]))
                plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=5 * (2 - 1 * float(not outriggers)), fill=True, color='w'))
        
        # plot small red annuli for auto_metrics
        if use_auto_metrics:
            if ant in auto_ex_ants:
                ant_has_flag = True                
                plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=5 * (2 - 1 * float(not outriggers)), fill=True, lw=0, color=autom_color)) 
        
        # plot black/white circles with black outlines for antennas
        plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=4 * (2 - 1 * float(not outriggers)), fill=True, color=['w', 'k'][ant_has_flag], ec='k'))
        if not ant_has_flag:
            unflagged_ants.append(ant)

        # label antennas, using apriori statuses if available
        try:
            bgc = matplotlib.colors.to_rgb(status_colors[a_priori_statuses[ant]])
            c = 'black' if (bgc[0]*0.299 + bgc[1]*0.587 + bgc[2]*0.114) > 186 / 256 else 'white'
        except:
            c = 'k'
            bgc='white'
        plt.text(hd.antpos[ant][0], hd.antpos[ant][1], str(ant), va='center', ha='center', color=c, backgroundcolor=bgc)

    # label nodes
    for node in sorted(set(list(nodes.values()))):
        if not np.isnan(node) and not np.all(np.isnan(node_centers[node])):
            plt.text(node_centers[node][0], node_centers[node][1], str(node), va='center', ha='center', bbox={'color': 'w', 'ec': 'k'})
    
    # build legend 
    legend_objs = []
    legend_labels = []
    
    # use circles for annuli 
    legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markeredgecolor='k', markerfacecolor='w', markersize=13))
    legend_labels.append(f'{len(unflagged_ants)} / {len(ants)} Total {["Core", "Outrigger"][outriggers]} Antennas Never Flagged')
    legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markerfacecolor='k', markersize=15))
    legend_labels.append(f'{len(ants) - len(unflagged_ants)} Antennas {["Core", "Outrigger"][outriggers]} Flagged for Any Reason')

    if use_auto_metrics:
        legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markeredgewidth=2, markeredgecolor=autom_color, markersize=15))
        legend_labels.append(f'{len([ant for ant in auto_ex_ants if ant in ants])} {["Core", "Outrigger"][outriggers]} Antennas Flagged by Auto Metrics')
    if use_ant_metrics: 
        legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markeredgewidth=2, markeredgecolor=antm_color, markersize=15))
        legend_labels.append(f'{np.round(np.sum([frac for ant, frac in ant_metrics_xants_frac_by_ant.items() if ant in ants]), 2)} Antenna-Nights on' 
                             f'\n{np.sum([frac > 0 for ant, frac in ant_metrics_xants_frac_by_ant.items() if ant in ants])} {["Core", "Outrigger"][outriggers]} Antennas '
                             'Flagged by Ant Metrics\n(alpha indicates fraction of time)')        
    if use_redcal:
        legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markeredgewidth=2, markeredgecolor=rc_color, markersize=15))
        legend_labels.append(f'{np.round(np.sum(list(redcal_flagged_frac.values())), 2)} Antenna-Nights on' 
                             f'\n{np.sum([frac > 0 for ant, frac in redcal_flagged_frac.items() if ant in ants])} {["Core", "Outrigger"][outriggers]} Antennas '
                             'Flagged by Redcal\n(alpha indicates fraction of time)')

    # use rectangular patches for a priori statuses that appear in the array
    for aps in sorted(list(set(list(a_priori_statuses.values())))):
        if aps != 'Not Found':
            legend_objs.append(plt.Circle((0, 0), radius=7, fill=True, color=status_colors[aps]))
            legend_labels.append(f'A Priori Status:\n{aps} ({[status for ant, status in a_priori_statuses.items() if ant in ants].count(aps)} {["Core", "Outrigger"][outriggers]} Antennas)')

    # label nodes as a white box with black outline
    if len(node_centers) > 0:
        legend_objs.append(matplotlib.patches.Patch(facecolor='w', edgecolor='k'))
        legend_labels.append('Node Number')

    if len(unused_ants) > 0:
        legend_objs.append(matplotlib.lines.Line2D([0], [0], marker='o', color='w', markerfacecolor='grey', markersize=15, alpha=.2))
        legend_labels.append(f'Anntenna Not In Data')
        
    
    plt.legend(legend_objs, legend_labels, ncol=2, fontsize='large', framealpha=1)
    
    if outriggers:
        pass
    else:
        plt.xlim([-200, 150])
        plt.ylim([-150, 150])        
       
    # set axis equal and label everything
    plt.axis('equal')
    plt.tight_layout()
    plt.title(f'Summary of {["Core", "Outrigger"][outriggers]} Antenna Statuses and Metrics on {JD}', size=20)    
    plt.xlabel("Antenna East-West Position (meters)", size=12)
    plt.ylabel("Antenna North-South Position (meters)", size=12)
    plt.xticks(fontsize=12)
    plt.yticks(fontsize=12)
    xlim = plt.gca().get_xlim()
    ylim = plt.gca().get_ylim()    
        
    # plot unused antennas
    plt.autoscale(False)    
    for ant in unused_ants:
        if nodes[ant] in node_centers:
            plt.plot([hd.antpos[ant][0], node_centers[nodes[ant]][0]], 
                     [hd.antpos[ant][1], node_centers[nodes[ant]][1]], 'k', alpha=.2, zorder=0)
        
        plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=4, fill=True, color='w', ec=None, alpha=1, zorder=0))
        plt.gca().add_artist(plt.Circle(tuple(hd.antpos[ant][0:2]), radius=4, fill=True, color='grey', ec=None, alpha=.2, zorder=0))
        if hd.antpos[ant][0] < xlim[1] and hd.antpos[ant][0] > xlim[0]:
            if hd.antpos[ant][1] < ylim[1] and hd.antpos[ant][1] > ylim[0]:
                plt.text(hd.antpos[ant][0], hd.antpos[ant][1], str(ant), va='center', ha='center', color='k', alpha=.2) 

Figure 1: Array Plot of Flags and A Priori Statuses¶

This plot shows all antennas, which nodes they are connected to, and their a priori statuses (as the highlight text of their antenna numbers). It may also show (depending on what is finished running):

  • Whether they were flagged by auto_metrics (red circle) for bandpass shape, overall power, temporal variability, or temporal discontinuities. This is done in a binary fashion for the whole night.
  • Whether they were flagged by ant_metrics (green circle) as either dead (on either polarization) or crossed, with the transparency indicating the fraction of the night (i.e. number of files) that were flagged.
  • Whether they were flagged by redcal (blue circle) for high chi^2, with the transparency indicating the fraction of the night (i.e. number of files) that were flagged.

Note that the last fraction does not include antennas that were flagged before going into redcal due to their a priori status, for example.

In [26]:
core_ants = [ant for ant in ants if ant < 320]
outrigger_ants = [ant for ant in ants if ant >= 320]
Plot_Array(ants=core_ants, unused_ants=unused_ants, outriggers=False)
if len(outrigger_ants) > 0:
    Plot_Array(ants=outrigger_ants, unused_ants=sorted(set(unused_ants + core_ants)), outriggers=True)

Metadata¶

In [27]:
from hera_qm import __version__
print(__version__)
from hera_cal import __version__
print(__version__)
2.0.4.dev44+g3962204
3.1.5.dev171+gc8e6162
In [ ]: