Calibration Smoothing¶
by Josh Dillon, last updated December 20, 2025
This notebook runs calibration smoothing to the gains coming out of file_calibration notebook. It removes any flags founds on by that notebook and replaces them with flags generated from full_day_rfi and full_day_antenna_flagging. It flags antennas with high relative difference between the original gains and smoothed gains. It also plots the results for a couple of antennas.
Here's a set of links to skip to particular figures and tables:
• Figure 1: Identifying and Blacklisting abscal Failures¶
• Figure 2: Antenna Phases with Identified Phase Flips¶
• Figure 3: Full-Day Gain Amplitudes Before and After smooth_cal¶
• Figure 4: Full-Day Gain Phases Before and After smooth_cal¶
• Figure 5: Full-Day $\chi^2$ / DoF Waterfall from Redundant-Baseline Calibration¶
• Figure 6: Average $\chi^2$ per Antenna¶
• Figure 7: Relative Difference Before and After Smoothing¶
In [1]:
import time
tstart = time.time()
In [2]:
import os
os.environ['HDF5_USE_FILE_LOCKING'] = 'FALSE'
import h5py
import hdf5plugin # REQUIRED to have the compression plugins available
import numpy as np
import glob
import copy
import warnings
import matplotlib
import matplotlib.pyplot as plt
from hera_cal import io, utils, smooth_cal
from hera_qm.time_series_metrics import true_stretches
%matplotlib inline
from IPython.display import display, HTML
Parse inputs¶
In [3]:
# get files
SUM_FILE = os.environ.get("SUM_FILE", None)
# SUM_FILE = "/lustre/aoc/projects/hera/h6c-analysis/IDR3/2459893/zen.2459893.25258.sum.uvh5"
SUM_SUFFIX = os.environ.get("SUM_SUFFIX", 'sum.uvh5')
CAL_SUFFIX = os.environ.get("CAL_SUFFIX", 'sum.omni.calfits')
SMOOTH_CAL_SUFFIX = os.environ.get("SMOOTH_CAL_SUFFIX", 'sum.smooth.calfits')
ANT_FLAG_SUFFIX = os.environ.get("ANT_FLAG_SUFFIX", 'sum.antenna_flags.h5')
RFI_FLAG_SUFFIX = os.environ.get("RFI_FLAG_SUFFIX", 'sum.flag_waterfall.h5')
FREQ_SMOOTHING_SCALE = float(os.environ.get("FREQ_SMOOTHING_SCALE", 30.0)) # MHz
TIME_SMOOTHING_SCALE = float(os.environ.get("TIME_SMOOTHING_SCALE", 1e4)) # seconds
EIGENVAL_CUTOFF = float(os.environ.get("EIGENVAL_CUTOFF", 1e-12))
PER_POL_REFANT = os.environ.get("PER_POL_REFANT", "False").upper() == "TRUE"
BLACKLIST_TIMESCALE_FACTOR = float(os.environ.get("BLACKLIST_TIMESCALE_FACTOR", 4.0))
BLACKLIST_RELATIVE_ERROR_THRESH = float(os.environ.get("BLACKLIST_RELATIVE_ERROR_THRESH", 1))
BLACKLIST_RELATIVE_WEIGHT = float(os.environ.get("BLACKLIST_RELATIVE_WEIGHT", 0.1))
FM_LOW_FREQ = float(os.environ.get("FM_LOW_FREQ", 87.5)) # in MHz
FM_HIGH_FREQ = float(os.environ.get("FM_HIGH_FREQ", 108.0)) # in MHz
SC_RELATIVE_DIFF_CUTOFF = float(os.environ.get("SC_RELATIVE_DIFF_CUTOFF", 0.2))
for setting in ['SUM_FILE', 'SUM_SUFFIX', 'CAL_SUFFIX', 'SMOOTH_CAL_SUFFIX', 'ANT_FLAG_SUFFIX',
'RFI_FLAG_SUFFIX', 'FREQ_SMOOTHING_SCALE', 'TIME_SMOOTHING_SCALE', 'EIGENVAL_CUTOFF',
'PER_POL_REFANT', 'BLACKLIST_TIMESCALE_FACTOR', 'BLACKLIST_RELATIVE_ERROR_THRESH',
'BLACKLIST_RELATIVE_WEIGHT', 'FM_LOW_FREQ', 'FM_HIGH_FREQ', 'SC_RELATIVE_DIFF_CUTOFF']:
if issubclass(type(eval(setting)), str):
print(f'{setting} = "{eval(setting)}"')
else:
print(f'{setting} = {eval(setting)}')
SUM_FILE = "/mnt/sn1/data2/2461054/zen.2461054.25247.sum.uvh5" SUM_SUFFIX = "sum.uvh5" CAL_SUFFIX = "sum.omni.calfits" SMOOTH_CAL_SUFFIX = "sum.smooth.calfits" ANT_FLAG_SUFFIX = "sum.antenna_flags.h5" RFI_FLAG_SUFFIX = "sum.flag_waterfall.h5" FREQ_SMOOTHING_SCALE = 10.0 TIME_SMOOTHING_SCALE = 600000.0 EIGENVAL_CUTOFF = 1e-12 PER_POL_REFANT = False BLACKLIST_TIMESCALE_FACTOR = 4.0 BLACKLIST_RELATIVE_ERROR_THRESH = 1.0 BLACKLIST_RELATIVE_WEIGHT = 0.1 FM_LOW_FREQ = 87.5 FM_HIGH_FREQ = 108.0 SC_RELATIVE_DIFF_CUTOFF = 0.2
Load files and select reference antenna(s)¶
In [4]:
hd = io.HERAData(SUM_FILE)
sum_glob = '.'.join(SUM_FILE.split('.')[:-3]) + '.*.' + SUM_SUFFIX
cal_files_glob = sum_glob.replace(SUM_SUFFIX, CAL_SUFFIX)
cal_files = sorted(glob.glob(cal_files_glob))
print(f'Found {len(cal_files)} *.{CAL_SUFFIX} files starting with {cal_files[0]}.')
Found 1758 *.sum.omni.calfits files starting with /mnt/sn1/data2/2461054/zen.2461054.25247.sum.omni.calfits.
In [5]:
rfi_flag_files_glob = sum_glob.replace(SUM_SUFFIX, RFI_FLAG_SUFFIX)
rfi_flag_files = sorted(glob.glob(rfi_flag_files_glob))
print(f'Found {len(rfi_flag_files)} *.{RFI_FLAG_SUFFIX} files starting with {rfi_flag_files[0]}.')
Found 1758 *.sum.flag_waterfall.h5 files starting with /mnt/sn1/data2/2461054/zen.2461054.25247.sum.flag_waterfall.h5.
In [6]:
ant_flag_files_glob = sum_glob.replace(SUM_SUFFIX, ANT_FLAG_SUFFIX)
ant_flag_files = sorted(glob.glob(ant_flag_files_glob))
print(f'Found {len(ant_flag_files)} *.{ANT_FLAG_SUFFIX} files starting with {ant_flag_files[0]}.')
Found 1758 *.sum.antenna_flags.h5 files starting with /mnt/sn1/data2/2461054/zen.2461054.25247.sum.antenna_flags.h5.
In [7]:
cs = smooth_cal.CalibrationSmoother(cal_files, flag_file_list=(ant_flag_files + rfi_flag_files),
ignore_calflags=True, pick_refant=False, load_chisq=True, load_cspa=True)
invalid value encountered in multiply
In [8]:
# Pick reference antenna(s) but don't let ants known to flip phases get picked as reference antennas
banned_refants = [(144, 'Jnn'), (121, 'Jee'), (71, 'Jnn')]
cs.refant = smooth_cal.pick_reference_antenna({ant: cs.gain_grids[ant] for ant in cs.gain_grids if ant not in banned_refants},
{ant: cs.flag_grids[ant] for ant in cs.gain_grids if ant not in banned_refants},
cs.freqs, per_pol=True, acceptable_candidate_frac=0.25, antpos=hd.antpos)
for pol in cs.refant:
print(f'Reference antenna {cs.refant[pol][0]} selected for smoothing {pol} gains.')
if not PER_POL_REFANT:
# in this case, rephase both pols separately before smoothing, but also smooth the relative polarization calibration phasor
overall_refant = smooth_cal.pick_reference_antenna({ant: cs.gain_grids[ant] for ant in cs.refant.values()},
{ant: cs.flag_grids[ant] for ant in cs.refant.values()},
cs.freqs, per_pol=False)
print(f'Overall reference antenna {overall_refant} selected.')
other_refant = [ant for ant in cs.refant.values() if ant != overall_refant][0]
relative_pol_phasor = cs.gain_grids[overall_refant] * cs.gain_grids[other_refant].conj() # TODO: is this conjugation right?
relative_pol_phasor /= np.abs(relative_pol_phasor)
abscal_refants = {cs.refant[pol]: cs.gain_grids[cs.refant[pol]] for pol in ['Jee', 'Jnn']}
Reference antenna 173 selected for smoothing Jee gains. Reference antenna 132 selected for smoothing Jnn gains.
Overall reference antenna (np.int64(132), 'Jnn') selected.
In [9]:
cs.rephase_to_refant(propagate_refant_flags=True)
In [10]:
lst_grid = utils.JD2LST(cs.time_grid) * 12 / np.pi
lst_grid[lst_grid > lst_grid[-1]] -= 24
Find consistent outliers in relative error after a coarse smoothing¶
These are typically a sign of failures of abscal.
In [11]:
relative_error_samples = {pol: np.zeros_like(cs.gain_grids[cs.refant[pol]], dtype=float) for pol in ['Jee', 'Jnn']}
sum_relative_error = {pol: np.zeros_like(cs.gain_grids[cs.refant[pol]], dtype=float) for pol in ['Jee', 'Jnn']}
per_ant_avg_relative_error = {}
# perform a 2D DPSS filter with a BLACKLIST_TIMESCALE_FACTOR longer timescale, averaging the results per-pol
for ant in cs.gain_grids:
if np.all(cs.flag_grids[ant]):
continue
filtered, _ = smooth_cal.time_freq_2D_filter(gains=cs.gain_grids[ant],
wgts=(~cs.flag_grids[ant]).astype(float),
freqs=cs.freqs,
times=cs.time_grid,
freq_scale=FREQ_SMOOTHING_SCALE,
time_scale=TIME_SMOOTHING_SCALE * BLACKLIST_TIMESCALE_FACTOR,
eigenval_cutoff=EIGENVAL_CUTOFF,
method='DPSS',
fit_method='lu_solve',
fix_phase_flips=True,
phase_flip_time_scale = TIME_SMOOTHING_SCALE / 2,
flag_phase_flip_ints=True,
skip_flagged_edges=True,
freq_cuts=[(FM_LOW_FREQ + FM_HIGH_FREQ) * .5e6],
)
relative_error = np.where(cs.flag_grids[ant], 0, np.abs(cs.gain_grids[ant] - filtered) / np.abs(filtered))
per_ant_avg_relative_error[ant] = np.nanmean(np.where(cs.flag_grids[ant], np.nan, relative_error))
relative_error_samples[ant[1]] += (~cs.flag_grids[ant]).astype(float)
sum_relative_error[ant[1]] += relative_error
# figure out per-antpol cuts for where to set weights to 0 for the main smooth_cal (but not necessarily flags)
cs.blacklist_wgt = BLACKLIST_RELATIVE_WEIGHT
for pol in ['Jee', 'Jnn']:
avg_rel_error = sum_relative_error[pol] / relative_error_samples[pol]
to_blacklist = np.where(relative_error_samples[pol] > 0, avg_rel_error > BLACKLIST_RELATIVE_ERROR_THRESH, False)
for ant in cs.ants:
if ant[1] == pol:
cs.waterfall_blacklist[ant] = to_blacklist
invalid value encountered in divide
In [12]:
def plot_relative_error():
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fig, axes = plt.subplots(1, 3, figsize=(14, 7))
extent = [cs.freqs[0] / 1e6, cs.freqs[-1] / 1e6, lst_grid[-1], lst_grid[0]]
cmap = plt.get_cmap('Greys', 256)
cmap.set_over('red')
for ax, pol in zip(axes[0:2], ['Jee', 'Jnn']):
to_plot = sum_relative_error[pol] / relative_error_samples[pol]
im = ax.imshow(np.where(np.isfinite(to_plot), to_plot, np.nan), aspect='auto', interpolation='none',
vmin=0, vmax=BLACKLIST_RELATIVE_ERROR_THRESH, extent=extent, cmap=cmap)
ax.set_title(pol)
ax.set_yticklabels(ax.get_yticks() % 24)
ax.set_ylabel('LST (hours)')
ax.set_xlabel('Frequency (MHz)')
plt.colorbar(im, ax=axes[0:2], location='top', extend='max', label='Average Relative Error on Initial Smoothing')
for pol in ['Jee', 'Jnn']:
axes[2].hist((sum_relative_error[pol] / relative_error_samples[pol]).ravel(), bins=np.arange(0,2,.01), alpha=.5, label=pol)
axes[2].set_yscale('log')
axes[2].set_ylabel('Number of Waterfall Pixels')
axes[2].set_xlabel('Relative Error')
axes[2].axvline(BLACKLIST_RELATIVE_ERROR_THRESH, ls='--', c='r', label='Blacklist Threshold')
axes[2].legend()
Figure 1: Identifying and Blacklisting abscal Failures¶
This plot highlights regions of the waterfall that are per-polarization blacklisted (i.e. given 0 weight in the main smooth_cal fit, but not necessarily flagged). This is usually a sign of problems with abscal and often occurs because
In [13]:
plot_relative_error()
In [14]:
# duplicate a small number of abscal gains for plotting
antnums = set([ant[0] for ant in cs.ants])
flags_per_antnum = [np.sum(cs.flag_grids[ant, 'Jnn']) + np.sum(cs.flag_grids[ant, 'Jee']) for ant in antnums]
larger_relative_error = np.array([np.max([per_ant_avg_relative_error.get((ant, pol), np.inf) for pol in ['Jee', 'Jnn']]) for ant in antnums])
refant_nums = [ant[0] for ant in cs.refant.values()]
# pick candidates that don't exhibit too many flags or non-smooth structure on first pass
candidate_ants = []
rel_error_factor = 1
while len(candidate_ants) < 6: # Select more candidates to ensure we have enough after potential flagging
candidate_ants = [ant for ant, nflags, rel_err in zip(antnums, flags_per_antnum, larger_relative_error)
if (ant not in refant_nums) and (nflags <= np.percentile(flags_per_antnum, 25))
and (rel_err <= SC_RELATIVE_DIFF_CUTOFF * rel_error_factor)
and not np.all(cs.flag_grids[ant, 'Jee']) and not np.all(cs.flag_grids[ant, 'Jnn'])]
rel_error_factor += .1
# choose antennas to plot: select up to 6 candidates, prioritizing diversity across antenna numbers
candidate_ants_sorted = sorted(candidate_ants)
step = max(1, len(candidate_ants_sorted) // 6) # spread them out
_candidates = sorted(candidate_ants_sorted[::step][:6])
ants_to_plot_candidates = [_candidates[i//2] if i % 2 == 0 else _candidates[-(i//2)-1] for i in range(len(_candidates))]
# Store abscal gains for all candidates
abscal_gains = {}
for pol in ['Jee', 'Jnn']:
for antnum in ants_to_plot_candidates:
if PER_POL_REFANT:
abscal_gains[antnum, pol] = cs.gain_grids[(antnum, pol)] * np.abs(abscal_refants[cs.refant[pol]]) / abscal_refants[cs.refant[pol]]
else:
abscal_gains[antnum, pol] = cs.gain_grids[(antnum, pol)] / np.abs(abscal_refants[cs.refant[pol]]) * abscal_refants[cs.refant[pol]]
abscal_gains[antnum, pol] *= np.abs(abscal_refants[overall_refant]) / abscal_refants[overall_refant]
Perform smoothing¶
In [15]:
if not PER_POL_REFANT:
# treat the relative_pol_phasor as if it were antenna -1
cs.gain_grids[(-1, other_refant[1])] = relative_pol_phasor
cs.flag_grids[(-1, other_refant[1])] = cs.flag_grids[overall_refant] | cs.flag_grids[other_refant]
cs.waterfall_blacklist[(-1, other_refant[1])] = cs.waterfall_blacklist[cs.ants[0][0], 'Jee'] | cs.waterfall_blacklist[cs.ants[0][0], 'Jnn']
In [16]:
meta = cs.time_freq_2D_filter(freq_scale=FREQ_SMOOTHING_SCALE,
time_scale=TIME_SMOOTHING_SCALE,
eigenval_cutoff=EIGENVAL_CUTOFF,
method='DPSS',
fit_method='lu_solve',
fix_phase_flips=True,
phase_flip_time_scale = TIME_SMOOTHING_SCALE / 2,
flag_phase_flip_ints=True,
skip_flagged_edges=True,
freq_cuts=[(FM_LOW_FREQ + FM_HIGH_FREQ) * .5e6],)
2 phase flips detected on antenna (np.int64(30), 'Jee'). A total of 1 integrations were phase-flipped relative to the 0th integration between 2461054.469007605 and 2461054.469007605.
10 phase flips detected on antenna (np.int64(42), 'Jee'). A total of 82 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.4712445675.
2 phase flips detected on antenna (np.int64(18), 'Jee'). A total of 1 integrations were phase-flipped relative to the 0th integration between 2461054.469007605 and 2461054.469007605.
8 phase flips detected on antenna (np.int64(201), 'Jee'). A total of 302 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.480416113.
6 phase flips detected on antenna (np.int64(50), 'Jee'). A total of 361 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.4835478603.
2 phase flips detected on antenna (np.int64(203), 'Jnn'). A total of 1 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.4608426928.
18 phase flips detected on antenna (np.int64(202), 'Jee'). A total of 263 integrations were phase-flipped relative to the 0th integration between 2461054.448315704 and 2461054.4790739357.
2 phase flips detected on antenna (np.int64(240), 'Jee'). A total of 1 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.4608426928.
12 phase flips detected on antenna (np.int64(204), 'Jee'). A total of 22 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.469007605.
6 phase flips detected on antenna (np.int64(198), 'Jee'). A total of 358 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.483212316.
10 phase flips detected on antenna (np.int64(220), 'Jnn'). A total of 151 integrations were phase-flipped relative to the 0th integration between 2461054.455362135 and 2461054.4730341374.
6 phase flips detected on antenna (np.int64(239), 'Jnn'). A total of 4 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.4617374777.
4 phase flips detected on antenna (np.int64(239), 'Jee'). A total of 109 integrations were phase-flipped relative to the 0th integration between 2461054.459053123 and 2461054.4712445675.
4 phase flips detected on antenna (np.int64(221), 'Jnn'). A total of 62 integrations were phase-flipped relative to the 0th integration between 2461054.460283452 and 2461054.4672180354.
14 phase flips detected on antenna (np.int64(221), 'Jee'). A total of 155 integrations were phase-flipped relative to the 0th integration between 2461054.4569280087 and 2461054.478290999.
6 phase flips detected on antenna (np.int64(82), 'Jee'). A total of 359 integrations were phase-flipped relative to the 0th integration between 2461054.4431706904 and 2461054.483436012.
6 phase flips detected on antenna (np.int64(83), 'Jee'). A total of 360 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.483436012.
6 phase flips detected on antenna (np.int64(68), 'Jee'). A total of 336 integrations were phase-flipped relative to the 0th integration between 2461054.4450721084 and 2461054.4827649235.
10 phase flips detected on antenna (np.int64(40), 'Jee'). A total of 217 integrations were phase-flipped relative to the 0th integration between 2461054.455473983 and 2461054.480192417.
8 phase flips detected on antenna (np.int64(269), 'Jnn'). A total of 162 integrations were phase-flipped relative to the 0th integration between 2461054.4550265907 and 2461054.4733696817.
8 phase flips detected on antenna (np.int64(281), 'Jee'). A total of 352 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.4826530754.
6 phase flips detected on antenna (np.int64(281), 'Jnn'). A total of 302 integrations were phase-flipped relative to the 0th integration between 2461054.4471972226 and 2461054.48119905.
8 phase flips detected on antenna (np.int64(269), 'Jee'). A total of 309 integrations were phase-flipped relative to the 0th integration between 2461054.445855045 and 2461054.4807516574.
6 phase flips detected on antenna (np.int64(57), 'Jee'). A total of 43 integrations were phase-flipped relative to the 0th integration between 2461054.464981073 and 2461054.4701260864.
6 phase flips detected on antenna (np.int64(253), 'Jee'). A total of 332 integrations were phase-flipped relative to the 0th integration between 2461054.444624716 and 2461054.4818701386.
8 phase flips detected on antenna (np.int64(103), 'Jee'). A total of 312 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.481534594.
2 phase flips detected on antenna (np.int64(203), 'Jee'). A total of 111 integrations were phase-flipped relative to the 0th integration between 2461054.459053123 and 2461054.4713564157.
14 phase flips detected on antenna (np.int64(201), 'Jnn'). A total of 209 integrations were phase-flipped relative to the 0th integration between 2461054.452565932 and 2461054.478290999.
18 phase flips detected on antenna (np.int64(222), 'Jee'). A total of 40 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.469007605.
8 phase flips detected on antenna (np.int64(37), 'Jee'). A total of 347 integrations were phase-flipped relative to the 0th integration between 2461054.4444010197 and 2461054.483436012.
8 phase flips detected on antenna (np.int64(237), 'Jee'). A total of 308 integrations were phase-flipped relative to the 0th integration between 2461054.4459668933 and 2461054.4808635055.
8 phase flips detected on antenna (np.int64(237), 'Jnn'). A total of 180 integrations were phase-flipped relative to the 0th integration between 2461054.4530133246 and 2461054.478290999.
8 phase flips detected on antenna (np.int64(238), 'Jnn'). A total of 104 integrations were phase-flipped relative to the 0th integration between 2461054.457039857 and 2461054.470797175.
4 phase flips detected on antenna (np.int64(220), 'Jee'). A total of 296 integrations were phase-flipped relative to the 0th integration between 2461054.4470853745 and 2461054.480192417.
2 phase flips detected on antenna (np.int64(51), 'Jnn'). A total of 333 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.483436012.
8 phase flips detected on antenna (np.int64(139), 'Jee'). A total of 339 integrations were phase-flipped relative to the 0th integration between 2461054.4444010197 and 2461054.4825412272.
10 phase flips detected on antenna (np.int64(268), 'Jnn'). A total of 242 integrations were phase-flipped relative to the 0th integration between 2461054.4508882104 and 2461054.478290999.
8 phase flips detected on antenna (np.int64(252), 'Jnn'). A total of 278 integrations were phase-flipped relative to the 0th integration between 2461054.4490986406 and 2461054.4806398093.
6 phase flips detected on antenna (np.int64(252), 'Jee'). A total of 351 integrations were phase-flipped relative to the 0th integration between 2461054.4432825386 and 2461054.4826530754.
10 phase flips detected on antenna (np.int64(254), 'Jee'). A total of 278 integrations were phase-flipped relative to the 0th integration between 2461054.448315704 and 2461054.4797450243.
6 phase flips detected on antenna (np.int64(177), 'Jee'). A total of 357 integrations were phase-flipped relative to the 0th integration between 2461054.4431706904 and 2461054.483212316.
6 phase flips detected on antenna (np.int64(216), 'Jee'). A total of 356 integrations were phase-flipped relative to the 0th integration between 2461054.4432825386 and 2461054.483212316.
8 phase flips detected on antenna (np.int64(233), 'Jee'). A total of 363 integrations were phase-flipped relative to the 0th integration between 2461054.4423877536 and 2461054.483212316.
6 phase flips detected on antenna (np.int64(282), 'Jee'). A total of 350 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.482317531.
6 phase flips detected on antenna (np.int64(217), 'Jee'). A total of 357 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.483100468.
10 phase flips detected on antenna (np.int64(219), 'Jnn'). A total of 227 integrations were phase-flipped relative to the 0th integration between 2461054.452230388 and 2461054.478290999.
8 phase flips detected on antenna (np.int64(219), 'Jee'). A total of 325 integrations were phase-flipped relative to the 0th integration between 2461054.4451839565 and 2461054.4817582904.
8 phase flips detected on antenna (np.int64(200), 'Jnn'). A total of 278 integrations were phase-flipped relative to the 0th integration between 2461054.4490986406 and 2461054.4806398093.
10 phase flips detected on antenna (np.int64(53), 'Jee'). A total of 313 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.4818701386.
6 phase flips detected on antenna (np.int64(139), 'Jnn'). A total of 307 integrations were phase-flipped relative to the 0th integration between 2461054.4471972226 and 2461054.4818701386.
8 phase flips detected on antenna (np.int64(159), 'Jee'). A total of 345 integrations were phase-flipped relative to the 0th integration between 2461054.4439536273 and 2461054.4828767716.
6 phase flips detected on antenna (np.int64(159), 'Jnn'). A total of 314 integrations were phase-flipped relative to the 0th integration between 2461054.4468616783 and 2461054.482093835.
2 phase flips detected on antenna (np.int64(250), 'Jnn'). A total of 326 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.4826530754.
24 phase flips detected on antenna (np.int64(285), 'Jee'). A total of 164 integrations were phase-flipped relative to the 0th integration between 2461054.453460717 and 2461054.478290999.
10 phase flips detected on antenna (np.int64(285), 'Jnn'). A total of 11 integrations were phase-flipped relative to the 0th integration between 2461054.460283452 and 2461054.4622967183.
8 phase flips detected on antenna (np.int64(197), 'Jee'). A total of 364 integrations were phase-flipped relative to the 0th integration between 2461054.4424996017 and 2461054.483436012.
6 phase flips detected on antenna (np.int64(234), 'Jee'). A total of 357 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.483100468.
4 phase flips detected on antenna (np.int64(164), 'Jee'). A total of 2 integrations were phase-flipped relative to the 0th integration between 2461054.466882491 and 2461054.469007605.
20 phase flips detected on antenna (np.int64(299), 'Jee'). A total of 66 integrations were phase-flipped relative to the 0th integration between 2461054.459276819 and 2461054.4701260864.
10 phase flips detected on antenna (np.int64(284), 'Jnn'). A total of 91 integrations were phase-flipped relative to the 0th integration between 2461054.457039857 and 2461054.4701260864.
10 phase flips detected on antenna (np.int64(284), 'Jee'). A total of 287 integrations were phase-flipped relative to the 0th integration between 2461054.4471972226 and 2461054.4797450243.
6 phase flips detected on antenna (np.int64(282), 'Jnn'). A total of 278 integrations were phase-flipped relative to the 0th integration between 2461054.4489867925 and 2461054.480192417.
6 phase flips detected on antenna (np.int64(196), 'Jnn'). A total of 340 integrations were phase-flipped relative to the 0th integration between 2461054.4454076528 and 2461054.4835478603.
8 phase flips detected on antenna (np.int64(196), 'Jee'). A total of 364 integrations were phase-flipped relative to the 0th integration between 2461054.4424996017 and 2461054.483436012.
8 phase flips detected on antenna (np.int64(283), 'Jee'). A total of 321 integrations were phase-flipped relative to the 0th integration between 2461054.4452958046 and 2461054.481534594.
8 phase flips detected on antenna (np.int64(218), 'Jee'). A total of 343 integrations were phase-flipped relative to the 0th integration between 2461054.443841779 and 2461054.482429379.
6 phase flips detected on antenna (np.int64(178), 'Jee'). A total of 340 integrations were phase-flipped relative to the 0th integration between 2461054.444624716 and 2461054.4827649235.
8 phase flips detected on antenna (np.int64(215), 'Jee'). A total of 364 integrations were phase-flipped relative to the 0th integration between 2461054.4424996017 and 2461054.483436012.
10 phase flips detected on antenna (np.int64(182), 'Jnn'). A total of 150 integrations were phase-flipped relative to the 0th integration between 2461054.456145072 and 2461054.4733696817.
6 phase flips detected on antenna (np.int64(179), 'Jee'). A total of 339 integrations were phase-flipped relative to the 0th integration between 2461054.444624716 and 2461054.4826530754.
2 phase flips detected on antenna (np.int64(65), 'Jnn'). A total of 350 integrations were phase-flipped relative to the 0th integration between 2461054.4449602603 and 2461054.4839952528.
4 phase flips detected on antenna (np.int64(71), 'Jee'). A total of 151 integrations were phase-flipped relative to the 0th integration between 2461054.4588294267 and 2461054.4770606696.
2 phase flips detected on antenna (np.int64(156), 'Jnn'). A total of 346 integrations were phase-flipped relative to the 0th integration between 2461054.4449602603 and 2461054.4835478603.
8 phase flips detected on antenna (np.int64(158), 'Jee'). A total of 355 integrations were phase-flipped relative to the 0th integration between 2461054.4432825386 and 2461054.483212316.
4 phase flips detected on antenna (np.int64(98), 'Jee'). A total of 368 integrations were phase-flipped relative to the 0th integration between 2461054.442723298 and 2461054.4838834046.
6 phase flips detected on antenna (np.int64(84), 'Jnn'). A total of 295 integrations were phase-flipped relative to the 0th integration between 2461054.4489867925 and 2461054.482093835.
2 phase flips detected on antenna (np.int64(52), 'Jnn'). A total of 125 integrations were phase-flipped relative to the 0th integration between 2461054.4595005154 and 2461054.4733696817.
10 phase flips detected on antenna (np.int64(105), 'Jee'). A total of 76 integrations were phase-flipped relative to the 0th integration between 2461054.4608426928 and 2461054.470797175.
10 phase flips detected on antenna (np.int64(198), 'Jnn'). A total of 201 integrations were phase-flipped relative to the 0th integration between 2461054.4548028945 and 2461054.478290999.
12 phase flips detected on antenna (np.int64(217), 'Jnn'). A total of 147 integrations were phase-flipped relative to the 0th integration between 2461054.455362135 and 2461054.4732578336.
10 phase flips detected on antenna (np.int64(216), 'Jnn'). A total of 170 integrations were phase-flipped relative to the 0th integration between 2461054.455138439 and 2461054.478290999.
4 phase flips detected on antenna (np.int64(55), 'Jnn'). A total of 115 integrations were phase-flipped relative to the 0th integration between 2461054.460171604 and 2461054.4730341374.
4 phase flips detected on antenna (np.int64(54), 'Jnn'). A total of 189 integrations were phase-flipped relative to the 0th integration between 2461054.457039857 and 2461054.478290999.
6 phase flips detected on antenna (np.int64(69), 'Jnn'). A total of 216 integrations were phase-flipped relative to the 0th integration between 2461054.4555858313 and 2461054.4798568725.
4 phase flips detected on antenna (np.int64(140), 'Jnn'). A total of 278 integrations were phase-flipped relative to the 0th integration between 2461054.4499934255 and 2461054.48119905.
8 phase flips detected on antenna (np.int64(83), 'Jnn'). A total of 318 integrations were phase-flipped relative to the 0th integration between 2461054.4468616783 and 2461054.4826530754.
6 phase flips detected on antenna (np.int64(144), 'Jee'). A total of 10 integrations were phase-flipped relative to the 0th integration between 2461054.463862592 and 2461054.466882491.
4 phase flips detected on antenna (np.int64(16), 'Jnn'). A total of 11 integrations were phase-flipped relative to the 0th integration between 2461054.4655403136 and 2461054.466770643.
6 phase flips detected on antenna (np.int64(138), 'Jnn'). A total of 321 integrations were phase-flipped relative to the 0th integration between 2461054.446526134 and 2461054.4826530754.
2 phase flips detected on antenna (np.int64(50), 'Jnn'). A total of 337 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.4838834046.
4 phase flips detected on antenna (np.int64(100), 'Jnn'). A total of 331 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.483436012.
2 phase flips detected on antenna (np.int64(176), 'Jnn'). A total of 333 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.483436012.
6 phase flips detected on antenna (np.int64(101), 'Jnn'). A total of 311 integrations were phase-flipped relative to the 0th integration between 2461054.4471972226 and 2461054.482317531.
2 phase flips detected on antenna (np.int64(117), 'Jnn'). A total of 333 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.483436012.
8 phase flips detected on antenna (np.int64(141), 'Jnn'). A total of 217 integrations were phase-flipped relative to the 0th integration between 2461054.4548028945 and 2461054.479297632.
4 phase flips detected on antenna (np.int64(179), 'Jnn'). A total of 298 integrations were phase-flipped relative to the 0th integration between 2461054.4485394 and 2461054.482093835.
2 phase flips detected on antenna (np.int64(157), 'Jnn'). A total of 330 integrations were phase-flipped relative to the 0th integration between 2461054.4463024377 and 2461054.483100468.
2 phase flips detected on antenna (np.int64(177), 'Jnn'). A total of 326 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.4827649235.
4 phase flips detected on antenna (np.int64(123), 'Jnn'). A total of 101 integrations were phase-flipped relative to the 0th integration between 2461054.4598360597 and 2461054.4711327194.
4 phase flips detected on antenna (np.int64(103), 'Jnn'). A total of 226 integrations were phase-flipped relative to the 0th integration between 2461054.4548028945 and 2461054.4800805687.
2 phase flips detected on antenna (np.int64(87), 'Jnn'). A total of 102 integrations were phase-flipped relative to the 0th integration between 2461054.460283452 and 2461054.471580112.
4 phase flips detected on antenna (np.int64(181), 'Jnn'). A total of 216 integrations were phase-flipped relative to the 0th integration between 2461054.4530133246 and 2461054.478514695.
4 phase flips detected on antenna (np.int64(68), 'Jnn'). A total of 263 integrations were phase-flipped relative to the 0th integration between 2461054.452230388 and 2461054.4816464423.
8 phase flips detected on antenna (np.int64(137), 'Jee'). A total of 354 integrations were phase-flipped relative to the 0th integration between 2461054.4433943867 and 2461054.483212316.
8 phase flips detected on antenna (np.int64(178), 'Jnn'). A total of 311 integrations were phase-flipped relative to the 0th integration between 2461054.4469735264 and 2461054.482205683.
6 phase flips detected on antenna (np.int64(267), 'Jnn'). A total of 270 integrations were phase-flipped relative to the 0th integration between 2461054.4501052736 and 2461054.4806398093.
2 phase flips detected on antenna (np.int64(53), 'Jnn'). A total of 102 integrations were phase-flipped relative to the 0th integration between 2461054.4605071484 and 2461054.471803808.
8 phase flips detected on antenna (np.int64(141), 'Jee'). A total of 311 integrations were phase-flipped relative to the 0th integration between 2461054.446414286 and 2461054.481422746.
4 phase flips detected on antenna (np.int64(135), 'Jnn'). A total of 353 integrations were phase-flipped relative to the 0th integration between 2461054.444624716 and 2461054.484107101.
8 phase flips detected on antenna (np.int64(66), 'Jnn'). A total of 214 integrations were phase-flipped relative to the 0th integration between 2461054.4555858313 and 2461054.4797450243.
2 phase flips detected on antenna (np.int64(160), 'Jnn'). A total of 106 integrations were phase-flipped relative to the 0th integration between 2461054.4595005154 and 2461054.4712445675.
10 phase flips detected on antenna (np.int64(15), 'Jnn'). A total of 109 integrations were phase-flipped relative to the 0th integration between 2461054.4605071484 and 2461054.4730341374.
8 phase flips detected on antenna (np.int64(36), 'Jnn'). A total of 325 integrations were phase-flipped relative to the 0th integration between 2461054.446526134 and 2461054.483100468.
4 phase flips detected on antenna (np.int64(155), 'Jee'). A total of 365 integrations were phase-flipped relative to the 0th integration between 2461054.4430588423 and 2461054.4838834046.
In [17]:
# calculate average chi^2 per antenna before additional flagging
avg_cspa_vs_time = {ant: np.nanmean(np.where(cs.flag_grids[ant], np.nan, cs.cspa_grids[ant]), axis=1) for ant in cs.ants}
avg_cspa_vs_freq = {ant: np.nanmean(np.where(cs.flag_grids[ant], np.nan, cs.cspa_grids[ant]), axis=0) for ant in cs.ants}
avg_cspa = {ant: np.nanmean(np.where(cs.flag_grids[ant], np.nan, cs.cspa_grids[ant])) for ant in cs.ants}
Mean of empty slice
Mean of empty slice
Mean of empty slice
In [18]:
# Pick out antennas with too high relative differences before and after smoothing and flag them.
avg_relative_diffs = {ant: np.nanmean(rel_diff) for ant, rel_diff in meta['freq_avg_rel_diff'].items()}
to_cut = sorted([ant for ant, diff in avg_relative_diffs.items() if ant[0] >= 0 and diff > SC_RELATIVE_DIFF_CUTOFF])
if len(to_cut) > 0:
for ant in to_cut:
print(f'Flagging antenna {ant[0]}{ant[1][-1]} with a relative difference before and after smoothing of {avg_relative_diffs[ant]:.2%} '
f'(compared to the {SC_RELATIVE_DIFF_CUTOFF:.2%} cutoff).')
cs.flag_grids[ant] |= True
else:
print(f'No antennas have a relative difference above the {SC_RELATIVE_DIFF_CUTOFF:.2%} cutoff.')
Flagging antenna 5n with a relative difference before and after smoothing of 33.04% (compared to the 20.00% cutoff). Flagging antenna 7n with a relative difference before and after smoothing of 27.00% (compared to the 20.00% cutoff). Flagging antenna 8n with a relative difference before and after smoothing of 23.20% (compared to the 20.00% cutoff). Flagging antenna 9n with a relative difference before and after smoothing of 20.44% (compared to the 20.00% cutoff). Flagging antenna 15n with a relative difference before and after smoothing of 39.27% (compared to the 20.00% cutoff). Flagging antenna 16n with a relative difference before and after smoothing of 34.10% (compared to the 20.00% cutoff). Flagging antenna 17n with a relative difference before and after smoothing of 30.75% (compared to the 20.00% cutoff). Flagging antenna 18e with a relative difference before and after smoothing of 31.04% (compared to the 20.00% cutoff). Flagging antenna 19n with a relative difference before and after smoothing of 22.83% (compared to the 20.00% cutoff). Flagging antenna 30e with a relative difference before and after smoothing of 31.59% (compared to the 20.00% cutoff). Flagging antenna 31n with a relative difference before and after smoothing of 24.12% (compared to the 20.00% cutoff). Flagging antenna 32e with a relative difference before and after smoothing of 24.65% (compared to the 20.00% cutoff). Flagging antenna 36n with a relative difference before and after smoothing of 55.69% (compared to the 20.00% cutoff). Flagging antenna 37e with a relative difference before and after smoothing of 49.63% (compared to the 20.00% cutoff). Flagging antenna 40e with a relative difference before and after smoothing of 43.41% (compared to the 20.00% cutoff). Flagging antenna 41n with a relative difference before and after smoothing of 35.76% (compared to the 20.00% cutoff). Flagging antenna 42e with a relative difference before and after smoothing of 37.20% (compared to the 20.00% cutoff). Flagging antenna 43n with a relative difference before and after smoothing of 29.38% (compared to the 20.00% cutoff). Flagging antenna 45n with a relative difference before and after smoothing of 22.15% (compared to the 20.00% cutoff). Flagging antenna 50e with a relative difference before and after smoothing of 53.16% (compared to the 20.00% cutoff). Flagging antenna 50n with a relative difference before and after smoothing of 54.23% (compared to the 20.00% cutoff). Flagging antenna 51n with a relative difference before and after smoothing of 52.80% (compared to the 20.00% cutoff). Flagging antenna 52n with a relative difference before and after smoothing of 47.49% (compared to the 20.00% cutoff). Flagging antenna 53e with a relative difference before and after smoothing of 47.07% (compared to the 20.00% cutoff). Flagging antenna 53n with a relative difference before and after smoothing of 45.95% (compared to the 20.00% cutoff). Flagging antenna 54n with a relative difference before and after smoothing of 43.54% (compared to the 20.00% cutoff). Flagging antenna 55n with a relative difference before and after smoothing of 39.94% (compared to the 20.00% cutoff). Flagging antenna 57e with a relative difference before and after smoothing of 34.62% (compared to the 20.00% cutoff). Flagging antenna 58n with a relative difference before and after smoothing of 28.19% (compared to the 20.00% cutoff). Flagging antenna 59n with a relative difference before and after smoothing of 23.65% (compared to the 20.00% cutoff). Flagging antenna 65n with a relative difference before and after smoothing of 55.94% (compared to the 20.00% cutoff). Flagging antenna 66n with a relative difference before and after smoothing of 48.97% (compared to the 20.00% cutoff). Flagging antenna 68e with a relative difference before and after smoothing of 46.77% (compared to the 20.00% cutoff). Flagging antenna 68n with a relative difference before and after smoothing of 46.71% (compared to the 20.00% cutoff). Flagging antenna 69n with a relative difference before and after smoothing of 44.59% (compared to the 20.00% cutoff). Flagging antenna 71e with a relative difference before and after smoothing of 40.29% (compared to the 20.00% cutoff). Flagging antenna 72n with a relative difference before and after smoothing of 33.92% (compared to the 20.00% cutoff). Flagging antenna 73n with a relative difference before and after smoothing of 28.63% (compared to the 20.00% cutoff). Flagging antenna 75e with a relative difference before and after smoothing of 24.72% (compared to the 20.00% cutoff). Flagging antenna 75n with a relative difference before and after smoothing of 20.51% (compared to the 20.00% cutoff). Flagging antenna 82e with a relative difference before and after smoothing of 52.30% (compared to the 20.00% cutoff). Flagging antenna 83e with a relative difference before and after smoothing of 50.60% (compared to the 20.00% cutoff). Flagging antenna 83n with a relative difference before and after smoothing of 50.70% (compared to the 20.00% cutoff). Flagging antenna 84n with a relative difference before and after smoothing of 48.63% (compared to the 20.00% cutoff). Flagging antenna 87n with a relative difference before and after smoothing of 39.31% (compared to the 20.00% cutoff). Flagging antenna 91n with a relative difference before and after smoothing of 22.36% (compared to the 20.00% cutoff). Flagging antenna 98e with a relative difference before and after smoothing of 54.64% (compared to the 20.00% cutoff). Flagging antenna 100n with a relative difference before and after smoothing of 52.15% (compared to the 20.00% cutoff). Flagging antenna 101n with a relative difference before and after smoothing of 50.00% (compared to the 20.00% cutoff). Flagging antenna 103e with a relative difference before and after smoothing of 46.32% (compared to the 20.00% cutoff). Flagging antenna 103n with a relative difference before and after smoothing of 44.84% (compared to the 20.00% cutoff). Flagging antenna 105e with a relative difference before and after smoothing of 39.33% (compared to the 20.00% cutoff). Flagging antenna 105n with a relative difference before and after smoothing of 36.81% (compared to the 20.00% cutoff). Flagging antenna 106n with a relative difference before and after smoothing of 34.78% (compared to the 20.00% cutoff). Flagging antenna 108n with a relative difference before and after smoothing of 23.74% (compared to the 20.00% cutoff). Flagging antenna 117n with a relative difference before and after smoothing of 54.81% (compared to the 20.00% cutoff). Flagging antenna 123n with a relative difference before and after smoothing of 40.14% (compared to the 20.00% cutoff).
Flagging antenna 126n with a relative difference before and after smoothing of 26.95% (compared to the 20.00% cutoff). Flagging antenna 135n with a relative difference before and after smoothing of 60.30% (compared to the 20.00% cutoff). Flagging antenna 137e with a relative difference before and after smoothing of 53.83% (compared to the 20.00% cutoff). Flagging antenna 138n with a relative difference before and after smoothing of 52.55% (compared to the 20.00% cutoff). Flagging antenna 139e with a relative difference before and after smoothing of 48.47% (compared to the 20.00% cutoff). Flagging antenna 139n with a relative difference before and after smoothing of 50.05% (compared to the 20.00% cutoff). Flagging antenna 140n with a relative difference before and after smoothing of 48.30% (compared to the 20.00% cutoff). Flagging antenna 141e with a relative difference before and after smoothing of 48.41% (compared to the 20.00% cutoff). Flagging antenna 141n with a relative difference before and after smoothing of 45.22% (compared to the 20.00% cutoff). Flagging antenna 144e with a relative difference before and after smoothing of 39.14% (compared to the 20.00% cutoff). Flagging antenna 145n with a relative difference before and after smoothing of 29.91% (compared to the 20.00% cutoff). Flagging antenna 146e with a relative difference before and after smoothing of 30.74% (compared to the 20.00% cutoff). Flagging antenna 155e with a relative difference before and after smoothing of 62.41% (compared to the 20.00% cutoff). Flagging antenna 156n with a relative difference before and after smoothing of 57.50% (compared to the 20.00% cutoff). Flagging antenna 157n with a relative difference before and after smoothing of 53.73% (compared to the 20.00% cutoff). Flagging antenna 158e with a relative difference before and after smoothing of 50.71% (compared to the 20.00% cutoff). Flagging antenna 159e with a relative difference before and after smoothing of 49.36% (compared to the 20.00% cutoff). Flagging antenna 159n with a relative difference before and after smoothing of 51.22% (compared to the 20.00% cutoff). Flagging antenna 160n with a relative difference before and after smoothing of 47.12% (compared to the 20.00% cutoff). Flagging antenna 164e with a relative difference before and after smoothing of 36.52% (compared to the 20.00% cutoff). Flagging antenna 164n with a relative difference before and after smoothing of 32.71% (compared to the 20.00% cutoff). Flagging antenna 165e with a relative difference before and after smoothing of 32.78% (compared to the 20.00% cutoff). Flagging antenna 165n with a relative difference before and after smoothing of 29.06% (compared to the 20.00% cutoff). Flagging antenna 166n with a relative difference before and after smoothing of 26.61% (compared to the 20.00% cutoff). Flagging antenna 167e with a relative difference before and after smoothing of 25.01% (compared to the 20.00% cutoff). Flagging antenna 176n with a relative difference before and after smoothing of 57.93% (compared to the 20.00% cutoff). Flagging antenna 177e with a relative difference before and after smoothing of 54.93% (compared to the 20.00% cutoff). Flagging antenna 177n with a relative difference before and after smoothing of 55.48% (compared to the 20.00% cutoff). Flagging antenna 178e with a relative difference before and after smoothing of 52.36% (compared to the 20.00% cutoff). Flagging antenna 178n with a relative difference before and after smoothing of 53.51% (compared to the 20.00% cutoff). Flagging antenna 179e with a relative difference before and after smoothing of 50.39% (compared to the 20.00% cutoff). Flagging antenna 179n with a relative difference before and after smoothing of 51.42% (compared to the 20.00% cutoff). Flagging antenna 181n with a relative difference before and after smoothing of 46.88% (compared to the 20.00% cutoff). Flagging antenna 182n with a relative difference before and after smoothing of 42.51% (compared to the 20.00% cutoff). Flagging antenna 184n with a relative difference before and after smoothing of 36.13% (compared to the 20.00% cutoff). Flagging antenna 185e with a relative difference before and after smoothing of 34.96% (compared to the 20.00% cutoff). Flagging antenna 185n with a relative difference before and after smoothing of 31.58% (compared to the 20.00% cutoff). Flagging antenna 186n with a relative difference before and after smoothing of 28.94% (compared to the 20.00% cutoff). Flagging antenna 187e with a relative difference before and after smoothing of 28.63% (compared to the 20.00% cutoff). Flagging antenna 187n with a relative difference before and after smoothing of 23.47% (compared to the 20.00% cutoff). Flagging antenna 188e with a relative difference before and after smoothing of 24.57% (compared to the 20.00% cutoff). Flagging antenna 196e with a relative difference before and after smoothing of 55.56% (compared to the 20.00% cutoff). Flagging antenna 196n with a relative difference before and after smoothing of 59.55% (compared to the 20.00% cutoff). Flagging antenna 197e with a relative difference before and after smoothing of 51.85% (compared to the 20.00% cutoff). Flagging antenna 198e with a relative difference before and after smoothing of 49.70% (compared to the 20.00% cutoff). Flagging antenna 198n with a relative difference before and after smoothing of 49.90% (compared to the 20.00% cutoff). Flagging antenna 200n with a relative difference before and after smoothing of 48.45% (compared to the 20.00% cutoff). Flagging antenna 201e with a relative difference before and after smoothing of 45.27% (compared to the 20.00% cutoff). Flagging antenna 201n with a relative difference before and after smoothing of 44.37% (compared to the 20.00% cutoff). Flagging antenna 202e with a relative difference before and after smoothing of 43.52% (compared to the 20.00% cutoff). Flagging antenna 203e with a relative difference before and after smoothing of 41.46% (compared to the 20.00% cutoff). Flagging antenna 203n with a relative difference before and after smoothing of 38.14% (compared to the 20.00% cutoff). Flagging antenna 204e with a relative difference before and after smoothing of 38.18% (compared to the 20.00% cutoff). Flagging antenna 204n with a relative difference before and after smoothing of 34.11% (compared to the 20.00% cutoff). Flagging antenna 205e with a relative difference before and after smoothing of 33.64% (compared to the 20.00% cutoff). Flagging antenna 205n with a relative difference before and after smoothing of 29.41% (compared to the 20.00% cutoff). Flagging antenna 206e with a relative difference before and after smoothing of 29.21% (compared to the 20.00% cutoff).
Flagging antenna 208e with a relative difference before and after smoothing of 21.30% (compared to the 20.00% cutoff). Flagging antenna 215e with a relative difference before and after smoothing of 54.19% (compared to the 20.00% cutoff). Flagging antenna 216e with a relative difference before and after smoothing of 53.11% (compared to the 20.00% cutoff). Flagging antenna 216n with a relative difference before and after smoothing of 53.42% (compared to the 20.00% cutoff). Flagging antenna 217e with a relative difference before and after smoothing of 50.25% (compared to the 20.00% cutoff). Flagging antenna 217n with a relative difference before and after smoothing of 50.82% (compared to the 20.00% cutoff). Flagging antenna 218e with a relative difference before and after smoothing of 48.27% (compared to the 20.00% cutoff). Flagging antenna 219e with a relative difference before and after smoothing of 46.67% (compared to the 20.00% cutoff). Flagging antenna 219n with a relative difference before and after smoothing of 46.55% (compared to the 20.00% cutoff). Flagging antenna 220e with a relative difference before and after smoothing of 45.06% (compared to the 20.00% cutoff). Flagging antenna 220n with a relative difference before and after smoothing of 43.09% (compared to the 20.00% cutoff). Flagging antenna 221e with a relative difference before and after smoothing of 42.80% (compared to the 20.00% cutoff). Flagging antenna 221n with a relative difference before and after smoothing of 40.60% (compared to the 20.00% cutoff). Flagging antenna 222e with a relative difference before and after smoothing of 40.07% (compared to the 20.00% cutoff). Flagging antenna 222n with a relative difference before and after smoothing of 36.25% (compared to the 20.00% cutoff). Flagging antenna 223e with a relative difference before and after smoothing of 36.89% (compared to the 20.00% cutoff). Flagging antenna 223n with a relative difference before and after smoothing of 32.86% (compared to the 20.00% cutoff). Flagging antenna 224e with a relative difference before and after smoothing of 32.24% (compared to the 20.00% cutoff). Flagging antenna 224n with a relative difference before and after smoothing of 28.99% (compared to the 20.00% cutoff). Flagging antenna 225e with a relative difference before and after smoothing of 27.12% (compared to the 20.00% cutoff). Flagging antenna 225n with a relative difference before and after smoothing of 23.77% (compared to the 20.00% cutoff). Flagging antenna 226e with a relative difference before and after smoothing of 23.69% (compared to the 20.00% cutoff). Flagging antenna 233e with a relative difference before and after smoothing of 53.78% (compared to the 20.00% cutoff). Flagging antenna 234e with a relative difference before and after smoothing of 51.90% (compared to the 20.00% cutoff). Flagging antenna 237e with a relative difference before and after smoothing of 46.76% (compared to the 20.00% cutoff). Flagging antenna 237n with a relative difference before and after smoothing of 46.13% (compared to the 20.00% cutoff). Flagging antenna 238n with a relative difference before and after smoothing of 42.79% (compared to the 20.00% cutoff). Flagging antenna 239e with a relative difference before and after smoothing of 42.52% (compared to the 20.00% cutoff). Flagging antenna 239n with a relative difference before and after smoothing of 39.17% (compared to the 20.00% cutoff). Flagging antenna 240e with a relative difference before and after smoothing of 39.39% (compared to the 20.00% cutoff). Flagging antenna 241e with a relative difference before and after smoothing of 34.12% (compared to the 20.00% cutoff). Flagging antenna 241n with a relative difference before and after smoothing of 31.04% (compared to the 20.00% cutoff). Flagging antenna 242e with a relative difference before and after smoothing of 30.46% (compared to the 20.00% cutoff). Flagging antenna 242n with a relative difference before and after smoothing of 27.96% (compared to the 20.00% cutoff). Flagging antenna 243e with a relative difference before and after smoothing of 26.31% (compared to the 20.00% cutoff). Flagging antenna 243n with a relative difference before and after smoothing of 24.73% (compared to the 20.00% cutoff). Flagging antenna 244e with a relative difference before and after smoothing of 22.89% (compared to the 20.00% cutoff). Flagging antenna 244n with a relative difference before and after smoothing of 21.59% (compared to the 20.00% cutoff). Flagging antenna 250n with a relative difference before and after smoothing of 57.27% (compared to the 20.00% cutoff). Flagging antenna 252e with a relative difference before and after smoothing of 48.23% (compared to the 20.00% cutoff). Flagging antenna 252n with a relative difference before and after smoothing of 50.76% (compared to the 20.00% cutoff). Flagging antenna 253e with a relative difference before and after smoothing of 46.95% (compared to the 20.00% cutoff). Flagging antenna 254e with a relative difference before and after smoothing of 45.21% (compared to the 20.00% cutoff). Flagging antenna 255n with a relative difference before and after smoothing of 40.64% (compared to the 20.00% cutoff). Flagging antenna 256n with a relative difference before and after smoothing of 38.00% (compared to the 20.00% cutoff). Flagging antenna 257e with a relative difference before and after smoothing of 37.10% (compared to the 20.00% cutoff). Flagging antenna 257n with a relative difference before and after smoothing of 33.86% (compared to the 20.00% cutoff). Flagging antenna 267n with a relative difference before and after smoothing of 55.18% (compared to the 20.00% cutoff). Flagging antenna 268n with a relative difference before and after smoothing of 50.03% (compared to the 20.00% cutoff). Flagging antenna 269e with a relative difference before and after smoothing of 46.17% (compared to the 20.00% cutoff). Flagging antenna 269n with a relative difference before and after smoothing of 47.18% (compared to the 20.00% cutoff). Flagging antenna 272e with a relative difference before and after smoothing of 40.79% (compared to the 20.00% cutoff). Flagging antenna 272n with a relative difference before and after smoothing of 36.83% (compared to the 20.00% cutoff). Flagging antenna 273n with a relative difference before and after smoothing of 32.29% (compared to the 20.00% cutoff). Flagging antenna 281e with a relative difference before and after smoothing of 51.81% (compared to the 20.00% cutoff). Flagging antenna 281n with a relative difference before and after smoothing of 55.16% (compared to the 20.00% cutoff). Flagging antenna 282e with a relative difference before and after smoothing of 50.10% (compared to the 20.00% cutoff).
Flagging antenna 282n with a relative difference before and after smoothing of 53.29% (compared to the 20.00% cutoff). Flagging antenna 283e with a relative difference before and after smoothing of 48.82% (compared to the 20.00% cutoff). Flagging antenna 284e with a relative difference before and after smoothing of 47.12% (compared to the 20.00% cutoff). Flagging antenna 284n with a relative difference before and after smoothing of 45.47% (compared to the 20.00% cutoff). Flagging antenna 285e with a relative difference before and after smoothing of 44.82% (compared to the 20.00% cutoff). Flagging antenna 285n with a relative difference before and after smoothing of 43.65% (compared to the 20.00% cutoff). Flagging antenna 299e with a relative difference before and after smoothing of 44.41% (compared to the 20.00% cutoff). Flagging antenna 299n with a relative difference before and after smoothing of 41.66% (compared to the 20.00% cutoff). Flagging antenna 301e with a relative difference before and after smoothing of 38.15% (compared to the 20.00% cutoff). Flagging antenna 301n with a relative difference before and after smoothing of 35.43% (compared to the 20.00% cutoff). Flagging antenna 302n with a relative difference before and after smoothing of 31.30% (compared to the 20.00% cutoff). Flagging antenna 306e with a relative difference before and after smoothing of 20.03% (compared to the 20.00% cutoff). Flagging antenna 313n with a relative difference before and after smoothing of 38.30% (compared to the 20.00% cutoff). Flagging antenna 315e with a relative difference before and after smoothing of 33.26% (compared to the 20.00% cutoff). Flagging antenna 316e with a relative difference before and after smoothing of 29.77% (compared to the 20.00% cutoff). Flagging antenna 316n with a relative difference before and after smoothing of 27.54% (compared to the 20.00% cutoff). Flagging antenna 317e with a relative difference before and after smoothing of 26.87% (compared to the 20.00% cutoff). Flagging antenna 319e with a relative difference before and after smoothing of 20.43% (compared to the 20.00% cutoff). Flagging antenna 319n with a relative difference before and after smoothing of 21.20% (compared to the 20.00% cutoff).
In [19]:
if not PER_POL_REFANT:
# put back in the smoothed phasor, ensuring the amplitude is 1 and that data are flagged anywhere either polarization's refant is flagged
smoothed_relative_pol_phasor = cs.gain_grids[(-1, other_refant[-1])] / np.abs(cs.gain_grids[(-1, other_refant[-1])])
for ant in cs.gain_grids:
if ant[0] >= 0 and ant[1] == other_refant[1]:
cs.gain_grids[ant] /= smoothed_relative_pol_phasor
cs.flag_grids[ant] |= (cs.flag_grids[(-1, other_refant[1])])
cs.refant = overall_refant
In [20]:
def phase_flip_diagnostic_plot():
'''Shows time-smoothed antenna avg phases after taking out a delay and filtering in time.'''
if not np.any([np.any(meta['phase_flipped'][ant]) for ant in meta['phase_flipped']]):
print("No antennas have phase flips identified. Nothing to plot.")
return
plt.figure(figsize=(14,4))
for ant in meta['phase_flipped']:
if np.any(meta['phase_flipped'][ant]):
to_plot = np.angle(np.exp(1.0j * (meta['phases'][ant] - meta['time_smoothed_phases'][ant])))
to_plot[to_plot < -np.pi / 2] += 2 * np.pi
plt.plot(cs.time_grid - int(cs.time_grid[0]), to_plot, label=f'{ant[0]}{ant[1][-1]}')
plt.legend(title='Antennas with Identified Phase Flips', ncol=4)
plt.xlabel(f'JD - {int(cs.time_grid[0])}')
plt.ylabel('Average Phase After Filtering (radians)')
plt.tight_layout()
Figure 2: Antenna Phases with Identified Phase Flips¶
In [21]:
phase_flip_diagnostic_plot()
Plot results¶
In [22]:
def amplitude_plot(ant_to_plot):
with warnings.catch_warnings():
warnings.simplefilter("ignore")
# Pick vmax to not saturate 90% of the abscal gains
vmax = np.max([np.percentile(np.abs(cs.gain_grids[ant_to_plot, pol][~cs.flag_grids[ant_to_plot, pol]]), 99) for pol in ['Jee', 'Jnn']])
display(HTML(f'<h2>Antenna {ant_to_plot} Amplitude Waterfalls</h2>'))
# Plot abscal gain amplitude waterfalls for a single antenna
fig, axes = plt.subplots(4, 2, figsize=(14,14), gridspec_kw={'height_ratios': [1, 1, .4, .4]})
for ax, pol in zip(axes[0], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
extent=[cs.freqs[0]/1e6, cs.freqs[-1]/1e6, lst_grid[-1], lst_grid[0]]
im = ax.imshow(np.where(cs.flag_grids[ant], np.nan, np.abs(cs.gain_grids[ant])), aspect='auto', cmap='inferno',
interpolation='nearest', vmin=0, vmax=vmax, extent=extent)
ax.set_title(f'Smoothcal Gain Amplitude of Antenna {ant[0]}: {pol[-1]}-polarized' )
ax.set_xlabel('Frequency (MHz)')
ax.set_ylabel('LST (Hours)')
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_yticklabels(ax.get_yticks() % 24)
plt.colorbar(im, ax=ax, orientation='horizontal', pad=.15)
# Now flagged plot abscal waterfall
for ax, pol in zip(axes[1], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
extent=[cs.freqs[0]/1e6, cs.freqs[-1]/1e6, lst_grid[-1], lst_grid[0]]
im = ax.imshow(np.where(cs.flag_grids[ant], np.nan, np.abs(abscal_gains[ant])), aspect='auto', cmap='inferno',
interpolation='nearest', vmin=0, vmax=vmax, extent=extent)
ax.set_title(f'Abscal Gain Amplitude of Antenna {ant[0]}: {pol[-1]}-polarized' )
ax.set_xlabel('Frequency (MHz)')
ax.set_ylabel('LST (Hours)')
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_yticklabels(ax.get_yticks() % 24)
plt.colorbar(im, ax=ax, orientation='horizontal', pad=.15)
# Now plot mean gain spectra
for ax, pol in zip(axes[2], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
nflags_spectrum = np.sum(cs.flag_grids[ant], axis=0)
to_plot = nflags_spectrum <= np.percentile(nflags_spectrum, 75)
ax.plot(cs.freqs[to_plot] / 1e6, np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.abs(abscal_gains[ant])), axis=0)[to_plot], 'r.', label='Abscal')
ax.plot(cs.freqs[to_plot] / 1e6, np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.abs(cs.gain_grids[ant])), axis=0)[to_plot], 'k.', ms=2, label='Smoothed')
ax.set_ylim([0, vmax])
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_xlabel('Frequency (MHz)')
ax.set_ylabel('|g| (unitless)')
ax.set_title(f'Mean Infrequently-Flagged Gain Amplitude of Antenna {ant[0]}: {pol[-1]}-polarized')
ax.legend(loc='upper left')
# Now plot mean gain time series
for ax, pol in zip(axes[3], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
nflags_series = np.sum(cs.flag_grids[ant], axis=1)
to_plot = nflags_series <= np.percentile(nflags_series, 75)
ax.plot(lst_grid[to_plot], np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.abs(abscal_gains[ant])), axis=1)[to_plot], 'r.', label='Abscal')
ax.plot(lst_grid[to_plot], np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.abs(cs.gain_grids[ant])), axis=1)[to_plot], 'k.', ms=2, label='Smoothed')
ax.set_ylim([0, vmax])
ax.set_xlabel('LST (hours)')
ax.set_ylabel('|g| (unitless)')
ax.set_title(f'Mean Infrequently-Flagged Gain Amplitude of Antenna {ant[0]}: {pol[-1]}-polarized')
ax.set_xticklabels(ax.get_xticks() % 24)
ax.legend(loc='upper left')
plt.tight_layout()
plt.show()
In [23]:
def phase_plot(ant_to_plot):
with warnings.catch_warnings():
warnings.simplefilter("ignore")
display(HTML(f'<h2>Antenna {ant_to_plot} Phase Waterfalls</h2>'))
fig, axes = plt.subplots(4, 2, figsize=(14,14), gridspec_kw={'height_ratios': [1, 1, .4, .4]})
# Plot phase waterfalls for a single antenna
for ax, pol in zip(axes[0], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
extent=[cs.freqs[0]/1e6, cs.freqs[-1]/1e6, lst_grid[-1], lst_grid[0]]
im = ax.imshow(np.where(cs.flag_grids[ant], np.nan, np.angle(cs.gain_grids[ant])), aspect='auto', cmap='inferno',
interpolation='nearest', vmin=-np.pi, vmax=np.pi, extent=extent)
refant = (cs.refant[pol] if isinstance(cs.refant, dict) else cs.refant)
ax.set_title(f'Smoothcal Gain Phase of Ant {ant[0]}{pol[-1]} / Ant {refant[0]}{refant[1][-1]}')
ax.set_xlabel('Frequency (MHz)')
ax.set_ylabel('LST (Hours)')
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_yticklabels(ax.get_yticks() % 24)
plt.colorbar(im, ax=ax, orientation='horizontal', pad=.15)
# Now plot abscal phase waterfall
for ax, pol in zip(axes[1], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
extent=[cs.freqs[0]/1e6, cs.freqs[-1]/1e6, lst_grid[-1], lst_grid[0]]
im = ax.imshow(np.where(cs.flag_grids[ant], np.nan, np.angle(abscal_gains[ant])), aspect='auto', cmap='inferno',
interpolation='nearest', vmin=-np.pi, vmax=np.pi, extent=extent)
refant = (cs.refant[pol] if isinstance(cs.refant, dict) else cs.refant)
ax.set_title(f'Abscal Gain Phase of Ant {ant[0]}{pol[-1]} / Ant {refant[0]}{refant[1][-1]}')
ax.set_xlabel('Frequency (MHz)')
ax.set_ylabel('LST (Hours)')
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_yticklabels(ax.get_yticks() % 24)
plt.colorbar(im, ax=ax, orientation='horizontal', pad=.15)
# Now plot median gain spectra
for ax, pol in zip(axes[2], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
nflags_spectrum = np.sum(cs.flag_grids[ant], axis=0)
to_plot = nflags_spectrum <= np.percentile(nflags_spectrum, 75)
ax.plot(cs.freqs[to_plot] / 1e6, np.nanmedian(np.where(cs.flag_grids[ant], np.nan, np.angle(abscal_gains[ant])), axis=0)[to_plot], 'r.', label='Abscal')
ax.plot(cs.freqs[to_plot] / 1e6, np.nanmedian(np.where(cs.flag_grids[ant], np.nan, np.angle(cs.gain_grids[ant])), axis=0)[to_plot], 'k.', ms=2, label='Smoothed')
ax.set_ylim([-np.pi, np.pi])
ax.set_xlim([cs.freqs[0]/1e6, cs.freqs[-1]/1e6])
ax.set_xlabel('Frequency (MHz)')
refant = (cs.refant[pol] if isinstance(cs.refant, dict) else cs.refant)
ax.set_ylabel(f'Phase of g$_{{{ant[0]}{pol[-1]}}}$ / g$_{{{refant[0]}{refant[1][-1]}}}$')
ax.set_title(f'Median Infrequently-Flagged Gain Phase of Ant {ant[0]}{pol[-1]} / Ant {refant[0]}{refant[1][-1]}')
ax.legend(loc='upper left')
# # Now plot median gain time series
for ax, pol in zip(axes[3], ['Jee', 'Jnn']):
ant = (ant_to_plot, pol)
nflags_series = np.sum(cs.flag_grids[ant], axis=1)
to_plot = nflags_series <= np.percentile(nflags_series, 75)
ax.plot(lst_grid[to_plot], np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.angle(abscal_gains[ant])), axis=1)[to_plot], 'r.', label='Abscal')
ax.plot(lst_grid[to_plot], np.nanmean(np.where(cs.flag_grids[ant], np.nan, np.angle(cs.gain_grids[ant])), axis=1)[to_plot], 'k.', ms=2, label='Smoothed')
ax.set_ylim([-np.pi, np.pi])
ax.set_xlabel('LST (hours)')
refant = (cs.refant[pol] if isinstance(cs.refant, dict) else cs.refant)
ax.set_ylabel(f'Phase of g$_{{{ant[0]}{pol[-1]}}}$ / g$_{{{refant[0]}{refant[1][-1]}}}$')
ax.set_title(f'Mean Infrequently-Flagged Gain Phase of Ant {ant[0]}{pol[-1]} / Ant {refant[0]}{refant[1][-1]}')
ax.set_xticklabels(ax.get_xticks() % 24)
ax.legend(loc='upper left')
plt.tight_layout()
plt.show()
In [24]:
# Select first 2 unflagged antennas from candidates for amplitude plotting
ants_to_plot = []
for ant_candidate in ants_to_plot_candidates:
if not (np.all(cs.flag_grids[ant_candidate, 'Jee']) and np.all(cs.flag_grids[ant_candidate, 'Jnn'])):
ants_to_plot.append(ant_candidate)
if len(ants_to_plot) >= 2:
break
Figure 3: Full-Day Gain Amplitudes Before and After smooth_cal¶
Here we plot abscal and smooth_cal gain amplitudes for both of the sample antennas. We also show means across time/frequency, excluding frequencies/times that are frequently flagged.
In [25]:
if len(ants_to_plot) == 0:
print("Warning: No unflagged antennas available for plotting.")
else:
for ant_to_plot in ants_to_plot:
amplitude_plot(ant_to_plot)
Antenna 79 Amplitude Waterfalls
Antenna 229 Amplitude Waterfalls
Figure 4: Full-Day Gain Phases Before and After smooth_cal¶
Here we plot abscal and smooth_cal phases relative to each polarization's reference antenna for both of the sample antennas. We also show medians across time/frequency, excluding frequencies/times that are frequently flagged.
In [26]:
# Use the same selected unflagged antennas for phase plotting
if len(ants_to_plot) == 0:
print("Warning: No unflagged antennas available for plotting.")
else:
for ant_to_plot in ants_to_plot:
phase_plot(ant_to_plot)
Antenna 79 Phase Waterfalls
Antenna 229 Phase Waterfalls
Examine $\chi^2$¶
In [27]:
def chisq_plot():
fig, axes = plt.subplots(1, 2, figsize=(14, 10), sharex=True, sharey=True)
extent = [cs.freqs[0]/1e6, cs.freqs[-1]/1e6, lst_grid[-1], lst_grid[0]]
for ax, pol in zip(axes, ['Jee', 'Jnn']):
refant = (cs.refant[pol] if isinstance(cs.refant, dict) else cs.refant)
im = ax.imshow(np.where(cs.flag_grids[refant], np.nan, cs.chisq_grids[pol]), vmin=1, vmax=5,
aspect='auto', cmap='turbo', interpolation='none', extent=extent)
ax.set_yticklabels(ax.get_yticks() % 24)
ax.set_title(f'{pol[1:]}-Polarized $\\chi^2$ / DoF')
ax.set_xlabel('Frequency (MHz)')
axes[0].set_ylabel('LST (hours)')
plt.tight_layout()
fig.colorbar(im, ax=axes, pad=.07, label='$\\chi^2$ / DoF', orientation='horizontal', extend='both', aspect=50)
Figure 5: Full-Day $\chi^2$ / DoF Waterfall from Redundant-Baseline Calibration¶
Here we plot $\chi^2$ per degree of freedom from redundant-baseline calibration for both polarizations separately. While this plot is a little out of place, as it was not produced by this notebook, it is a convenient place where all the necessary components are readily available. If the array were perfectly redundant and any non-redundancies in the calibrated visibilities were explicable by thermal noise alone, this waterfall should be all 1.
In [28]:
chisq_plot()
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
In [29]:
def cspa_vs_time_plot():
fig, axes = plt.subplots(2, 1, figsize=(14, 6), sharex=True, sharey=True, gridspec_kw={'hspace': 0})
for ax, pol in zip(axes, ['Jee', 'Jnn']):
detail_cutoff = np.percentile([np.nanmean(m) for ant, m in avg_cspa_vs_time.items()
if ant[1] == pol and np.isfinite(np.nanmean(m))], 95)
for ant in avg_cspa_vs_time:
if ant[1] == pol and not np.all(cs.flag_grids[ant]):
if np.nanmean(avg_cspa_vs_time[ant]) > detail_cutoff:
ax.plot(lst_grid, avg_cspa_vs_time[ant], label=str((int(ant[0]), ant[1])), zorder=100)
else:
ax.plot(lst_grid, avg_cspa_vs_time[ant], c='grey', alpha=.2, lw=.5)
ax.legend(title=f'{pol[1:]}-Polarized', ncol=2)
ax.set_ylabel('Mean Unflagged $\\chi^2$ per Antenna')
ax.set_xlabel('LST (hours)')
ax.set_xticklabels(ax.get_xticks() % 24)
plt.ylim([1, 5.4])
plt.tight_layout()
In [30]:
def cspa_vs_freq_plot():
fig, axes = plt.subplots(2, 1, figsize=(14, 6), sharex=True, sharey=True, gridspec_kw={'hspace': 0})
for ax, pol in zip(axes, ['Jee', 'Jnn']):
detail_cutoff = np.percentile([np.nanmean(m) for ant, m in avg_cspa_vs_freq.items()
if ant[1] == pol and np.isfinite(np.nanmean(m))], 95)
for ant in avg_cspa_vs_freq:
if ant[1] == pol and not np.all(cs.flag_grids[ant]):
if np.nanmean(avg_cspa_vs_freq[ant]) > detail_cutoff:
ax.plot(cs.freqs / 1e6, avg_cspa_vs_freq[ant], label=str((int(ant[0]), ant[1])), zorder=100)
else:
ax.plot(cs.freqs / 1e6, avg_cspa_vs_freq[ant], c='grey', alpha=.2, lw=.5)
ax.legend(title=f'{pol[1:]}-Polarized', ncol=2)
ax.set_ylabel('Mean Unflagged $\\chi^2$ per Antenna')
ax.set_xlabel('Frequency (MHz)')
plt.ylim([1, 5.4])
plt.tight_layout()
In [31]:
def avg_cspa_array_plot():
hd = io.HERAData(SUM_FILE)
fig, axes = plt.subplots(1, 2, figsize=(14, 8), sharex=True, sharey=True, gridspec_kw={'wspace': 0})
for pol, ax in zip(['Jee', 'Jnn'], axes):
ants_here = [ant for ant in avg_cspa if np.isfinite(avg_cspa[ant]) and ant[1] == pol if ant[0] in hd.antpos]
avg_chisqs = [avg_cspa[ant] for ant in ants_here]
xs = [hd.antpos[ant[0]][0] for ant in ants_here]
ys = [hd.antpos[ant[0]][1] for ant in ants_here]
names = [ant[0] for ant in ants_here]
im = ax.scatter(x=xs, y=ys, c=avg_chisqs, s=200, vmin=1, vmax=3, cmap='turbo')
ax.set_aspect('equal')
for x,y,n in zip(xs, ys, names):
ax.text(x, y, str(n), va='center', ha='center', fontsize=8)
ax.set_title(pol)
ax.set_xlabel('East-West Antenna Position (m)')
axes[0].set_ylabel('North-South Antenna Position (m)')
plt.tight_layout()
plt.colorbar(im, ax=axes, location='top', aspect=60, pad=.04, label='Mean Unflagged $\\chi^2$ per Antenna', extend='both')
Figure 6: Average $\chi^2$ per Antenna¶
Here we plot $\chi^2$ per antenna from redundant-baseline calibration, separating polarizations and averaging the unflagged pixels in the waterfalls over frequency or time. The worst 5% of antennas are shown in color and highlighted in the legends, the rest are shown in grey. We also show time- and frequency-averaged $\chi^2$ for each antennas as a scatter plot with array position.
In [32]:
cspa_vs_freq_plot()
cspa_vs_time_plot()
avg_cspa_array_plot()
Mean of empty slice
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. Mean of empty slice set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
Examine relative differences before and after smoothing¶
In [33]:
def time_avg_diff_plot():
fig, axes = plt.subplots(2, 1, figsize=(14, 6), sharex=True, sharey=True, gridspec_kw={'hspace': 0})
for ax, pol in zip(axes, ['Jee', 'Jnn']):
detail_cutoff = np.percentile([np.nanmean(diff) for ant, diff in meta['time_avg_rel_diff'].items()
if ant[1] == pol and np.isfinite(np.nanmean(diff))], 95)
for ant, rel_diff in meta['time_avg_rel_diff'].items():
if ant[0] >= 0 and ant[1] == pol and np.any(np.isfinite(rel_diff)):
if np.nanmean(rel_diff) > detail_cutoff:
if np.all(cs.flag_grids[ant]):
ax.plot(cs.freqs / 1e6, rel_diff, label=str((int(ant[0]), ant[1])), zorder=99, ls='--', c='r', lw=.5)
else:
ax.plot(cs.freqs / 1e6, rel_diff, label=str((int(ant[0]), ant[1])), zorder=100)
else:
ax.plot(cs.freqs / 1e6, rel_diff, c='grey', alpha=.2, lw=.5)
med_rel_diff = np.nanmedian([diff for ant, diff in meta['time_avg_rel_diff'].items() if ant[1] == pol], axis=0)
ax.plot(cs.freqs / 1e6, med_rel_diff, 'k--', label='Median')
ax.set_ylim([0, 1.05])
ax.legend(title=f'{pol[1:]}-Polarized', ncol=2)
ax.set_ylabel('Time-Averaged Relative Difference\nBefore and After Smoothing')
ax.set_xlabel('Frequency (MHz)')
plt.tight_layout()
In [34]:
def freq_avg_diff_plot():
fig, axes = plt.subplots(2, 1, figsize=(14, 6), sharex=True, sharey=True, gridspec_kw={'hspace': 0})
for ax, pol in zip(axes, ['Jee', 'Jnn']):
detail_cutoff = np.percentile([np.nanmean(m) for ant, m in meta['freq_avg_rel_diff'].items()
if ant[1] == pol and np.isfinite(np.nanmean(m))], 95)
for ant, rel_diff in meta['freq_avg_rel_diff'].items():
if ant[0] >= 0 and ant[1] == pol and np.any(np.isfinite(rel_diff)):
if np.nanmean(rel_diff) > detail_cutoff:
if np.all(cs.flag_grids[ant]):
ax.plot(lst_grid, rel_diff, label=str((int(ant[0]), ant[1])), zorder=99, ls='--', c='r', lw=.5)
else:
ax.plot(lst_grid, rel_diff, label=str((int(ant[0]), ant[1])), zorder=100)
else:
ax.plot(lst_grid, rel_diff, c='grey', alpha=.2, lw=.5)
med_rel_diff = np.nanmedian([diff for ant, diff in meta['freq_avg_rel_diff'].items() if ant[1] == pol], axis=0)
ax.plot(lst_grid, med_rel_diff, 'k--', label='Median', zorder=101)
ax.set_ylim([0, 1.05])
ax.legend(title=f'{pol[1:]}-Polarized', ncol=2)
ax.set_ylabel('Frequency-Averaged Relative Difference\nBefore and After Smoothing')
ax.set_xlabel('LST (hours)')
ax.set_xticklabels(ax.get_xticks() % 24)
plt.tight_layout()
In [35]:
def avg_difference_array_plot():
hd = io.HERAData(SUM_FILE)
fig, axes = plt.subplots(1, 2, figsize=(14, 8), sharex=True, sharey=True, gridspec_kw={'wspace': 0})
for pol, ax in zip(['Jee', 'Jnn'], axes):
avg_diffs = [np.nanmean(meta['time_avg_rel_diff'][ant]) for ant in meta['time_avg_rel_diff'] if ant[1] == pol if ant[0] in hd.antpos]
xs = [hd.antpos[ant[0]][0] for ant in meta['time_avg_rel_diff'] if ant[1] == pol if ant[0] in hd.antpos]
ys = [hd.antpos[ant[0]][1] for ant in meta['time_avg_rel_diff'] if ant[1] == pol if ant[0] in hd.antpos]
names = [ant[0] for ant in meta['time_avg_rel_diff'] if ant[1] == pol if ant[0] in hd.antpos]
im = ax.scatter(x=xs, y=ys, c=avg_diffs, s=200, vmin=0, vmax=.25, cmap='turbo')
ax.set_aspect('equal')
for x,y,n in zip(xs, ys, names):
color = ('w' if np.all(cs.flag_grids[n, pol]) else 'k')
ax.text(x, y, str(n), va='center', ha='center', fontsize=8, c=color)
ax.set_title(pol)
ax.set_xlabel('East-West Antenna Position (m)')
axes[0].set_ylabel('North-South Antenna Position (m)')
plt.tight_layout()
plt.colorbar(im, ax=axes, location='top', aspect=60, pad=.04, label='Average Relative Difference Before and After Smoothing', extend='max')
Figure 7: Relative Difference Before and After Smoothing¶
Similar to the above plots, here we show the relative difference before and after smoothing, compared to the magnitude of the smoothed calibration solution. Totally flagged antennas (because they are above the SC_RELATIVE_DIFF_CUTOFF) are red in the first two plots, and their numbers are white in the last plot.
In [36]:
time_avg_diff_plot()
freq_avg_diff_plot()
avg_difference_array_plot()
All-NaN slice encountered
All-NaN slice encountered set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. All-NaN slice encountered set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
Save Results¶
In [37]:
add_to_history = 'Produced by calibration_smoothing notebook with the following environment:\n' + '=' * 65 + '\n' + os.popen('conda env export').read() + '=' * 65
In [38]:
cs.write_smoothed_cal(output_replace=(CAL_SUFFIX, SMOOTH_CAL_SUFFIX), add_to_history=add_to_history, clobber=True)
Mean of empty slice
invalid value encountered in multiply invalid value encountered in divide
Metadata¶
In [39]:
for repo in ['hera_cal', 'hera_qm', 'hera_filters', 'hera_notebook_templates', 'pyuvdata']:
exec(f'from {repo} import __version__')
print(f'{repo}: {__version__}')
hera_cal: 3.7.7.dev97+gc2668d3f7 hera_qm: 2.2.1.dev4+gf6d02113b hera_filters: 0.1.7
hera_notebook_templates: 0.0.1.dev1313+g92178a09c pyuvdata: 3.2.5.dev1+g5a985ae31
In [40]:
print(f'Finished execution in {(time.time() - tstart) / 60:.2f} minutes.')
Finished execution in 48.67 minutes.