In [1]:
import os
os.environ['HDF5_USE_FILE_LOCKING'] = 'FALSE'
import h5py
import hdf5plugin
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import FormatStrFormatter
import matplotlib.patches as mpatches
import matplotlib.gridspec as gridspec
import numpy as np
from pyuvdata import UVCal, UVData
import sys
import glob
import uvtools as uvt
from astropy.time import Time
from astropy.coordinates import EarthLocation, AltAz, Angle
from astropy.coordinates import SkyCoord as sc
import pandas
import warnings 
import copy
from hera_notebook_templates import utils
import hera_qm
from hera_mc import cm_hookup
import importlib
from scipy import stats
from IPython.display import display, HTML
#warnings.filterwarnings('ignore')

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
#get data location
data_path = os.environ['DATA_PATH']
print(f'DATA_PATH = "{data_path}"')
statuses = os.environ['APRIORI_STATUSES']
print(f'APRIORI_STATUSES = {statuses}')
JD = os.environ['JULIANDATE']
print(f'JULIANDATE = {JD}')
utc = Time(JD, format='jd').datetime
print(f'Date = {utc.month}-{utc.day}-{utc.year}')
DATA_PATH = "/mnt/sn1/2460276"
APRIORI_STATUSES = dish_maintenance,dish_ok,RF_maintenance,RF_ok,digital_ok,digital_maintenance,calibration_maintenance,calibration_triage,calibration_ok
JULIANDATE = 2460276
Date = 11-27-2023
In [3]:
# Load in data
HHfiles, difffiles, HHautos, diffautos, uvdx, uvdy = utils.load_data(data_path,JD)
    
uvd = UVData()
unread = True
readInd=0
while unread and readInd<len(HHautos):
    try:
        uvd.read(HHautos[readInd])
        unread = False
    except:
        readInd += 1
        continue
use_ants = utils.get_use_ants(uvd,statuses,JD)
print(f'This day contains {len(use_ants)} antennas of the given status category.')
uvd.read(HHautos[::10], skip_bad_files=True, antenna_nums = use_ants)
lsts = uvd.lst_array
uvdx.select(antenna_nums=use_ants)
uvdy.select(antenna_nums=use_ants)
1675 sum files found between JDs 2460276.25452 and 2460276.64756
1675 diff files found between JDs 2460276.25452 and 2460276.64756
1674 sum auto files found between JDs 2460276.25452 and 2460276.64756
1674 diff auto files found between JDs 2460276.25452 and 2460276.64756
This day contains 184 antennas of the given status category.

Sky Coverage Map¶

Map of the sky (made using the Haslam 408MHz map). The RA/DEC range covered by this night of observation is shaded based on a 12 degree FWHM of the beam. Horizontal dashed lines represent the stripe that HERA can observe, while the shaded region is what was observed on this night. Vertical lines represent the beginning and ending LSTs of this observation. Selected sources are labelled, sources included are those in the GLEAM 4Jy catalog with a flux >10.9 Jy. Note that the map is clipped at the northern horizon.

In [4]:
sources = utils.gather_source_list()
utils.plot_sky_map(uvd,dec_pad=55,ra_pad=55,clip=False,sources=sources)
No description has been provided for this image

LST Coverage¶

Shows the LSTs (in hours) and JDs for which data is collected. Green represents data, red means no data.

In [5]:
utils.plot_lst_coverage(uvd)
No description has been provided for this image

Autocorrelations for a single file¶

This plot shows autocorrelations for one timestamp of each antenna that is active and each polarization. For each node, antennas are ordered by SNAP number, and within that by SNAP input number. The antenna number label color corresponds to the a priori status of that antenna.

In [6]:
### plot autos
utils.plot_autos(uvdx, uvdy)
No description has been provided for this image

Waterfalls of Autocorrelation Amplitudes for each Antenna and Each polarization¶

These plots show autocorrelation waterfalls of each antenna that is active and whose status qualifies for this notebook. For each node, antennas are ordered by SNAP number, and within that by SNAP input number. The antenna number label color corresponds to the a priori status of that antenna.

In [7]:
utils.plot_wfs(uvd, pol = 0)
No description has been provided for this image
In [8]:
utils.plot_wfs(uvd, pol = 1)
No description has been provided for this image

Correlation Metrics¶

The first plot shows the correlation metric (described below) for a set of baseline types, as calculated at several times throughout the night. It is expected that longer baselines (darker color) will exhibit lower values than the short baselines.

The matrices show the phase correlation between antennas. Using the even and odd visibilities, each pixel is calculated as (even/abs(even)) * (conj(odd)/abs(odd)), and then averaged across time and frequency. If the phases are noise-like, this value will average down to zero. If the antennas are well correlated, the phases should not be noise-like, and this value should average to 1. The lines denoting node boundaries are intended to help confirm that inter-node correlations are functioning - if they aren't, this plot will appear block-diagonal.

This metric has shown to be LST locked - when comparing to other nights, be sure to compare for the same LST. It is expected that some LSTs will look much better or worse than others.

Note: Within each node, the order of antennas is determined by snap, and within that by snap input number.

In [9]:
badAnts = []
badAnts = utils.plotNodeAveragedSummary(uvd,HHfiles,JD,use_ants,mat_pols=['xx','yy','xy','yx'])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [9], line 2
      1 badAnts = []
----> 2 badAnts = utils.plotNodeAveragedSummary(uvd,HHfiles,JD,use_ants,mat_pols=['xx','yy','xy','yx'])

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:811, in plotNodeAveragedSummary(uv, HHfiles, jd, use_ants, pols, mat_pols, baseline_groups, removeBadAnts, plotRatios, plotSummary)
    807 if baseline_groups == []:
    808     baseline_groups = [(14,0,'14m E-W'),(14,-11,'14m NW-SE'),(14,11,'14m SW-NE'),(29,0,'29m E-W'),(29,22,'29m SW-NE'),
    809                    (44,0,'44m E-W'),(58.5,0,'58m E-W'),(73,0,'73m E-W'),(87.6,0,'88m E-W'),
    810                   (102.3,0,'102m E-W')]
--> 811 nodeMedians,lsts,badAnts=get_correlation_baseline_evolutions(uv,HHfiles,jd,use_ants,pols=pols,mat_pols=mat_pols,
    812                                                             bl_type=baseline_groups,removeBadAnts=removeBadAnts,
    813                                                             plotRatios=plotRatios)
    814 pols = mat_pols
    815 if plotSummary is False:

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:1477, in get_correlation_baseline_evolutions(uv, HHfiles, jd, use_ants, badThresh, pols, bl_type, removeBadAnts, plotMatrix, mat_pols, plotRatios)
   1444 def get_correlation_baseline_evolutions(uv,HHfiles,jd,use_ants='auto',badThresh=0.35,pols=['xx','yy'],bl_type=(14,0,'14m E-W'),
   1445                                         removeBadAnts=False, plotMatrix=True,mat_pols=['xx','yy','xy','yx'],plotRatios=False):
   1446     """
   1447     Calculates the average correlation metric for a set of redundant baseline groups at one hour intervals throughout a night of observation.
   1448     
   (...)
   1475         Antenna numbers flagged as bad based on badThresh parameter.
   1476     """
-> 1477     files, lsts, inds = get_hourly_files(uv, HHfiles, jd)
   1478     if use_ants == 'auto':
   1479         use_ants = uv.get_ants()

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:1388, in get_hourly_files(uv, HHfiles, jd)
   1386 try:
   1387     dat = UVData()
-> 1388     dat.read(file, read_data=False)
   1389 except KeyError:
   1390     continue

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:12716, in UVData.read(self, filename, axis, file_type, read_data, skip_bad_files, background_lsts, ignore_name, use_future_array_shapes, allow_rephase, make_multi_phase, antenna_nums, antenna_names, ant_str, bls, catalog_names, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, keep_all_metadata, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, check_autos, fix_autos, phase_type, projected, correct_lat_lon, calc_lst, fix_old_proj, fix_use_ant_pos, use_model, data_column, pol_order, ignore_single_chan, raise_error, read_weights, allow_flex_pol, multidim_index, remove_flex_pol, blt_order, blts_are_rectangular, time_axis_faster_than_bls, data_array_dtype, use_aoflagger_flags, remove_dig_gains, remove_coarse_band, correct_cable_len, correct_van_vleck, cheby_approx, flag_small_auto_ants, propagate_coarse_flags, flag_init, edge_width, start_flag, end_flag, flag_dc_offset, remove_flagged_ants, phase_to_pointing_center, nsample_array_dtype, corrchunk, receivers, sidebands, mir_select_where, apply_tsys, apply_flags, apply_dedoppler, pseudo_cont, rechunk, recompute_nbls)
  12697     self.read_ms(
  12698         filename,
  12699         data_column=data_column,
   (...)
  12712         use_future_array_shapes=use_future_array_shapes,
  12713     )
  12715 elif file_type == "uvh5":
> 12716     self.read_uvh5(
  12717         filename,
  12718         antenna_nums=antenna_nums,
  12719         antenna_names=antenna_names,
  12720         ant_str=ant_str,
  12721         bls=bls,
  12722         frequencies=frequencies,
  12723         freq_chans=freq_chans,
  12724         times=times,
  12725         time_range=time_range,
  12726         lsts=lsts,
  12727         lst_range=lst_range,
  12728         polarizations=polarizations,
  12729         blt_inds=blt_inds,
  12730         phase_center_ids=phase_center_ids,
  12731         catalog_names=catalog_names,
  12732         read_data=read_data,
  12733         data_array_dtype=data_array_dtype,
  12734         keep_all_metadata=keep_all_metadata,
  12735         multidim_index=multidim_index,
  12736         remove_flex_pol=remove_flex_pol,
  12737         background_lsts=background_lsts,
  12738         run_check=run_check,
  12739         check_extra=check_extra,
  12740         run_check_acceptability=run_check_acceptability,
  12741         strict_uvw_antpos_check=strict_uvw_antpos_check,
  12742         fix_old_proj=fix_old_proj,
  12743         fix_use_ant_pos=fix_use_ant_pos,
  12744         check_autos=check_autos,
  12745         fix_autos=fix_autos,
  12746         use_future_array_shapes=use_future_array_shapes,
  12747         time_axis_faster_than_bls=time_axis_faster_than_bls,
  12748         blts_are_rectangular=blts_are_rectangular,
  12749         recompute_nbls=recompute_nbls,
  12750     )
  12751     select = False
  12753 if select:

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:11536, in UVData.read_uvh5(self, filename, antenna_nums, antenna_names, ant_str, bls, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, catalog_names, keep_all_metadata, read_data, data_array_dtype, multidim_index, remove_flex_pol, background_lsts, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, fix_old_proj, fix_use_ant_pos, check_autos, fix_autos, use_future_array_shapes, time_axis_faster_than_bls, blts_are_rectangular, blt_order, recompute_nbls)
  11529     raise ValueError(
  11530         "Reading multiple files from class specific "
  11531         "read functions is no longer supported. "
  11532         "Use the generic `uvdata.read` function instead."
  11533     )
  11535 uvh5_obj = uvh5.UVH5()
> 11536 uvh5_obj.read_uvh5(
  11537     filename,
  11538     antenna_nums=antenna_nums,
  11539     antenna_names=antenna_names,
  11540     ant_str=ant_str,
  11541     bls=bls,
  11542     frequencies=frequencies,
  11543     freq_chans=freq_chans,
  11544     times=times,
  11545     time_range=time_range,
  11546     lsts=lsts,
  11547     lst_range=lst_range,
  11548     polarizations=polarizations,
  11549     blt_inds=blt_inds,
  11550     phase_center_ids=phase_center_ids,
  11551     catalog_names=catalog_names,
  11552     data_array_dtype=data_array_dtype,
  11553     keep_all_metadata=keep_all_metadata,
  11554     read_data=read_data,
  11555     multidim_index=multidim_index,
  11556     remove_flex_pol=remove_flex_pol,
  11557     background_lsts=background_lsts,
  11558     run_check=run_check,
  11559     check_extra=check_extra,
  11560     run_check_acceptability=run_check_acceptability,
  11561     strict_uvw_antpos_check=strict_uvw_antpos_check,
  11562     fix_old_proj=fix_old_proj,
  11563     fix_use_ant_pos=fix_use_ant_pos,
  11564     check_autos=check_autos,
  11565     fix_autos=fix_autos,
  11566     use_future_array_shapes=use_future_array_shapes,
  11567     recompute_nbls=recompute_nbls,
  11568     blt_order=blt_order,
  11569     blts_are_rectangular=blts_are_rectangular,
  11570     time_axis_faster_than_bls=time_axis_faster_than_bls,
  11571 )
  11572 self._convert_from_filetype(uvh5_obj)
  11573 del uvh5_obj

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvh5.py:1659, in UVH5.read_uvh5(self, filename, antenna_nums, antenna_names, ant_str, bls, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, catalog_names, keep_all_metadata, read_data, data_array_dtype, multidim_index, remove_flex_pol, background_lsts, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, fix_old_proj, fix_use_ant_pos, check_autos, fix_autos, use_future_array_shapes, blt_order, blts_are_rectangular, time_axis_faster_than_bls, recompute_nbls)
   1657 # check if object has all required UVParameters set
   1658 if run_check:
-> 1659     self.check(
   1660         check_extra=check_extra,
   1661         run_check_acceptability=run_check_acceptability,
   1662         strict_uvw_antpos_check=strict_uvw_antpos_check,
   1663         allow_flip_conj=True,
   1664         check_autos=check_autos,
   1665         fix_autos=fix_autos,
   1666     )
   1668 return

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:3293, in UVData.check(self, check_extra, run_check_acceptability, check_freq_spacing, strict_uvw_antpos_check, allow_flip_conj, check_autos, fix_autos)
   3290 # Check internal consistency of numbers which don't explicitly correspond
   3291 # to the shape of another array.
   3292 if self.Nants_data != self._calc_nants_data():
-> 3293     raise ValueError(
   3294         "Nants_data must be equal to the number of unique "
   3295         "values in ant_1_array and ant_2_array"
   3296     )
   3298 if self.Nbls != len(np.unique(self.baseline_array)):
   3299     raise ValueError(
   3300         "Nbls must be equal to the number of unique "
   3301         f"baselines in the data_array. Got {self.Nbls}, not"
   3302         f"{len(np.unique(self.baseline_array))}"
   3303     )

ValueError: Nants_data must be equal to the number of unique values in ant_1_array and ant_2_array

Visibility amplitude spectra for a set of redundant baselines, labeled by inter vs. intranode baselines. The red and blue should exhibit the same bandpass shape - if the red are consistently different from the blue, this indicates an issue with internode correlations.

Note: antennas that were identified as bad by the correlation matrix have been removed from this plot.

In [10]:
utils.plotVisibilitySpectra(HHfiles[len(HHfiles)//2+1], JD, use_ants, badAnts=[])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/utils.py:4094, in get_baseline_redundancies(baselines, baseline_vecs, tol, include_conjugates)
   4093 try:
-> 4094     bl_gps = find_clusters(baselines, baseline_vecs, tol, strict=True)
   4095 except ValueError as exc:

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/utils.py:4033, in find_clusters(location_ids, location_vectors, tol, strict)
   4031 adj = _adj_list(location_vectors, tol)  # adj = list of sets
-> 4033 loc_gps = _find_cliques(adj, strict=strict)
   4034 loc_gps = [np.sort(location_ids[gp]).tolist() for gp in loc_gps]

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/utils.py:3999, in _find_cliques(adj, strict)
   3998     if not all(sorted(st) in loc_gps for st in adj):
-> 3999         raise ValueError("Non-isolated cliques found in graph.")
   4001 return loc_gps

ValueError: Non-isolated cliques found in graph.

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In [10], line 1
----> 1 utils.plotVisibilitySpectra(HHfiles[len(HHfiles)//2+1], JD, use_ants, badAnts=[])

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:873, in plotVisibilitySpectra(file, jd, use_ants, badAnts, pols)
    871 uv.read_uvh5(file)
    872 x = cm_hookup.get_hookup('default')
--> 873 baseline_groups = get_baseline_groups(uv,use_ants="auto")
    874 freqs = uv.freq_array[0]/1000000
    875 loc = EarthLocation.from_geocentric(*uv.telescope_location, unit='m')

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:1427, in get_baseline_groups(uv, bl_groups, use_ants)
   1410 """
   1411 Generate dictionary containing baseline groups.
   1412 
   (...)
   1423     Dictionary containing list of lists of redundant baseline numbers, formatted as bls[group label]
   1424 """
   1426 bls={}
-> 1427 baseline_groups,vec_bin_centers,lengths = uv.get_redundancies(use_antpos=False,include_autos=False)
   1428 for i in range(len(baseline_groups)):
   1429     bl = baseline_groups[i]

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:9980, in UVData.get_redundancies(self, tol, use_antpos, include_conjugates, include_autos, conjugate_bls)
   9977 baseline_vecs = np.take(self.uvw_array, unique_inds, axis=0)
   9978 baselines = np.take(self.baseline_array, unique_inds)
-> 9980 return uvutils.get_baseline_redundancies(
   9981     baselines, baseline_vecs, tol=tol, include_conjugates=include_conjugates
   9982 )

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/utils.py:4096, in get_baseline_redundancies(baselines, baseline_vecs, tol, include_conjugates)
   4094     bl_gps = find_clusters(baselines, baseline_vecs, tol, strict=True)
   4095 except ValueError as exc:
-> 4096     raise ValueError(
   4097         "Some baselines are falling into multiple"
   4098         " redundant groups. Lower the tolerance to resolve ambiguity."
   4099     ) from exc
   4101 n_unique = len(bl_gps)
   4102 vec_bin_centers = np.zeros((n_unique, 3))

ValueError: Some baselines are falling into multiple redundant groups. Lower the tolerance to resolve ambiguity.
<Figure size 640x480 with 0 Axes>

Even and Odd File Checks¶

A waterfall showing the ratio between the even and odd visibilities. The purpose of this is to highlight xengine failures, which will cause this value to fall to zero or go to infinity. If things are working properly, this value should be stable at 1. The boundaries between different x-engines are shown by the vertical white lines.

In [11]:
if len(HHautos) == len(diffautos):
    uvd_diff = UVData()
    uvd_diff.read(diffautos[::10], skip_bad_files=True, antenna_nums=use_ants)
    rat = utils.plotEvenOddWaterfalls(uvd,uvd_diff)
else:
    uvd_diff = UVData()
    use_diffs = [f for f in diffautos if '%s/zen.%s.%s.sum.autos.uvh5' % (data_path,f.split('.')[1],f.split('.')[2]) in HHautos[::10]]
    uvd_diff.read(use_diffs, skip_bad_files=True, antenna_nums = use_ants)
    uvd_sum = uvd.select(times=np.unique(uvd_diff.time_array),inplace=False)
    rat = utils.plotEvenOddWaterfalls(uvd_sum,uvd_diff)
No description has been provided for this image

Crossed Antenna Check¶

These are differences between different panels of the correlation matrices shown above (see panel titles for specifics). Antennas showing as consistently blue are ones which are correlating stronger in the cross pols than in the auto pols, indicating that the antenna polarizations are likely crossed.

In [12]:
crossedAnts = utils.plotNodeAveragedSummary(uvd,HHfiles,JD,use_ants,mat_pols=['xx','yy','xy','yx'],plotRatios=True,
                                       plotSummary=False)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [12], line 1
----> 1 crossedAnts = utils.plotNodeAveragedSummary(uvd,HHfiles,JD,use_ants,mat_pols=['xx','yy','xy','yx'],plotRatios=True,
      2                                        plotSummary=False)

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:811, in plotNodeAveragedSummary(uv, HHfiles, jd, use_ants, pols, mat_pols, baseline_groups, removeBadAnts, plotRatios, plotSummary)
    807 if baseline_groups == []:
    808     baseline_groups = [(14,0,'14m E-W'),(14,-11,'14m NW-SE'),(14,11,'14m SW-NE'),(29,0,'29m E-W'),(29,22,'29m SW-NE'),
    809                    (44,0,'44m E-W'),(58.5,0,'58m E-W'),(73,0,'73m E-W'),(87.6,0,'88m E-W'),
    810                   (102.3,0,'102m E-W')]
--> 811 nodeMedians,lsts,badAnts=get_correlation_baseline_evolutions(uv,HHfiles,jd,use_ants,pols=pols,mat_pols=mat_pols,
    812                                                             bl_type=baseline_groups,removeBadAnts=removeBadAnts,
    813                                                             plotRatios=plotRatios)
    814 pols = mat_pols
    815 if plotSummary is False:

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:1477, in get_correlation_baseline_evolutions(uv, HHfiles, jd, use_ants, badThresh, pols, bl_type, removeBadAnts, plotMatrix, mat_pols, plotRatios)
   1444 def get_correlation_baseline_evolutions(uv,HHfiles,jd,use_ants='auto',badThresh=0.35,pols=['xx','yy'],bl_type=(14,0,'14m E-W'),
   1445                                         removeBadAnts=False, plotMatrix=True,mat_pols=['xx','yy','xy','yx'],plotRatios=False):
   1446     """
   1447     Calculates the average correlation metric for a set of redundant baseline groups at one hour intervals throughout a night of observation.
   1448     
   (...)
   1475         Antenna numbers flagged as bad based on badThresh parameter.
   1476     """
-> 1477     files, lsts, inds = get_hourly_files(uv, HHfiles, jd)
   1478     if use_ants == 'auto':
   1479         use_ants = uv.get_ants()

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/hera_notebook_templates/utils.py:1388, in get_hourly_files(uv, HHfiles, jd)
   1386 try:
   1387     dat = UVData()
-> 1388     dat.read(file, read_data=False)
   1389 except KeyError:
   1390     continue

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:12716, in UVData.read(self, filename, axis, file_type, read_data, skip_bad_files, background_lsts, ignore_name, use_future_array_shapes, allow_rephase, make_multi_phase, antenna_nums, antenna_names, ant_str, bls, catalog_names, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, keep_all_metadata, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, check_autos, fix_autos, phase_type, projected, correct_lat_lon, calc_lst, fix_old_proj, fix_use_ant_pos, use_model, data_column, pol_order, ignore_single_chan, raise_error, read_weights, allow_flex_pol, multidim_index, remove_flex_pol, blt_order, blts_are_rectangular, time_axis_faster_than_bls, data_array_dtype, use_aoflagger_flags, remove_dig_gains, remove_coarse_band, correct_cable_len, correct_van_vleck, cheby_approx, flag_small_auto_ants, propagate_coarse_flags, flag_init, edge_width, start_flag, end_flag, flag_dc_offset, remove_flagged_ants, phase_to_pointing_center, nsample_array_dtype, corrchunk, receivers, sidebands, mir_select_where, apply_tsys, apply_flags, apply_dedoppler, pseudo_cont, rechunk, recompute_nbls)
  12697     self.read_ms(
  12698         filename,
  12699         data_column=data_column,
   (...)
  12712         use_future_array_shapes=use_future_array_shapes,
  12713     )
  12715 elif file_type == "uvh5":
> 12716     self.read_uvh5(
  12717         filename,
  12718         antenna_nums=antenna_nums,
  12719         antenna_names=antenna_names,
  12720         ant_str=ant_str,
  12721         bls=bls,
  12722         frequencies=frequencies,
  12723         freq_chans=freq_chans,
  12724         times=times,
  12725         time_range=time_range,
  12726         lsts=lsts,
  12727         lst_range=lst_range,
  12728         polarizations=polarizations,
  12729         blt_inds=blt_inds,
  12730         phase_center_ids=phase_center_ids,
  12731         catalog_names=catalog_names,
  12732         read_data=read_data,
  12733         data_array_dtype=data_array_dtype,
  12734         keep_all_metadata=keep_all_metadata,
  12735         multidim_index=multidim_index,
  12736         remove_flex_pol=remove_flex_pol,
  12737         background_lsts=background_lsts,
  12738         run_check=run_check,
  12739         check_extra=check_extra,
  12740         run_check_acceptability=run_check_acceptability,
  12741         strict_uvw_antpos_check=strict_uvw_antpos_check,
  12742         fix_old_proj=fix_old_proj,
  12743         fix_use_ant_pos=fix_use_ant_pos,
  12744         check_autos=check_autos,
  12745         fix_autos=fix_autos,
  12746         use_future_array_shapes=use_future_array_shapes,
  12747         time_axis_faster_than_bls=time_axis_faster_than_bls,
  12748         blts_are_rectangular=blts_are_rectangular,
  12749         recompute_nbls=recompute_nbls,
  12750     )
  12751     select = False
  12753 if select:

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:11536, in UVData.read_uvh5(self, filename, antenna_nums, antenna_names, ant_str, bls, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, catalog_names, keep_all_metadata, read_data, data_array_dtype, multidim_index, remove_flex_pol, background_lsts, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, fix_old_proj, fix_use_ant_pos, check_autos, fix_autos, use_future_array_shapes, time_axis_faster_than_bls, blts_are_rectangular, blt_order, recompute_nbls)
  11529     raise ValueError(
  11530         "Reading multiple files from class specific "
  11531         "read functions is no longer supported. "
  11532         "Use the generic `uvdata.read` function instead."
  11533     )
  11535 uvh5_obj = uvh5.UVH5()
> 11536 uvh5_obj.read_uvh5(
  11537     filename,
  11538     antenna_nums=antenna_nums,
  11539     antenna_names=antenna_names,
  11540     ant_str=ant_str,
  11541     bls=bls,
  11542     frequencies=frequencies,
  11543     freq_chans=freq_chans,
  11544     times=times,
  11545     time_range=time_range,
  11546     lsts=lsts,
  11547     lst_range=lst_range,
  11548     polarizations=polarizations,
  11549     blt_inds=blt_inds,
  11550     phase_center_ids=phase_center_ids,
  11551     catalog_names=catalog_names,
  11552     data_array_dtype=data_array_dtype,
  11553     keep_all_metadata=keep_all_metadata,
  11554     read_data=read_data,
  11555     multidim_index=multidim_index,
  11556     remove_flex_pol=remove_flex_pol,
  11557     background_lsts=background_lsts,
  11558     run_check=run_check,
  11559     check_extra=check_extra,
  11560     run_check_acceptability=run_check_acceptability,
  11561     strict_uvw_antpos_check=strict_uvw_antpos_check,
  11562     fix_old_proj=fix_old_proj,
  11563     fix_use_ant_pos=fix_use_ant_pos,
  11564     check_autos=check_autos,
  11565     fix_autos=fix_autos,
  11566     use_future_array_shapes=use_future_array_shapes,
  11567     recompute_nbls=recompute_nbls,
  11568     blt_order=blt_order,
  11569     blts_are_rectangular=blts_are_rectangular,
  11570     time_axis_faster_than_bls=time_axis_faster_than_bls,
  11571 )
  11572 self._convert_from_filetype(uvh5_obj)
  11573 del uvh5_obj

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvh5.py:1659, in UVH5.read_uvh5(self, filename, antenna_nums, antenna_names, ant_str, bls, frequencies, freq_chans, times, time_range, lsts, lst_range, polarizations, blt_inds, phase_center_ids, catalog_names, keep_all_metadata, read_data, data_array_dtype, multidim_index, remove_flex_pol, background_lsts, run_check, check_extra, run_check_acceptability, strict_uvw_antpos_check, fix_old_proj, fix_use_ant_pos, check_autos, fix_autos, use_future_array_shapes, blt_order, blts_are_rectangular, time_axis_faster_than_bls, recompute_nbls)
   1657 # check if object has all required UVParameters set
   1658 if run_check:
-> 1659     self.check(
   1660         check_extra=check_extra,
   1661         run_check_acceptability=run_check_acceptability,
   1662         strict_uvw_antpos_check=strict_uvw_antpos_check,
   1663         allow_flip_conj=True,
   1664         check_autos=check_autos,
   1665         fix_autos=fix_autos,
   1666     )
   1668 return

File ~/mambaforge/envs/RTP/lib/python3.10/site-packages/pyuvdata/uvdata/uvdata.py:3293, in UVData.check(self, check_extra, run_check_acceptability, check_freq_spacing, strict_uvw_antpos_check, allow_flip_conj, check_autos, fix_autos)
   3290 # Check internal consistency of numbers which don't explicitly correspond
   3291 # to the shape of another array.
   3292 if self.Nants_data != self._calc_nants_data():
-> 3293     raise ValueError(
   3294         "Nants_data must be equal to the number of unique "
   3295         "values in ant_1_array and ant_2_array"
   3296     )
   3298 if self.Nbls != len(np.unique(self.baseline_array)):
   3299     raise ValueError(
   3300         "Nbls must be equal to the number of unique "
   3301         f"baselines in the data_array. Got {self.Nbls}, not"
   3302         f"{len(np.unique(self.baseline_array))}"
   3303     )

ValueError: Nants_data must be equal to the number of unique values in ant_1_array and ant_2_array

Antenna Positions¶

Antennas outlined in black here have been identified by the correlation matrix as bad antennas. Antennas with a colorful outline correspond to their status as identified by ant_metrics (see above plot). Faded antennas are those not meeting the apriori status requirement for this notebook run. Gold stars are node box locations.

In [13]:
uvd1 = UVData()
uvd1.read(HHfiles[readInd], skip_bad_files=True)
utils.plot_antenna_positions(uvd1, badAnts=badAnts,use_ants=use_ants)
No description has been provided for this image

Observer Inspection Plots¶

Antennas of status digital_OK or better that are flagged as bad by any of the above metrics are plotted here so observers can inspect their failures in more detail. Additionally, a 'good' template has been used to identify outliers. The upper line plots are averages over the whole observation, and the lower line plots are slices of a single time in the middle of the observation. These plots are recommended diagnostics for demoting antennas to lower statuses or reporting issues. If the plots below look OK, check other plots in notebook to hunt why the antenna was flagged. NOTE: The colorbar/power scales in these plots are NOT locked between antennas OR polarizations so that the detail will be visible on all plots. Be sure to check for reasonable power levels, as this may be the reason the antenna was flagged for inspection.

In [14]:
d, tempAnts = utils.flag_by_template(uvd,HHautos,JD,use_ants=use_ants,pols=['XX','YY'],plotMap=False)
No template for lst=0.01 - skipping
No template for lst=1.0 - skipping
No template for lst=2.0 - skipping
No template for lst=3.0 - skipping
No template for lst=4.0 - skipping
In [15]:
inspectAnts = utils.plot_inspect_ants(uvd,JD,badAnts=badAnts,use_ants=use_ants,
                                     tempAnts=tempAnts,crossedAnts=crossedAnts)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In [15], line 2
      1 inspectAnts = utils.plot_inspect_ants(uvd,JD,badAnts=badAnts,use_ants=use_ants,
----> 2                                      tempAnts=tempAnts,crossedAnts=crossedAnts)

NameError: name 'crossedAnts' is not defined

Mean-Subtracted Waterfalls¶

Here the mean value in each frequency bin has been subtracted out. This effectively subtracts out the bandpass shape, making time variations more visible.

In [16]:
utils.plot_wfs(uvd,0,mean_sub=True,jd=JD)
utils.plot_wfs(uvd,1,mean_sub=True,jd=JD)
No description has been provided for this image
No description has been provided for this image
In [ ]: