Tables and catalogs#

Catalogs, photometry, light curves and instrument curves open in a table viewer that scrolls smoothly through millions of rows. As with images, the table stays on the server: sorting, filtering, statistics and plots are worked out there, and the page only receives the rows on screen.

The examples use stars, a pandas table of the 600 stars in the sample field, with their sky positions (ra, dec), pixel positions (x, y), magnitude and color. It is also saved as stars.csv (stars_csv).

Note

A table viewer needs its server, so the tables on this page are pictures. Everything described here works in a notebook.

Opening a table#

pc.table takes a file or a table in the kernel. Files can be CSV, TSV or ECSV (gzipped too), Parquet, VOTable (.vot, .votable, or an .xml one given to pc.table), Arrow/Feather, or FITS tables, given as a path, an s3:// URI or an https:// URL. pc.show recognizes table files too, FITS files that hold tables and no image among them, so you don’t have to remember which to call.

import pixel_clarity as pc

pc.show(stars_csv, height=440)
A table viewer

From the kernel, pc.table takes astropy, pandas and polars tables and NumPy structured arrays. They are written once to a temporary file on the server: Parquet when pyarrow is installed (which keeps units, masks and times), otherwise ECSV for astropy tables and CSV for pandas.

from astropy.table import Table

pc.table(Table.read("catalog.ecsv"))
pc.table(stars)                     # a pandas DataFrame

A Parquet, Arrow or FITS table too large to load whole (more than a quarter of the server’s memory) asks which columns you want first.

Files with several tables#

A FITS file can hold several tables, and a VOTable too. The viewer opens the first and lists the others in a chooser in its toolbar. To open on another, name it: a FITS table by its EXTNAME or HDU number, a VOTable’s by its place in the file, from 0.

pc.table("survey.fits", table="SOURCES")
pc.table("survey.fits", table=2)           # its second table, BRIGHT

A FITS file with an image and a catalog opens on the image; the catalog is listed with the file’s images, and Open table there shows it in the table viewer, in a tab of its own.

Filtering and sorting#

Type a filter in the bar above the rows and press Enter. Filters are Python-like expressions over the column names: comparisons, and, or, not, arithmetic, and functions such as abs, log10 and sqrt. The row count shows how many of the table’s rows pass.

Here we keep the bright, red stars (mag < 18 and color > 0) and sort them by magnitude:

pc.table(stars, height=440)
A table viewer

You can also filter from the data itself: right-click a cell to keep the rows above, below or equal to its value, pick a range on a column’s histogram, or draw a box on the plot. Click a column’s header to sort by it, and Shift-click another to add a second key.

Columns can be moved, resized, pinned and hidden, and the Columns panel adds computed columns from an expression, mag - 5 * log10(dist / 10) for example.

Inspecting a column or a row#

Select a column to see its statistics in the side panel: count, nulls, range, mean, median and a histogram, plus the most common values. Select a row to see it as a list, which is easier to read for a wide catalog. Units and descriptions from ECSV and Parquet files show in the headers and the panel.

Plotting#

plot=True opens the table with its plot pane showing. Choose the columns for the axes, and optionally one to color by. Large tables are drawn as a density map rather than millions of points, so a plot stays quick however long the table is. Log and time axes, lines and histograms are in the plot’s menu.

Here, magnitude against color, with the magnitude axis flipped so bright stars are at the top:

pc.table(stars, plot=True, height=500)
A table viewer

Overlaying a catalog on an image#

With an image open in another viewer, Export → Overlay on image draws the table’s rows (the filtered ones) on the image viewer you used last. The dialog asks which columns hold the positions, sky or pixel, the circles’ radius and an optional label column. The line under the table says which viewer it went to.

From Python, you have more control: pass the rows you want to the image’s add_regions (see Coordinates and regions).

Catalogs over the view#

The catalog button in an image viewer’s toolbar (on images with sky coordinates, with the online setting on, as it is by default) overlays Gaia DR3, 2MASS or SIMBAD with one click, or any VizieR table by its ID, over what the view shows. Hover a marker for its name and magnitude, or click it for the whole row. The queries go to CDS in Strasbourg. Gaia and 2MASS come in a few seconds: the viewer’s process asks VizieR’s TAP service for them (a browser can’t read its answers), and your browser asks VizieR’s usual service as well if that is slow, the first answer used. SIMBAD and other VizieR tables are asked from your browser.

In the catalog list, the plot button opens the catalog’s rows in the table viewer: they are saved as an ECSV file next to the notebook (named after the catalog) and open in a tab of their own. Through the kernel the viewer says how to open the file from Python instead, and it is a file like any other: Table.read("Gaia DR3 ….ecsv") reads it too.

Exporting#

Export writes the rows as they are now (filtered, sorted, with the columns shown) as CSV, TSV, ECSV, or a region file, next to the notebook (or to the viewer’s out_dir). Select a block of cells (drag, or Shift with the arrow keys) and right-click for Copy or Copy with header, ready to paste into a spreadsheet.

Opening it the same way again#

A table viewer’s state holds its filters, sort, columns and plot. Pass it to another viewer to open that one the same way: the same catalog next week, or a second catalog with the same columns.

t = pc.table("catalog.parquet")
# ... filter and arrange it in the viewer ...
pc.table("catalog_v2.parquet", state=t.state)