Data Setup and Outputs

The ulens_lsst package relies on external datasets for specific configurations and generates simulation outputs during pipeline execution. This section details the required input datasets, their download and configuration, and the structure of the generated output datasets.

Required Input Datasets

For certain pipeline configurations (e.g., when sources_catalog=”TRILEGAL” in config.yaml), external datasets such as chunks_TRILEGAL_Genulens are required to simulate TRILEGAL sources.

Downloading and Configuring Datasets

  1. TRILEGAL Datasets: - Source: Download the chunks_TRILEGAL_Genulens datasets from the project repository (data/chunks_TRILEGAL_Genulens/) or a designated external source (e.g., Zenodo DOI to be provided). - Placement: Store the datasets in the ulens_lsst/data/chunks_TRILEGAL_Genulens/ directory within your project. - Alternative: Specify a custom CSV file path if using a different source catalog.

  2. Genulens Datasets: Microlensing event parameters. - Source: Download the chunks_TRILEGAL_Genulens datasets from the project repository (data/chunks_TRILEGAL_Genulens/). - Placement: Store the datasets in the ulens_lsst/data/chunks_TRILEGAL_Genulens/ directory within your project. - Alternative: There is a plan to add a custom option in future version.

  3. Configuration: - Copy the example configuration file:

    cp ulens_lsst/config/config_example.yaml config.yaml

    • Update config.yaml with the appropriate settings: .. code-block:: yaml

      sources_catalog: “TRILEGAL” TRILEGAL_Genulens_path: “data/chunks_TRILEGAL_Genulens/”

    • For a custom CSV file: .. code-block:: yaml

      sources_catalog: “path/to/custom.csv”

Note: These input datasets are not bundled with the package to maintain a lightweight installation. Users must download them separately or generate them using the pipeline.

Generated Simulation Outputs

When executing the SimPipeline with a configured config.yaml, the pipeline creates an output directory named after the name parameter (under main_path, defaulting to /runs/). This directory contains temporary files and final simulation outputs, which are stored as Parquet files.

Dataset Structure

The generated datasets are organized into three main Parquet files, each serving a specific purpose in the simulation pipeline. Below are detailed descriptions of their columns:

Photometry Datasets

  • photometry_*.parquet: - Contains ideal simulated light curves before injection into LSST-like data. - Columns:

    • event_id: Unique identifier for the microlensing event (int32).

    • time: Observation time in Modified Julian Date (MJD) (float64).

    • band: Filter band (e.g., ‘u’, ‘g’, ‘r’, ‘i’, ‘z’, ‘y’) (string).

    • ideal_mag: Ideal magnitude without noise (float32).

    • meas_mag: Measured magnitude with noise (float32, post-injection if applicable).

    • meas_mag_err: Error in measured magnitude (float32).

    • meas_flux: Measured flux (float32).

    • meas_flux_err: Error in measured flux (float32).

    • magnification: Magnification factor applied to the light curve (float32).

    • injection_flag: Status of injection (e.g., ‘none’, ‘injected’) (string).

    • measure_flag: Status of measurement (e.g., ‘none’, ‘measured’) (string).

  • calexps-photometry_*.parquet: - Contains LSST-like simulated light curves after injection into calibrated exposures (calexps). - Columns: Identical to photometry_*.parquet, with additional noise and blending effects from the LSST simulation.

Events Dataset

  • data-events_*.parquet: - Stores detailed metadata about simulated microlensing events. - Columns:

    • event_id: Unique identifier (int32).

    • ra: Right ascension in degrees (float64).

    • dec: Declination in degrees (float64).

    • model: Microlensing model type (e.g., ‘PSPL’ for Point Source Point Lens) (string).

    • system_type: System classification (e.g., ‘microlensing’) (string).

    • points: Number of photometry points (int).

    • logL: Log luminosity (float).

    • logTe: Log effective temperature (float).

    • D_L: Lens distance in parsecs (float).

    • D_S: Source distance in parsecs (float).

    • mu_rel: Relative proper motion in mas/year (float).

    • nearby_object_ra: RA of nearby object in degrees (float).

    • nearby_object_dec: Dec of nearby object in degrees (float).

    • nearby_object_objId: Object ID of nearby object (string).

    • nearby_object_distance: Distance to nearby object in arcseconds (float).

    • cadence_noise: Source of cadence and noise (e.g., ‘DP0’, ‘rubin_sim’) (string).

    • peak_time: Peak time of the event in MJD (float).

    • nearby_object_mag_{band}: Magnitude of nearby object per band (e.g., mag_u, mag_g) (float).

    • nearby_object_fwhm_{band}: FWHM of nearby object per band (e.g., fwhm_u, fwhm_g) (float).

    • param_{key}: Model parameters (e.g., param_t0, param_u0) (float).

    • param-pylima_{key}: pyLIMA-specific parameters if applicable (e.g., param-pylima_tE) (float).

Additional Information

  • Format: All datasets are stored in Parquet format for efficiency with large datasets.

  • Generation: Datasets are generated by running the pipeline with appropriate config.yaml settings (e.g., steps: simulate,load_nearby,process_photometry).

  • Limitations: Requires the LSST stack (e.g., lsst-scipipe-10.0.0) for full compatibility. Dataset size and version may vary with pipeline updates.

Notebook Investigation

A Jupyter notebook, tutorials/dataset_investigation.ipynb, explores the generated datasets, including column analysis and visualization. To use it:

  • Ensure the notebook is in docs/tutorials/.

  • Run locally with Jupyter: jupyter notebook docs/tutorials/dataset_analysis.ipynb.

  • The notebook covers initial data inspection.