Performing Dual-Domain Filtering¶

After selecting high-quality subsets in both the real and Fourier domains, you can intersect them to identify micrographs that exhibit strong features in both spaces.

This step combines the complementary strengths of real-space and frequency-space representations to yield the most reliable set of high-quality images for downstream analysis.

Tip

Run this step on a remote cluster for large datasets. Transfer the .parquet files from your local Phoenix session to the cluster before starting.

Activate the Environment¶

conda activate prismpyp

Create an Output Directory¶

mkdir intersection

Compute the Intersection¶

Run the following command to take the intersection between real and Fourier domain selections:

prismpyp intersect \
--parquet-files output_dir/fft/fft_good_export.parquet output_dir/real/real_good_export.parquet \
--output-folder intersection \
--link-type soft \
--webp-path metadata/webp

Tip

Use --link-type soft to create symbolic links or --link-type hard to copy the files instead.

Output Files¶

The following outputs will be written to the intersection/ directory:

File	Description
`intersection.parquet`	Metadata table containing information for all intersected micrographs
`files_in_common.txt`	List of intersected file names
`files`	The `.webp` file for the actual intersected micrographs (either symlinked or copied)

This subset represents the most consistent and high-quality micrographs across both domains, suitable for subsequent refinement or reconstruction workflows.

Uploading Filtered Subsets for Futher Processing¶

Once you have the outputs of prismPYP, you can upload your good subsets back into your favorite processing program to continue with the rest of the steps in the cryo-EM pipeline. Currently, we support uploads to nextPYP and cryoSPARC.

Uploading to nextPYPUploading to cryoSPARC

We can make a new mrcs folder for uploading to nextPYP in order to perform subsequent processing, using the outputs generated by the previous intersect command to help save some re-computation.

Create a New mrcs Folder for the Subset

Run the following command to generate a new mrcs folder that can be re-imported to your cryoSPARC project:

prismpyp upload_nextpyp \
--intersection-folder intersection \
--link-type soft \
--mrc-path /path/to/original/mrcs

Output Files

The following output will be written to the intersection/ directory:

File	Description
`mrcs`	The `.mrc` files for the actual intersected micrographs (either symlinked or copied)

Importing Results Back Into nextPYP¶

Create an Import Raw Data block and supply the path to the intersected .mrc files as the Location field: /path/to/prismpyp/intersection/mrcs
Fill out the other microscope parameters accordingly
Run the Import Raw Data block
Connect the outputs of the Import Raw Data block to a Pre-Processing block
Run the Pre-Procesing block
Continue with other jobs as needed

We can make a new .cs file for uploading to cryoSPARC in order to perform subsequent processing, using the outputs generated by the previous intersect command to help save some re-computation.

Create a New .cs File for the Subset

Run the following command to generate a new .cs file that can be re-imported to your cryoSPARC project:

prismpyp upload_cryosparc \
--output-folder intersection \
--link-type soft \
--cs-path /path/to/latest/job/file.cs

The choice of --cs-path depends on the last pre-processing job you ran in your cryoSPARC project before switching to prismPYP. For example, if you want to keep the pre-computed CTF parameters from a Manually Curate Exposures job (J4), then you would supply the exported .cs file from the Manually Curate Exposures job, like so:

prismpyp upload_cryosparc \
--output-folder intersection \
--link-type soft \
--cs-path /cryosparc/output/dir/exports/groups/J4_exposures_accepted/J4_exposures_accepted_exported.cs

Output Files

The following output will be written to the intersection/ directory:

File	Description
`filtered.cs`	CryoSPARC metadata file containing the good subset of micrographs that are suitable for downstream processing.

Importing Results Back Into CryoSPARC¶

The upload_cryosparc command will produce a file filtered.cs that contains only the statistics for the micrographs in intersection/files_in_common.txt. You can then use the filtered.cs file in cryoSPARC by using a Import Results Group job:

cryosparc import results group

This job takes the absolute path to a .csg file as input. See the cryoSPARC documentation here for more information.

In order to re-import this subset into cryoSPARC, you will need to do the following:

(Optional but highly recommended) Save a backup of the original .cs file:

cp /cryosparc/output/dir/exports/groups/J4_exposures_accepted/J4_exposures_accepted_exported.cs \
    /cryosparc/output/dir/exports/groups/J4_exposures_accepted/J4_exposures_accepted_exported.cs.bak

Copy the output file from intersection/filtered.cs to your cryoSPARC project directory (e.g., /cryosparc/output/dir) under the same file name as the original .cs file:
```
    cp intersection/filtered.cs \
    /cryosparc/output/dir/exports/groups/J4_exposures_accepted/J4_exposures_accepted_exported.cs
```
Create an Import Results Group job in cryoSPARC and supply the .csg file associated with the Manually Curate Exposures job: /cryosparc/output/dir/exports/groups/J4_exposures_accepted/J4_exposures_accepted_exported.csg
Now, you can proceed with next steps in cryoSPARC.