#######################################
Single-particle tutorial (EMPIAR-10025)
#######################################
This tutorial shows how to convert raw movies from `EMPIAR-10025 (T20S proteasome) `_ into a ~3A resolution structure.
Total running time required to complete this tutorial: 45 min.
We first use the command line to download and decompress a tbz file containing a subset of 20 movies, the gain reference, and an initial model:
.. code-block:: bash
# cd to a location in the shared file system and run:
wget https://nextpyp.app/files/data/nextpyp_spr_tutorial.tbz
tar xvfz nextpyp_spr_tutorial.tbz
Open your browser and navigate to the url of your ``nextPYP`` instance (e.g., ``https://nextpyp.myorganization.org``).
Step 1: Create a new project
----------------------------
.. dropdown:: Data processing runs are organized into projects. We will create a new project for this tutorial
:container: + shadow
:title: bg-primary text-white text-left
:open:
* The first time you login into ``nextPYP``, you should see an empty **Dashboard**:
.. figure:: ../images/dashboard_empty.webp
:alt: Create new project
* Click on :badge:`Create new project,badge-primary`, give the project a name, and select :badge:`Create,badge-primary`
.. figure:: ../images/tutorial_spa_new.webp
:alt: Create new project
* Select the new project from the **Dashboard** and click :badge:`Open,badge-primary`
.. figure:: ../images/tutorial_spa_open.webp
:alt: Select new project
* The newly created project will be empty and a **Jobs** panel will appear on the right
.. figure:: ../images/tutorial_spa_empty.webp
:alt: Empty project
Step 2: Import raw movies
-------------------------
.. dropdown:: Import the raw movies downloaded above (:fa:`stopwatch` <1 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click :badge:`Import Data,badge-primary` and select :badge:`Single Particle (from Raw Data),badge-primary`
.. figure:: ../images/tutorial_spa_import_dialog.webp
:alt: Import dialog
* A form to enter parameters will appear:
.. figure:: ../images/tutorial_spa_import_data.webp
:alt: File browser
* Go to the **Raw data** tab:
.. tabbed:: Raw data
- Set the ``Location`` of the raw data clicking on the icon :fa:`search,text-primary` and browsing to the directory where the you downloaded the raw data
- Type ``14*.tif`` in the filter box (lower right) and click on the icon :fa:`filter,text-primary` to verify your selection. 20 matches should be displayed
- Click :badge:`Choose File Pattern,badge-primary`
- Click on the **Gain reference** tab
.. figure:: ../images/tutorial_spa_import_browser.webp
:alt: File browser
.. tabbed:: Gain reference
- Set the ``Location`` of the gain reference by clicking the icon :fa:`search,text-primary` and navigating to the directory where you downloaded the data for the tutorial. Select the file ``Gain.mrc`` and click :badge:`Choose File,badge-primary`
- Check ``Flip vertically``
- Click on the **Microscope parameters** tab
.. figure:: ../images/tutorial_spa_import_gain.webp
:alt: File browser
.. tabbed:: Microscope parameters
- Set ``Pixel size (A)`` to 0.66
- Set ``Acceleration voltage (kV)`` to 300
.. figure:: ../images/tutorial_spa_import_scope.webp
:alt: Project dashboard
* Click :badge:`Save,badge-primary` and the new block will appear on the project page
.. figure:: ../images/tutorial_spa_import_modified.webp
:alt: Project dashboard
* The block is in the modified state (indicated by the :fa:`asterisk` sign, top bar) and is ready to be executed
* Clicking the button :badge:`Run,badge-primary` will show another dialog where you can select which blocks to run. Since there is only block available, simply click on :badge:`Start Run for 1 block,badge-primary`. This will launch a process that reads the first movie, applies the gain reference and displays a thumbnail inside the :badge:`Single Particle (from Raw Data),badge-secondary` block
.. figure:: ../images/tutorial_spa_import_done.webp
:alt: Gain thumbnail
.. tip::
Click inside the :badge:`Single Particle (from Raw Data),badge-secondary` block to see a larger version of the image
Step 3: Pre-processing
----------------------
.. dropdown:: Movie frame alignment, CTF estimation and particle picking (:fa:`stopwatch` 2 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on :guilabel:`Movies` (output of :badge:`Single Particle (from Raw Data),badge-secondary` block) and select :badge:`Pre-processing,badge-primary`
.. figure:: ../images/tutorial_spa_pre_process_dialog.webp
:alt: File browser
* Go to the **Particle detection** tab:
.. tabbed:: Particle detection
* Set ``Particle radius (A)`` to 65
* Set ``Detection method`` to all
* Set ``Min distance (pixels)`` to 40
* Click on the **Resources** tab
.. tabbed:: Resources
* Set ``Threads per task`` to 7
* Set ``Memory per task`` to 14
* Set other runtime parameters as needed (see :doc:`Computing resources<../reference/computing>`)
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary`. You can monitor the status of the run using the **Jobs** panel
.. figure:: ../images/tutorial_spa_pre_process_modified.webp
:alt: File browser
* Click inside the :badge:`Pre-processing,badge-secondary` block to inspect the results (you don't need to wait until processing is done to do this). Results will be grouped into tabs:
.. tabbed:: Plots
.. figure:: ../images/tutorial_spa_pre_process_page.webp
:alt: Dataset statistics
.. tabbed:: Table
.. figure:: ../images/tutorial_spa_pre_process_table.webp
:alt: Table view
.. tabbed:: Gallery
.. figure:: ../images/tutorial_spa_pre_process_gallery.webp
:alt: Gallery view
.. tabbed:: Micrograph
Data processing details (particle picking, drift trajectory, CTF profile, power spectrum)
.. figure:: ../images/tutorial_spa_pre_process_micrographs.webp
:alt: Micrograph view
.. tip::
While on the **Micrographs** tab, use the navigation bar at the top of the page to look at the results for other micrographs
Step 4: Reference-based refinement
----------------------------------
.. dropdown:: Reference-based particle alignment (:fa:`stopwatch` 3 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on :guilabel:`Particles` (output of :badge:`Pre-processing,badge-secondary` block) and select :badge:`Particle refinement,badge-primary`
.. figure:: ../images/tutorial_spa_coarse_dialog.webp
:alt: File browser
* Go to the **Sample** tab:
.. tabbed:: Sample
- Set ``Molecular weight (kDa)`` to 700
- Set ``Particle radius (A)`` to 80
- Set ``Symmetry`` to D7
- Click on the **Extraction** tab
.. figure:: ../images/tutorial_spa_coarse_sample.webp
:alt: File browser
.. tabbed:: Extraction
- Set ``Box size (pixels)`` to 128
- Set ``Image binning`` to 4
- Click on the **Refinement** tab
.. figure:: ../images/tutorial_spa_coarse_extract.webp
:alt: File browser
.. tabbed:: Refinement
- Set the location of the ``Initial model`` by clicking on the icon :fa:`search, text-primary`, navigating to the folder where you downloaded the data for the tutorial, selecting the file `EMPIAR-10025_init_ref.mrc`, and clicking :badge:`Choose File,badge-primary`
- Set ``Max resolution (A)`` to 8:7:6
- Check ``Use signed correlation``
- Set ``Last iteration`` to 5
- Check ``Use alignment priors``
- Click on the **Reconstruction** tab
.. figure:: ../images/tutorial_spa_coarse_refinement.webp
:alt: File browser
.. tabbed:: Reconstruction
- Set ``Fraction of particles`` to 0
.. figure:: ../images/tutorial_spa_coarse_reconstruction.webp
:alt: File browser
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary`
The new block will appear on the **Dashboard** and a thumbnail will be displayed inside after the run is finished
.. figure:: ../images/tutorial_spa_coarse_modified.webp
:alt: File browser
This process executes four rounds of global orientation search (iterations 2-5). The fraction of good particles at each iteration will be determined automatically (``Fraction of particles`` = 0) and used for reconstruction
* Click inside the :badge:`Pre-processing,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_coarse_iter5.webp
:alt: Iteration 5
Step 5: Filter bad particles
----------------------------
.. dropdown:: Identify particles with low alignment scores (:fa:`stopwatch` 1 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on :guilabel:`Particles` (output of :badge:`Particle refinement,badge-secondary` block) and select :badge:`Particle filtering,badge-primary`
.. figure:: ../images/tutorial_spa_fine_dialog.webp
:alt: File browser
* Go to the **Particle filtering** tab:
.. tabbed:: Particle filtering
- Check ``Automatic score threshold``
- Set ``Min distance between particles (A)`` to 20
- Select the ``Input parameter file`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-coarse-refinement-*_r01_05.par.bz2`
- Check ``Generate reconstruction after filtering``
- Click on the **Refinement** tab
.. tabbed:: Refinement
- Select the ``Initial model`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-coarse-refinement-*_r01_05.mrc`
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary` to execute particle cleaning and produce a reconstruction with only the clean particles
* Click inside the :badge:`Filter particles,badge-secondary` block to look at the reconstruction after cleaning:
.. figure:: ../images/tutorial_spa_cleaning_iter2.webp
:alt: Iteration 2
Step 6 Permanently remove bad particles
---------------------------------------
.. dropdown:: Permanently remove bad particles to improve efficiency of steps downstream (:fa:`stopwatch` <1 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Edit the settings of the :badge:`Particle filtering,badge-secondary` block by clicking on the icon :fa:`bars, text-primary` and selecting the :fa:`edit, text-primary` Edit option
* Go to the **Particle filtering** tab
.. tabbed:: Particle filtering
- Check ``Permanently remove particles``
- Uncheck ``Generate reconstruction after filtering``
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary` to launch the job
Step 7: Particle refinement
---------------------------
.. dropdown:: Reconstruction and additional refinement using 2x binned particles (:fa:`stopwatch` 9 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on :guilabel:`Particles` (output of :badge:`Filter particles,badge-secondary` block) and select :badge:`Particle refinement,badge-primary`
.. figure:: ../images/tutorial_spa_fine_dialog.webp
:alt: File browser
* Go to the **Extraction** tab:
.. tabbed:: Extraction
- Set ``Box size (pixels)`` to 256
- Set ``Image binning`` to 2
- Click on the **Refinement** tab
.. tabbed:: Refinement
- Select the ``Initial model`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-fine-refinement-*_r01_02.mrc`
- Select the ``Input parameter file`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-fine-refinement-*_r01_02_clean.par.bz2`
- Set ``Max resolution (A)`` to 6:4:3
- Check ``Use signed correlation``
- Set ``Last iteration`` to 6
- Set ``Search mode`` to local
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary` to launch the job
* Click inside the :badge:`Particle refinement,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_fine_iter2.webp
:alt: Iteration 2
.. tip::
Use the navigation bar at the top left of the page to look at the results for different iterations
Step 8: Create shape mask
-------------------------
.. dropdown:: Use most recent reconstruction to build a shape mask (:fa:`stopwatch` <1 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on :guilabel:`Particles` (output of :badge:`Particle refinement,badge-secondary` block) and select :badge:`Masking,badge-primary`
* Enter parameter values for the **Masking** tab:
.. tabbed:: Masking
- Select the ``Input map`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-coarse-refinement-*_r01_06.mrc`
- Set ``Threshold for binarization`` to 0.3
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary` to launch the job
* Click on the icon :fa:`bars, text-primary` of the :badge:`Masking,badge-secondary` block, select the :badge:`Show Filesystem Location` option, and :badge:`Copy,badge-primary` the location of the block in the filesystem (we will use this in the next step))
* Click inside the :badge:`Masking,badge-secondary` block to inspect the results of masking
Step 9: Local refinement
------------------------
.. dropdown:: Additional refinement iterations using 2x binned data (:fa:`stopwatch` 2 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Go one block upstream to the :badge:`Particle refinement,badge-secondary` block, click on the icon :fa:`bars, text-primary` and select the :fa:`edit, text-primary` Edit option from the menu
* Go to the **Refinement** tab:
.. tabbed:: Refinement
- Set ``Last iteration`` to 7
- Select the ``Shape mask`` by clicking on the icon :fa:`search, text-primary`, navigating to the path of the :badge:`Masking,badge-secondary` block copied above, and selecting the file `frealign/maps/mask.mrc`
* Click :badge:`Save,badge-primary`, then :badge:`Run,badge-primary`. We now need to uncheck the box for the :badge:`Masking,badge-secondary` block (since we don't want to re-run this block), then click :badge:`Start Run for 1 block,badge-primary`
* Click inside the :badge:`Particle refinement,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_fine_iter7.webp
:alt: Iteration 7
Step 10: Particle-based CTF refinement
--------------------------------------
.. dropdown:: Per-particle CTF refinement using most recent reconstruction (:fa:`stopwatch` 9 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on the menu icon :fa:`bars, text-primary` from the :badge:`Particle refinement,badge-secondary` block and choose the :fa:`edit, text-primary` Edit option.
* Go to the **Refinement** tab:
.. tabbed:: Refinement
- Set ``Last iteration`` to 8
- Click on the **Constrained refinement** tab
.. tabbed:: Constrained refinement
- Set ``Number of regions`` to 8,8
- Check ``Refine CTF per-particle``
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary`
* Click inside the :badge:`Particle refinement,badge-secondary` block to inspect the results
Step 11: Movie frame refinement
-------------------------------
.. dropdown:: Particle-based movie-frame alignment and data-driven exposure weighting (:fa:`stopwatch` 8 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click :guilabel:`Particle set` (output of :badge:`Particle refinement,badge-secondary` block) and select :badge:`Movie refinement,badge-primary`
* Go to the **Refinement** tab:
.. tabbed:: Refinement
- Select the ``Initial model`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-coarse-refinement-*_r01_07.mrc`
- Select the ``Input parameter`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-coarse-refinement-*_r01_07.par.bz2`
- Set ``Max resolution (A)`` to 3
- Set ``Last iteration`` to 3
- Check ``Skip refinement``
- Go to the **Constrained refinement** tab
.. tabbed:: Constrained refinement
- Set ``Last exposure for refinement`` to 60
- Check ``Movie frame refinement``
- Check ``Regularize translations``
- Set ``Spatial sigma`` to 15
- Go to the **Exposure weighting** tab
.. tabbed:: Exposure weighting
- Check ``Dose weighting``
* Click :badge:`Save,badge-primary`, then :badge:`Run,badge-primary` to launch Movie refinement. Uncheck the box for the :badge:`Masking,badge-secondary` block and click :badge:`Start Run for 1 block,badge-primary`
* Click inside the :badge:`Movie refinement,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_movie_iter3.webp
:alt: Iteration 3
Step 12: Refinement after movie frame refinement
------------------------------------------------
.. dropdown:: Additional refinement using new frame alignment parameters (:fa:`stopwatch` 8 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click on the menu icon :fa:`bars, text-primary` from the :badge:`Movie refinement,badge-secondary` block and choose the :fa:`edit, text-primary` Edit option.
* Go to the **Refinement** tab:
.. tabbed:: Refinement
- Set ``Last iteration`` to 4
- Uncheck ``Skip refinement``
- Click on the **Constrained refinement** tab
.. tabbed:: Constrained refinement
- Uncheck ``Movie frame refinement``
* Click :badge:`Save,badge-primary`, :badge:`Run,badge-primary`, and :badge:`Start Run for 1 block,badge-primary`
* Click inside the :badge:`Movie refinement,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_final_map.webp
:alt: Final map
Step 13: Map sharpening
-----------------------
.. dropdown:: Apply B-bactor weighting in frequency space (:fa:`stopwatch` <1 min)
:container: + shadow
:title: bg-primary text-white text-left
:open:
* Click :guilabel:`Frames` (output of :badge:`Movie refinement,badge-secondary` block) and select :badge:`Post-processing,badge-primary`
* Go to the **Post-processing** tab:
.. tabbed:: Post-processing
- Select the ``First half map`` by clicking on the icon :fa:`search, text-primary` and selecting the file `sp-flexible-refinement-*_r01_half1.mrc`
- Set ``Automask threshold`` to 0.5
- Set ``Adhoc B-factor (A^2)`` to -50
* Click :badge:`Save,badge-primary`, then :badge:`Run,badge-primary`. Uncheck the box for the :badge:`Masking,badge-secondary` block and click :badge:`Start Run for 1 block,badge-primary`
* Click inside the :badge:`Map sharpening,badge-secondary` block to inspect the results:
.. figure:: ../images/tutorial_spa_post_processing.webp
:alt: Post processing
.. note::
Running times were measured running micrographs in parallel on nodes with 124 vCPUs, 720GB RAM, and 3TB of local SSDs
.. seealso::
* :doc:`Single-particle session`
* :doc:`Tomography tutorial`
* :doc:`Classification tutorial`
* :doc:`Tomography session`