======Extrapolation of Missing Craniofacial Skeletal Structure via Statistical Shape Models======
**Last updated: May 9, 2014; 22:46 EDT**
======Summary======
Statistical Shape Models are the primary topic of this project, however several sub-topics include surface and volumetric deformable registration, volumetric segmentation, feature extraction, and surface extrapolation. The goal is to design and implement a method for extrapolating missing anatomical craniofacial skeletal structure with the use of a statistical shape model of the human cranium.
* **Students:** Robert Grupp (robert.grupp@gmail.com), Hsin-Hong Chiang (cblta@hotmail.com)
* **Mentor(s):** Dr. Yoshito Otake (otake@jhu.edu), Dr. Russell Taylor (rht@jhu.edu), Dr. Mehran Armand (Mehran.Armand@jhuapl.edu), Ryan Murphy (Ryan.Murphy@jhuapl.edu)
[{{ :courses:446:2014:446-2014-01:extrap_overview.png?600 |Figure 1: Basic Processing Flow}}]
======Background, Specific Aims, and Significance======
We intend to apply the algorithms developed for this project to the field of craniofacial surgery. The procedure of interest is craniofacial transplantation, which is the process of transplanting a donor’s craniofacial soft tissue, and possibly bone structure, onto a patient that has been subject to some severe craniofacial deformation. The surgery aims to restore lost functionality to the patient, such as the ability to smell, speak, or eat solid food [6]. By allowing the patient to participate in society as a “normal” individual, the surgery may help alleviate psychosocial traumas developed by the patient upon their disfiguration [6]. Figure 2 shows preoperative and postoperative views of a transplant recipient. Once a potential donor has been identified, the decision to perform surgery must be made within a very short time frame (24-36 hours)[7]. Amongst other factors, the skeletal structure of the patient and donor is compared for compatibility via cephalometric measurements [7]. We propose an attempt to estimate the skeletal structure of the patient’s face prior to disfiguration/injury. Once this estimated structure is completed for the patient, it may be matched quickly for compatibility with the donor’s skeletal structure and provide additional insight related to the patient-donor compatibility. If this skeletal comparison is accurate and useful, the surgeons may reprioritize their available time with additional refinement of the surgical plan or evaluation of other compatibility issues. It should be noted that the estimation of the patient’s ideal skeletal structure is an attempt to maximize the aesthetic quality of the surgical result, and makes no guarantee regarding the postoperative biomechanics of the patient. It is plausible that higher aesthetic quality will imply “good” biomechanics, but this will need to be the topic of further study. By leveraging techniques developed in the forensic facial reconstruction community [4], combined with the estimate of the patient’s true craniofacial skeletal structure, it may also be possible to create a model reflecting patient’s true physical appearance.
[{{ :courses:446:2014:446-2014-01:face_transplant_gordon.png?700 |Figure 2: A craniofacial transplant recipient. Left preoperative photograph, middle preop- erative CT, right postoperative photograph.}}]
======Deliverables======
* **Minimum:** (Expected by April 4, 2014) (//Achieved May 4, 2014//)
- Manual segmentation of the skeletal regions in the cranial CT images (//Achieved March 16, 2014//)
- A deformable registration of each CT image, or the surface mesh represented by each CT image, to a chosen template (//Achieved March 10, 2014//)
- Creation and evaluation of the cranial CT atlas using the segmented CT images and deformable registration outputs (//Achieved April 28, 2014//)
- Creation and evaluation of the method to extrapolate missing skeletal data utilizing the atlas (//Achieved May 4, 2014//)
* **Expected:** (Expected by April 20, 2014) (//Achieved May 6, 2014//)
- Creation and evaluation of an atlas via a bootstrapping technique (//Achieved May 6, 2014//)
- Development and evaluation of a similarity metric between estimated patient surface and the donor surface
* (Eliminated as the below deliverable was indicated to be more important)
- Development of a realistic patient "disfigurement" (//Achieved April 13, 2014//)
* **Maximum:** (Expected by May 5, 2014)
- Aborted due to lack of time
- Design of a method to use the estimated surface of the patient to assist in surgical planning
- Create a system architecture for the future use of this system
======Technical Approach======
This project consists of the following high-level technical components:
* Development of the atlas creation pipeline
* CT Segmentation
* Deformable Registration
* Algorithm development for the extrapolation of missing shape data
* Design a method for incorporation into surgical planning
* Design of a future system architecture
Once an atlas of skeletal surfaces has been created, the patient's skeletal surface will be deformably registered to the atlas, yielding an estimate of the patient's surface without abnormal pathology. The region of the patient's skeletal surface containing the abnormalities will be replaced with the corresponding regions from the atlas estimate. Additional processing will be required to remove any "jagged edges," or discontinuities, as a result of the replacement.
Figure 1 depicts a high level overview of the proposed reconstruction algorithm. Further details for each component are described in this section.
For this project, manual segmentation of the skeletal structure of the cranium will be performed. Open source, freely available, tools such as MITK, ITK Snap, and 3D Slicer may be used for this purpose [14][15][12]. This will most likely result in a more accurate segmentation compared to an automated method, and also avoid the “black art” of segmentation parameter tuning.
The general method we propose for atlas creation consists of the following steps and is a derivation of that found in Chintalapani, et al. [2]:
- Given a collection of CT images (not including the patient, **I**P), choose one as the template, **I**T, one as a test image, **I**Test, and denote the remaining //N// images as: **I**1, **I**2, ..., **I**//N//.
- Construct a set of geometrically aligned and topologically consistent set of meshes for images **I**T, **I**1, **I**2, ..., **I**//N//, denote them **M**T, **M**1, **M**2, ..., **M**//N//.
- Compute the mean mesh, **M**~, from **M**T, **M**1, **M**2, ..., **M**//N// and perform Principle Component Analysis (PCA) on the displacement vectors between the corresponding vertices on each mesh and **M**~.
- Using the eigenvectors output by PCA as variational modes, evaluate the atlas' accuracy representing the test subject with respect to the number of modes utilized, //NM//, for //NM// in { 1, 2, ..., N, N+1 }. This is commonly referred to as the "leave one out" method.
- Choose //NM//* as the number of the modes to use in the atlas, where //NM//* represents the starting point of "diminishing returns" with respect to accuracy gained with increasing number of modes.
This approach differs most noticeably from the approach in [2], in that we will utilize only surface based mesh structures and operate on shape information only. We anticipate adequate results, even without the use of image intensity values, since the bone density of the skull is not of immediate importance.
We are currently evaluating two distinct methods for completing step 2 (creating **M**T, **M**1, **M**2, ..., **M**//N//) in the general atlas creation creation method. The first method is summarized in the following steps:
- For i in {1, 2, ..., //N//}, perform a volumetric deformable registration from **I**//i// to **I**//T//. The output is a displacement field from exact voxels in **I**//T// to sub-voxels in **I**//i//, denote it //Di//.
- Segment **I**//T// and create its surface mesh, **M**//T//.
- For i in {1, 2, ..., //N//}, utilize //Di// to deform **M**//T// to **M**//i////'//.
- For i in {1, 2, ..., //N//}, perform a rigid registration from **M**//i////'// to **M**//T// to obtain **M**//i//.
The second method under consideration is summarized as follows:
- Segment **I**T, **I**1, **I**2, ..., **I**//N// and create surface meshes: **M**//T//, **M**//1////'//, **M**//2////'//, ..., **M**//N////'//.
- Manually identify the locations of (a priori determined) anatomical landmarks on each mesh: **L**//T//, **L**1, **L**2, ..., **L**//N//.
- For i in {1, 2, ..., N}, perform a surface feature-based deformable registration, //Di//, from **L**//i// to **L**//T//.
- For i in {1, 2, ..., N}, apply //Di// to **M**//T// to obtain **M**//i//.
We plan to utilize Matlab, ITK, 3D Slicer, and MeshLab for mesh creation and modification operations [11][9][12][3].
After initial development of an atlas creation pipeline and structural extrapolation technique, we hope to integrate an iterative "bootstrapping" mechanism into the atlas creation process. We plan on following a similar approach as [2], with the exception of using only surface data.
In order to simulate a patient with some missing, or deformed, skeletal anatomy we plan on using MeshLab [3] to perform a straightforward cutting, or cropping, of the skeletal surface. We intend to perform an initial estimate of the patient’s surface using the "Statistical Atlas-Based Extrapolation of CT Data" method described in [1]. We refer to the existing skeletal structure of the patient, with deformities or missing structure, as the "known" structure, and any estimates using the atlas as the "estimated" structure. Denote the known surface mesh of the patient as //M//; it may be partitioned into two regions as shown in Equation 1, with one region representing the deformed region of the patient (//M//Transplant) and the other representing the unchanged region of the patient (//M//Keep). We can then perform a deformable surface registration of M to the previously created atlas; this yields an estimate of the "normal" patient (//M'//) in terms of the mean shape (**v**0) and the modes (**v**//i// for //i// = 1, ..., //NM//) as shown in Equation 2. We intend to use the method developed as part of CIS I Programming Assignment 5 to perform this registration [8]. An analogous partitioning of //M'// may then be created to obtain the estimate of the patient’s missing structure, shown in Equation 3. Replacing //M//Transplant with //M'//Transplant in //M// yields, an estimate of the patient's true skeletal structure, //M//∗, as shown in Equation 4.
[{{ :courses:446:2014:446-2014-01:eq1.png?nolink&150 |Equation 1}}]
[{{ :courses:446:2014:446-2014-01:eq2.png?nolink&150 |Equation 2}}]
[{{ :courses:446:2014:446-2014-01:eq3.png?nolink&150 |Equation 3}}]
[{{ :courses:446:2014:446-2014-01:eq4.png?nolink&150 |Equation 4}}]
In most cases //M//∗ is an insufficient estimate, due to a discontinuity on the edge joining //M//Keep with //M'//Transplant [1], therefore some "smoothing" process is desirable. We hope to build off of existing reconstructive techniques in forensic anthropology [13]. Most of these techniques utilize thin plate spline (TPS) interpolation/extrapolation. Figure 3 depicts a "toy" one-dimensional example we have created for creating a smooth surface from two discontinuous inputs. We used a cubic-spline for interpolation in this example, which is the one-dimensional analog to the TPS. Matlab or ITK may be utilized for TPS implementations [11][9]. //**The smooth extrapolation was not implemented this semester due to insufficient time**//. Given sufficient time, we would also like to simulate a more realistic trauma to the patient (as shown in the middle of figure 1), and evaluate it’s registration to the atlas, and subsequent surface reconstruction.
[{{ :courses:446:2014:446-2014-01:toy_interp.png?direct&700 |Figure 3: A toy problem for reconstructing a surface smoothly using cubic splines. (a) the known surface and landmarks (blue) and the estimated surface and landmarks (red); note the discontinuity between the two surfaces. (b) the smoothly reconstructed surface ignoring landmarks in overlap regions. (c) zoomed version of one overlap region from (b). (d) the smoothly reconstructed surface using a weighted combination of landmarks from each surface in overlap regions. (e) zoomed version of one overlap region from (d).}}]
Additionally, we plan on developing a similarity metric between an estimated patient skeletal structure and a potential donor's skeletal structure. This could initially be completed via a rigid registration and a computation of the Euclidian distance from the donor's surface to the estimated patient's surface within the transplant region of interest.
A joint effort between Johns Hopkins Hospital and Johns Hopkins Applied Physics Laboratory (APL) is currently underway to utilize preoperative CT data of the patient and donor, along with intraoperative processing, to obtain superior surgical outcomes [5]. We would like to extend this reconstruction method to assist in the craniofacial surgical planning phase of the APL project. The estimated skeletal structure may be able to enhance the selection of cutting planes for a transplant with minimal skeletal discontinuities.
We are also interested in developing a future system architecture that would incorporate a shared atlas amongst several participating hospitals, and an automated ability for obtaining a patient's skeletal reconstruction. Opposed to our manual segmentation of the skeletal regions of each cranium, an automated method should be employed or developed. Recent segmentation techniques, such Krach's "Sheetless Segmentation," should be able to provided sufficient results [10]. An automated method of re-bootstrapping the atlas will need to be designed as well, so that the atlas may represent a larger segment of the population. If patient-donor compatibility is a concern, then a system of matching potential donors to patients, as the donors become available, would be another component. This would be relevant for automatically notifying the patient's surgeon when an ideal match for the patient becomes available in a different geographic locale; this could allow the donor to be immediately transported to the patient's institution before it becomes inviable for transplant. This system architecture would depend on consistent surgical planning, execution, and postoperative procedures across all participating institutions.
======Sample Results======
[{{ :courses:446:2014:446-2014-01:ssm_3modes.png?400 |Figure 4: Mean Shape and Modes 1-3 of the Atlas}}]
[{{ :courses:446:2014:446-2014-01:multivar_extrap_1.png?400 |Figure 5: Extrapolation example with multivariate Gaussian regression; extrapolated area highlighted in green}}]
[{{ :courses:446:2014:446-2014-01:chad_patient_recon_full_fig.png?400 |Figure 6: A craniofacial transplant candidate. Top: current skeletal structure. Middle: Atlas reconstruction. Bottom: Heat map of residual surface errors in normal regions}}]
[{{ :courses:446:2014:446-2014-01:disfigure_4.png?400 |Figure 7: A synthetic disfigurement}}]
[{{ :courses:446:2014:446-2014-01:rand_bone_bootstrap_mean_vert.png?400 |Figure 8: Mean vertex errors for 0-3 iterations of deformable registration bootstrapping}}]
======Dependencies======
* **Obtaining the Cranial CT Data**
* //**__Dependency met__**// via the use of the TCIA dataset (see below).
* We have an initial set of 6 cadaver head CT images from Dr. Otake, this is enough to get started with segmentation and registration evaluation
* The Cancer Imaging Archive (TCIA) has two datasets with head CT images of 77 and 91 patients, respectively; the data is freely available with no usage restrictions
* If TCIA data is insufficient, then we can request additional data from Dr. Armand, however it may require IRB approval.
* If neither of these plans work, then the fallback would be to use existing pelvis CT data with no schedule impact.
* **Access to Mentors**
* //**__Dependency met__**// via:
* Recurring weekly meeting with Dr. Otake.
* Additionally, the recurring weekly BIGSS Lab meeting can be used to address issues with other mentors.
* **Access to Fast Computer**
* //**__Dependency met__**// via the use of the new BIGSS Lab server.
======Milestones and Status ======
**//__Summary:__//** As of May 9, 2014, we have met our minimum and expected deliverables.
[[External Link]]======= Detailed Schedule =======
{{:courses:446:2014:446-2014-01:gant_chart.png?750|Project Milestones Chart}}
* Green task bars indicate a task completed on, or before, schedule
* Blue task bars indicate a task that is in not yet started, but not affecting the schedule, or a task that is in progress and on schedule
* Red task bars indicate a task that is over schedule
* The red arrows indicate schedule slips
* A purple diamond indicates an achieved milestone and a yellow diamond indicates a planned milestone
======Reports and presentations======
* Project Plan
* {{:courses:446:2014:446-2014-01:grupp_chiang_cis2_proposal_v02.pdf| Project plan presentation}}
* {{:courses:446:2014:446-2014-01:grupp_chiang_proposal_v1.pdf|Project plan proposal}}
* Project Background Reading
* See Bibliography below for links.
* Project Checkpoint
* {{:courses:446:2014:446-2014-01:cis2_checkpoint_v04.pdf| Project checkpoint presentation}}
* Paper Seminar Presentations
* {{:courses:446:2014:446-2014-01:grupp_seminar_v1.pdf| Seminar Presentation (Grupp)}}
* {{:courses:446:2014:446-2014-01:chiang_seminar_v2.pdf| Seminar Presentation (Chiang)}}
* {{:courses:446:2014:446-2014-01:grupp_seminar.pdf| Seminar Paper Summary (Grupp)}}
* {{:courses:446:2014:446-2014-01:chiang_seminar_writeup.pdf| Seminar Presentation Summary(Chiang)}}
* {{:courses:446:2014:446-2014-01:Paper1_Chiang.pdf| Seminar Presentation Paper1(Chiang)}}
* {{:courses:446:2014:446-2014-01:Paper2_Chiang.pdf| Seminar Presentation Paper2(Chiang)}}
* {{:courses:446:2014:446-2014-01:1-s2.0-s0278239110002582-main.pdf| Seminar Presentation Paper1 (Grupp)}}
* {{:courses:446:2014:446-2014-01:sgp06.pdf| Seminar Presentation Paper2 (Grupp)}}
* Project Mini-Checkpoint
* {{:courses:446:2014:446-2014-01:2014-04-24_mini_checkpoint.pdf| Project mini-checkpoint presentation}}
* Project Final Poster
* {{:courses:446:2014:446-2014-01:project1_poster.pdf|PDF of Poster}}
* Project Final Report
* {{:courses:446:2014:446-2014-01:project_1_final_report.pdf|Final Report}}
======Project Bibliography=======
* **Project Reading List**
* //__Craniofacial Surgery Background__//
* James A. McNamara Jr., A method of cephalometric evaluation, American Journal of Orthodontics, Volume 86, Issue 6, December 1984, Pages 449-469.
* F. V. Tenti., Cephalometric analysis as a tool for treatment planning and evaluation, Eur J Orthod (1981) 3 (4): 241-245.
* Cutting, Court M.D., et al., Three-Dimensional Computer-Assisted Design of Craniofacial Surgical Procedures: Optimization and Interaction with Cephalometric and CT-Based Models. Plastic & Reconstructive Surgery. 77(6):877-885, June 1986.
* Gordon, Chad R. DO, et al., The Cleveland Clinic FACES Score: A Preliminary Assessment Tool for Identifying the Optimal Face Transplant Candidate. Journal of Craniofacial Surgery. 20(6):1969-1974, November 2009.
* Chopra, Karan MD, et al., Clinical Application of the FACES Score for Face Transplantation. Journal of Craniofacial Surgery. 25(1): 64-69, January 2014.
* Gordon, Chad R. DO, et al., The World’s Experience With Facial Transplantation: What Have We Learned Thus Far? Annals of Plastic Surgery. 63(5):572-578, November 2009.
* Gordon, Chad R. DO, et al., Le Fort-Based Maxillofacial Transplantation: Current State of the Art and a Refined Technique Using Orthognathic Applications. Journal of Craniofacial Surgery. 23(1):81-87, January 2012.
* //__Atlas Creation__//
* T.F. Cootes, et al., Active Shape Models-Their Training and Application, Computer Vision and Image Understanding, Volume 61, Issue 1, January 1995, Pages 38-59.
* Chintalapani, Gouthami, et al., Statistical Atlases of Bone Anatomy: Construction, Iterative Improvement and Validation, Medical Image Computing and Computer-Assisted Intervention MICCAI 2007, 4791:499-506, 2007.
* Chintalapani, Gouthami. Statistical Atlases of Bone Anatomy and Their Applications. Thesis (Ph. D.)–Johns Hopkins University, 2010.
* Sadowsky, O.. Image registration and hybrid volume reconstruction of bone anatomy using a statistical shape atlas. Thesis (Ph. D.)–Johns Hopkins University, 2009.
* Stefan Zachow, et al., Reconstruction of mandibular dysplasia using a statistical 3D shape model, International Congress Series, Volume 1281, May 2005, Pages 1238-1243.
* //__Deformable Registration (for Atlas Creation)__//
* Sotiras, A., et al., Deformable Medical Image Registration: A Survey, Medical Imaging, IEEE Transactions on , vol.32, no.7, pp.1153,1190, July 2013.
* Brian B. Avants, et al., A reproducible evaluation of ANTs similarity metric performance in brain image registration, NeuroImage, Volume 54, Issue 3, 1 February 2011, Pages 2033-2044.
* //__Surface Reconstruction/Estimation__//
* Bookstein, Fred L., Principal warps: thin-plate splines and the decomposition of deformations, Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.11, no.6, pp.567,585, Jun 1989.
* Stefano Benazzi, et al., A new OH5 reconstruction with an assessment of its uncertainty, Journal of Human Evolution, Volume 61, Issue 1, July 2011, Pages 75-88.
* Senck, Sascha, et al., Virtual Reconstruction of Very Large Skull Defects Featuring Partly and Completely Missing Midsagittal Planes, The Anatomical Record. 296(5):745-758, May 2013.
* Kazhdan, Michael, et al., Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing. 2006.
* **Page References**
* [1] G. Chintalapani. Statistical atlases of bone anatomy and their applications. Johns Hopkins University, 2010.
* [2] G. Chintalapani, L. Ellingsen, O. Sadowsky, J. Prince, and R. Taylor. Statistical atlases of bone anatomy: Construction, iterative improvement and validation. In N. Ayache, S. Ourselin, and A. Maeder, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2007, volume 4791 of Lecture Notes in Computer Science, pages 499–506. Springer Berlin Heidelberg, 2007.
* [3] P. Cignoni, M. Corsini, and G. Ranzuglia. Meshlab: an open-source 3d mesh processing system. Ercim news, 73:45–46, 2008.
* [4] P. Claes, D. Vandermeulen, S. De Greef, G. Willems, J. G. Clement, and P. Suetens. Computerized craniofacial reconstruction: conceptual framework and review. Forensic science international, 201(1):138–145, 2010.
* [5] C. R. Gordon, R. J. Murphy, D. Coon, E. Basafa, Y. Otake, M. Al Rakan, E. Rada, S. Susarla, E. Swanson, E. Fishman, et al. Preliminary development of a workstation for craniomaxillofacial surgical procedures: Introducing a computer-assisted planning and execution system. Journal of Craniofacial Surgery, 25(1):273–283, 2014.
* [6] C. R. Gordon, M. Siemionow, F. Papay, L. Pryor, J. Gatherwright, E. Kodish, C. Paradis, K. Coffman, D. Mathes, S. Schneeberger, et al. The world’s experience with facial transplantation: what have we learned thus far? Annals of plastic surgery, 63(5):572– 578, 2009.
* [7] C. R. Gordon, S. M. Susarla, Z. S. Peacock, L. B. Kaban, and M. J. Yaremchuk. Le fort–based maxillofacial transplantation: Current state of the art and a refined technique using orthognathic applications. Journal of Craniofacial Surgery, 23(1):81– 87, 2012.
* [8] R. Grupp and H.-H. Chiang. Programming assignment 5, December 2013. EN.600.445 Coursework.
* [9] L. Ibanez, W. Schroeder, L. Ng, and J. Cates. The ITK Software Guide. Kitware, Inc., http://www.itk.org/ItkSoftwareGuide.pdf, third edition, 2013.
* [10] M. Krcah, G. Szekely, and R. Blanc. Fully automatic and fast segmentation of the femur bone from 3d-ct images with no shape prior. In Biomedical Imaging: From Nano to Macro, 2011 IEEE International Symposium on, pages 2087–2090, March 2011.
* [11] MATLAB. version 8.1.0.604 (R2013a). The MathWorks Inc., Natick, Massachusetts, 2013.
* [12] S. Pieper, M. Halle, and R. Kikinis. 3d slicer. In Biomedical Imaging: Nano to Macro, 2004. IEEE International Symposium on, pages 632–635. IEEE, 2004.
* [13] S. Senck, M. Coquerelle, G. W. Weber, and S. Benazzi. Virtual reconstruction of very large skull defects featuring partly and completely missing midsagittal planes. The Anatomical Record, 296(5):745–758, 2013.
* [14] I. Wolf, M. Vetter, I. Wegner, M. Nolden, T. Bottger, M. Hastenteufel, M. Schobinger, T. Kunert, and H.-P. Meinzer. The medical imaging interaction toolkit (mitk): a toolkit facilitating the creation of interactive software by extending vtk and itk. In Medical Imaging 2004, pages 16–27. International Society for Optics and Photonics, 2004.
* [15] P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho, J. C. Gee, and G. Gerig. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage, 31(3):1116–1128, 2006.
======Other Resources and Project Files======
* Source code is hosted by the LCSR GitLab server in the following repositories (please email Rob if access is required):
* https://git.lcsr.jhu.edu/rgrupp3/cis2-atlas-extrapolation
* https://git.lcsr.jhu.edu/rgrupp3/grupp