Fusion of Images with Prior Data

One common theme in research in our laboratory and elsewhere is combinining prior information about an individual patient or patient populations with new images to produce an improved model of the patient.

Examples may be found throughout our research writeups(e.g., Sparse X-Ray Reconstruction, 3D ACL Tunnel Position Estimation). Some more examples may be found below.

Deformable registration to statistical models

Deformable 2D-3D registration of statistical model of pelvis from sparse set of x-rays

Just as other groups, we have a long history of creating statistical models of patient anatomy and using the results for such purposes as assisting the segmentation of 3D images by means of deformable registration of these models to patient-specific images (e.g., [Yao 2003a], [Ellingsen 2009], [Chintalapani 2007], [Chintalapani 2009], [Seshamani 2011]) and to create estimates of 3D models from sparse sets of 2D data (e.g., [Yao 2002a], [Yao 2003b], [Sadowsky 2006], [Sadowsky 2007], [Chintalapani 2009b], [Chintalapani 2010]).

Model Completion, Given Partial CT + X-rays

Information may be fused from multiple sources to produce a model. For example, Chintalapani et al. have combined preoperative CT images of a limited portion of the pelvis with a statistical atlas and a limited number of x-ray images to create a model that may be used for preoperative planning and intraoperative registration in periacetabular osteotomies [Chintalapani 2010]. A fuller description of the overall system may be found in [Arminger 2007] and [Lepisto 2008].

3D Biomechanics Calculations from a Sparse Set of DXA Images

Another project has involved the use of a statistical atlas constructed from Quantitative CT (QCT) images of the proximal femur [Wilson 2007] to construct “volumetric DXA (VXA)” models from a sparse set of dual-energy (DXA) images [Ahmad 2010]. These VXA models are then used to compute biomechanical measures that may be used to predict fracture risk.

Fusion of video with segmented preoperative models

Other projects have emphasized the fusion of segmented preoperative models with intraoperative endoscopic video. In one example [Su 2009],video from a DaVinci stereo endoscope was used to create a 3D surface model of a kidney undergoing laparoscopic partial nephrectomy. This model was then registered to a segmented preoperative CT model to create stereoscopic video overlay visualizations. In other projects (e.g., [Burschka 2005], [Mirota 2009], [Mirota 2011], [Uneri 2012]), video has been fused with segmented volumetric models for skull base and head and neck surgery.

Anatomical Model Completion via Statistical Shape Models

The current application of interest is Total Face Transplant Surgery. The primary intent of this project is to estimate missing anatomical structure using a statistical shape model (SSM). This may be applied in the presence of an incomplete medical image or when a patient has undergone severe trauma. With an estimate of the patient's true anatomy, we believe that a more complete and accurate surgical plan may be used. A known, and non-traumatized, portion of the transplant candidate's cranial structure is matched to a SSM of the entire cranial structure. The regions instantiated from the SSM corresponding to unknown, or traumatized, regions of the patient's anatomy are used to estimate the entire cranial structure of the patient. The major issue with the completion approach described in [Chintalapani 2010] is the presence of a non-smooth transition from the known region to the unknown region. We hoped that by modeling this estimation problem as a Multivariate Gaussian Regression problem would yield a smooth boundary, however this was not the case. We believe that asymmetries resulting from the pose of the patient's mandible are not accurately modeled by the primarily symmetrical “known” region of the neurocranium used as prior data. In order to avoid this problem, work is currently underway to create separate SSMs for the neurocranium and mandible.

Figure 1: Extrapolation example with multivariate Gaussian regression; extrapolated area highlighted in green

Details are currently on the intranet page.


  • [Ahmad 2010] O. Ahmad, K. Ramamurthi, K. Wilson, K. Engelke, R. Prince, and R. Taylor, “Volumetric DXA (VXA) – A New Method to Extract 3D Information from Multiple In Vivo DXA Images”, Journal of Bone and Mineral Research (JBMR), vol. 25- 12, pp. 2468-2475, 2010. http://onlinelibrary.wiley.com/doi/10.1002/jbmr.140/full DOI 10.1002/jbmr.140, PMC20533301.
  • [Arminger 2007] R. Armiger, M. Armand, J. Lepisto, D. Minhas, K. Tallroth, S. Mears, M. Waites, and R. H. Taylor, “Evaluation of a Computerized Measurement Technique for Joint Alignment Before and During Periacetabular Osteotomy”, Computer Aided Surgery,, vol. 12- 4, pp. 215-224, 2007.
  • [Burschka 2005] D. Burschka, M. Li, R. Taylor, G. D. Hager, and M. Ishii, “Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery”, Medical Image Analysis, vol. 9- 5, pp. 413-439, October, 2005.
  • [Chintalapani 2007a] G. Chintalapani, L. M. Ellingsen, O. Sadowsky, J. L. Prince, and R. H. Taylor, “Statistical Atlases of Bone Anatomy: Construction, Iterative Improvement and Validation”, in Proc. 10th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Brisbane, Australia, October, 2007. pp. 499-506.
  • [Chintalapani 2007b] G. Chintalapani, O. Sadowsky, L. M. Ellingsen, J. L. Prince, and R. H. Taylor, “Integrating Statistical Models of Bone Density into Shape Based 2D-3D Registration Framework.”, in PMMIA: A workshop in Conjunction with MICCAI '09, London, 2009. pp. 151-161.
  • [Chintalapani 2009] G. Chintalapani and R. H. Taylor., “Construction of Multi-Component Statistical Shape Models of Bone Anatomy (Abstract in Proceedings)”, in Computer Assisted Orthopaedic Surgery (CAOS), 2009.
  • [Chintalapani 2010] G. Chintalapani, R. Murphy, R. Armiger, J. Lepistos, Y. Otake, N. Sugano, R. H. Taylor, and M. Armand, “Statistical Atlas Based Extrapolation of CT Data for Planning Periacetabular Osteotomy”, in Medical Imaging 2010: Visualization, Image-Guided Procedures, and Modeling, 2010. pp. 539-548. DOI:10.1117/12.845570
  • [Ellingsen 2009] L. M. Ellingsen, G. Chintalapani, R. H. Taylor, and J. L. Prince, “Robust deformable image registration using prior shape information for atlas to patient registration”, Computer Medical Imaging and Graphics, vol. 34- 1, pp. 79-90, 2009.
  • [Lepisto 2008] J. Lepisto, M. Armand, and R. Armiger, “Periacetabular osteotomy in adult hip dysplasia – developing a computer aided real-time biomechanical guiding system (BGS)”, Suomen Ortopedia ja Traumatologia, vol. 31-, pp. 186-190, February, 2008.
  • [Mirota 2009] D. Mirota, H. Wang, R. Taylor, M. Ishii, and H. Wang, “Toward Video-Based Navigation for Endoscopic Endonasal Skull Base Surgery”, in Medical Image Computing and Computer-Assisted Intervention (MICCAI), London, Sept 20-24, 2009. pp. 91-99. PMID: Pending
  • [Mirota 2011] D. J. Mirota, A. Uneri, S. Schafer, S. Nithiananthan, D. D. Reh, G. L. Gallia, R. H. Taylor, G. D. Hager, and J. H. Siewerdsen, “High-accuracy 3D image-based registration of endoscopic video to C-arm cone-beam CT for image-guided skull base surgery”, in Medical Imaging 2011: Visualization, Image-Guided Procedures, and Modeling, Orlando, 79640J-1 to 79640J-10, 2011.
  • [Sadowsky 2006a] O. Sadowsky, J. Cohen, and R. H. Taylor “Projected Tetrahedra Revisited: A Barycentric Formulation Applied to Digital Radiograph Reconstruction Using Higher-Order Attenuation Functions”, IEEE Transactions On Visualization and Computer Graphics, vol. 12- 4, pp. 461-473, July/August, 2006.
  • [Sadowsky 2006b] O. Sadowsky, K. Ramamurthi, L. M. Ellingsen, G. Chintalapani, J. L. Prince, and R. H. Taylor “Atlas-Assisted Tomography: Registration of a Deformable Atlas to Compensate for Limited-Angle Cone-Beam Trajectory”, in ISBI, Arlington, Va., April, 2006.
  • [Sadowsky 2007] O. Sadowsky, G. Chintalapani, and R. H. Taylor, “Deformable 2D-3D Registration of the Pelvis with a Limited Field of View, Using Shape Statistics”, in MICCAI, Brisbane, Australia, 2007. pp. 519–526. PMID: 18044608
  • [Sadowsky 2010] O. Sadowsky, J. Lee, E. G. Sutter, S. J. Wall, J. L. Prince, and R. H. Taylor, ” Hybrid Cone-Beam Tomographic Reconstruction: Incorporation of Prior Anatomical Models to Compensate for Missing Data”, IEEE Trans. on Medical Imaging, vol. In press-Epub date July 26-, 2010. Pubmed ID 20667807.
  • [Seshamani 2011] S. Seshamani, G. Chintalapani, and R. Taylor, “Iterative Refinement of Point Correspondences for 3D Statistical Shape Models”, in Medical Image Computing and Computer-Assisted Intervention (MICCAI), Toronto, September 18-22, 2011. pp. 417-425.
  • [Su 2009] L.-M. Su, B. P. Vagvolgyi, R. Agarwal, C. E. Reiley, R. H. Taylor, and G. D. Hager, “Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration”, Urology, vol. 73- 4, pp. 896-900, Apr, 2009. PMID: 1919340
  • [Uneri 2012] A. Uneri, S. Schafer, D. Mirota, S. Nithiananthan, Y. Otake, R. H. Taylor, G. Gallia, A. Khanna, S. Lee, D. Reg, and J. H. Siewerdsen, “TREK: An Integrated System Architecture for Intraoperative Cone-Beam CT Guided Surgery”, Int J Comput Assist Radiol Surg vol. 7- 1, pp. 159-73, Jul. 9 (epub), 2012. 0.1007/s11548-011-0636-7
  • [Wilson 2007] K. Wilson, O. Ahmad, K. Ramamurthy, K. Engelke, and R. Taylor, “A statistical atlas of the proximal femur”, Journal of Clinical Densitometry, vol. 10- 2, p. S223, 6 May, 2007.
  • [Yao 2002] J. Yao and R. H. Taylor, “Deformable registration between a statistical bone density atlas and X-ray images”, in Second International Conference on Computer Assisted Orthopaedic Surgery (CAOS 2002), Santa Fe, June 18-23, 2002.
  • [Yao 2003a] J. Yao and R. H. Taylor, “Non-Rigid Registration and Correspondence in Medical Image Analysis Using Multiple-Layer Flexible Mesh Template Matching”, International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), vol. 17- 7, pp. 1145-1165, 2003.
  • [Yao 2003b] J. Yao and R. H. Taylor, “Assessing Accuracy Factors in Deformable 2D/3D Medical Image Registration Using a Statistical Pelvis Model”, in Ninth Int. Conference on Computer Vision, Nice, October 12-16, 2003. pp. 1329-1334.
research.prior_knowledge.txt · Last modified: 2014/07/07 12:51 by rgrupp3@johnshopkins.edu