Last updated: Feb 26, 14:54

Development of a method to calibrate sensors placed on the back of the spine to assist in determining whether a given patient is eligible for spinal surgery. The method for doing this is by using a monocular camera and computer vision to detect and identify key points on the spine and mapping those locations with the IMU data via the transfer function. Part of this project entails setting up an AWS system that pipelines the data through uploading, storage, processing, and output for the doctor to allow for this technique to be easily utilized.

• Students: Damiano Marsilli, Arijit Nukala, Jonathan Young
• Mentor(s): Dr. Nicholas Theodore MD, Evan Haas & Antony Fuleihan, Dr. Youseph Yazdi, Drs. John Williams and Siri Khalsa

The spine is a mobile structure that allows the body to bend, twist, and lift. However, spine surgeons currently do not have a method of quantitative motion analysis for diagnosis and surgery pre-planning. Current spinal imaging techniques involve X-Rays, CT scans, and MRIs, which provide static images that are not representative of the spinal curvature of the patient during routine daily activities. As a result of limited imaging options, spinal fusion surgeries are often improperly prescribed. Our mentor’s team, Curve Assure, prototyped a device that can be placed on the back of patients to give doctors an insight into the behavior of the spine when the patient is moving over a longer period of time. However, to provide actionable insights for doctors, sensor readings must be augmented with spinal imaging techniques to produce a personalized quantitative motion model.

Figure 2 - Patient With multiple revisions [4]* In the example above, a patient had to undergo multiple surgeries, as the first two surgeries did not produce the desired outcome.

Our project aims to augment Curve Assure’s spinal IMUs with computer vision techniques. Our project has multiple phases. The first phase entails the development of a method capable of estimating an approximate spinal curve of the patient using monocular video. A pre-trained pose-estimation model will be adapted and trained using a transfer learning approach to produce spinal key points. Spinal key points, paired with a volumetric pose estimation model, will be used to produce an approximate spinal curve using splines in the pose estimation model. The second phase corresponds to the development of a transfer function that maps IMU data to spinal keypoint angles. This transfer function will be used to correlate IMU sensor data with the estimated spinal curve, producing a quantitative model of the patient's movement in 3D space. Lastly, the last phase entails deploying our method on the cloud and developing a pipeline that enables users to upload video and sensor data to the cloud and receive a quantitative model of the patient's spine.

Figure 1 - Phases of This Project

Current methods of accessing eligibility for spinal fusion surgeries are complex and unclear. Up to 1.62 million instrumented spinal fusion procedures occur each year [1]. However, these surgeries suffer from complication and revision rates as high as 50% and 36% respectively [2]. One contribution to this shocking statistic is that these procedures are often improperly prescribed. 17% of spinal fusion surgeries are performed on patients who should not have been recommended for the procedure in the first place [3]. In fact, the variability regarding the eligibility assessment can drastically change for patients. One study showed that even expert assessments can drastically change for a given patient from one day to another. These patients often suffer from negative outcomes as well as costly medical bills that are unnecessary.

- Minimum (Expected delivery: April 7)

1. Simple transfer function for translating IMU data and spinal keypoints to estimated spinal joint angles.
2. Spinal estimation model capable of detecting key points on the spine from monocular video.
3. Detailed documentation of interface and code structure between the two major components (transfer function & spinal estimation)

- Expected (Expected delivery: April 28)

1. Spinal estimation model can estimate spinal curvature and segment spinal sections through volumetric estimation and interpolation from monocular video.
2. Amazon Web Services (AWS) workflow and infrastructure for model training and testing for future development.
3. Detailed procedure for clinical use of the product including patient orientation, patient movement, and sensor location designed to produce the most descriptive spinal model of the patient.

- Maximum (Expected delivery: May 11)

1. AWS pipeline that allows for the remote upload of patient videos and IMU data and performs analysis in the cloud, returning the spinal model's output.
2. Detailed procedure and method for sensor calibration, so as to reduce the impact of sensor drift and position prediction errors over the 48-hour data collection period.

The task of spinal curve estimation is divided into two sub-tasks, spinal keypoint estimation, and spinal curve interpolation. The integration of these two components can be found in Figure 3. We discuss each separately.

Spinal Keypoint Estimation

The task of spinal keypoint estimation involves the consistent detection, and tracking, of spinal key points on a patient from monocular video. Formally, given a video, represented as a tensor of shape V = (F, C, H, W) where F is the number of frames, C is the number of channels (3 for RGB, 1 for grayscale), and H and W are the height and width of the frames respectively. Then, we must produce a function f such that f(V) = K, where is a tensor of shape (4, F, 2), corresponding to 4 key points found for each frame for both axes (u,b) in image coordinates.

To produce f, we leverage deep learning and the paradigm of transfer learning. Specifically, we will take a pre-trained pose-estimation model (OpenPose [10], PoseNet [11], DensePose [12], etc.) and extract its feature-detection backbone. As virtually all pose-estimation models are CNN-based, the backbones learned by these models will provide a latent image or video representation that encapsulates information useful to the task of pose estimation. Then, we will concatenate our own spinal-keypoint head, which will be used for the downstream task of spinal keypoint prediction. The spinal keypoint head will consist of a shallow series of fully connected layers. The specific depth and width of these layers, as well as the loss and activation functions involved, are yet to be specified and will be determined empirically with their impact on downstream accuracy.

We will explore various transfer learning approaches to train our spinal keypoint head. First, we will simply freeze the pose-estimation backbone and train the spinal keypoint head in a standard supervised setting. Then, we will explore various state-of-the-art few-shot transfer learning techniques, such as SpotTune [8] and surgical finetuning [9], and compare the results empirically.

Spinal Curve Interpolation

The task of spinal curve interpolation involves the interpolation of spinal key points on a volumetric mesh to produce a function approximating the spinal curve. Formally, given a tensor of key points K as described above, and a set of points P representing a volumetric mesh of the patient, we must estimate the function g(K,P) that best interpolates the points K, constrained by the mesh P.

To produce g, we will explore a variety of approaches. First, we will begin with simple bilinear interpolation between the spinal key points and the points produced by the volumetric mesh. This approach will serve as a baseline for comparison of future results. Next, we will explore more sophisticated approaches to interpolation, such as spline interpolation, inverse distance weighting, and radial-basis function interpolation [5][6][7].

• Documentation: Spinal Keypoint Estimation Documentation

We will develop a transfer function that utilizes patient IMU data in conjunction with estimated spinal key points to produce clinically relevant spinal range of motion (ROM) data. The transfer function takes inputs from the computer vision system’s predicted 3D spine key points, represented by the tensor in the shape (4, F, 2) as previously mentioned, and the IMUs orientation data, represented by the quaternion q_t = (q₁, q₂, q₃, q₄), to compute estimated joint angles for each of the four major spinal regions in the sagittal and coronal planes. We will implement the function in two phases, first using a simple transfer function and later applying a robust transfer function.

Simple Transfer Function

To test our initial approach to the transfer function, we will use an iPad with LIDAR capabilities to capture RGB-D datasets that we can use to estimate the spinal keypoint locations in 3D. These 3D locations will allow us to define a frame for the keypoint/box, enabling defined transforms from a global frame to the camera to local frames along the spine. To begin, we will measure the keypoint frames as an estimate of where the IMU is, defining the transforms _G^Fq as the rotation of the fixed reference frame F to the global reference frame G, _F^Kq as the rotation of the keypoint frame K to the fixed reference frame F, and the rotation _K^Kq as the rotations of the keypoint frames K with respect to each other. We can obtain the joint angles through quaternion multiplication, measuring the angle between two keypoints as the quaternion multiplication of the rotation between the global frame and the sensor, _G^Kq. These angles can then be transformed into clinical angles, yielding the flexion/extension angle F, the lateral flexion angle L, and the axial rotation angle T.

Robust Transfer Function

To reduce potential errors, we plan on segmenting the IMUs in parallel with the spinal keypoints, allowing us to find the transformations between the IMUs, which may be improperly placed, and the spinal keypoints. These transformations will enable better estimates of anatomy. We can follow the same method as we did in the simple transfer function to compute the clinical angles, adding an additional step in the quaternion multiplication as we will have the IMU as an intermediary and use its transform to the keypoint K, _IMU^Kq to find _G^Kq.

• Documentation: Transfer_Function_Documentation

There are two phases associated with cloud deployment.

The first phase is the test phase, where both large-scale versions of the spinal recognition model and the transfer function will be developed on EC2 instances on AWS. The reason for this is that the EC2s possess far greater computing power than personal computers, making it easier to train. Furthermore, this test data will be uploaded onto S3 buckets, making them easier to access for multiple applications.

• Design documentation for first phase can be found here - AWS_Modeling_Pipeline_Documentation_V1

The second phase is the deployment phase, where the data gets uploaded, stored, and processed, and a result is generated for the doctor. To do this, the data is uploaded to S3 via some sort of trigger (TBD), which is then fed into an EC2 instance to process the information. Then, the result of the EC2 is dumped into another EC2 or service, also TBD, to generate a report for the doctor. Then, this report is generated for the doctor to use.

• Design documentation for the second phase can be found here - AWS_Pipeline_Documentation_V1

Website deployment. A method of uploading and retrieving the files is necessary to interact with the system itself. Documentation of this process can be found here - AWS_Web_Based_Documentation_V1

A key point, however, is adding user authentication. Due to the limitations of the scope of the project that is not feasible. However, since our team is using AWS, next steps would involve AWS Cognito.

Summary Documentation can be found here: Summary Documentation

If dependency deadlines are not met, one of two actions will be taken: alternative solutions will be implemented or checkpoints will be pushed back.

Software

- Computing power will be acquired from the following:

• Personal laptops used by each team member provide sufficient computing power and support for processing the video while also processing information from the sensor data. Deadline not applicable.
• Amazon Web Services (AWS) EC2 instances, virtual machines with configurable hardware, will be utilized in case our personal computer’s processing power is insufficient. However, in order to run on AWS platforms, our team requires AWS credits.
  • Deadline to obtain: March 6th
  • Contingency: In general, if the credits cannot be obtained, our team plans to depend on AWS minimally or completely resort to our own computers.

- Cloud-based services are required to assist in processing and capturing data. The following AWS services may be utilized:

• S3 - storage, Sagemaker - Machine learning development space, Lambda/SQS - triggers to activate other AWS services, Amazon Kinesis Video Streams - processing video for machine learning.
• Our team needs to ensure that each member has the appropriate IAM roles in order to access these services, alongside having the necessary AWS credits to utilize the above services. Deadline to Obtain: March 6th, Contingency: In the event, our team cannot obtain AWS credits, our team will use a personal AWS using our own finances, but only in a limited capacity.
• Access to pre-trained pose models - Our team needs to have access to pre-trained pose estimation models used for transfer learning and volumetric estimation. Most of these models are open source, and will simply need to be installed and compiled.
  • Deadline to Obtain: Mar 6th

Footage for testing and validation:

• Access to videos (train & test data) - In order to train our model and validate our results, there must be access to videos that feature patients in a series of environments. The videos will need to be visually diverse to ensure model generalization.
  • Deadline to Obtain: March 6th.
  • Contingency: Our team will film our own videos.
• IMU Data - Data from the IMUs needs to be acquired, specifically those that come from the sample individuals moving according to the defined procedure for movement.
  • Deadline to Obtain: March 6th

Testing

• IMU Sensors - IMUs will be acquired from Evan once the team has felt that it is appropriate to proceed to the test phase. The current scheduling for testing is around the second to last week of April. Deadline to Acquire: April 17th

1. Milestone name: Obtain Train/Test Data for Keypoint Estimation
   • Planned Date: March 6th
   • Expected Date: March 11th
   • Status: Acquired April 24th
2. Milestone name: Produce Spinal Keypoint Estimation Model
   • Planned Date: April 9th
   • Expected Date: April 9th
   • Status: 100%
3. Milestone name: Determine Spinal Angles from Transfer Function
   • Planned Date: April 16th
   • Expected Date: April 27th
   • Status: 100%
4. Milestone name: Produce Spinal Curve Estimation Model
   • Planned Date: April 22nd
   • Expected Date: April 22nd
   • Status: 100%
5. Milestone name: AWS Pipeline Integration for all Models
   • Planned Date: April 30th
   • Expected Date: April 30th
   • Status: 80%

• 3/20: Codebase architecture and documentation completed

(https://github.com/CurveAssure/CISII/tree/main/CurveEstimation). Training loop and loss functions defined for both the bounding box and keypoint prediction tasks.

• 3/25: Pytorch Dataset objects created for both tasks.

• 3/27: Dataloader objects defined for both tasks. Model objects defined for keypoint task - ready to train keypoint detection model.

• 4/10: Documentation Added
• 5/5: Keypoint model created. Additional refinement of the model in progress.

• 3/27: Codebase architecture start and uploaded

(https://github.com/CurveAssure/CISII/tree/main/TransferFunction). Main Matlab code for the computation of the spinal range of motion complete.

• 4/10: Documentation Added
• 5/5: Ongoing development of transfer function.

• 3/16: AWS Sagemaker tool for keypoint labeling spinal keypoints and bounding boxes were deployed and mentors are actively labeling.
• 3/20: AWS EC2 resources requested for machine learning processing. One machine is provisioned for use, specifically, a P3 instance.
• 3/27: AWS Pipeline for processing input data and output data from an architectural standpoint (infrastructure and communication between services via SQS) is finished (set for maximum deliverable). Code can be found here - https://github.com/CurveAssure/CISII/tree/main
• 4/10: Additional configuration for client use is created via AWS Cognito. Webpages for interfaces in progress. Documentation Added.
• 4/19: Additional documentation for webpage client use is created. Webpages for interfaces are complete, but do not have the capability to verify the user due to an lack of expertise in programming.
• 5/5: Summary Documentation finished. MATLAB instance created and provisioned. Dynamic scaling pipeline completed. Waiting final model and transfer function for deployment.

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • A novel IMU-based clinical assessment protocol for Axial Spondyloarthritis: a protocol validation study
  • SpotTune: Transfer Learning through Adaptive Fine-tuning
  • A Cloud Scalable Platform for DICOM Image Analysis as a Tool for Remote Medical Support
• Project Checkpoint
  • Checkpoint Presentation
  • System Requirements Documentation
  • Testing Plan
• Paper Seminar Presentations
  • Critical Reading Presentation
• Paper Seminar Report
  • Critical Review Paper Report
• Poster Teaser Presentation
  • Poster Teaser Presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

Zhang, Y. et al. Therapeutics for enhancement of spinal fusion: A mini review. J. Orthop. Transl. 31, 73–79 (2021).

Rajaee, S. S., Kanim, L. E. A. & Bae, H. W. National trends in revision spinal fusion in the USA. Bone Jt. J. 96-B, 807–816 (2014).

Dhillon, K. Spinal Fusion for Chronic Low Back Pain: A ‘Magic Bullet’ or Wishful Thinking? Malays. Orthop. J. 10, 61–68 (2016).

Diebo BG, Shah NV, Stroud SG, Paulino CB, Schwab FJ, Lafage V. Realignment surgery in adult spinal deformity: Prevalence and risk factors for proximal junctional kyphosis. Orthopade. Hornbeck, Haysn. “Fast cubic spline interpolation.” arXiv preprint arXiv:2001.09253 (2020).

Lu, George Y., and David W. Wong. “An adaptive inverse-distance weighting spatial interpolation technique.” Computers & geosciences 34.9 (2008): 1044-1055.

Wright, Grady Barrett. Radial basis function interpolation: numerical and analytical developments. University of Colorado at Boulder, 2003.

Guo, Yunhui, et al. “Spottune: transfer learning through adaptive fine-tuning.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

Lee, Yoonho, et al. “Surgical fine-tuning improves adaptation to distribution shifts.” arXiv preprint arXiv:2210.11466 (2022).

Osokin, Daniil. “Real-time 2d multi-person pose estimation on cpu: Lightweight openpose.” arXiv preprint arXiv:1811.12004 (2018).

Kendall, Alex, Matthew Grimes, and Roberto Cipolla. “Posenet: A convolutional network for real-time 6-dof camera relocalization.” Proceedings of the IEEE international conference on computer vision. 2015.

Güler, Rıza Alp, Natalia Neverova, and Iasonas Kokkinos. “Densepose: Dense human pose estimation in the wild.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-08).