Surgeon Maps

The Challenge

On October 20, 2026, at The MedTech Conference in Boston (AdvaMed), four teams of four students had one day to go from a clinical problem to a working prototype and a pitch in front of industry judges. The competition was sponsored by Stryker and run by Moonshot, which provided the surgical robot and the software platform.

How might we better integrate patient-specific preoperative planning into intraoperative surgical decision-making for hip FAI?

Femoroacetabular impingement (FAI) is abnormal contact between the femoral head-neck junction and the rim of the hip socket, caused by extra bone. It is commonly treated arthroscopically: the surgeon removes the excess bone through small portals, guided by a camera. The difficulty is to remove enough bone without damaging the articular cartilage or the labrum, while translating a preoperative plan, often CT-based, into a live 2D view.

Teams chose a focus among FIND (localize the target and the protected structures), REACH (move the tool to a chosen location) and PROTECT (avoid what must be preserved). We chose FIND: without knowing where the target and the protected structures are, neither robotic positioning nor a safety warning has any context.

Surgeon Maps

The idea I proposed, which the team adopted, works like Google Maps for the surgeon: show where to drill and where not to. The surgical camera is mounted along the drill shaft, so the drill tip is always at the center of the image. A segmentation model outlines the excess bone to remove in green and the cartilage and labrum to avoid in red, and the operator steers the robot by hand with a controller, relative to that overlay. The AI only reads the scene; every movement comes from the person holding the controller.

Data and Model

We simulated FAI lesions on the hip model with clay and filmed them with the surgical camera, collecting more than 1,000 images. They were auto-annotated with SAM 3 in Roboflow, then all four of us reviewed the labels, splitting the images between us. Each of us launched training runs with more or less data: Olivia trained the first model to get a working version, and I trained the final one used in the demo, a 3-class RF-DETR segmentation model (excess bone, cartilage, labrum).

These numbers come from a single physical hip model in which each structure is a distinct color of clay, so the model largely recognizes colors. Real bone, cartilage and labrum do not look like this.

Getting It to Run Live

Late in the day, with about 30 minutes left before pitch preparation, the model was trained but nothing ran live. My teammates, Olivia in particular, had already tried to debug the two blockers; while they started on the pitch, I kept working on them.

• The surgical camera crashed the software, a problem only our team had. I traced it to an Intel graphics-driver component failing during frame decoding in OpenCV's Windows media backend, not to the platform or the camera. Disabling hardware transforms stopped it from loading, and the camera kept streaming at 1280x720 at the same frame rate.
• No team could reach the vision model. Two causes: the model configuration pointed to a name that could not be loaded, and no local inference server was running. I corrected the configuration and used a setting that tries a local server first, then falls back to Roboflow's hosted endpoint.

The organizers asked me to push both fixes so they could integrate them, and to share them with every team, which I did.

The Navigation Layer

With the pipeline running, we added a navigation window on top, written from a detailed specification the team put together. Every frame, it segments the excess bone, cartilage and labrum, marks the target centroid with a crosshair, and shows the target confidence and the model latency (about 0.2 to 0.4 s). Statuses are smoothed over several frames so that one noisy frame cannot flip them, and a warning about proximity to a protected structure overrides everything else.

Two rules shaped the interface. The wording never says "safe to cut": every status describes position relative to the approved plan. And a permanent banner reads "Research prototype | Human controlled | Not for clinical use | Sends no robot commands".

Live Demo and Judging

Aadhira, Silmi and Olivia built and delivered the pitch, and I ran the code and operated the robot during the presentations: with the segmentation running on screen, the robot removed the clay lesion under the surgical drape. One of the judges, Karolina Wolf of ResMed, took the controller and drove the robot herself.

Almost every judge asked about next steps:

• Audio guidance, like a car's parking sensors, so the surgeon does not have to look away from the field.
• A robot lockout near protected structures, which would require near-perfect accuracy and raises hard regulatory questions about accountability when AI acts.
• Drill depth, raised by Robert Cohen of Stryker: estimating how deep the drill is, not only where it is in 2D.
• AI adoption in medtech, discussed with Shaun Braun of Zimmer Biomet, who saw regulation, more than company inertia, as the reason the industry moves carefully.

Limits

The model works because each structure is a different color of clay. Real bone, cartilage and labrum look nothing like that, and the guidance is in 2D pixel space, with no millimeter calibration and no registration to a CT-based plan. Doing this for real would take large annotated datasets of arthroscopic images, each patient's CT scan and many patients. The roadmap in our pitch went step by step: robustness to lighting, angle and occlusion, metric calibration, registration with the CT plan, realistic imagery, then a clinical study, increasing the model's responsibility only as the evidence improves.

This was a feasibility demo on a model hip, not evidence that the approach would work on patients.

Team

Rany Fouad (Columbia University), Aadhira Nair (Northeastern University), Silmi Sultaan Ali (UC Berkeley) and Olivia Bartels (Milwaukee School of Engineering).