Agricultural Robotics and 3D Reconstruction
From July to October 2025 I was a Research Intern at the Shanghai Artificial Intelligence Research Institute (SAIRI), also affiliated with MindSpore Shanghai, on a project run with the Shanghai Academy of Agricultural Sciences. The setting was controlled indoor agriculture: plants grown without soil on stacked shelves, under managed light and nutrients. The goal of the larger project was a plant-state model that could assess growth and guide the growing environment. My part was the upstream sensing chain that feeds it: I built the robotic-arm control, the mobile platform, the multi-view RGB-D acquisition, and the three-dimensional reconstruction, and I led the leaf-area estimation and root measurement work with colleagues.
What the chain had to deliver
The system measures the same plant repeatedly over its growth without destroying it. That sets the requirements. A measurement is only useful if it is comparable with the one taken a week earlier and if it answers a biological question, so the chain has to be repeatable end to end: the platform returns to the same station, the camera visits the same viewpoints, the reconstruction is computed the same way, and the quantities extracted from it are defined with the agronomists rather than by what happens to be easy to compute. Precision is traded against acquisition time, image count, processing cost, and the plant's own variation, and those trade-offs have to be made explicitly, per quantity.
Platform and motion
The mobile platform moves between growing stations and has to arrive at each one in a repeatable pose, because the arm's viewpoint sequence is defined relative to the station. I wrote the layer around the platform's existing navigation stack—motion commands, pose handling, and the coordination with image acquisition—and read enough of the underlying methods to know where the stack would and would not hold.
The arm carries the camera. For each plant I designed a viewpoint sequence around it: a set of camera poses spread over azimuth and elevation with enough overlap between neighbours for the reconstruction to link them. Every pose has to be reachable and every transition has to clear the plant and the shelf, so reachability and collision constraints are part of the sequence design rather than something checked afterwards. Arm motion, camera triggering, and the recording of images and poses run as coordinated ROS nodes, so that each captured frame carries the pose it was taken from.
Acquisition
Capture uses an Intel RealSense D435i, recording colour, depth, and the robot pose together. The forward-kinematic pose of the arm gives every frame a known position in the arm's base frame, which is what lets a purely visual reconstruction be brought back to physical units—a requirement, since leaf area and root length are reported in centimetres, not in an arbitrary scale. Lighting is controlled so that batches are comparable, and the dataset is organized by plant, growth stage, and batch, with the acquisition metadata recorded alongside the images.
Reconstruction
Reconstruction runs as a scripted COLMAP pipeline: feature extraction and matching, sparse structure-from-motion, undistortion, dense multi-view stereo, fusion, and meshing. Plants are a demanding subject for this—thin, self-similar, weakly textured—and the pipeline's parameters are tuned for that subject: more and better-localized features per image, a matcher that keeps only geometrically consistent pairs, and a mapper that is allowed to produce several sub-models so that the largest consistent one can be selected automatically.
Before reconstruction, the subject is separated from the scene. A mask built from depth and colour keeps what is close to the camera and green, which removes the shelf, the neighbouring plants, and the room from the images that reach the matcher. After reconstruction, the point cloud is cleaned: outliers are removed, the background is segmented away, and the result is checked for pose consistency, reprojection error, and coverage before anything is measured from it. The fused point cloud and the Delaunay mesh are the artifacts measurements are taken from; the Poisson mesh is kept for visualization.
Measurement
Two quantities were extracted. Leaf area is computed by segmenting individual leaves on the reconstruction and integrating their surface; root geometry is measured from the reconstruction of the exposed root system. The definitions of both were settled with the agricultural collaborators, so that what the system reports is a quantity an agronomist would act on and not only one that is geometrically self-consistent.
Two decisions that shaped the system
The plants were grown densely. Neighbouring leaves blocked the viewpoints the reconstruction needed, leaves reflected light, crossed one another, and changed shape as the plant grew, and the reconstruction of a plant in place could not see enough of it. The decisive change was not to the algorithm but to the scene: I added a second arm whose job is to lift a plant out of the growing area so that the camera arm can observe it alone. Once the plant stands by itself, the neighbours that blocked the useful viewpoints are gone, and the camera can go where the measurement needs it to be rather than where the shelf allows. This is the configuration the system ran in.
Lifting the plant also exposed its roots. Whether measuring them was worth the effort was an agricultural question, not a geometric one; I raised it with a collaborator from the agricultural side, and root measurement was added to the workflow as a result. Together these two changes reorganized the chain around the information the downstream model needs: the measurement decides the coverage, resolution, and precision the acquisition has to provide, and the acquisition is designed backwards from it.
Outcome
I delivered the complete upstream chain—platform, arm, acquisition, and reconstruction— as working components of the system, with the leaf-area and root measurements defined and running on top of it. The part of the reconstruction and inspection workflow that can be released is packaged as two toolkits, VisRecon and Multiview Reconstruction. They are what is public of the code behind this page, not the whole of it.
The longer account of how this project changed the way I think about engineering research is in How Agricultural Robotics Changed My View of Engineering Research.
Resources
Not everything from this project is published here. Some of it is still being organised, and some cannot be released.