At NOAA Fisheries, optical technology plays a major role in collecting data to better understand our marine and coastal ecosystems. Optics uses tools such as advanced underwater cameras to generate video and images that scientists use to better understand biodiversity and track species behavior.
To collect data, many surveys have traditionally relied on time spent aboard a NOAA ship. Researchers usually sail to the area where the survey is taking place, but they are limited to collecting data in areas where the vessel can navigate. Time spent on board a vessel can be very costly and researchers must plan their efforts based on the time of year the vessel is available.
Optics increases our ability to collect this same data using video and image data instead of human observation. This is particularly useful in the deeper parts of the ocean, which can be difficult to access due to lack of light and extreme pressure. We are able to “see” the underwater ecosystems we protect while ensuring safer operations. Integrating this information into our research provides a better understanding of valuable marine resources.
The results have a direct impact on management decisions for the conservation of habitats and species. In warmer climates, we use optics to study corals as part of the National Coral Reef Monitoring Program. At science centers across the country, we use remote underwater video cameras to identify marine resources.
In colder climates, such as remote areas of Alaska, we identify marine mammals using drone images and infrared optics. And in Antarctica, we attach cameras to autonomous underwater vehicles such as gliders to collect information on krill and plankton.
Data Challenges
However, as researchers rely more and more on optics, they face a growing problem: The videos and images they collect generate enormous amounts of data. Dr. Matthew Campbell, Head of the Gulf and Caribbean Reef Fish Branch at the Southeast Fisheries Science Center, shared his perspective on the challenge:
“Once the investigation is complete, we will have collected thousands of hours of footage which will need to be reviewed. Many investigations deploy hundreds, if not thousands, of cameras. The amount of data can be overwhelming.
For many surveys, transforming raw data into usable information can take a long time. Each data point must be manually annotated for identification and all fish counts must be recorded. The process often involves correctly marking thousands of data points in a video or image, one by one.
Each survey team has a limited number of resources available, particularly in terms of personnel, so a bottleneck may occur during the post-processing phase. Once the annotation stage is complete, researchers can then begin the data analysis process, which can also be time-consuming. Complete processing and analysis of optical data can take a year or more, depending on the amount of data collected.
Once collected, storing the data can also be a challenge. Many researchers choose to store data locally, meaning it is limited to the computing power of the local network. Additionally, only those who have access to the local network can work on the dataset. Therefore, the data is not available for others to contribute to the analysis process or conduct related scientific research.
Of course, working with our survey data requires highly technical skills. Flexibility in which software to use is limited. Many fisheries biology researchers must first learn programming languages and technical software before analyzing data. Although user capabilities range from beginner to expert, more staff with advanced computer skills are needed.
To address today’s challenges in data collection and optical processing and analysis, we are focusing our investments in three main categories: data acquisition, powerful processing tools and increased accessibility.
Data Acquisition
We will use small, inexpensive optical devices that can run on multiple platforms to collect images. By integrating these devices into our unmanned systems already in place, we can cover areas that ships cannot cover. Dr. Campbell explained:
“Guaranteeing delivery time is a difficult process. We carefully plan our work at sea within the limited vessel time allotted for a specific investigation. Using optical technology, we are reducing our reliance on NOAA ships by using unmanned systems such as remotely deployed gliders.
In regions where data collection capabilities are limited, we will begin collecting data. High-resolution satellite imagery will extend our survey reach to these regions. Aerial imagery will be validated using unmanned systems on ships and aircraft.
Powerful processing tools
We will use a cloud-based system to efficiently manage our large data sets. This will allow us to collaborate remotely and increase IT requirements without investing in expensive physical infrastructure. We will also improve the tools we already use to ensure they handle the data formats of all fisheries science centers.
We will process data more efficiently with AI, machine learning and edge computing capabilities that automatically analyze images. This saves time, reduces resources and leaves more availability for other projects.
Increased accessibility
We will develop state-of-the-art classifiers, detectors, trackers and segmentation models for marine life. Our labeled optical image tools and data sources will be accessible to our scientists in various regions. Easier access through web portals will help data quickly turn into products that inform management decisions across the country.
On the staff side, we are recruiting data scientists and engineers to create easy-to-use journeys. For our current staff, we will create and share resources that make it more efficient to learn, experiment, and adapt models for specific applications. In addition to participating in user groups and rotational assignments, this will allow staff to increase their analyst and developer skills.
Optics in action
Some of these efforts to modernize our optical technology are already underway. One such case is the Gulf Fishery Independent Survey of Habitat and Ecosystem Resources, which began field work in 2020. Currently funded for 10 years, it brings together three historic surveys in the Gulf region under a single, global based on habitat. Dr. Campbell highlighted the success of this project:
“We have a cloud-based automated processing system with automated workflows and shared technical knowledge. This type of system benefits not only the scientists working directly on a project, but also those outside of the labs where the data was created.
Each year, this major project deploys approximately 2,000 cameras and maps approximately 3,000 linear nautical miles. The survey results are essential to help assess the status of managed reef fishes throughout the Gulf of Mexico. This 30-year time series of camera data is used to estimate fish abundance and habitat composition in the northern Gulf of Mexico.
A key part of the survey involves collecting three types of still video data: single-camera view, 360-degree view, and stereo view. The cameras are baited, which encourages fish to explore the surroundings and bring them closer to the cameras. The single camera and 360-degree camera views allow us to identify fish species and abundance, while the stereo camera view allows us to estimate fish length. Using this optical data, we then leverage machine learning to support the fish counting process.
We train computer models for fish detection and classification using open source software and data from our manually annotated image and video library. Optical conditions, fish density and habitat complexity can impact model accuracy. This method allows us to quickly generate semi-automated fish counts, which helps reduce human effort and results in a more efficient fish counting process.
The future of optics
In the future, we will continue to harness the power of optical technology. Through Funding for the Inflation Reduction ActWe will conduct coupled video-acoustic studies to improve data collection on reef fishes, particularly red snapper, in the Gulf of Mexico. We will use a combination of underwater video data, imaging sonar, split beam echo sounder and x-rays. We will develop new data collection and automation methods to help us better understand the whole red snapper populations in the Gulf. These projects are planned for the first half of 2025.
The future of marine research is closely linked to continued advances in optical technology. By investing in innovative tools and collaborative efforts, we can address the challenges of data collection and analysis while opening new possibilities for understanding and protecting our oceans. Through these efforts, we not only improve the way we study marine life, but also help preserve our vital marine resources.