Since the Net Gains Alliance began our work in 2017, we’ve called for implementation of unique trip identifiers (UTIDs) as a way to quickly and reliably integrate commercial fishing data. Ideally, unique trip identifiers are machine-generated at the first point of data collection and automatically propagated to link all of the information generated by a trip such as trip reports, dealer reports, observer information, and Vessel Monitoring System (VMS) tracks. UTIDs are often lauded as the key to data interoperability, and they could be if we were designing our fisheries data systems with a clean slate. The reality is that we’re trying to reverse engineer interoperability into distributed data systems that have evolved over decades. Solving this puzzle requires a more stepwise approach.
The need for improved interoperability is urgent. Efficient data integration reduces the amount of valuable time, talent and resources spent on tedious data matching and cleanup, and would ensure timely and comprehensive data are available to scientists, managers, and fishing businesses. Every U.S. region is facing challenges that clarify the stakes and the costs of inefficiency. On the West Coast, wind energy development is driving the need for a comprehensive picture of the spatial footprint and value of fishing effort. On the East Coast, climate change impacts to fishing effort and stock distributions require the ability to compile and share information across state and federal management offices. The immediacy and data-rich experience of consumer technology stands in stark contrast to the reality of fisheries data systems, where it can take weeks or even months to make data available.
As NGA reflects on the data modernization work of the last 7 years, a core theme we want to recognize is the value of incremental progress, where small strategic steps add up to big interoperability gains over time. Over the past year NGA held conversations with data experts at NMFS Headquarters, NMFS regional offices and science centers, and regional Fishery Information Networks (FINs). Here we share the key messages we’ve heard about UTIDs and interoperability, where the experts are seeing progress, and what’s needed to see this work through.
Four key messages
Focus on interoperability, not UTIDs
Interoperability is the goal and UTIDs are one tool. Other approaches to identifying and connecting data streams can provide the same utility. UTIDs can help facilitate data integration, but only once data systems are capable of utilizing them. To paraphrase one expert: UTIDs were initially thought of as a number; now the term describes the broader conversation about how data can be integrated across systems growing at different speeds. Focusing on UTIDs as a simple solution can downplay the complexity of the challenge, and investment needed to modernize data systems and work toward greater interoperability.
UTIDs can still be a useful tool. The Alaska Region automatically creates UTIDs to link records (including EM, landings reports, observer reports and biological data, and VMS) after the records have been created and received by the region. While these identifiers are not generated prior to each trip this approach serves the same purpose of automating the matching process and minimizing the amount of manual reconciliation and review needed.
Integration happens through data modernization.
We heard UTIDs described as the “top of the data modernization pyramid.” Data interoperability can’t be imposed upon siloed data systems that can’t easily communicate. The heavy lifting of interoperability happens through the step by step work of updating, streamlining, and connecting data sources while keeping these systems functional and not breaking what’s already in place. The tactical steps that advance interoperability take many forms such as continuing to implement electronic reporting, having consistent definitions and taxonomies for ports, permits, and vessels; modernizing legacy technology, and improving data quality. UTIDs were described as a final cleanup step or finishing touch.
Solutions need to be regionally tailored.
The work of integrating data streams is different for every region, fishery, and even gear types, each of which may have a different combination of data sources and levels of coverage (such as the proportion of trips observed). In the Greater Atlantic Region, a subset of vessels submit pre-trip notifications and electronic Vessel Trip Reports, providing defined start and end points to a commercial fishing trip. On the West Coast, different agency perspectives on defining the endpoint of a trip has been one of the challenges to data integration. While there is no top-down solution that works for all regions, there is still value in sharing experience and approaches across regions, as well as developing tools (such as the open source electronic monitoring web framework described below) that can be adapted to suit the needs of different monitoring programs.
Progress is happening.
Each step toward modernizing the data ecosystem makes it easier for the next piece to fall into place. Progress is cumulative, and NMFS and its partners have learned from past efforts and are now investing in more effective projects. Our conversations also emphasized the importance of institutional knowledge. Interoperability isn’t simply a technical challenge that can be solved with an IT contract; problem solving requires people with in-depth knowledge of how fisheries operate and fishery data systems currently work.
Where we’re seeing progress
Our conversations highlighted examples of big steps forward toward interoperability. Much of this work is supported through NMFS’ Fishery Information Systems program, including funding for discrete projects as well as Professional Specialty Groups that foster collaboration among experts.
Several recent and ongoing East Coast projects illustrate how different but complementary data modernization projects can help move interoperability forward.
Provide a single, comprehensive data source: The Northeast Fisheries Science Center and Greater Atlantic Regional Office co-developed the Catch Accounting and Monitoring System (CAMS), which compiles and provides a comprehensive source of commercial landings and discards for science and management purposes in the region. Previously, analysts and scientists could employ different ways of defining datasets for different purposes. The CAMS system provides a single source for all data users, so that data used for different purposes are more easily reconciled and more transparent to the public.
Improve data quality and timeliness for easier matching: Linking vessel and dealer reports is the key step toward faster, easier data integration. In 2021, the Greater Atlantic region fully transitioned to electronic vessel trip reports (eVTRs), which has improved data quality by reducing the potential for small errors, such as a blank field, that can cause matching errors and require more manual effort. GARFO is now building on this work by modernizing the dealer reporting system in conjunction with ACCSP, the Atlantic FIN. This will include developing a new data model and shifting outdated reporting methods (e.g. file uploads) to API-based submissions, which will provide a similar point of data validation to facilitate faster matching and could help support a UTID.
Streamline the development of new electronic reporting apps: NMFS can reduce the time that it takes to develop and deploy new reporting applications with an open source development framework that can be customized to different fisheries and reporting requirements. A cross-regional team of experts from the Greater Atlantic and Southeast Regions and Atlantic Highly Migratory Species Program are collaborating to develop the source code, documentation, and governance policy for this new framework, drawing on lessons learned from the GARFO’s Fish Online reporting application. This framework will support more consistency across the suite of reporting apps in use, enable the agency to make updates to source code (e.g., security updates), and enable developers to focus on program-specific needs. This project will also produce a proof of concept for the for-hire fishery in the Southeast.
Consistently identify vessels and permits: On the East Coast, vessels may hold permits administered by three separate offices (Greater Atlantic, Southeast, Atlantic Highly Migratory Species) each with their own permitting structure. It can be time consuming and require a lot of manual intervention to share information across these permitting structures, as the same information may not be collected consistently across systems. A new FIS project for 2024 will develop a comprehensive vessel registry to easily identify a vessel, its permits, and reporting obligations. Each region maintains their own permitting system while being able to efficiently match information across these systems. The Office of Science and Technology will host the registry and allow access to the partners such as the Fishery Information Networks.
Our takeaways and recommendations
Net Gains commends the work of NMFS, staff, and partners to advance data interoperability and the positive examples we’re seeing of strategic investment and collaboration across roles and regions. While we believe UTIDs are achievable and desirable in some circumstances, there is also an important story to tell about how stepwise improvements can make big, ambitious changes possible.
Our recommendations to NMFS and its partners include:
Communicate progress. We encourage NMFS to identify metrics to help communicate stepwise progress toward improved data interoperability, including through the agency’s development of data modernization metrics and reporting on Inflation Reduction Act funds.
Dedicate resources and staffing. Sustained progress toward interoperability requires NMFS and its partners to have the resources to engage in multi-year planning and problem solving. This includes hiring and retaining FTEs with institutional knowledge, and continuing to invest in the FIS program and Professional Specialty Groups.
Institutionalize better internal data sharing practices. Interoperability is not just a technical challenge; there can also be cultural and procedural impediments to data sharing within offices within the agency and with partners (e.g. procedures for sharing business information, harmonizing multiple data sharing agreements and MOUs.) Part of achieving better data interoperability is enabling secure, streamlined internal data sharing, particularly as moving data to the cloud makes it logistically easier to share and access data.