As part of a series looking into future trends and developments in shipping, Norwegian Hull Club has been identifying some new and interesting developments with automation and autonomous concept vessels. This first insight provides a round up of the current state of development and raises some questions on possible impacts such developments may have.
The benefits of automation and autonomy to varying degrees have become self evident in many areas of commercial activity including transportation. Many cities world wide have automated and partially autonomous rail or metro systems. Fully automated container terminals are well established as efficient ways to manage the ship-to-shore interface. Airline travel now depends in large part on automation within flight systems for certain stages of the flight program.
Driverless car technology is also now at a relatively advanced stage with large major cap companies like Google and Tesla pioneering R&D along with many of the conventional car manufacturers. Google is currently testing its driverless car technology out in California in areas the car’s computer has pre-studied, with on-board computers sharing data between vehicles but at limited speeds (25 mph) and still with a driver on board for safety. Meanwhile in Michigan at MCity, a 32-acre testing ground at the University of Michigan, fifteen companies including Ford, General Motors and Nissan have cooperated to help build a facility to conduct research into how a fully autonomous car would perform in an unpredictable (real) world environment. Volvo states that they will be ready to test 100 of their driverless cars on public highways in Gothenburg in conjunction with Swedish authorities by 2017.
Whilst automation and autonomy in shipping (unmanned vessels) has been considered for some time with various systems already being field tested, it is still seen as a relatively long way off and remains mired in uncertainty over a multitude of technical, practical and regulatory concerns. That said, the continued drive for extracting cost efficiencies and increasing competitiveness within the transportation chain continues to be a powerful driver in this field.
Plymouth University in Devon, England has recently announced its new MARS project (Mayflower Autonomous Research Ship). They describe the issue as follows;
“While advances in technology have propelled land and air-based transport to new levels of intelligent autonomy, it has been a different story on the sea. The civilian maritime world has struggled to keep pace with technology due to a combination of cultural and cost factors. The autonomous drone technology that has been used so effectively in situations considered unsuitable for humans has not been harnessed by the shipping industry, which continues to steer the conservative course, its diesel engines pumping out carbon emissions and its manned crews at risk from piracy.
It begs the question, if we can put a rover on Mars and have it autonomously conduct research, why can’t we sail an unmanned vessel across the Atlantic Ocean and, ultimately, around the globe?”
Developments in technology and computing would appear now to provide positive proof that the concept could be achievable (even towards higher degrees of autonomy/intelligence) if various applications can be successfully brought together, allied to breaking through current data transmission limitations not normally experienced by shore-based applications.
Rødseth & Burmeister in their paper ‘Developments toward the unmanned ship’ describing the EU funded project ‘MUNIN’ (Maritime Unmanned Navigation through Intelligence in Networks) illustrate quite nicely the practical differences between various system models along with the underlying determinism concerns that are coupled to increased autonomy. The more autonomy that is assigned to a robot, the less controllable it is.
Image Source: Rødseth & Burmeister, ‘Developments toward the unmanned ship’
Autonomy in the maritime sphere is described by MUNIN as;
“Next generation modular control systems and communications technology [that] will enable wireless monitoring and control functions both on and off board. These will include advanced decision support systems to provide a capability to operate ships remotely under semi or fully autonomous control.”
The limitations of fully autonomous ships are clearly understood and integrating them into existing maritime transport systems has not been fully reconciled. As Rødseth & Burmeister point out, ‘a fully intelligent ship will have limited commercial utility as safety is difficult to guarantee and control of speed, fuel consumption and arrival times is more uncertain. MUNIN will therefore develop the principles for a basically automatic ship, but with some capability to handle certain unplanned situations within defined constraints.’
Automatic avoidance of detected and recognized targets in accordance with good seamanship and established rules such as ColRegs could still be quite a way off then since ticking this box off requires significantly advanced hardware and software (signal processing) solutions.
DARPA (the US government’s Defense Advanced Research Projects Agency), used to dealing with hard-problems, is tackling both the hardwear and software issues head-on.
In a recently issued ‘RFI’ (Request for Information) on ‘Hardware and Software for Unmanned Vessel Perception’ with responses due by April 28th 2015, DARPA’s Tactical Technology Office seeks to identify currently available sensor systems and image processing technology to support automatic real-time surface vessel detection and classification from passive, i.e., electro-optical/infrared (EO/IR), or non-radar active, i.e., light detection and ranging (LIDAR), sensors.
DARPA state that their
“ACTUV program is developing an experimental unmanned surface vessel (USV) for a variety of potential U.S. Navy missions  and the U.S. Navy is developing smaller USVs in the 7 to 12 meter size range. Such vessels will rely on a suite of active and passive sensors for general situational assessment; in particular, such vessels need to comply with International Regulations for Preventing Collisions at Sea, or ColRegs . While active sensor systems like radar and Automatic Identification System (AIS) can provide significant relevant information about vessel traffic, it is desirable to augment USVs with multiple sensor modalities. In particular, sensors such as EO/IR and LIDAR offer unique capabilities to detect and classify surface vessels and surface vessel states.”
DARPA continue that it is interested in hardware and software solutions that enable an autonomous lookout from a surface vessel; specifically, information on technologies that automate a “lookout by sight” from sea surface vessels. ColRegs Rule 5 states, “Every vessel shall at all times maintain a proper look-out by sight and hearing as well as by all available means appropriate in the prevailing circumstances and conditions so as to make a full appraisal of the situation and of the risk of collision.”
The RFI seeks information in three areas: 1) Maritime Perception Sensors (passive optical or non-radar active), 2) Maritime Perception Software (algorithms and software to support Maritime Perception Software for detection, tracking and classification from passive optical or non-radar active imagers that can address near-real-time command and control), and 3) Classification Software for Day-Shapes/Navigation Lights (algorithms and software to support detection, tracking and classification for passive optical or non-radar active imagers for the purpose of detecting day shapes and lights in a marine environment).
Should DARPA succeed with the ACTUV program they will have gone beyond the autonomy objectives of the MUNIN program, albeit limited to military applications however this an oft used route for incubating commercial applications to the wider marketplace. To use the Rødseth & Burmeister model, they will have an intelligent vessel with ‘full freedom’ determinism.
Dealing with the software side of things requires development of computing powers capable of dealing with the plethora of raw data likely produced by the many hard-wear sensors feeding inputs into the control system. This is where augmented cognition, the study of the augmenting of human mental functions by computer programs, comes into play.
Here also, distinct limitations have long been recognized. Mary Cummins (associate professor in the Massachusetts Institute of Technology Aeronautics and Astronautics department, director of the MIT Humans and Automation Laboratory, and a former U.S. Navy fighter pilot) pointed this out clearly in a 2010 paper titled ‘Technology Impedances to Augmented Cognition’. She states when attempting to address the ‘elephant-in-the-room’ that;
“Significant uncertainty is inherently present in command-and-control domains, where the bulk of Aug-Cog research has taken place. When environmental uncertainty is coupled with the uncertainty inherent in all psychophysiologic measures and their subsequent analyses, the outcomes of any resultant predictive models are predictions that are so broad that they are not useful, or the predictions carry so much uncertainty they cannot be trusted.”
“A clearly stated goal of the Aug-Cog community is to enhance human information-processing capabilities through the design of adaptive interfaces via cognitive state estimation. This work is important and highly relevant today with the U.S. Department of Defense’s increasing use of supervisory control systems, particularly unmanned systems. However, although researchers have demonstrated some interesting proofs of concept and made incremental progress in terms of hardware and software advances, the results are preliminary at best. Moreover, they do not suggest that the ultimate desired results are achievable in the near term. For the general field of augmented cognition to make critical advances, significant focus (and funding) should be placed on hardware development and associated signal-processing efforts. Without critical advances in EEG and other neurologic and physiologic technologies, the Aug-Cog effort cannot make significant progress or be operationalized. Furthermore, significant additional research is needed in the development of decision theoretic models and predictive algorithms in dynamic, highly uncertain domains for open-loop systems with noisy sensor data.”
DARPA has had a Strategic Computing Program in place since 1983 with its stated aims including development of autonomous systems for military application and intelligent functional capabilities for natural language, vision, speech, expert systems, navigation, planning and reasoning. Since Mary Cummins paper in 2010 a great many strides have been made in the Aug-Cog space.
Dario Gil (Director of Symbiotic Cognitive Systems at IBM Research) at TED in October 2014 in a talk titled: ‘Cognitive systems and the future of expertise’ summarised these main developments; ‘computers were thought to lack the capability for strategy and Deep Blue went on to beat Gary Casparov at chess. Then computers were thought to be unable to handle the nuances of human language and Watson went on to beat Ken Jennings and Brad Rutter at the ultimate game of language and knowledge, Jeopardy’.
Now, machine learning algorithms allow Watson to learn from the raw data to solve complex problems at high speeds with wide ranging applications. Indeed Dario cites people working in collaboration with cognitive computing is the future in expertise, particularly when it comes to excluding human biases whilst utilizing analysis and discovery capabilities, unsurpassed in their ability to find connections and bring evidence across all available digital knowledge.
IBM’s new Watson based cognitive computing powers have already been leveraged into the legal space with the creation of ROSS. IBM define ROSS as follows;
“ROSS is an artificially intelligent attorney to help you power through legal research. ROSS improves upon existing alternatives by actually understanding your questions in natural sentences like - "When can a debtor reject a collective bargaining agreement as per the US bankruptcy code?"
ROSS then provides you an instant answer with citations and suggests highly topical readings from a variety of content sources.”
SoundHound have recently released a new speech assistant app ‘Hound’. According to the company, Hound's Speech-to-Meaning engine delivers unrivalled speed and accuracy where speech recognition and natural language understanding is done simultaneously and in real time. It can handle multiple criteria questions. In addition to understanding natural language sentences with specific details and criteria Hound can also understand exclusions and negation.
Should such gains within the Aug-Cog space be fully capitalized on within the current maritime r&d programs, then in theory a higher degree of determinism ought to be achievable given the improvements in hard-wear sensors. Data transmission issues still prevail though. The amount of bandwidth required for the real time processing and computing that the likes of Watson, Hound, ROSS and Google’s driverless cars do when they talk to one another is significant and not currently available in a truly offshore capacity.
The same constraints limit development within the Autonomous Underwater Vehicle (AUV) space. These are ideal sonar platforms for acoustic and optical imaging in a variety of water depths and are now widely used. Due to the limited bandwidth available from acoustic modems though, real-time full resolution sensor data from the AUV is not available and operators must wait until the AUV is recovered in order to process the complete dataset. In the context of commercial applications, this severely limits the operational benefits of the AUV and may not fit well into the modern workflow.
Access to full-resolution, real-time data, typically collected by many AUV’s is a critical requirement for military mine warfare applications, which may also be twinned with USV’s under development within the ACTUV program.
Referring back again to the Rødseth & Burmeister model, the bandwidth limitation issues prevailing in the offshore arena will bite anytime a system is designed to be controlled/monitored by shore-based personnel – at least insofar as any real time functionality is concerned.
Ship-to-shore use of data transmission though is undergoing constant development and improvement.
Maersk is one example of how a major shipping line is now trying to leverage ‘big-data’ and maximize its ability to transmit data to shore from its ocean going vessels in order enhance and optimise their services. The below infographic shows how the Maersk Line now transmits around 30 terra-bytes of data via satellite links every month through its fleet.
Source: Maersk's Facebook page
Maersk via a situation room in Mumbai, monitor the Maersk Line’s fleet through GPS and satellite 24 hours a day, seven days a week. The team is using complex, real-time data to plan and execute the most efficient voyages for the vessels with the result being an optimised network, less fuel consumed and less CO2 emissions.
The airline industry is also enhancing its tracking capabilities in the wake of several recent tragedies involving downed planes in remote off-shore regions by utilizing the ‘electronic-handshake’. Rockwell Collins Inc. and nine carriers are testing a tracking system that uses similar, little-known electronic communications to trace aircraft flying over oceans and beyond the reach of radar, say’s its Chief Executive Officer Kelly Ortberg. Tests of the ARINC Multi-Link are ongoing and the product should be ready to be commercialized by the end of 2015.
Facebook however has taken a different approach to connectivity and recently unveiled its ‘Aquila’ project, a full-scale drone, which it plans to use to provide internet access in remote parts of the world.
Jay Parikh, Facebook’s vice-president of engineering has said: “Our mission is to connect everybody in the world. This is going to be a great opportunity for us to motivate the industry to move faster on this technology.”
This could well provide a glimpse of a solution to the data transmission limitations presently holding back full use of real time command and control systems. If the currently available real-time augmented cognitive computing powers could be applied on board in a truly offshore sense, the software solutions that unlock specifically, technologies that automate a “lookout by sight” from sea surface vessels required by DARPA would be a closer reality. Ships could talk to and learn from one another like the Google driverless cars. As compared though to conventional GPS, satellite communication services it could however be a very costly and unwieldy approach.
IBM Research is invested heavily into developing ‘cognitive environments’ and describe how these work as follows;
“As people inhabit and move across many physical environments, we see a fluid, coherent computing experience through space and time, connected by an ecosystem of cognitive environments inhabited by a society of specialized software agents called cogs. Cogs work in a mutually beneficial partnership with humans to enable better complex data-driven decision-making. We call these partnerships Symbiotic Cognitive Systems.
A cognitive environment is an infrastructure inhabited by the society of cogs and the devices that let them behave as one shared integrated resource, enabling “human-computer collaboration at the speed of thought.” Cognitive Environments can look and feel very different (from decision rooms in the workplace, to cars, to homes, to mobile), but by being connected to one another they will feel seamless.”
If an unmanned vessel with hard-wear sensor systems capable of maintaining and meeting the ColReg 5 requirements of a lookout by sight had an on-board cognitive computer, just as IBM illustrate above for cars, homes, offices etc., full freedom determinism could be achieved, albeit to comply with overarching strategies, pre-programmed say at commencement of the voyage before departure.
The vessel would be autonomous and under its own ‘intelligent’ command and control but being monitored by a shore based organisation via conventional means (AIS for example) until it reaches the littoral coastline of its planned destination whereupon, once in range, full bandwidth cog-to-cog or cog-to-human real time communication and control could take over (if needed). Offshore bandwidth limitations could possibly be circumvented altogether.
So what might these vessels look like?
The Classification Society DNV-GL has recently announced its revolutionary concept for an unmanned, zero-emission, shortsea vessel. “Named the ReVolt, this vessel is 60 metres long and is fully battery powered and autonomous – it requires no crew.”
Image source: DNV GL(c)Toftenes Multivisjon AS
Shipping, despite what many think, is changing. Quite how such drastic changes, if they are fully realized, will impact the industry generally as well as ancillary service providers (legal, insurance, broking etc) remains to be seen. Conventional approaches to data systems, communication and provision of response services will likely be clunky and unwieldy insofar as such concept systems are concerned. New cyber-security issues will also likely come to the fore. Other questions also spring to mind.
What will happen if a fully autonomous vessel is unable to avoid a collision with a traditional vessel and is rendered holed, grounded or otherwise unresponsive and uncommunicative? In the immediate aftermath of a casualty, the actions and information reporting of the crew on-board are invaluable to obtaining situational awareness for insurance companies and authorities acting as ‘first responders. This is so that appropriate responses and resources (salvage teams, tugs etc) can be marshalled and mobilized. Who will facilitate this on an unmanned vessel?
What kind of ship-to-ship communication would there have been between the conventional, manned vessel and the unmanned vessel in the immediate lead up to a casualty/incident? Usually there is VHF Channel 16 communication between each vessel’s bridge team and/or Pilots, all recorded on the Voyage Data Recorder (VDR), the shipping equivalent of a ‘black box’.
Would such vessels be compatible with the use of Pilots? Pilots are traditionally tasked with manoeuvring ships through dangerous or congested waters, such as harbours or river mouths where local knowledge of currents and hidden hazards are often key. They conduct and complete the berthing and un-berthing of ships by directly controlling the ship's manoeuvrability via use of tugs and shore linesmen over a radio. Would Pilots be needed at all for such ships?
Would such vessels be privately owned or merely an extension of a States import/export policy requirements, communicating seamlessly with one another to ship products and goods as needed?
Will such vessels require similar regulatory compliance as conventional manned vessels (STCW requirements would arguably be obsolete) such as Flag, Class, Oil and financial response requirements etc and who will implement it?
If there are no crew required on vessels anymore, what will happen to the pool of available experience, traditionally obtained from individuals with prior seagoing experience essential for shore-based legal, insurance and broking companies?
The drive for extracting cost efficiencies and increasing competitiveness though the development and use of unmanned vessels seems to be continuing apace. It does however raise some interesting questions and have the potential for some unintended consequences if not managed properly and in an orderly way.
Source: Norwegian Hull Club
Relevant past articles: