LiDAR-Camera Fusion Assistance Systems for Crane Operators

Client

A European integrator that supplies cranes for construction sites and cargo ports.

Challenge

To develop an AI-based engineering solution to improve crane operation efficiency, accuracy and safety.

Solution

1. Concept Design

To meet this challenge, we decided to develop a LiDAR-camera fusion assistance system for crane operators combining the capabilities of LiDAR sensors and video cameras.

LiDAR technology (light detection and ranging) uses laser beams to measure distances and create detailed 3D maps of the surroundings. Camera technology complements LiDAR by capturing visual information and providing additional contextual information, such as colour, texture, and fine details.​

_{Instant AI-powered real-time alerts for crane operators for their quick response}

2. System Design and Its Benefits

We have developed a prototype of the LiDAR-camera fusion assistance system, which will enable new features and benefits:

1. Improved safety and risk mitigation by providing comprehensive situational awareness and detecting potential hazards:​

• Obstacle detection helps detect nearby structures, equipment, or personnel that pose risks during crane operations.
• Collision avoidance by providing instant alerts to the operator whenever the system detects an imminent collision with an obstacle, such as a structure or a person.​
• Real-time warnings and proximity monitoring are integrated into the operator's interface, ensuring prompt response.
• Adverse condition awareness, even in challenging weather or lighting conditions, reduces the risk of accidents caused by lack of visibility.​

_{A system prototype with a camera on the crane operator's cab to detect unauthorised objects}

2. Enhanced efficiency and productivity

LiDAR-camera fusion systems reduce human error and streamline operations by taking advantage of data-driven decision-making:​

• Automated object recognition. Advanced algorithms allow operators to quickly identify structures, equipment, or objects that may impact crane movements.
• Precise positioning assistance in real-time by leveraging the accurate distance measurements from LiDAR and camera visual feedback. ​
• Virtual overlays and augmented reality. It includes distance markers, load paths, safety zones, or other relevant data.

_{This compact LiDAR can be mounted on a crane arm to provide a detailed 3D site map}

3. Real-time data and analytics

The LiDAR-camera fusion system enables monitoring, performance analysis, and operational insights:​

• Real-time data monitoring with constant data collection on the environment and crane operations for alerts and statistics.
• Performance metrics and analysis: load positioning accuracy, time taken for specific tasks, or frequency of obstacles encountered.This data helps improve operational efficiency, identify potential bottlenecks and make data-driven decisions.​
• Workflow optimisation. Insights gained from the sensor fusion system lead to improved task sequencing and better coordination with other activities on the worksite.​
• Operator behaviour analysis. A system can be equipped with an additional camera looking at the operator to analyse their behaviour, including distraction, bad habits (smoking, phone), and emotional state.

Business Value

Our client has received a LiDAR-camera fusion assistance system prototype for crane operators on construction sites and cargo ports.

Our software engineers analysed the user interface of crane operators to seamlessly integrate our platform with real-time alerts, video streaming with an augmented reality layer, and analytics for more efficient operation.

Integrating LiDAR-camera fusion assistance systems can provide a competitive edge and further develop the new system through machine learning and optimising algorithms for data-driven decisions. This technology results in lower costs and a significant increase in safety.

Here is an overview of the features and business value of the new system: