`get_batch` hang in some condition. I can provide the reproduce repo.

[vipcxj]

Please clone this repo

It's very strange, using get_batch read some frames cause hanging. I have locate the special frames. I have not found any strange of these frame. I don't know how to debug you project to find the problem. So I create a reop which reproducing the issue. Please help me find the problem.
By the way. the video in the repo have a wrong timestamps, but its may not relate to the issue. Because I have many video with such wrong timestamps work fine.

[dmenig]

Same issue with gpu utilization.

[iceflame89]

same issue

[JarvisFei]

same issue

[yjhong89]

• same issue
• hanging never ends until I forcefully stopped the container
  

[zijunwa]

It seems that in certain cases, the get_batch function may hang or freeze at the underlying C++ implementation, possibly due to a deadlock. However, these videos do not exhibit such issues during normal playback. When using decord for video reading, this can cause the entire training process to stop, and the GPU memory occupied by the model will not be released. Even manually terminating it with ctrl+c is insufficient; you still need to use kill [pid] to forcefully terminate the corresponding process to free GPU memory.

The SafeVideoReader class I wrote below might help solve this problem. It uses a multithreading approach, creating a new class and a new thread when calling decord.VideoReader.get_batch. If the thread does not respond within 10 seconds, it falls back to OpenCV for reading. This ensures that the training process does not come to a halt due to decord-related reading issues. However, when the original get_batch thread encounters a freeze or deadlock, there does not seem to be a good way to terminate these processes at the Python level. As a result, they turn into zombie processes during the execution of the main program. By setting thread.daemon = True, these hanging processes can be killed once the main program terminates.

import threading
import logging

import cv2
import torch
import numpy as np
import decord

logger = logging.getLogger(__name__)

class SafeVideoReader(decord.VideoReader):
    def __init__(self, uri, ctx=decord.cpu(0), width=-1, height=-1, num_threads=0, fault_tol=-1):
        super().__init__(uri, ctx, width, height, num_threads, fault_tol)
        self.uri = uri
        self._width = width if width > 0 else None
        self._height = height if height > 0 else None
        
    def get_batch(self, indices):
        result = []
        exception = []
        
        def target():
            decord.bridge.set_bridge('torch')
            try:
                res = super(SafeVideoReader, self).get_batch(indices)
                result.append(res)
            except Exception as e:
                exception.append(e)
        
        thread = threading.Thread(target=target)
        thread.daemon = True
        thread.start()
        thread.join(timeout=10)
        
        if thread.is_alive():
            logger.warning(f"Timeout reading {self.uri} with decord, switching to OpenCV")
            return self._get_batch_opencv(indices)
        elif exception:
            logger.warning(f"Error using decord: {exception[0]}, switching to OpenCV")
            return self._get_batch_opencv(indices)
        else:
            return result[0]
    
    def _get_batch_opencv(self, indices):
        cap = cv2.VideoCapture(self.uri)
        frames = []
        original_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
        original_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
        
        # Determine final resolution
        target_width = self._width if self._width is not None else original_width
        target_height = self._height if self._height is not None else original_height
        
        for idx in sorted(indices):  # Sort indices to improve reading efficiency
            cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
            ret, frame = cap.read()
            
            if not ret:
                # Generate a placeholder black frame
                frame = np.zeros((target_height, target_width, 3), dtype=np.uint8)
            else:
                # Convert color space and resize if necessary
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                if (target_width, target_height) != (original_width, original_height):
                    frame = cv2.resize(frame, (target_width, target_height))
            
            frames.append(frame)
        
        cap.release()
        
        # Convert to a torch tensor while maintaining consistent dimensions
        frames_tensor = torch.stack([torch.from_numpy(f) for f in frames])
        return frames_tensor

There doesn’t seem to be a good solution for handling zombie processes yet, but the SafeVideoReader above should work as a temporary fix.

[xavierhou]

anyone fix this issue? same problem

[cydiachen]

It seems that in certain cases, the get_batch function may hang or freeze at the underlying C++ implementation, possibly due to a deadlock. However, these videos do not exhibit such issues during normal playback. When using decord for video reading, this can cause the entire training process to stop, and the GPU memory occupied by the model will not be released. Even manually terminating it with ctrl+c is insufficient; you still need to use kill [pid] to forcefully terminate the corresponding process to free GPU memory.

The SafeVideoReader class I wrote below might help solve this problem. It uses a multithreading approach, creating a new class and a new thread when calling decord.VideoReader.get_batch. If the thread does not respond within 10 seconds, it falls back to OpenCV for reading. This ensures that the training process does not come to a halt due to decord-related reading issues. However, when the original get_batch thread encounters a freeze or deadlock, there does not seem to be a good way to terminate these processes at the Python level. As a result, they turn into zombie processes during the execution of the main program. By setting thread.daemon = True, these hanging processes can be killed once the main program terminates.

import threading
import logging

import cv2
import torch
import numpy as np
import decord

logger = logging.getLogger(name)

class SafeVideoReader(decord.VideoReader):
def init(self, uri, ctx=decord.cpu(0), width=-1, height=-1, num_threads=0, fault_tol=-1):
super().init(uri, ctx, width, height, num_threads, fault_tol)
self.uri = uri
self._width = width if width > 0 else None
self._height = height if height > 0 else None

def get_batch(self, indices):
    result = []
    exception = []
    
    def target():
        decord.bridge.set_bridge('torch')
        try:
            res = super(SafeVideoReader, self).get_batch(indices)
            result.append(res)
        except Exception as e:
            exception.append(e)
    
    thread = threading.Thread(target=target)
    thread.daemon = True
    thread.start()
    thread.join(timeout=10)
    
    if thread.is_alive():
        logger.warning(f"Timeout reading {self.uri} with decord, switching to OpenCV")
        return self._get_batch_opencv(indices)
    elif exception:
        logger.warning(f"Error using decord: {exception[0]}, switching to OpenCV")
        return self._get_batch_opencv(indices)
    else:
        return result[0]

def _get_batch_opencv(self, indices):
    cap = cv2.VideoCapture(self.uri)
    frames = []
    original_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    original_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    
    # Determine final resolution
    target_width = self._width if self._width is not None else original_width
    target_height = self._height if self._height is not None else original_height
    
    for idx in sorted(indices):  # Sort indices to improve reading efficiency
        cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
        ret, frame = cap.read()
        
        if not ret:
            # Generate a placeholder black frame
            frame = np.zeros((target_height, target_width, 3), dtype=np.uint8)
        else:
            # Convert color space and resize if necessary
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            if (target_width, target_height) != (original_width, original_height):
                frame = cv2.resize(frame, (target_width, target_height))
        
        frames.append(frame)
    
    cap.release()
    
    # Convert to a torch tensor while maintaining consistent dimensions
    frames_tensor = torch.stack([torch.from_numpy(f) for f in frames])
    return frames_tensor

There doesn’t seem to be a good solution for handling zombie processes yet, but the SafeVideoReader above should work as a temporary fix.

Thank you for your solution. It seems that after adopting safevideoreader we can avoid some timeout issues. But It occurs to me that particular video in the dataset causes the stuck.

[josephzpng]

Same problem.

[ohmyitscasper]

Potential fix: #341

[Bizilizi]

Version 0.6.0 Issue is still there.

@zhreshold Its been there for years, and it renders the core functionality to useless.