How to enable mic audio with file streaming

[bhavinatharva]

Is your feature request related to a problem? Please describe.

Currently, there is no support for streaming microphone audio together with audio from a video or audio file source. This makes it difficult to build applications where a recorded video is played while the user speaks live over it to explain or comment on the content.

For example, when explaining a tutorial, presentation, educational topic, or demo video, both the original video audio and the live microphone voice need to be captured and streamed simultaneously.

Describe the solution you'd like

I would like support for streaming microphone audio together with file/video audio in real time. The feature should allow:

• Playing a recorded video or audio file
• Capturing live microphone voice simultaneously
• Streaming both audio sources together in sync
• Option to mix both into a single output stream or access them separately
• Maintaining low latency and audio synchronization

Describe alternatives you've considered

One alternative is recording the microphone voice separately and merging it with the video later during editing. However, this does not support real-time streaming and adds additional processing steps.

Another option is implementing custom audio routing and mixing at the native platform level, but this increases development complexity.

Additional context

This feature would be useful for:

• Educational video explanations
• Tutorial walkthroughs
• Live commentary on recorded videos
• Presentation narration
• AI-powered video analysis with voice explanation

[pedroSG94]

Hello,

I understand the case.
Currently the best way is use mixAudioSource. This way you can capture de device sound if the user start to reproduce a video.

[bhavinatharva]

Thanks for the clarification.

I am currently using a generic file stream for video streaming instead of playing the video directly through the library’s media source/player pipeline.

Because of that, I’m not sure how mixAudioSource would integrate in this case. Is there any recommended way to inject or mix audio from a custom file stream with microphone audio before sending it to the stream?

My goal is to stream:

• Video from a custom file stream
• Original file audio
• Microphone audio simultaneously (mixed)

Any guidance or example for this setup would be very helpful.

[pedroSG94]

If you don't want use MixAudioSource to get microphone and device audio mixed that should works if you reproduce the video with sound in the device (this way the audio is captured from the device)

The only way is create a new Audio source yourself.
You can check MicrophoneSource and AudioFileSource classes and try to mix the audio like MixAudioSource but it is really difficult to sync and avoid noise. Also you need control that the audio from the file have the same sample rate or you will have audio problems

[bhavinatharva]

I create those files

package com.atharvasystem.quickscreenrecorder.ui.streams;

/**
 * A highly optimized, garbage-collection free Circular Byte Buffer.
 * Solves the O(N) memory copy bottleneck caused by standard ByteBuffer.compact().
 */
public class CircularByteBuffer {
    private final byte[] buffer;
    private int head = 0;
    private int tail = 0;
    private int available = 0;
    private final int capacity;

    public CircularByteBuffer(int capacity) {
        this.capacity = capacity;
        this.buffer = new byte[capacity];
    }

    /**
     * Puts bytes into the ring buffer. If the buffer is full, it automatically
     * drops the oldest audio frames to prevent memory overflow and keep audio realtime.
     */
    public synchronized void put(byte[] src, int offset, int length) {
        if (length > capacity) return; // Ignore chunks larger than total capacity
        
        if (length > capacity - available) {
            // Buffer overflow: drop oldest audio data to keep it realtime
            int overflow = length - (capacity - available);
            head = (head + overflow) % capacity;
            available -= overflow;
        }
        
        int firstPart = Math.min(length, capacity - tail);
        System.arraycopy(src, offset, buffer, tail, firstPart);
        
        int secondPart = length - firstPart;
        if (secondPart > 0) {
            System.arraycopy(src, offset + firstPart, buffer, 0, secondPart);
        }
        
        tail = (tail + length) % capacity;
        available += length;
    }

    /**
     * Reads up to `length` bytes from the ring buffer.
     */
    public synchronized int get(byte[] dst, int offset, int length) {
        int readLen = Math.min(length, available);
        if (readLen == 0) return 0;
        
        int firstPart = Math.min(readLen, capacity - head);
        System.arraycopy(buffer, head, dst, offset, firstPart);
        
        int secondPart = readLen - firstPart;
        if (secondPart > 0) {
            System.arraycopy(buffer, 0, dst, offset + firstPart, secondPart);
        }
        
        head = (head + readLen) % capacity;
        available -= readLen;
        return readLen;
    }

    public synchronized int available() {
        return available;
    }

    public synchronized void clear() {
        head = 0;
        tail = 0;
        available = 0;
    }
}

/*
 * Copyright (C) 2024 pedroSG94.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.atharvasystem.quickscreenrecorder.ui.streams;

import android.content.Context;
import android.media.AudioFormat;
import android.media.AudioManager;
import android.media.AudioTrack;
import android.media.MediaCodec;
import android.media.MediaFormat;
import android.media.MediaRecorder;
import android.net.Uri;
import android.os.Build;

import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import androidx.annotation.RequiresApi;

import com.pedro.common.AudioCodec;
import com.pedro.common.TimeUtils;
import com.pedro.common.VideoCodec;
import com.pedro.encoder.CodecErrorCallback;
import com.pedro.encoder.TimestampMode;
import com.pedro.encoder.Frame;
import com.pedro.encoder.audio.AudioEncoder;
import com.pedro.encoder.audio.GetAudioData;
import com.pedro.encoder.input.audio.GetMicrophoneData;
import com.pedro.encoder.input.audio.MicrophoneManager;
import com.pedro.encoder.input.audio.AudioUtils;
import com.pedro.encoder.input.decoder.AudioDecoder;
import com.pedro.encoder.input.decoder.AudioDecoderInterface;
import com.pedro.encoder.input.decoder.BaseDecoder;
import com.pedro.encoder.input.decoder.DecoderInterface;
import com.pedro.encoder.input.decoder.Extractor;
import com.pedro.encoder.input.decoder.VideoDecoder;
import com.pedro.encoder.input.decoder.VideoDecoderInterface;
import com.pedro.encoder.utils.CodecUtil;
import com.pedro.encoder.video.FormatVideoEncoder;
import com.pedro.encoder.video.GetVideoData;
import com.pedro.encoder.video.VideoEncoder;
import com.pedro.library.base.recording.RecordController;
import com.pedro.library.util.AndroidMuxerRecordController;
import com.pedro.library.util.FpsListener;
import com.pedro.library.util.IORunnable;
import com.pedro.library.util.streamclient.StreamBaseClient;
import com.pedro.library.view.GlInterface;
import com.pedro.library.view.GlStreamInterface;
import com.pedro.library.view.OpenGlView;

import java.io.FileDescriptor;
import java.io.IOException;
import java.nio.ByteBuffer;

@RequiresApi(api = Build.VERSION_CODES.JELLY_BEAN_MR2)
public abstract class CustomFromFileBase {

  private static final String TAG = "CustomFromFileBase";

  protected VideoEncoder videoEncoder;
  private AudioEncoder audioEncoder;
  private GlInterface glInterface;
  private boolean streaming = false;
  protected RecordController recordController;
  private final FpsListener fpsListener = new FpsListener();

  private VideoDecoder videoDecoder;
  private AudioDecoder audioDecoder;

  private MicrophoneManager microphoneManager;
  private boolean isMicAudio = false;
  private boolean isMixAudio = false;

  protected boolean videoEnabled = false;
  private boolean audioEnabled = false;
  private AudioTrack audioTrackPlayer;
  private VideoDecoderInterface videoDecoderInterface;
  private AudioDecoderInterface audioDecoderInterface;

  // PCM Mixer variables
  private CircularByteBuffer micBufferStore = new CircularByteBuffer(1024 * 512); // 512KB Ring Buffer
  private CircularByteBuffer fileBufferStore = new CircularByteBuffer(1024 * 512); // 512KB Ring Buffer
  private Thread audioMixerThread;
  private boolean isMixerRunning = false;
  
  private byte[] mixBufferTmp1 = new byte[8192];
  private byte[] mixBufferTmp2 = new byte[8192];
  private byte[] mixBufferOut = new byte[8192];
  private AudioUtils audioUtils = new AudioUtils();

  public CustomFromFileBase(VideoDecoderInterface videoDecoderInterface,
      AudioDecoderInterface audioDecoderInterface) {
    init(videoDecoderInterface, audioDecoderInterface);
  }

  public CustomFromFileBase(Context context, VideoDecoderInterface videoDecoderInterface,
      AudioDecoderInterface audioDecoderInterface) {
    glInterface = new GlStreamInterface(context);
    init(videoDecoderInterface, audioDecoderInterface);
  }

  public CustomFromFileBase(OpenGlView openGlView, VideoDecoderInterface videoDecoderInterface,
      AudioDecoderInterface audioDecoderInterface) {
    glInterface = openGlView;
    init(videoDecoderInterface, audioDecoderInterface);
  }

  private void init(VideoDecoderInterface videoDecoderInterface,
      AudioDecoderInterface audioDecoderInterface) {
    this.videoDecoderInterface = videoDecoderInterface;
    this.audioDecoderInterface = audioDecoderInterface;
    videoEncoder = new VideoEncoder(getVideoData);
    audioEncoder = new AudioEncoder(getAudioData);
    videoDecoder = new VideoDecoder(videoDecoderInterface, decoderInterfaceVideo);
    audioDecoder = new AudioDecoder(getFilePCMData, audioDecoderInterface, decoderInterfaceAudio);
    recordController = new AndroidMuxerRecordController();
    microphoneManager = new MicrophoneManager(getMicPCMData);
  }

  public void setTimestampMode(TimestampMode timestampModeVideo, TimestampMode timestampModeAudio) {
    videoEncoder.setTimestampMode(timestampModeVideo);
    audioEncoder.setTimestampMode(timestampModeAudio);
  }

  public void setFpsListener(FpsListener.Callback callback) {
    fpsListener.setCallback(callback);
  }

  public boolean prepareVideo(String filePath, int bitRate, int rotation, int profile,
      int level) throws IOException {
    if (!videoDecoder.initExtractor(filePath)) return false;
    return finishPrepareVideo(bitRate, rotation, profile, level);
  }

  public boolean prepareVideo(FileDescriptor fileDescriptor, int bitRate, int rotation, int profile,
      int level) throws IOException {
    if (!videoDecoder.initExtractor(fileDescriptor)) return false;
    return finishPrepareVideo(bitRate, rotation, profile, level);
  }

  public boolean prepareVideo(Context context, Uri uri, int bitRate, int rotation, int profile,
      int level) throws IOException {
    if (!videoDecoder.initExtractor(context, uri)) return false;
    return finishPrepareVideo(bitRate, rotation, profile, level);
  }

  public boolean prepareVideo(String filePath, int bitRate, int rotation) throws IOException {
    return prepareVideo(filePath, bitRate, rotation, -1, -1);
  }

  public boolean prepareVideo(FileDescriptor fileDescriptor, int bitRate, int rotation) throws IOException {
    return prepareVideo(fileDescriptor, bitRate, rotation, -1, -1);
  }

  public boolean prepareVideo(String filePath) throws IOException {
    return prepareVideo(filePath, 1200 * 1024, 0);
  }

  public boolean prepareVideo(Context context, Uri uri, int bitRate, int rotation) throws IOException {
    return prepareVideo(context, uri, bitRate, rotation, -1, -1);
  }

  public boolean prepareVideo(Context context, Uri uri) throws IOException {
    return prepareVideo(context, uri, 1200 * 1024, 0);
  }

  public boolean prepareVideo(FileDescriptor fileDescriptor) throws IOException {
    return prepareVideo(fileDescriptor, 1200 * 1024, 0);
  }

  private boolean finishPrepareVideo(int bitRate, int rotation, int profile,  int level) {
    boolean result = videoEncoder.prepareVideoEncoder(videoDecoder.getWidth(), videoDecoder.getHeight(), videoDecoder.getFps(),
            bitRate, rotation, 2, FormatVideoEncoder.SURFACE, profile, level);
    if (!result) return false;
    result = videoDecoder.prepareVideo(videoEncoder.getInputSurface());
    videoEnabled = result;
    return result;
  }

  // ==== Audio Setup Methods ==== //

  // 1. Prepare for File Audio Only
  public boolean prepareAudio(String filePath, int bitRate) throws IOException {
    if (!audioDecoder.initExtractor(filePath)) return false;
    return finishPrepareAudio(bitRate);
  }

  public boolean prepareAudio(FileDescriptor fileDescriptor, int bitRate) throws IOException {
    if (!audioDecoder.initExtractor(fileDescriptor)) return false;
    return finishPrepareAudio(bitRate);
  }

  public boolean prepareAudio(Context context, Uri uri, int bitRate) throws IOException {
    if (!audioDecoder.initExtractor(context, uri)) return false;
    return finishPrepareAudio(bitRate);
  }

  private boolean finishPrepareAudio(int bitRate) {
    isMicAudio = false;
    isMixAudio = false;
    audioDecoder.prepareAudio();
    boolean result = audioEncoder.prepareAudioEncoder(bitRate, audioDecoder.getSampleRate(),
        audioDecoder.isStereo());
    onAudioInfoImp(audioDecoder.isStereo(), audioDecoder.getSampleRate());
    audioEnabled = result;
    return result;
  }

  // 2. Prepare for Mic Audio Only
  public boolean prepareAudioMic(int bitrate, int sampleRate, boolean isStereo, boolean echoCanceler, boolean noiseSuppressor) {
      isMicAudio = true;
      isMixAudio = false;
      if (!microphoneManager.createMicrophone(MediaRecorder.AudioSource.DEFAULT, sampleRate, isStereo, echoCanceler, noiseSuppressor)) {
          return false;
      }
      onAudioInfoImp(isStereo, sampleRate);
      audioEnabled = audioEncoder.prepareAudioEncoder(bitrate, sampleRate, isStereo);
      return audioEnabled;
  }

  public boolean prepareAudioMic(int bitrate, int sampleRate, boolean isStereo) {
      return prepareAudioMic(bitrate, sampleRate, isStereo, false, false);
  }

  public boolean prepareAudioMic() {
      return prepareAudioMic(64 * 1024, 32000, true, false, false);
  }

  // 3. Prepare for MIXED Audio (File + Mic)
  public boolean prepareAudioMix(String filePath, int bitrate, boolean echoCanceler, boolean noiseSuppressor) throws IOException {
      if (!audioDecoder.initExtractor(filePath)) return false;
      return finishPrepareAudioMix(bitrate, echoCanceler, noiseSuppressor);
  }

  public boolean prepareAudioMix(Context context, Uri uri, int bitrate, boolean echoCanceler, boolean noiseSuppressor) throws IOException {
      if (!audioDecoder.initExtractor(context, uri)) return false;
      return finishPrepareAudioMix(bitrate, echoCanceler, noiseSuppressor);
  }

  public boolean prepareAudioMix(String filePath) throws IOException {
      return prepareAudioMix(filePath, 64 * 1024, false, false);
  }

  public boolean prepareAudioMix(Context context, Uri uri) throws IOException {
      return prepareAudioMix(context, uri, 64 * 1024, false, false);
  }

  private boolean finishPrepareAudioMix(int bitrate, boolean echoCanceler, boolean noiseSuppressor) {
      audioDecoder.prepareAudio();
      
      int sampleRate = audioDecoder.getSampleRate();
      boolean isStereo = audioDecoder.isStereo();
      
      if (!microphoneManager.createMicrophone(MediaRecorder.AudioSource.DEFAULT, sampleRate, isStereo, echoCanceler, noiseSuppressor)) {
          return false; // Mic failed
      }
      
      isMicAudio = false; // Using mix
      isMixAudio = true;
      
      onAudioInfoImp(isStereo, sampleRate);
      audioEnabled = audioEncoder.prepareAudioEncoder(bitrate, sampleRate, isStereo);
      return audioEnabled;
  }


  // ==== PCM Mixing Callbacks ==== //

  private float micVolume = 1.0f;
  private float fileVolume = 1.0f;

  public void setAudioMixVolume(float micVol, float fileVol) {
      this.micVolume = micVol;
      this.fileVolume = fileVol;
  }

  private void ensureMixBuffersCapacity(int size) {
    if (mixBufferTmp1.length < size) {
      mixBufferTmp1 = new byte[size];
      mixBufferTmp2 = new byte[size];
      mixBufferOut = new byte[size];
    }
  }

  private void tryMixAndEncode(long timestamp) {
    while (true) {
        int micAvailable = micBufferStore.available();
        int fileAvailable = fileBufferStore.available();
        
        int mixSize = Math.min(micAvailable, fileAvailable);
        mixSize = mixSize - (mixSize % 4); 
        
        if (mixSize > 0) {
          ensureMixBuffersCapacity(mixSize);
          
          micBufferStore.get(mixBufferTmp1, 0, mixSize);
          fileBufferStore.get(mixBufferTmp2, 0, mixSize);
          
          audioUtils.applyVolumeAndMix(mixBufferTmp1, micVolume, mixBufferTmp2, fileVolume, mixBufferOut);
          audioEncoder.inputPCMData(new Frame(mixBufferOut, 0, mixSize, timestamp));
          
        } else if (micAvailable > 256000) { 
          // File is dead or severely lagging ( > 1.5 seconds out of sync ). Drain mic.
          int drainSize = micAvailable - (micAvailable % 4);
          ensureMixBuffersCapacity(drainSize);
          micBufferStore.get(mixBufferTmp1, 0, drainSize);
          audioEncoder.inputPCMData(new Frame(mixBufferTmp1, 0, drainSize, timestamp));
          
        } else if (fileAvailable > 256000) { 
          // Mic is dead or severely lagging. Drain file.
          int drainSize = fileAvailable - (fileAvailable % 4);
          ensureMixBuffersCapacity(drainSize);
          fileBufferStore.get(mixBufferTmp2, 0, drainSize);
          audioEncoder.inputPCMData(new Frame(mixBufferTmp2, 0, drainSize, timestamp));
        } else {
          // Wait for more data to arrive
          break;
        }
    }
  }

  private void startMixerThread() {
      isMixerRunning = true;
      audioMixerThread = new Thread(() -> {
          while (isMixerRunning) {
              tryMixAndEncode(TimeUtils.getCurrentTimeMicro());
              try {
                  Thread.sleep(10); // Polling interval
              } catch (InterruptedException e) {
                  Thread.currentThread().interrupt();
                  break;
              }
          }
      }, "AudioMixerThread");
      audioMixerThread.start();
  }

  private void stopMixerThread() {
      isMixerRunning = false;
      if (audioMixerThread != null) {
          audioMixerThread.interrupt();
          audioMixerThread = null;
      }
  }

  private final GetMicrophoneData getMicPCMData = frame -> {
    if (!isMixAudio && isMicAudio) { 
      // ONLY MIC MODE
      audioEncoder.inputPCMData(frame);
      return;
    }
    if (!isMixAudio) return; // Ignore mic if File Only Mode
    
    // MIX MODE (Lock-free push)
    micBufferStore.put(frame.getBuffer(), frame.getOffset(), frame.getSize());
  };

  private final GetMicrophoneData getFilePCMData = frame -> {
    if (audioTrackPlayer != null && !isMicAudio && !isMixAudio) {
      // ONLY play audio to device speakers if we are in FILE ONLY mode
      audioTrackPlayer.write(frame.getBuffer(), frame.getOffset(), frame.getSize());
    }
    
    if (!isMixAudio && !isMicAudio) { 
      // ONLY FILE MODE
      audioEncoder.inputPCMData(frame);
      return;
    }
    if (!isMixAudio) return; // Ignore file if Mic Only Mode

    // MIX MODE (Lock-free push)
    fileBufferStore.put(frame.getBuffer(), frame.getOffset(), frame.getSize());
  };

  public void muteAudio() {
    if (isMixAudio) {
      if (microphoneManager != null) microphoneManager.mute();
      if (audioDecoder != null) audioDecoder.mute();
    } else if (isMicAudio) {
      if (microphoneManager != null) microphoneManager.mute();
    } else {
      if (audioDecoder != null) audioDecoder.mute();
    }
  }

  public void unMuteAudio() {
    if (isMixAudio) {
      if (microphoneManager != null) microphoneManager.unMute();
      if (audioDecoder != null) audioDecoder.unMute();
    } else if (isMicAudio) {
      if (microphoneManager != null) microphoneManager.unMute();
    } else {
      if (audioDecoder != null) audioDecoder.unMute();
    }
  }

  public boolean isAudioMuted() {
    if (isMixAudio) {
      return microphoneManager != null && microphoneManager.isMuted();
    } else if (isMicAudio) {
      return microphoneManager != null && microphoneManager.isMuted();
    } else {
      return audioDecoder != null && audioDecoder.isMuted();
    }
  }

  public boolean isAudioDeviceEnabled() {
    return audioTrackPlayer != null
        && audioTrackPlayer.getPlayState() == AudioTrack.PLAYSTATE_PLAYING;
  }

  public void setEncoderErrorCallback(CodecErrorCallback encoderErrorCallback) {
    videoEncoder.setEncoderErrorCallback(encoderErrorCallback);
    audioEncoder.setEncoderErrorCallback(encoderErrorCallback);
  }

  public void playAudioDevice() {
    if (audioEnabled && !isMicAudio && !isMixAudio) {
      if (isAudioDeviceEnabled()) {
        audioTrackPlayer.stop();
      }
      int channel =
          audioDecoder.isStereo() ? AudioFormat.CHANNEL_OUT_STEREO : AudioFormat.CHANNEL_OUT_MONO;
      int buffSize = AudioTrack.getMinBufferSize(audioDecoder.getSampleRate(), channel,
          AudioFormat.ENCODING_PCM_16BIT);
      audioTrackPlayer =
          new AudioTrack(AudioManager.STREAM_MUSIC, audioDecoder.getSampleRate(), channel,
              AudioFormat.ENCODING_PCM_16BIT, buffSize, AudioTrack.MODE_STREAM);
      audioTrackPlayer.play();
    }
  }

  public void stopAudioDevice() {
    if (audioEnabled && isAudioDeviceEnabled()) {
      audioTrackPlayer.stop();
      audioTrackPlayer = null;
    }
  }

  public boolean prepareAudio(String filePath) throws IOException {
    return prepareAudio(filePath, 64 * 1024);
  }

  public boolean prepareAudio(Context context, Uri uri) throws IOException {
    return prepareAudio(context, uri, 64 * 1024);
  }

  protected abstract void onAudioInfoImp(boolean isStereo, int sampleRate);

  public void forceCodecType(CodecUtil.CodecType codecTypeVideo, CodecUtil.CodecType codecTypeAudio) {
    if (videoEnabled) videoEncoder.forceCodecType(codecTypeVideo);
    if (audioEnabled) audioEncoder.forceCodecType(codecTypeAudio);
  }

  public void startRecord(@NonNull String path, @Nullable RecordController.Listener listener) throws IOException {
    RecordController.RecordTracks tracks = RecordController.RecordTracks.ALL;
    if (!videoEnabled) tracks = RecordController.RecordTracks.AUDIO;
    else if (!audioEnabled) tracks = RecordController.RecordTracks.VIDEO;
    recordController.setRequestKeyFrame(this::requestKeyFrame);
    recordController.startRecord(path, listener, tracks);
    if (!streaming) startEncoders();
  }

  public void startRecord(@NonNull final String path) throws IOException {
    startRecord(path, null);
  }

  @RequiresApi(api = Build.VERSION_CODES.O)
  public void startRecord(@NonNull final FileDescriptor fd, @Nullable RecordController.Listener listener) throws IOException {
    RecordController.RecordTracks tracks = RecordController.RecordTracks.ALL;
    if (!videoEnabled) tracks = RecordController.RecordTracks.AUDIO;
    else if (!audioEnabled) tracks = RecordController.RecordTracks.VIDEO;
    recordController.setRequestKeyFrame(this::requestKeyFrame);
    recordController.startRecord(fd, listener, tracks);
    if (!streaming) startEncoders();
  }

  @RequiresApi(api = Build.VERSION_CODES.O)
  public void startRecord(@NonNull final FileDescriptor fd) throws IOException{
    startRecord(fd, null);
  }

  public void stopRecord() {
    recordController.stopRecord();
    if (!streaming) stopStream();
  }

  protected abstract void startStreamImp(String url);

  public void startStream(String url) {
    streaming = true;
    if (!recordController.isRunning()) {
      startEncoders();
    } else {
      if (videoEnabled) requestKeyFrame();
    }
    startStreamImp(url);
  }

  private void startEncoders() {
    long startTs = TimeUtils.getCurrentTimeMicro();
    if (videoEnabled) videoEncoder.start(startTs);
    if (audioTrackPlayer != null) audioTrackPlayer.play();
    if (audioEnabled) audioEncoder.start(startTs);
    if (videoEnabled) prepareGlView();
    if (videoEnabled) videoDecoder.start();
    
    if (audioEnabled) {
      if (isMixAudio) {
        // Clear old buffers before starting
        micBufferStore.clear();
        fileBufferStore.clear();
        startMixerThread();
        microphoneManager.start();
        audioDecoder.start();
      } else if (isMicAudio) {
        microphoneManager.start();
      } else {
        audioDecoder.start();
      }
    }
  }

  public void setDecoderErrorCallback(CodecErrorCallback codecErrorCallback) {
    audioDecoder.setCodecErrorCallback(codecErrorCallback);
    videoDecoder.setCodecErrorCallback(codecErrorCallback);
  }

  public void replaceView(Context context) {
    replaceGlInterface(new GlStreamInterface(context));
  }

  public void replaceView(OpenGlView openGlView) {
    replaceGlInterface(openGlView);
  }

  private void replaceGlInterface(GlInterface glInterface) {
    if (this.glInterface != null && videoEnabled) {
      if (isStreaming() || isRecording()) {
        videoDecoder.pauseRender();
        this.glInterface.removeMediaCodecSurface();
        this.glInterface.stop();
        this.glInterface = glInterface;
        prepareGlView();
        videoDecoder.resumeRender();
      } else {
        this.glInterface = glInterface;
      }
    }
  }

  private void prepareGlView() {
    if (glInterface != null) {
      int w = videoEncoder.getWidth();
      int h = videoEncoder.getHeight();
      boolean isPortrait = false;
      if (videoEncoder.getRotation() == 90 || videoEncoder.getRotation() == 270) {
        h = videoEncoder.getWidth();
        w = videoEncoder.getHeight();
        isPortrait = true;
      }
      glInterface.setEncoderSize(w, h);
      if (glInterface instanceof GlStreamInterface) {
        GlStreamInterface glStreamInterface = (GlStreamInterface) glInterface;
        glStreamInterface.setPreviewResolution(w, h);
        glStreamInterface.setIsPortrait(isPortrait);
      }
      glInterface.setRotation(0);
      glInterface.start();
      if (videoEncoder.getInputSurface() != null) {
        videoDecoder.changeOutputSurface(glInterface.getSurface());
        glInterface.addMediaCodecSurface(videoEncoder.getInputSurface());
      }
    }
  }

  public void requestKeyFrame() {
    if (videoEncoder.isRunning()) {
      if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
        videoEncoder.requestKeyframe();
      } else {
        if (glInterface != null) {
          glInterface.removeMediaCodecSurface();
        }
        videoEncoder.reset();
        if (glInterface != null) {
          glInterface.addMediaCodecSurface(videoEncoder.getInputSurface());
        } else {
          videoDecoder.changeOutputSurface(videoEncoder.getInputSurface());
        }
      }
    }
  }

  protected abstract void stopStreamImp();

  public void stopStream() {
    if (streaming) {
      streaming = false;
      stopStreamImp();
    }
    if (!recordController.isRecording()) {
      if (glInterface != null) {
        glInterface.removeMediaCodecSurface();
        glInterface.stop();
      }
      if (videoEnabled) videoDecoder.stop();
      
      if (audioEnabled) {
        if (isMixAudio) {
          stopMixerThread();
          microphoneManager.stop();
          audioDecoder.stop();
        } else if (isMicAudio) {
          microphoneManager.stop();
        } else {
          audioDecoder.stop();
        }
      }
      
      if (audioEnabled && isAudioDeviceEnabled()) {
        audioTrackPlayer.stop();
      }
      audioTrackPlayer = null;
      if (videoEnabled) videoEncoder.stop();
      if (audioEnabled) audioEncoder.stop();
      recordController.resetFormats();
      videoEnabled = false;
      audioEnabled = false;
    }
  }

  public void setLoopMode(boolean loopMode) {
    videoDecoder.setLoopMode(loopMode);
    audioDecoder.setLoopMode(loopMode);
  }

  public void reSyncFile() {
    if (isStreaming() && videoEnabled && audioEnabled) audioDecoder.moveTo(videoDecoder.getTime());
  }

  public GlInterface getGlInterface() {
    if (glInterface != null) {
      return glInterface;
    } else {
      throw new RuntimeException("You can't do it. You are not using Opengl");
    }
  }

  public int getBitrate() {
    return videoEncoder.getBitRate();
  }

  public int getResolutionValue() {
    return videoEncoder.getWidth() * videoEncoder.getHeight();
  }

  public int getStreamWidth() {
    return videoEncoder.getWidth();
  }

  public int getStreamHeight() {
    return videoEncoder.getHeight();
  }

  @RequiresApi(api = Build.VERSION_CODES.KITKAT)
  public void setVideoBitrateOnFly(int bitrate) {
    videoEncoder.setVideoBitrateOnFly(bitrate);
  }

  public void forceFpsLimit(boolean enabled) {
    int fps = enabled ? videoEncoder.getFps() : 0;
    videoEncoder.setForceFps(fps);
    if (glInterface != null) glInterface.forceFpsLimit(fps);
  }

  public boolean resetVideoEncoder() {
    if (glInterface != null) {
      glInterface.removeMediaCodecSurface();
      boolean result = videoEncoder.reset();
      if (!result) return false;
      glInterface.addMediaCodecSurface(videoEncoder.getInputSurface());
    } else {
      boolean result = videoEncoder.reset();
      if (!result) return false;
      if (videoDecoder.isRunning()) videoDecoder.changeOutputSurface(videoEncoder.getInputSurface());
    }
    return true;
  }

  public boolean resetAudioEncoder() {
    return audioEncoder.reset();
  }

  public boolean isStreaming() {
    return streaming;
  }

  public boolean isRecording() {
    return recordController.isRunning();
  }

  public void pauseRecord() {
    recordController.pauseRecord();
  }

  public void resumeRecord() {
    recordController.resumeRecord();
  }

  public RecordController.Status getRecordStatus() {
    return recordController.getStatus();
  }

  public double getVideoTime() {
    return videoDecoder.getTime();
  }

  public double getAudioTime() {
    return audioDecoder.getTime();
  }

  public double getVideoDuration() {
    return videoDecoder.getDuration();
  }

  public double getAudioDuration() {
    return audioDecoder.getDuration();
  }

  public void forceBt709Color(boolean enabled) {
    videoEncoder.forceBt709Color(enabled);
  }

  public void replaceAudioFile(String filePath) throws IOException {
    resetAudioDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(filePath)) throw new IOException("Extraction failed");
    });
  }

  public void replaceAudioFile(Context context, Uri uri) throws IOException {
    resetAudioDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(context, uri)) throw new IOException("Extraction failed");
    });
  }

  public void replaceAudioFile(FileDescriptor fileDescriptor) throws IOException {
    resetAudioDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(fileDescriptor)) throw new IOException("Extraction failed");
    });
  }

  public void replaceVideoFile(String filePath) throws IOException {
    resetVideoDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(filePath)) throw new IOException("Extraction failed");
    });
  }

  public void replaceVideoFile(Context context, Uri uri) throws IOException {
    resetVideoDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(context, uri)) throw new IOException("Extraction failed");
    });
  }

  public void replaceVideoFile(FileDescriptor fileDescriptor) throws IOException {
    resetVideoDecoder((BaseDecoder decoder) -> {
      if (!decoder.initExtractor(fileDescriptor)) throw new IOException("Extraction failed");
    });
  }

  private void resetVideoDecoder(IORunnable runnable) throws IOException {
    int width = videoDecoder.getWidth();
    int height = videoDecoder.getHeight();
    boolean wasRunning = videoDecoder.isRunning();
    VideoDecoder videoDecoder = new VideoDecoder(videoDecoderInterface, decoderInterfaceVideo);
    videoDecoder.setExtractor(this.videoDecoder.getExtractor());
    runnable.run(videoDecoder);
    if (width != videoDecoder.getWidth() || height != videoDecoder.getHeight()) throw new IOException("Resolution must be the same that the previous file");
    this.videoDecoder.stop();
    this.videoDecoder = videoDecoder;
    videoDecoder.prepareVideo(glInterface.getSurface());
    if (wasRunning) videoDecoder.start();
  }

  private void resetAudioDecoder(IORunnable runnable) throws IOException {
    int sampleRate = audioDecoder.getSampleRate();
    boolean isStereo = audioDecoder.isStereo();
    boolean wasRunning = audioDecoder.isRunning();
    AudioDecoder audioDecoder = new AudioDecoder(getFilePCMData, audioDecoderInterface, decoderInterfaceAudio);
    audioDecoder.setExtractor(this.audioDecoder.getExtractor());
    runnable.run(audioDecoder);
    if (sampleRate != audioDecoder.getSampleRate()) throw new IOException("SampleRate must be the same that the previous file");
    if (isStereo != audioDecoder.isStereo()) throw new IOException("Channels must be the same that the previous file");
    this.audioDecoder.stop();
    this.audioDecoder = audioDecoder;
    audioDecoder.prepareAudio();
    if (wasRunning) audioDecoder.start();
  }

  public void moveTo(double time) {
    if (videoEnabled) videoDecoder.moveTo(time);
    if (audioEnabled) audioDecoder.moveTo(time);
  }

  public void setVideoExtractor(Extractor extractor) {
    videoDecoder.setExtractor(extractor);
  }

  public void setAudioExtractor(Extractor extractor) {
    audioDecoder.setExtractor(extractor);
  }

  protected abstract void onVideoInfoImp(ByteBuffer sps, ByteBuffer pps, ByteBuffer vps);

  protected abstract void getVideoDataImp(ByteBuffer videoBuffer, MediaCodec.BufferInfo info);

  protected abstract void getAudioDataImp(ByteBuffer audioBuffer, MediaCodec.BufferInfo info);

  public void setRecordController(RecordController recordController) {
    if (!isRecording()) {
      recordController.updateInfo(this.recordController.getVideoCodec(), this.recordController.getAudioCodec());
      this.recordController = recordController;
    }
  }

  private final GetAudioData getAudioData = new GetAudioData() {
    @Override
    public void getAudioData(@NonNull ByteBuffer audioBuffer, @NonNull MediaCodec.BufferInfo info) {
      recordController.recordAudio(audioBuffer, info);
      if (streaming) getAudioDataImp(audioBuffer, info);
    }

    @Override
    public void onAudioFormat(@NonNull MediaFormat mediaFormat) {
      recordController.setAudioFormat(mediaFormat);
    }
  };

  private final GetVideoData getVideoData = new GetVideoData() {
    @Override
    public void onVideoInfo(@NonNull ByteBuffer sps, @Nullable ByteBuffer pps, @Nullable ByteBuffer vps) {
      onVideoInfoImp(sps.duplicate(),  pps != null ? pps.duplicate(): null, vps != null ? vps.duplicate() : null);
    }

    @Override
    public void getVideoData(@NonNull ByteBuffer videoBuffer, @NonNull MediaCodec.BufferInfo info) {
      fpsListener.calculateFps();
      recordController.recordVideo(videoBuffer, info);
      if (streaming) getVideoDataImp(videoBuffer, info);
    }

    @Override
    public void onVideoFormat(@NonNull MediaFormat mediaFormat) {
      recordController.setVideoFormat(mediaFormat);
    }
  };

  private final DecoderInterface decoderInterfaceVideo = new DecoderInterface() {

    @Override
    public void onLoop() {
      if (videoDecoder.isRunning()) videoDecoder.reset(glInterface.getSurface());
    }
  };

  private final DecoderInterface decoderInterfaceAudio = new DecoderInterface() {

    @Override
    public void onLoop() {
      if (audioDecoder.isRunning()) audioDecoder.reset(null);
    }
  };

  public abstract StreamBaseClient getStreamClient();

  public void setVideoCodec(VideoCodec codec) {
    setVideoCodecImp(codec);
    recordController.setVideoCodec(codec);
  }

  public void setAudioCodec(AudioCodec codec) {
    setAudioCodecImp(codec);
    recordController.setAudioCodec(codec);
  }

  protected abstract void setVideoCodecImp(VideoCodec codec);

  protected abstract void setAudioCodecImp(AudioCodec codec);
}

/*
 * Copyright (C) 2024 pedroSG94.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.atharvasystem.quickscreenrecorder.ui.streams

import android.content.Context
import android.media.MediaCodec
import android.os.Build
import androidx.annotation.RequiresApi
import com.pedro.common.AudioCodec
import com.pedro.common.ConnectChecker
import com.pedro.common.VideoCodec
import com.pedro.common.onMainThreadHandler
import com.pedro.encoder.input.decoder.AudioDecoderInterface
import com.pedro.encoder.input.decoder.VideoDecoderInterface
import com.pedro.library.util.streamclient.GenericStreamClient
import com.pedro.library.util.streamclient.RtmpStreamClient
import com.pedro.library.util.streamclient.RtspStreamClient
import com.pedro.library.util.streamclient.SrtStreamClient
import com.pedro.library.util.streamclient.StreamClientListener
import com.pedro.library.util.streamclient.UdpStreamClient
import com.pedro.library.view.OpenGlView
import com.pedro.rtmp.rtmp.RtmpClient
import com.pedro.rtsp.rtsp.RtspClient
import com.pedro.srt.srt.SrtClient
import com.pedro.udp.UdpClient
import java.nio.ByteBuffer

@RequiresApi(api = Build.VERSION_CODES.JELLY_BEAN_MR2)
class CustomGenericFromFile: CustomFromFileBase {

  private val streamClientListener = object: StreamClientListener {
    override fun onRequestKeyframe() {
      requestKeyFrame()
    }
  }
  private lateinit var connectChecker: ConnectChecker
  private lateinit var rtmpClient: RtmpClient
  private lateinit var rtspClient: RtspClient
  private lateinit var srtClient: SrtClient
  private lateinit var udpClient: UdpClient
  private lateinit var streamClient: GenericStreamClient
  private enum class StreamType { NONE, RTMP, RTSP, SRT, UDP }
  private var connectedType = StreamType.NONE

  constructor(
    openGlView: OpenGlView, connectChecker: ConnectChecker,
    videoDecoderInterface: VideoDecoderInterface, audioDecoderInterface: AudioDecoderInterface
  ) : super(openGlView, videoDecoderInterface, audioDecoderInterface) {
    init(connectChecker)
  }

  constructor(
    context: Context, connectChecker: ConnectChecker,
    videoDecoderInterface: VideoDecoderInterface, audioDecoderInterface: AudioDecoderInterface
  ) : super(context, videoDecoderInterface, audioDecoderInterface) {
    init(connectChecker)
  }

  constructor(
    connectChecker: ConnectChecker,
    videoDecoderInterface: VideoDecoderInterface, audioDecoderInterface: AudioDecoderInterface
  ) : super(videoDecoderInterface, audioDecoderInterface) {
    init(connectChecker)
  }

  private fun init(connectChecker: ConnectChecker) {
    this.connectChecker = connectChecker
    rtmpClient = RtmpClient(connectChecker)
    rtspClient = RtspClient(connectChecker)
    srtClient = SrtClient(connectChecker)
    udpClient = UdpClient(connectChecker)
    streamClient = GenericStreamClient(
      RtmpStreamClient(rtmpClient, streamClientListener),
      RtspStreamClient(rtspClient, streamClientListener),
      SrtStreamClient(srtClient, streamClientListener),
      UdpStreamClient(udpClient, streamClientListener),
    )
  }

  override fun getStreamClient(): GenericStreamClient = streamClient

  override fun setVideoCodecImp(codec: VideoCodec) {
    if (codec != VideoCodec.H264 && codec != VideoCodec.H265) {
      throw IllegalArgumentException("Unsupported codec: ${codec.name}. Generic only support video ${VideoCodec.H264.name} and ${VideoCodec.H265.name}")
    }
    rtmpClient.setVideoCodec(codec)
    rtspClient.setVideoCodec(codec)
    srtClient.setVideoCodec(codec)
    udpClient.setVideoCodec(codec)
  }

  override fun setAudioCodecImp(codec: AudioCodec) {
    if (codec != AudioCodec.AAC) {
      throw IllegalArgumentException("Unsupported codec: ${codec.name}. Generic only support audio ${AudioCodec.AAC.name}")
    }
    rtmpClient.setAudioCodec(codec)
    rtspClient.setAudioCodec(codec)
    srtClient.setAudioCodec(codec)
    udpClient.setAudioCodec(codec)
  }

  override fun onAudioInfoImp(isStereo: Boolean, sampleRate: Int) {
    rtmpClient.setAudioInfo(sampleRate, isStereo)
    rtspClient.setAudioInfo(sampleRate, isStereo)
    srtClient.setAudioInfo(sampleRate, isStereo)
    udpClient.setAudioInfo(sampleRate, isStereo)
  }

  override fun startStreamImp(url: String) {
    streamClient.connecting(url)
    if (url.startsWith("rtmp", ignoreCase = true)) {
      connectedType = StreamType.RTMP
      if (videoEncoder.rotation == 90 || videoEncoder.rotation == 270) {
        rtmpClient.setVideoResolution(videoEncoder.height, videoEncoder.width)
      } else {
        rtmpClient.setVideoResolution(videoEncoder.width, videoEncoder.height)
      }
      rtmpClient.setFps(videoEncoder.fps)
      rtmpClient.connect(url)
    } else if (url.startsWith("rtsp", ignoreCase = true)) {
      connectedType = StreamType.RTSP
      rtspClient.connect(url)
    } else if (url.startsWith("srt", ignoreCase = true)) {
      connectedType = StreamType.SRT
      srtClient.connect(url)
    } else if (url.startsWith("udp", ignoreCase = true)) {
      connectedType = StreamType.UDP
      udpClient.connect(url)
    } else {
      onMainThreadHandler {
        connectChecker.onConnectionFailed("Unsupported protocol. Only support rtmp, rtsp and srt")
      }
    }
  }

  override fun stopStreamImp() {
    when (connectedType) {
      StreamType.RTMP -> rtmpClient.disconnect()
      StreamType.RTSP -> rtspClient.disconnect()
      StreamType.SRT -> srtClient.disconnect()
      StreamType.UDP -> udpClient.disconnect()
      else -> {}
    }
    connectedType = StreamType.NONE
  }

  override fun getAudioDataImp(audioBuffer: ByteBuffer, info: MediaCodec.BufferInfo) {
    when (connectedType) {
      StreamType.RTMP -> rtmpClient.sendAudio(audioBuffer, info)
      StreamType.RTSP -> rtspClient.sendAudio(audioBuffer, info)
      StreamType.SRT -> srtClient.sendAudio(audioBuffer, info)
      StreamType.UDP -> udpClient.sendAudio(audioBuffer, info)
      else -> {}
    }
  }

  override fun onVideoInfoImp(sps: ByteBuffer, pps: ByteBuffer?, vps: ByteBuffer?) {
    rtmpClient.setVideoInfo(sps, pps, vps)
    rtspClient.setVideoInfo(sps, pps, vps)
    srtClient.setVideoInfo(sps, pps, vps)
    udpClient.setVideoInfo(sps, pps, vps)
  }

  override fun getVideoDataImp(videoBuffer: ByteBuffer, info: MediaCodec.BufferInfo) {
    when (connectedType) {
      StreamType.RTMP -> rtmpClient.sendVideo(videoBuffer, info)
      StreamType.RTSP -> rtspClient.sendVideo(videoBuffer, info)
      StreamType.SRT -> srtClient.sendVideo(videoBuffer, info)
      StreamType.UDP -> udpClient.sendVideo(videoBuffer, info)
      else -> {}
    }
  }
}

now it is working this custom files