CameraView/docs/_posts/2018-12-20-frame-processing.md

layout	title	subtitle	description	category	order	date	disqus
page	Frame Processing	Process each frame in real time	Process each frame in real time	docs	5	2018-12-20 20:45:42	1

We support frame processors that will receive data from the camera preview stream. This is a useful feature with a wide range of applications. For example, the frames can be sent to a face detector, a QR code detector, the Firebase Machine Learning Kit, or any other frame consumer.

cameraView.addFrameProcessor(new FrameProcessor() {
    @Override
    @WorkerThread
    public void process(@NonNull Frame frame) {
        byte[] data = frame.getData();
        int rotation = frame.getRotation();
        long time = frame.getTime();
        Size size = frame.getSize();
        int format = frame.getFormat();
        // Process...
    }
}

For your convenience, the FrameProcessor method is run in a background thread so you can do your job in a synchronous fashion. Once the process method returns, internally we will re-use the Frame instance and apply new data to it. So:

• you can do your job synchronously in the process() method. This is recommended.
• if you must hold the Frame instance longer, use frame = frame.freeze() to get a frozen instance that will not be affected. This is discouraged because it requires copying the whole array.

Process synchronously

Processing synchronously, for the duration of the process() method, is the recommended way of using processors, because it solves different issues:

• avoids the need of calling frame = frame.freeze() which is a very expensive operation
• the engine will automatically drop frames if the process() method is busy, so you'll only receive frames that you can handle
• we have already allocated a thread for you, so there's no need to create another

Some frame consumers might have a built-in asynchronous behavior. But you can still block the process() thread until the consumer has returned.

@Override
@WorkerThread
public void process(@NonNull Frame frame) {
    
    // EXAMPLE 1:
    // Firebase and Google APIs will often return a Task.
    // You can use Tasks.await() to complete the task on the current thread.
    // Read: https://developers.google.com/android/guides/tasks#blocking
    try {
        result = Tasks.await(firebaseDetector.detectInImage(firebaseImage));
    catch (Exception e) {
        // Firebase task failed.
    }
    
    
    // EXAMPLE 2:
    // For other async consumers, you can use, for example, a CountDownLatch.
    
    // Step 1: create the latch.
    final CountDownLatch latch = new CountDownLatch(1);
    
    // Step 2: launch async processing here...
    // When processing completes or fails, call latch.countDown();
    
    // Step 3: after launching, block the current thread.
    latch.await();
}

Related APIs

Frame API	Type	Description
camera.addFrameProcessor(FrameProcessor)	-	Register a FrameProcessor.
frame.getData()	byte[]	The current preview frame, in its original orientation.
frame.getTime()	long	The preview timestamp, in System.currentTimeMillis() reference.
frame.getRotation()	int	The rotation that should be applied to the byte array in order to see what the user sees.
frame.getSize()	Size	The frame size, before any rotation is applied, to access data.
frame.getFormat()	int	The frame ImageFormat. This will always be ImageFormat.NV21 for now.
frame.freeze()	Frame	Clones this frame and makes it immutable. Can be expensive because requires copying the byte array.
frame.release()	-	Disposes the content of this frame. Should be used on frozen frames to release memory.