Salesforce AI Introduces FOFPred: A Language-Driven Future Optical Flow Prediction Framework

Salesforce AI Introduces FOFPred: A Language-Driven Future Optical Flow Prediction Framework

Salesforce AI researchers have unveiled FOFPred, a novel language-driven framework for predicting future optical flow, enabling advancements in both robot control and video generation. The system predicts 4 future optical flow frames from input images and text instructions, offering a compact motion-only representation.

Future optical flow offers a more efficient approach to motion prediction than predicting full RGB frames, reducing computational complexity and focusing on essential movement data for tasks like robot control. Traditional methods struggle with accurately predicting motion in dynamic environments, often leading to unstable or inefficient robot behavior.

Key Insights

• Qwen2.5-VL & Flux.1 Integration (2026): FOFPred leverages a frozen Qwen2.5-VL vision language model and Flux.1 VAE for efficient encoding and decoding of visual and textual data.
• Relative Optical Flow Calculation: The framework addresses noisy video data by calculating relative optical flow, removing camera motion to isolate object-centric movements.
• Diffusion Transformer Architecture: FOFPred utilizes a diffusion transformer (DiT) trained on latent representations, enabling high-quality motion prediction and generation.

Working Example

# Example of encoding optical flow as RGB (conceptual)
import numpy as np
import cv2

def flow_to_rgb(flow_x, flow_y):
  """Converts optical flow components to an RGB image."""
  magnitude, angle = cv2.cartToPolar(flow_x, flow_y)
  hsv = np.zeros_like(flow_x, dtype=np.uint8)
  hsv[..., 0] = angle * 180 / np.pi / 2  # Hue
  hsv[..., 1] = 255  # Saturation
  hsv[..., 2] = cv2.normalize(magnitude, None, 0, 255, cv2.NORM_MINMAX) # Value
  rgb = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
  return rgb

# Example Usage (assuming flow_x and flow_y are numpy arrays)
# rgb_flow = flow_to_rgb(flow_x, flow_y)
# cv2.imwrite("optical_flow.png", rgb_flow)

Practical Applications

• Robotics: FOFPred enhances robot manipulation by providing accurate motion forecasts, enabling robots to perform complex tasks with greater precision and efficiency.
• Pitfall: Relying on raw optical flow without camera motion compensation can lead to inaccurate predictions and robotic failures, especially in egocentric video scenarios.

References:

• https://www.marktechpost.com/2026/01/21/salesforce-ai-introduces-fofpred-a-language-driven-future-optical-flow-prediction-framework-that-enables-improved-robot-control-and-video-generation/