Interactive Data Exploration for Computer Vision Projects with Rerun

instruments for information visualization and exploration have allowed me to iterate means faster on my pc imaginative and prescient tasks, particularly when the issues I’m confronted with usually are not simple and I have to make algorithm or design selections based mostly on dynamic time-varying alerts. These alerts can usually be difficult to research by simply quickly altering numbers plotted on the display screen or saved in a desk.

Whereas engaged on a few of these issues, I explored the built-in interactive parts from OpenCV, however other than some sliders the choices there are very restricted, particularly when attempting to combine some animated plots. There’s some hacky methods to get dynamic plots from matplotlib into OpenCV, which I explored on this post.

I additionally explored completely different UI frameworks like tkinter, which I utilized in my last project for a sentiment evaluation visualization. I constructed a customized element that allowed me to show dynamic frames. Nevertheless it nonetheless didn’t actually really feel like the correct device for the duty, particularly when attempting to work with interactive plots.

After which I stumbled upon Rerun. Each every so often I uncover a device or framework that actually excites me, and this was precisely a kind of instances. Rerun is an open supply device for visualizing information usually discovered within the robotics area starting from easy time sequence information and static photos to complicated 3D level clouds, video streams or different sorts of multi-modal information. The demos look actually spectacular and the setup and code samples are so easy, I used to be instantly hooked.

So I made a decision to transform my ball monitoring demo from a earlier mission and plot the information utilizing rerun. Let me present you the way simple it’s to make use of and create interactive purposes!

Quickstart

You’ll be able to set up rerun in any of your python tasks utilizing pip or uv:

pip set up rerun-sdk
uv add rerun-sdk

You can begin the viewer after putting in the sdk by merely operating it from the command line:

rerun

The viewer will likely be your important window the place your experiments will likely be proven. You’ll be able to depart it open or shut it between your experiments.

To instantiate a rerun viewer from a python script, you have to spawn an occasion with an experiment identify:

import rerun as rr

rr.init("ball_tracking", spawn=True)

Ball Monitoring Demo

Rerun experiment recordings may be saved to and loaded from .rrd recording recordsdata. You’ll be able to obtain the recording file for the ball monitoring demo from here. Press Ctrl + O or choose Open... within the menu on the highest left of the rerun viewer and cargo the downloaded recording file.

You will notice the ball monitoring demo playback as soon as after which the video stops. On the backside of the viewer, you’ve the timeline. You’ll be able to scrub by means of the timeline by clicking and dragging the deal with.

These recording recordsdata solely include the information of the experiment together with the video, its annotations and the time sequence of the monitoring. The format of the viewer is saved individually in a .rbl blueprint file. Obtain the blueprint file for the demo here and open it on high of the prevailing information file.

Now we’ve a barely higher overview with the place, velocity and acceleration plots separated and the video prominently centered.

Within the video body you may click on on the annotations and within the left Blueprint panel you may disguise or present them individually.

Time Sequence Plots

To investigate a selected plot, you may click on on the increase view button on the high proper of any window, for instance the place plot.

This can be a TimeSeriesView. This view can be utilized to plot information in a 2D chart with the x-axis representing the time area, in our case the discrete body index of the video. On this ball monitoring demo, we iterate over the video frames in a loop, the place we will set the body index of our timeline explicitly.

frame_index = 0

whereas True:
    ret, body = cap.learn()
    if not ret:
        break

    frame_index += 1
    if frame_index >= num_frames:
        break

    rr.set_time("frame_idx", sequence=frame_index)

To create the plot for the place, you have to log a Scalar worth for the place of the tracked ball at each body index. On this case after we calculate the place we will merely log it to rerun:

rr.log("ball/position_y", rr.Scalars(pos))

To configure the type of this plot, we additionally have to log one thing to the identical entity path (ball/position_y), however because the type doesn’t change we will log it as soon as earlier than the loop and provide a static argument.

rr.log(
    "ball/position_y",
    rr.SeriesLines(colours=[0, 128, 255], names="pos y"),
    static=True,
)

To outline the axis vary that’s seen per default, we have to specify a format element.

view_pos = rrb.TimeSeriesView(
    origin="ball/position_y",
    axis_y=rrb.ScalarAxis(vary=(0, 700)),
)
format = rrb.Blueprint(view_pos)
rr.send_blueprint(format)

Video Stream

Equally we will create a view for the video frames by logging the picture to rerun. Since rerun expects an RGB photos however OpenCV ues BGR for its shade channel ordering, we have to convert the frames from BGR to RGB earlier than passing them to rerun.

frame_rgb: np.ndarray = cv2.cvtColor(body, cv2.COLOR_BGR2RGB)
rr.log("body", rr.Picture(frame_rgb))

So as to add annotations to the picture view we have to log spatial parts to a sub path of the required entity path. For instance, we will draw the middle of the tracked ball:

rr.log(
    "body/factors",
    rr.Points2D(
        positions=[center],
        colours=[0, 0, 255],
        radii=4.0,
    ),
)

To incorporate the video body view within the format, we will use a Spatial2DView node:

view_frame = rrb.Spatial2DView(origin="body")

Then we will stack the plot from earlier than vertically with the body view by utilizing a Vertical format element:

format = rrb.Blueprint(
    rrb.Vertical(
        view_pos,
        view_frame,
        row_shares=[0.33],
    ),
)
rr.send_blueprint(format)

The row shares specifies how a lot every of the rows occupies in percentages. We will omit the second row share entry for the body view because the shares have so as to add as much as 1.

Limitations

Whereas engaged on this mission, I bumped into some limitations of Rerun. Within the unique mission I visualized a prediction of the trajectory at each timestep, however that is presently not doable in a time sequence view. Additionally the layouts and configuration of the plotted information is proscribed, for instance there’s no built-in means to attract a circle with out fill. However because the mission could be very actively being developed, there’s a very good probability that a few of these is likely to be doable sooner or later.

Conclusion

The developer expertise with this device as a pc imaginative and prescient engineer is extraordinarily good, the consumer interface hundreds virtually immediately and the timeline scrubbing may be extremely useful for understanding or debugging alerts in time sequence plots or in movies. I’ll positively preserve utilizing and exploring this mission and may solely advocate you to attempt it out for your self!

For extra particulars and the complete implementation, checkout the supply code of this mission below src/ball_tracking/trajectory_rerun.py.

https://github.com/trflorian/ball-tracking-live-plot

All visualizations on this publish had been created by the writer.