ARTICLE

How Canon's Free Viewpoint Video System brings a new perspective to Rugby World Cup 2019™ action

A representation of Canon's Free Viewpoint Video System showing the virtual camera moving freely around the playing field as a rugby player runs with the ball and others chase him.

Canon's Free Viewpoint Video System captures Rugby World Cup 2019™ action from every angle using an array of cameras around the ground, then combines all that information to create a totally immersive experience.

From the well-established Hawk-Eye system in tennis to the Video Assistant Referee in football, we're getting used to technology bringing new perspectives to sports. Now Canon's revolutionary Free Viewpoint Video System is taking tech to the next level, creating a thrilling and immersive new way to view sporting events. It makes it possible to watch the action from any angle, from the highest seat in the stadium to behind the try line or amongst the players, change viewpoint freely while the action is ongoing, and even switch to slow-motion, dramatically transforming the way you experience sports.

Standard stadium broadcast configurations use a combination of fixed cameras, roving operators on the sidelines, cable-suspended units and discreetly placed 'action cameras' to provide video feeds from a number of angles. The Free Viewpoint Video System also makes use of multiple cameras positioned around a stadium, but a much greater number of them, and the high-resolution feeds are combined and converted into 3D spatial data, generating a comprehensive three-dimensional map of the action moment by moment. This is stored on servers and can be subsequently navigated using a 'virtual camera' in order to view the action from any perspective, freely move around as the action is ongoing to get alternative angles on key moments, and put viewers closer to the action than ever before.

Following its debut with a J.LEAGUE Cup Final in 2016, the Free Viewpoint Video System is being put through its paces at Rugby World Cup 2019™. Canon is providing the International Games Broadcast Services (IGBS) with highlight videos of six matches being played at the 72,000-capacity International Stadium Yokohama – the largest stadium in Japan, and the venue for the tournament's semi-finals and final.

A diagram showing the Canon Free Viewpoint Video System shooting a match from all angles, the data being processed, and the footage being output to multiple formats.

A representative Free Viewpoint Video System setup. It requires some serious processing of the huge quantity of data being captured by the array of cameras, but the system has the potential to transform sports broadcasting and revolutionise training and performance analysis with almost unlimited angles on any play.

Capturing all the action

The Free Viewpoint Video System has seen numerous enhancements since its introduction. Image quality has increased, while the time it takes to generate the final video has decreased, enabling highlights to be delivered on the same day as the match. Additionally, the area covered by the system has been expanded. Rather than just focusing on the area of in front of the goal, as it did originally, the system now captures the entire pitch, allowing high-definition Free Viewpoint Video content to be generated wherever the action is happening.

To produce such comprehensive coverage of a rugby or football match requires numerous cameras to be placed at a range of angles around the pitch – the number of cameras required varies according to the size of the venue. Despite the amount of equipment involved, the team took just two to three days to install the cameras on pillars all the way around the International Stadium Yokohama.

A frame from a Canon Free Viewpoint Video System video of the Rugby World Cup 2019 New Zealand v South Africa match.

Experience the New Zealand v South Africa match at Rugby World Cup 2019™ as never before – the Canon Free Viewpoint Video System makes it possible to watch a match from any angle and manoeuvre a virtual camera around the 3D pitch while the action is in progress.

A high-level installation of three Canon Cinema EOS cameras, with weatherproof protective covers, viewed from underneath.

Just one of the camera installations on pillars around International Stadium Yokohama for Rugby World Cup 2019™, capturing all the action. Footage from the many cameras around the stadium is fed to high-performance servers to be combined and converted into 3D spatial data.

Converting video into 3D data

All the cameras are connected to a network and synchronised, which is no small task – in order for the 3D data to be generated correctly, all of the cameras have to be frame-perfect, which means all of them must be running in sync. Several members of staff are on-site to both monitor the system and produce the finished videos.

As the feeds from all the cameras are captured, the players, ball and other objects are extracted from the image and processed to generate 3D data. Naturally, the huge amount of data that's being created requires some equally prodigious processing, which is achieved with the aid of high-performance servers installed in a production truck located just outside the stadium.

Once the 3D data has been generated, the operator can set or move the position of the virtual camera within the 3D space. The development team has created a controller that not only permits the virtual camera to be positioned at precisely the desired area on the pitch, but also allows match time to be manipulated, speeding up the action or slowing it down.

Being able to move the viewpoint seamlessly while slowing down the action makes it possible to deconstruct and analyse the game-changing points in a match in even greater depth. It's easy to see the potential for the system to be used by coaches and managers to enhance the training of their athletes.

The final stage of the process is to output the video data in a format that can be viewed and shared. During Rugby World Cup 2019™, Canon has provided IGBS with highlight videos from each designated match within an hour of the final whistle.

In a frame from the Canon Free Viewpoint Video System clip of the Ireland v Scotland match at Rugby World Cup 2019, an Irish player takes a penalty kick.

As Ireland take a penalty kick in the Ireland v Scotland match at Rugby World Cup 2019™...

In a frame from the Canon Free Viewpoint Video System clip of the Ireland v Scotland match at Rugby World Cup 2019, the ball flies over the crossbar in a bird's-eye view.

...The virtual camera follows the flight of the ball and gives you a bird's-eye view as it travels through the goal posts.

The future of video capture

Currently, the Free Viewpoint Video System is being used to produce curated highlights after a match, but a new phase of development will see the technology being applied to a video replay or a live broadcast. To facilitate this, Canon is ramping up the system to support real-time video generation.

Of course, there is a whole world of sports out there beyond rugby and football. Free Viewpoint Video content could be used to enhance baseball or American football broadcasts, and the system could be applied to sports that take place in venues that are smaller than stadiums, such as basketball and judo. Canon's technology also has an exciting future outside of sport – concerts, commercials, music promos and movies are some of the areas that might benefit from the unique perspectives offered by the Free Viewpoint Video System. Watch this space – from any angle!

• Canon is an official sponsor of Rugby World Cup 2019™, taking place in Japan from 20 September to 2 November.

كتابة Marcus Hawkins