This repository contains the main software component of the AR Field Project. The project aims to simplify robot debugging and enhance gameplay in FIRST Tech Challenge (FTC) through the use of Augmented Reality technology. See an introduction to the project below:
Augmented Reality for FIRST Tech Challenge - Electric Mayhem Robotics
The AR Field requires a few key elements to provide a full Augmented Reality experience, as well as some optional hardware that is not supported by this application.
Projection System
The main visual component is provided by a single projector or projector array, arranged such that the combined projection area fully covers a standard FTC competition field (12x12 ft). Depending on the environment and budget of a particular deployment, the projector(s) may be hung from ceiling/truss supports, or held using a tripod/tripods. The projectors should be chosen to ensure full coverage and adequate brightness for the competition/lab environment. Having a lighting-controlled, indoor environment greatly helps with projection visibility and allows the use of more budget-friendly projectors. For best results, ensure the projection area exceeds the bounds of the field, and that projectors in an array have overlapping projection areas.
To find the optimal mounting position for a projector or set of projectors, we find that it is beneficial to model the projector's frustum (3D projection volume) by taking measurements of the projection area at different throw distances. By placing one or more of these frustum models into a scene with the FTC field CAD model, they can be positioned to allow for full field coverage and overlap (if applicable).
LED Ring
The AR Compositor software also supports a secondary visual component, an array of addressable RGB LEDs (WS2812B or similar) that span the perimeter of the field. While the LEDs or projection can be used alone, the software is best designed for their combined use. The particular setup for an LED strip may vary, as the system is able to accommodate any pixel density, but ~500 LEDs per field side is recommended. Additionally, a sufficiently rated power supply and an ART-NET compatible LED strip controller is needed to interface with MadMapper, the projection control software.
To mount the strips, it is easiest to use a pre-made LED Strip channel cut to the length of the field wall. A diffuser strip is usually included with these channel kits, and is highly recommended to make the LED light softer and less harsh for those standing field-side. Depending on the chosen attachment configuration, custom printed mounts may be used to place the lights on the top, side, or bottom of the upper lip of the field wall. The corners of the strips must be joined together to send signal and power to all sides, which will likely involve cutting/soldering the strips together. For the least brightness loss along the strip, and for compatibility with some pixel controllers, it is best to have half of the field (two walls in series) in parallel with the other half, in terms of power and signal respectively. Depending on the configuration, this may require two power supplies at opposite parts of the field perimeter.
Computer Hardware
The AR Field requires a central computer to handle communication with the FTC Live system, recieve external video inputs, run the AR Compositor, and process graphics for the other visual components. As such, it should have hardware powerful enough to handle intense video encoding, as well as a network interface and a good amount of other I/O. At minimum, use a system with Intel UHD Graphics 630 or equivalent, 16GB of DDR4-2400 RAM, and Windows 10. Ensure at least one Ethernet LAN Adapter, and 2 or more USB 3.1 ports.
Aside from the computer itself, a keyboard, mouse, and display (or a keyboard and touchscreen) is required to set up and control the software. For video input, an HDMI capture device (USB or PCIe) is required. The method of video output depends on the number of projectors being used. For a single projector, a computer's internal HDMI/DisplayPort connection can be used. For multiple projectors, a computer's discrete graphics card may be used if present. If the computer does not have a discrete graphics card, or it does not have enough display outputs, a DisplayLink adapter can be used to provide additional outputs without the need for another video card.
[!NOTE] If using a DisplayLink adapter, the DisplayLink driver must be installed on the computer.
The computer installation depends on the environment that the AR Field is being used in. For lab environments, the computer setup, along with all cabling and hardware mounting, can be permanant. It is best to keep the computer easily accessible, and installed near the field. For competition settings, the computer should still remain near the field, ideally at the scorekeeping table.
Optional Components
Optionally, the same projection software also supports various show lasers (such as LaserCube, Pangolin, or KVANT), with a paid upgrade package. Laser graphics are not generated by the compositor application, but can be accessed through MadMapper. This makes the laser add-on less useful for events, especially during match play, but can be used in lab environments for content or showcase.
[!CAUTION] Show lasers can cause eye damage if misused. Before operating a show laser, it is extremely important to become familiar with the safety information of the laser, as well as local regulations on public use. This is especially true for competitions/events, where special care must be taken to ensure the safety of spectators and competitors. As a general rule-of-thumb, mount the show laser such that the beams cannot be directed at audience members, and use attenuation zones to block out areas as needed.
In addition to the LED ring, any professional lighting equipment that supports DMX512, Art-Net, or sACN. These include all manner of dimmable stage lights, LED fixtures, and more. Note that any lighting fixtures controlled by MadMapper will sample the projection graphics.
A good part of any competition setup, an audio system is recommended for use with the AR Field. Not only can it provide the audio cues generated by the FTC Live system during match play, but it can also take audio from a connected video source to allow for multimedia content to be presented on the field. As is standard at FTC events, a speaker system can also be used for music and public address. The AR Compositor app does not produce sound itself, and all match-related audio should come from the FTC Live timing display software.
The AR Field project has been tested with a fairly simple hardware setup, which can be seen in the video above. The setup is meant to be relatively inexpensive, as it was designed as a proof of concept for the project. It is a permanant installation, but can easily be adapted for temporary deployment. Below is a list of the materials used and the cost estimate. The section above provides details on what each component is used for.
| Item | Quantity | Price |
|---|---|---|
| PROJ VOPLLS Smart Mini Projector | 4 | $160 |
| LED BTF-LIGHTING WS2812B ECO RGB LED Strip, 16.4FT 60LED/m | 4 | $14 |
| LED BTF-LIGHTING AC100-240V to DC5V20A Switching Power Supply | 1 | $23 |
| LED Besseto 10Pack 6.6Ft Black LED Aluminum Channel w/ Diffuser | 1 | $70 |
| LED LANDIN Artnet/sACN To SPI Pixel Controller 1360Pixels | 1 | $105 |
| COMP Dell Optiplex 5060 Micro | 1 | $258 |
| COMP Keychron B33 2.4 GHz Wireless Bluetooth Keyboard | 1 | $35 |
| COMP EVICIV Portable Monitor Touchscreen 15.6” | 1 | $140 |
| COMP Acer USB 3.0 HDMI Video Capture Card | 1 | $27 |
| COMP StarTech USB-C to Quad DisplayPort Workstation Dock | 1 | $135 |
| OPT LaserCube WiFi 1.2W | 1 | $1499 |
Total: $1489 ($2988 with optional LaserCube)
Note
Prices subject to change, cost estimate does not include mounting hardware, cables, tools, or software cost.
Warning
While this configuration functions as a proof of concept, the projection and compute hardware are fairly poor. The computer meets only the minimum requirements for the AR Field, which results in the mapping software being sluggish. The DisplayLink adapter can exhibit screen tearing, and all inputs may not be completely synchronized. The projectors aren't very reliable and are too dim for environments without proper lighting control. Still, it is a very usable setup for a lab environment.
The AR Field primarily relies on two pieces of software. The AR Compositor handles communication with the FTC Live system and provides AR graphics for match play, including dynamic scoring elements and LED effects. The graphics are sent to the projection mapping software, MadMapper, through the Spout protocol. MadMapper handles all projection, ensuring that the image is correctly sized to the field, and that a multi-projector setup can display one stitched-together image. The software sends signals to each of the projectors individually, and also communicates with the LEDs over ArtNet. If using the MadLaser add-on with a show laser, the MadMapper software also handles the laser output and control.
The MadMapper software is not free. A trial version exists, with a watermarked projection output and periodic blanking of all other outputs. It is suitable for testing and experimenting, but a full deployment of the AR Field system should use the licensed version of the software. A permanent license of the current version costs $560 ($860 with MadLaser) and includes 2 seats, while a temporary license is 50/seat/month ($80/seat/month with MadLaser).
The setup for MadMapper varies greatly depending on the use case, projectors, adapter, projection arrangement, laser addition, and software version. As such, no setup information can be provided here. Consult the official MadMapper resources to become familiar with the software before applying it to a specific deployment.
The general operation of the AR Compositor software is detailed in the video above, although it showcases an older development build of the software. After launching the application executable, enter in the connection information for the FTC Live system into the configuration menu. Then, wait for the application to confirm the connection. Once connected to the FTC Live server, enable the Spout stream to send graphics to MadMapper. The guides on screen will show the area containing projection graphics, and the area containing LED graphics. Map these according to the field setup. Once MadMapper is configured, disable the guides in the control panel. Then, enable any of the composite layers to display them on the field. Once configuration is complete, close the control panel with the button onscreen, or by pressing the C key. In the case of a technical issue, use the debug view to see internal timers, WebSocket messages, and match data.
For other input sources, such as a webcam or video capture card, check the MadMapper video input list. External NDI and Spout streams are also shown in MadMapper, which can be used for wireless or networked playback. Shaders and procedural animations can be accessed for showcase purposes, and are compatible with show lasers. For all graphics types, consult the MadMapper resources for information on how to properly map them to a given setup.
Note
Some composite layers may be invisible when not in a match, or exhibit dynamic behaviour while a match is in progress. See the layer description for information on what will be displayed, or run a test match in the scoring software.