Artificial ANPR/LPR Image Generation

A number of months ago, I was implementing some complex transaction forming functionality in a work project, and found I was spending more time creating test scenarios and testing my implementation than I could spend developing, as a consequence of the complex nature of the data, and real-time processing components. Traditionally, we would create data using real and configured  hardware, or load a video into one of our ANPR (Automatic Number Plate Recognition) engine providers, to simulate a camera. Although I had previously made tools to automatically create videos for simulation from a csv of licence plate data, it wasn’t practical to make synchronised videos for > 2 ANPR lanes.

artificial-image-generation-01

The transaction forming and read processing systems are idiosyncratic in the industry, and offer a lot of flexibility in ANPR applications. A number plate would typically be detected by our ANPR/LPR hardware in free flow (a vehicle passes through the camera and its presence is automatically acknowledged by the system), or triggered (another device or mechanism prompts the engine to analyse the currently buffered field of view and extract a licence plate read), and passed by its provider to the integrator, which would abstract the provider’s function and configuration, and facilitate a conduit to the processing hub and data repositories. The hub is then able to accept the read, analyse the circumstances from which it arrived, and determine the next stage to pass it to for further processing.

Stages could do anything, from post-correcting a numberplate (changing a low confidence ‘i’ to a ‘1’ in an Australian state with a plate format that has a higher probability of presenting a ‘1’ in this position), or transposing the read from its physical origin to a virtual lane or camera based on complex rules or traffic direction, to forming transactions and journeys from the existing reads in conjunction with this read, representing a visit in a car park, or passage through a defined zone.

When a stage is developed or modified, it typically relies on complex aggregated read data. Understandably using real hardware is quite burdensome to set up, configure, and track. The alternative is to artificially create data; this might otherwise result in setting up integration tests or seeding a data store. These again only go so far, especially whilst debugging in real time. I created a solution at home to simplify the end to end process, with a generation system that could create an artificial looking number plate and attach it to a drawn vehicle. Although petty, the image of the vehicle is important because this makes it easy to keep track of a certain car and notice greater system flaws during integration.

artificial-image-generation-02

My solution mounted as a provider inside our existing integrator, and was effortlessly absorbed by the IOC (inversion of control) resolver, (ensuring compatibility back to 2012!), and presented a WebAPI 2.0 set of endpoints. The endpoints permitted triggering a free flow read or queueing data at a camera/lane/engine level to be dequeued should a trigger arrive externally. This further extended to generate a random licence plate from the permissible stand-issue Victorian plates, or to generate a car for a known plate in the correct format, to providing the exact data desired for the read; including metadata, to images.

Note that most ANPR/LPR hardware uses two lenses. One to capture the vehicle in a nice format for display, and one through a lenses with an infrared filter applied, to improve accuracy with a higher contrast image.
Note that most ANPR/LPR hardware uses two lenses. One to capture the vehicle in a nice format for display, and one through a lenses with an infrared filter applied, to improve accuracy with a higher contrast image.

This ultimately allowed for running real-time scenarios from JavaScript or a tool like postman, with the exact data needed, whilst supporting end-to-end debugging of the entire transaction flow. This resulted in reducing a confusing and irritating 6 day job, to a 2 day task which was pleasurable to work on.

Although I haven’t updated this specific tool in a long time, I have expanded components for many other projects with features such as Taxi and Bus support, multiple state standard issue plate support, consistent car generation for any particular plate text, and fun value adds, like derelict looking vehicles to attach to watch lists, and people rendered driving the cars. I am now at a point where I would like to formalise the generation, rendering, and web interface components, to support my LEAP (LPR Engine Accuracy Platform) family of projects, which will eventually contribute towards developing my own ANPR/LPR engine from the ground up. This would include:

  • A portable definition file for a licence plate format, so that it can be created in an editor by a graphic designer or engineer, and integrated with any other LEAP component.
  • A portable definition file for a vehicle, or content generally. Supporting multiple colours, layers, and rules. Again, the goal would be to extract the engineering/design work so that it can be performed by someone more effective in this domain.
  • An ANPR/LPR virtual engine, with full metadata, such as plate coordinates, and an inaccuracy factor to simulate a real engine in software.
  • An abstracted ANPR/LPR Engine interface, with a provider to utilise my virtual implementation.
  • A presentable web interface to wrap and present all the virtual engine functions, and configure the instance.
  • Documentation.

As the study semester concludes and my free time becomes more available, this will be something I work towards, after first defining the shape of a generic ANPR/LPR read (LEAP.ReadIO). A new and exciting project has arisen at work which would largely benefit from such a system, and I’m confident time in orders of magnitude could be saved in development and testing.

artificial-image-generation-04

ANPR/LPR Heat Map Analysis

I created a simple utility today to assist in evaluating the alignment and environment factors of an LPR camera installation,  based on metadata produced by an LPR engine.

LPR is almost an art – having a great quality image, a well-tuned engine, and ideal ambient conditions are not enough to ensure best performance. Quite often, the engineering aspects of an installation far outweigh any ability to improve accuracy in software. Consequently, domain knowledge and experience is imperative to ensure that an LPR installation performs to the best of its capability.

By analysing the positions of a set of number plates in a camera’s field of view (FOV) with respect to confidence, correctness, and status; a lot of information regarding the camera can be deduced which in turn helps improve system performance.

Projects-Heatmap-01

Manual sampling can go a long way with the right knowledge. From the image above, it is easy to see that alignment could likely be changed in such a way that the region in which number plates present themselves is more central to the camera. This of course would help improve capture rate – one of the standard benchmark vectors of an LPR system. Doing so would likely just introduce reads prematurely, and adversely impact performance. This would then require the engine to be retuned; but the nature of how to do so can be difficult to deduce. Hence analysis can be quite complex, and problems can quickly compound and become overwhelming.

Projects-Heatmap-02

When viewed on a heat map, this requirement for alignment becomes immediately obvious. It is no longer necessary to be consciously aware of what’s going on during an accuracy audit – the necessary actions can be deduced entirely from the data in this image and the statistics that accompany it.

Projects-Heatmap-03

Different scenarios present different problems of course – like in this image it can be observed that the system read accuracy is quite good, as implied by lots of green. However, the congregation of these reads could be centralised in the FOV, permitting plates which might be uncaptured presently to be captured as they pass through a currently excluded region.

Projects-Heatmap-06

False positives can impact different systems in different ways. In some systems that rely on inferred transactions, they can cause havoc; creating invalid transactions which may have devastating roll-on effects. In other systems, they might go completely unnoticed. Regardless, as with any real-time transactional system that deals with images, space is a premium, and wasting it on useless data is an expensive penalty. A ‘next-step’ from considering a map with similar formations to this could be to filter reads that fall in the yellow regions (false positives), and possibly realign the device to re-centralise the hot spot. It may even justify considering a read-confidence filter to stop them existing in the first place.

Projects-Heatmap-04

An even distribution of misreads might indicate that the engine is not tuned or simply not capable enough to read the plate formats being presented. That is not the case in this scenario however, as the congregation of red regions central to the image, of a small size, tells us that the plate is too small to optimally read, and that a size filter will likely improve the system’s performance dramatically. Furthermore, a region filter could be applied to eliminate almost all the false positives with confidence, and minimal side effects.

Overall, I am really happy with how this process makes it easier to form educated decisions, with lowered risk. I will likely develop this tool further, to interface with more systems and engines to make it easier to quickly assess the quality of an installation with respect to the engineering decisions made, and of course, rectify any issues that may be present.