Month: January 2022

A sparse point cloud created during the Structure from Motion (SfM) step

Mapware’s Photogrammetry Pipeline, Part 3 of 6: Structure from Motion

The first two steps in Mapware’s photogrammetry pipeline dealt with 2D images of a landscape. By the third step, structure from motion (SfM), Mapware finally begins to see that landscape in 3D.

Purpose of structure from motion

Until this point, Mapware’s photogrammetry pipeline has worked exclusively with 2D images. During the keypoint extraction step, Mapware identified distinctive features visible in each image. Then, during the homography step, Mapware sorted those images into pairs that overlap one another on the same feature. It also calculated the linear transformation (mathematical differences in scale and orientation) between the images in each pair.

As a photogrammetry tool, Mapware must eventually make the leap from 2D to 3D. This starts to happen in the third step of its pipeline, called structure from motion (SfM). In this step, Mapware feeds all the image pairs it identified during homography into a low-resolution, sparse 3D representation of the landscape.

What does “structure from motion” mean?

The term “structure from motion” alludes to motion parallax, one of the ways humans perceive the world in 3D. As we move through space, our eyes see the same objects from different vantage points and our brains use that info to construct a mental 3D map of our environment.

In aerial photogrammetry, drones serve as a computer’s eyes—“walking” the computer through a landscape by taking overlapping photographs from above. And in the structure from motion step, Mapware serves as the computer’s brain by merging those images into a 3D model.

How Mapware creates sparse 3D models

To turn 2D photos into a sparse 3D model, Mapware must take the following steps:

Calculating linear transformation

Remember that, as drones fly over a landscape, they photograph the same feature multiple times from different distances and angles. Mapware addressed these scale/orientation differences differently in the first two steps of the photogrammetry pipeline.

In the keypoint extraction step, Mapware’s algorithms had to ignore any differences in scale and orientation. They had to know, for instance, that a feature in one image taken at close range was the same feature in another image taken from farther away. This way, it could assign both images nearly identical keypoints.

In the homography step, Mapware took a different approach and mathematically calculated these differences. As it identified image pairs with similar keypoints, it determined mathematically how the features in one image would have to be rotated and resized to perfectly match the features in another image. This calculation is called the linear transformation and is stored as a group of homography matrices.

The structure from motion step reuses this data. As Mapware adds each image pair to its 3D model, it uses the above homography matrices to undo differences in scale and orientation between the images. This ensures that the resulting 3D model will have a consistent scale.

Accessing camera metadata

In addition to calculating linear transformations between images, Mapware accesses metadata within each image to determine the camera’s focal length and to figure out its direction and distance from the subject. With this data, Mapware can accurately position the photos in 3D space.

Performing bundle adjustments

Even with the above homography calculations and camera data, Mapware is still estimating the locations of those drone cameras in 3D space. This is subject to what’s called the reprojection error and can seriously corrupt a model if not corrected. To minimize reprojection error, Mapware performs bundle adjustments at several intervals during the process.

After each image is added to the model, Mapware revisits the existing images and adjusts their 3D positions based on new information gained from the latest addition. These adjustments result in the best possible consensus between all images in the model. In other words, the entire model gets more accurate when it accepts new data.

Mapware’s SfM process

Here’s a basic look at the process from start to finish:

  1. Mapware selects a pair of images to serve as the basis of its model. It uses the homography matrices calculated before to try to pose the images together in the same scale/orientation, and then projects their keypoints into 3D space.

  2. Mapware sorts the remaining image pairs by priority and attempts to add them sequentially to the model. Higher priority is given to images with keypoints that are spread out across their surface, and to images with keypoints that match existing points in the model.**

  3. Mapware performs a cleanup step to remove bad data. This could mean images that failed several attempts at insertion to the model, pairs that were mistakenly matched during homography, points in the model that only occur in one image, or other outliers that could degrade the quality of the 3D model.

** NOTE: As mentioned above, Mapware performs bundle adjustments at several intervals during model reconstruction, including after each image is added and again after the final image has been added.

A landscape represented as a sparse point cloud (left) and 3D digital twin (right)
Note the difference in quality between these two models of Sandwich, Massachusetts. The sparse point cloud on the left was constructed during SfM. Mapware has to get through several more steps in the photogrammetry pipeline before it can generate the final 3D mesh on the right.

What comes next?

At this point, Mapware has a low-resolution, sparse 3D representation of the environment captured by drone photos. This is referred to as a sparse point cloud, and is the first indication about whether the drone image set will result in a high-quality 3D digital twin at the end of the pipeline.

But Mapware will improve the accuracy of its sparse 3D model by creating depth maps—topographical maps showing elevation of each point in each image. We will discuss the depth mapping step in the next blog in this series.

A drone executes a grid flight path over a target area

What are Drone Flight Planning Apps for Photogrammetry?

Why should you use a drone flight planning app for photogrammetry?

As UAV mapping grows in popularity, aspiring drone pilots are bound to hear about automated flight planning, flight control, and image capture apps. (We’ll call them flight apps for brevity.)

This is likely to be a new concept for first-time commercial pilots, and maybe even for businesses specializing in manual flight operations like photography or newsgathering. But a lot of the literature just assumes everyone knows what a flight app is and why it’s critical for photogrammetry. Read on if you missed the memo.

This isn’t going to be another of those “best apps for drone flying” lists, but an explanation of what flight apps are, how they work in the field, and why pilots absolutely need them for commercial UAV mapping.

What is a flight app?

A flight app is a computer program that lets you plan, execute, and monitor automated drone flights from your computer or mobile device. Many flight apps also allow you to configure photogrammetry mission settings such as camera angle and image overlap, further simplifying your job.

Since they exist on your PC, laptop, tablet, or smartphone, flight apps don’t typically come packaged with commercial-off-the-shelf drones. You’ll have to buy one separately and download it.

A drone, laptop, tablet, smartphone, and FPV headset
With a flight app, you run the mission from a laptop, tablet, or smartphone instead of your manual controller.

How do they work in the field?

No two flight apps are exactly alike. But here’s a hypothetical example of how you might use one to conduct an automated photogrammetry flight:

Preflight Planning

You arrive at the mission site with your drone and a tablet on which you’ve installed your flight app. When you boot up your tablet, the flight app displays a GPS map of the surrounding airspace. You draw directly on this map—either manually selecting waypoints for the drone to fly over or designating a coverage area and letting the app design a suitable flight pattern. Then you open other menus to set your drone camera angle, the number of photos it should take, and the amount of overlap between adjacent photos.

Flight

With everything set, you tap a button on the tablet and the drone takes off on its own towards the mission area. At this point, you probably put down your tablet and pick up your drone’s control stick if only to assume manual control in an emergency. Ideally, the drone follows its preprogrammed flight pattern, takes the necessary photos, and lands without your intervention.

Post-Flight

With your drone safely on the ground, the flight app begins automatically downloading photos from your drone’s SD card to your tablet and may even kick off an image processing job.

Why are they critical for UAV mapping?

Flight apps make professional UAV mappers more effective at their jobs. Here are a few reasons why you need them:

Precision

While a skilled pilot can achieve a lot, it’s very hard to do professional photogrammetry under manual flight control. Clients expect lifelike 3D models. But most photogrammetry software is unforgiving to variances in photo resolution caused by even slight changes in a drone’s airspeed, altitude, and pitch. Furthermore, most photogrammetry missions call for taking photos with a precise front and side overlap (70%, for instance). This is virtually impossible for a human to achieve with consistency. But computers do it effortlessly.

Efficiency

A good flight app not only executes missions with inhuman precision, but also with inhuman efficiency. The first time you use a flight app, it may shock you to see how quickly it completes the mission. With modern drone batteries lasting no more than 30 minutes, automating the flight can save you valuable time and let you cover much larger areas.

Reduced Pilot/Observer Workloads

If you want to run a single-pilot UAV mapping business, flight apps are essential. Anyone who has tried conducting a manual photogrammetry flight solo knows they risk breaking Part 107 regulations every time they glance momentarily from their drone to their camera to line up a shot. More importantly, the danger of actually losing visual line-of-sight increases with the size of the mission area. (You’d think by now that all drone manufacturers would paint their airframes bright orange for visibility, but for some reason they love gray.)

By using a flight app, you can look directly at your drone throughout the flight while the app handles the camera work. This means you can conceivably photograph large-scale areas safely without having to hire a visual observer.

All-In-One UAV Mapping Tool

Think about the complex workflow you must follow to conduct a single photogrammetry flight. First, you need to check an app like NOTAM Search or B4UFly for flight restrictions. After each flight, you need to manually transfer your photos from your drone to your smartphone or PC, and then from there to a desktop or cloud-based photogrammetry provider.

A good flight app is like a Swiss army knife for photogrammetry, handling automated flight planning, restricted airspace awareness, data capture, and maybe even image processing all in one application.

A drone transfers its image data to a laptop and then to the cloud for processing
After a flight, some apps may automatically transfer your images for photogrammetry processing.

Is all UAV mapping automated?

In general, manual piloting skills still matter in UAV mapping. Most flight apps build what pilots refer to as “lawnmower” flight paths, sweeping back and forth over the target area at a relatively consistent altitude and maintaining a fixed camera angle. While this is great for capturing flat, multi-acre sites, it doesn’t give the drone a clear view of vertical structures. Using a flight app over a forest might result in a 3D model of trees with clear canopies but blurry, distorted trunks. To get around this, pilots may still supplement their automated flights with manual flights—letting the app survey the entire area from above, and then conducting manual flights that orbit tall structures to capture their side views or even fly underneath their overhanging surfaces.

That said, some companies have begun offering flight apps designed specifically for vertical structures, and they are likely to become more commonplace as the tech evolves.

How to choose your UAV mapping flight app?

There are many flight apps on the market, but they aren’t all necessarily suited to every UAV mapping mission. If you decide to purchase a flight app for your business, review these common but important features:

Drone Compatibility

Before choosing a flight app, make sure your drone is compatible. Some drones, especially new models, may not support all available apps. For example, the DJI Mini 2 was released in November 2020, but its users had to wait for a January 2022 firmware update before they could use any automated flight app. As of this writing, many app developers are still playing catch-up.

Waypoint Flight Planning

This is a core feature of flight apps, but it’s worth mentioning. Your app should have an intuitive map that lets you draw the target area using lines, points, and polygons, and monitor the drone’s real-time position during the flight.

Camera/Overlap Control

Flight apps tailored for UAV mapping also let you configure camera settings like angle, shutter speed, and ISO, as well as the overlap between photos.

Flight Restriction Overlays

This important safety feature lets you see TFRs and other no-fly zones on the map as you plan your flight and may even prevent you from plotting waypoints in illegal airspace.

Integration with Photogrammetry Software

The best flight apps are part of a larger UAV mapping suite and can save you time by automatically transferring your images from the drone’s SD card to your mobile device and from there to your photogrammetry engine for processing.

Optional: Vertical Structures

As stated above, most UAV mappers can supplement their automated flights with manual flights to capture the occasional tower, tree, and building. But if a significant part of your business is modeling vertical structures, it might be worth finding an app that offers more than the standard “lawnmower” flight paths. Be aware that you may have to pay a steeper price.

Final thoughts

We hope this post has demystified the concept of flight apps and convinced you of their value in UAV mapping/photogrammetry. Let us know if we’ve missed any important app features. Most of all, fly safely.

How Hard is it for Manned Pilots to Pass the Part 107 Exam?

You’re a traditional manned (crewed) pilot who wants to fly drones for hire. How ready are you? What and how much do you need to study? In this post, we will help you harness your previous flight training to tackle the drone exam. 

Introduction

In a previous post, we outlined the steps non-pilots should take to study for, schedule, and pass the Part 107 Remote Pilot License exam. We also noted that licensed private pilots can instead take the FAA’s short online course thanks to their preexisting knowledge. But there’s a caveat: only pilots who meet the recency requirements of 14 CFR Part 61 are eligible to bypass the knowledge exam. If you’ve been out of the game for years, either because you retired from active service or simply too busy to fly, you’ll have to take the test like everyone else. 

This puts you in an odd spot. On one hand, you’re right to think the process should be shorter and easier for you due to your previous flight training. On the other hand, flying drones is certainly not like flying a Cessna. Just how ready are you to waltz into that exam room?  

Below, we reexamine the Part 107 curriculum through the eyes of rusty pilots like you who aren’t sure where to jump back in. Let us help you crush the Part 107 exam in short order.  

Brief overview: Part 107 versus Part 61

First thing’s first. The entire Part 107 certification process is the equivalent of what your Part 61 CFI referred to as “ground school.” There are no practical skills requirements. Whereas Part 61 training prescribes mandatory flight-hour minimums (including dual, solo, and night), an aeromedical evaluation, and a practical flight test, Part 107 students will never even speak with a DPE. Once you pass the written exam, you earn your commercial drone pilot license with all the rights and responsibilities it entails. It’s going to feel weird. 

Second, you probably know most of what will be on your Part 107 exam because the material is very similar to what you learned before. You might even recognize some of the test questions from your old ASA Private Pilot Test Prep Book. But there is still a sizeable amount of new material to absorb, and we will go over that below. 

NOTE: Should you still enroll in a drone flight-training program? There are many companies out there offering practical, hands-on training, and they are probably worth your investment. But unlike traditional flight school, they are not required to earn your license. We suggest passing the knowledge exam separately and learning practical skills—whether through a paid program or self-led practice—on your own time. 

Step 1: Review the FAA’s Part 107 training materials

Previously, we told you where to find the FAA’s Part 107 training materials. Now, we’ll help you prioritize each one in your rusty pilot training plan. 

#1: Title 14 CFR Part 107 – Small Unmanned Aircraft Systems 

Read this first. It contains virtually all the new information you will have to learn—specifically, how to fly drones for hire legally in the United States. Part 107 is mercifully short compared to other sections of the FAR-AIM, but you still need to know it well to pass the test. This is especially true if you start your career as an independent or freelance pilot. Remember: when you go solo, you are the expert. Nobody else will warn you when you are about to break a law. 

#2: AC 107-2 – Small Unmanned Aircraft Systems (SUAS) 

Read this next. It’s a supplement to Part 107, providing additional guidance on certain regulations. Some highlights include: 

  • Chapter 2, which provides a useful list of FAA reference materials in addition to those we’ve listed here 
  • Chapter 5, which offers practical techniques for communication and transfer of controls between pilots, estimating groundspeed and altitude, and operating over people or from moving vehicle 
  • Appendix C, which provides a neat preflight inspection condition chart for people who aren’t maintenance experts (useful for making go-no-go decisions) 

#3: FAA-G-8082-22 – Remote Pilot – Small Unmanned Aircraft Systems Study Guide 

Once you’re comfortable with Part 107, open this document to begin reviewing familiar ground school topics. While this study guide can help you reframe your old studies in the context of drone operations, do not assume it’s comprehensive. At best, it spotlights just some of the topics you will have to know for the exam. To get a sense of the full list, you need the ACS. 

#4: FAA-S-ACS-10B – Remote Pilot ‒ Small Unmanned Aircraft Systems Airman Certification Standards 

NOTE: If your original flight training concluded before 2015, you may not be familiar with the concept of Aeronautical Certification Standards. In short, ACS is an attempt by the FAA to improve the practical value of ground school by tying theoretical knowledge topics directly to pilot certification requirements. In crewed flight training, that means exchanging rote-based memorization for scenario-based training that teaches pilots to think outside the box. 

The ACS is particularly useful if you fail your exam. Read its section “Using the ACS” for more information on how you can figure out which topics you need to study more.  

More importantly, the ACS document provides you, the rusty pilot, with a roadmap of topics you will need to revisit from your previous flight training. Skim its list of learning objectives and you’ll realize that the Part 107 exam tests you on most of the same ground school concepts as it does crewed pilots, except maybe for flight instrumentation, airport signage, and engine mechanics. This should help you plan your ground school refresher. 

Step 2: Plan your ground school refresher

Option #1: Pilot’s Handbook of Aeronautical Knowledge 

Using the ACS objectives as your guide, skim the chapters in this book related to flight school topics that you’ve forgotten the most. Pay special attention to information-heavy topics like weather, aeronautical charts, and performance diagrams. You don’t want to be blindsided on exam day by an obscure question about microbursts. And make sure you can problem solve. For example, in keeping with the ACS philosophy, the exam may ask you to look at a spot on a Sectional Aeronautical Chart to determine whether your client’s requested flight is safe and legal. 

Option #2: Pilot Training System’s Drone Pilot Training Videos 

Instead of reading your notes or the Pilot’s Handbook of Aeronautical Knowledge, you can watch this YouTube series to cover the same topics in roughly 1.5 hours. Each video covers a particular topic, so you can pick and choose.  

Final thoughts

As a crewed pilot, you already have at least 75% of the requisite knowledge, and should easily pass the Part 107 exam with an abbreviated self-study plan like the one above. But don’t let that lull you into complacency. The FAA will expect you to behave responsibly like any other commercial pilot. Make sure you remember everything you learned to earn your first wings. Aerodynamics. Performance. ADM. NOTAMs. Airspace. Aeronautical charts. Weather. And once your new cert comes in the mail, do the public a favor and maintain your old high standards.  

Best of luck!