This article, authored by Anthony Salvi, was published on the Allegorithmic blog in 2016.
Following on the announcements that were made during the Substance Days keynote in July, among which the intent for Allegorithmic to deepen its roots into the production and delivery of pre-set content (textures, materials, etc.), we would like to share today some knowledge that we gathered about the use of photogrammetry in order to scan materials surrounding us.
As a member of Allegorithmic Labs, I spend my time experimenting with crazy ideas, new gear, and prototype software. When you combine that with my background as a professional Digital Operator (here), I could only find the subject of photogrammetry fascinating. But while the process of utilizing expansive cameras and optics to capture your data is fairly documented, it was unclear we could do this with a simple smartphone.
I decided to dig into the subject and give it a try and because, well, “it works!”, the goal of this blog post is to provide you with all you need to set up a low-cost, DIY pipeline for high-quality material scanning using photogrammetry.
As you will see, there is no magic involved, only the right tools (Substance of course), so I encourage everyone to go for it and make their own scanned materials. Let’s scan the world!
Shooting
In the case of capturing on a mobile device, it’s important to choose a photo app that can save uncompressed photos and provides manual exposure controls. In our case, we used a Samsung Galaxy S6 and the FV-5 camera app.
On a photogrammetry shoot, the goal is to capture a picture as sharp as possible and without any noise. The lens’s aperture on a smartphone is often locked or not easily adjustable, so we need to find an exposure setting that will suit our needs. The exposure needs to be good enough to keep the ISO as low as possible (close to 100) and the shutter speed high enough to avoid motion blur. Also, you need to set the White Balance on manual to avoid color shift between images.
In terms of light quality for the photogrammetry shoot, it’s best to wait for the lighting to be diffused such as on a cloudy day. Alternatively, you can use a diffuser disc such as the Photoflex LiteDisc. The diffused light will minimize harsh shadows, which will produce a better color map. In our case, we took advantage of the overcast sky for the shoot.
For a tree, we need to take a series of photographs around the trunk. We move around the trunk 360 degrees while taking a new photo at equidistant intervals.
Don’t forget to keep enough overlap between each photo as it will help the algorithms to provide a better result.
Intensive shooting can quickly drain the battery, so do not hesitate to pack an external battery for backup 😉
Photogrammetry
Back at the office, we will continue our work using a specific software for photogrammetry.
There are different software on the market, but for our example, we have chosen PhotoScan Standard from Agisoft.
The typical process for reconstructing the data is as follows: import the images, working with the Point Cloud data, and finally the Mesh reconstruction. In this example, we will be focusing only on one part of the tree. At the end of the process, we’ll have a High Poly Mesh, around 15M polygons, and a Color Texture in 8K format. As you can see, data captured using a mobile phone can still provide good results for photogrammetry.
Post-process
The High Poly Mesh is around 15M polygons and to fit into a standard workflow, we have to transfer the High Poly detail to a Low Poly Mesh. Here you can use the integrated bakers in Substance Designer or Substance Painter. You can also use your favorite 3D package for the Low Poly Mesh creation process.
In our case, we used Blender, for the Low Poly Mesh and the Normal / Color Baking. You can find a very nice tutorial from Darrin Lile here.
If you would like to work with a Height map, you can use Substance Designer to convert your Normal Texture into a Height Texture. In Substance Designer, you simply connect your Normal Texture into the Normal to Height node to produce the height data. Finally, go to Menu Bar, Tools/Preferences…/General/Cooker/Cooking size limit, type 8192, then set the Output Size to 13 to produce an 8K texture. And voilà, you have your Height Texture.
Now, we have our Low Poly Mesh, which can be used in any 3D application, but we have some holes and artifacts.
In this case, it’s possible to fix these issues using Substance Painter.
Substance Painter can import and export 8K maps. It’s possible to import our Color Texture in 8K, work on it in 4K and export the result in 8K as Substance Painter doesn’t upscale the textures. Substance Painter will actually recompute the texture to a lossless 8K size.
After importing the Mesh, Color, Normal and Height map, we simply add a Fill Layer with all textures. You can then add any additional channels that may be needed and set the resolution to 4K.
Next, we can use the Clone Tool to fix issues. By working on a new layer, you separate the clone data from the base photogrammetry.
Just, don’t forget to switch the blending mode for all channels to PassThrough. By using the PassThrough blending mode, all channels below the Clone Layer are copied.
And like others Brushes in Substance Painter, the Clone Tool works across all channels in a single stroke, you can clone the Color, as well as the Height and/or the Normal at the same time. It’s a huge time saver.
To remove as much of the lighting visible in the Color Texture as possible, we can use a filter we created in Substance Designer. This filter is a Work In Progress and will be available in a future release. (Click here to download it).
This filter can remove the lighting based on a Color Reference and the Ambient Occlusion (computed with the High Poly Mesh). The end result is a Color Texture that is more versatile and neutral, and can be used in different lighting conditions.
Finishing
Now, everything is fixed. We can start working on an augmented material, aka adding manual or procedural details to the scan.
Let’s start by baking all the additional maps, like World Space Normal, Ambient Occlusion, Curvature, Position, and Thickness. Substance Painter can bake this 15M polygon Mesh very quickly.
For example, we can add some moss on our tree. We found on Substance Share a Moss material, and imported it into Substance Painter.
We are off to a good start, simply by adding this material to a Fill Layer and using the Moss Smart Mask. If we play with the Mask Builder Parameters and the World Space Normal, it’s easy to put more moss on one side, like if it was on the North Side of our tree. We can add more details by adding another Smart Mask and finally use the “Organic Spread” Particle Brush to add some final touches.
For another example, we can burn our tree.
Just added a new Fill Layer with a smoke material, add a Mask and use the “Burn” Particle Brush. If you choose a large Brush Size, you can get a nice result.
Finally, you can export all Textures and Additional Maps in 8K and render the results in your favorite renderer. As you can see in this example, the 8K export from Substance Painter is lossless and is the same quality as the Color Texture from the Photogrammetric Software.
We’re now entering an era of scanning, and nothing should prevent everyone to be able to produce and customize their own scans. So I invite you all to go outside and just take pictures of materials you like. The knowledge and tools are here today to help you turn them into production-ready textures.
In the coming weeks, we will also publish a second part to this tutorial covering how to convert a scan like this into a seamless PBR material Substance Painter and Substance Designer.
Anyways, here you go, I hope you liked it! This blog is a first, little step for Allegorithmic and we will soon share more info and announcements. Stay tuned 🙂
