This tutorial will demonstrate how to extract a clean mask from a photo of a tree set against a blue sky, using channels and levels, and then export the final result as a transparent .png. The general concepts may be intuitively applied to other applications.

First, download the tree photo, and open it in Adobe Photoshop.

Balancing the Channels

Open the Channels palette to inspect the various color channels.

Color channels tell how much of each color is in the image. An RGB image like this photo will contain 3 separate color channels, one indicating the amount of Red, one for Green and one for Blue. White in a channel means that the color value is full (255), and black is empty (0).

Look for the channel with the highest contrast edge outlining the tree. The tree should be a flat color of very low contrast within the edge, and the backdrop should be white or quite light, also with low contrast. This way it will be easy to define the tree with an Alpha channel.

Because the sky is blue, the Blue channel will work well to start the mask.

Press Ctrl+3 to inspect the Blue channel.

On closer inspection, there are parts of the trunk which are considerably lighter than other parts of the tree. They are almost as light as parts of the sky. Only the sky should be white, otherwise parts of the trunk will appear transparent once the mask is applied.

Since the color of the bark is generally more Red than the rest of the scene, use the Red channel to push back the mask.

a) Make a copy of the Blue channel by dragging it onto the New Channel ( ) button.

Double click on the new channel and rename it Tree Alpha.

b) Then make a copy of the Red channel, and make sure that it is selected. (Ctrl+5)

c) Adjust levels on the Red copy channel:

Image > Adjustments > Levels... (Ctrl+L)

Input Levels: 50 / 0.85 / 222

Now the idea here is to isolate the bark-regions which are too bright in the Blue channel. Then this channel will be used as a selection to darken the Tree Alpha channel.

d) Drag the Red copy channel onto the Load channel as selection ( ) button.

e) Click on the Tree Alpha channel, and then press Ctrl+H to hide the selection so we can see what's happening.

f) Now significantly darken the bark region.

Image > Adjustments > Levels... (Ctrl+L)

Input Levels: 120 / 1.2 / 255

With this method we use the Blue channel to create transparency, and the Red channel to make it more opaque.

When compositing the tree into another scene, the sky will want to be totally transparent, and the tree solid. So bump the shadows to black, and the sky to pure white. To do this:

g) Press Ctrl+D to clear the previous selection.

h) Adjust levels on the Tree Alpha channel:

Image > Adjustments > Levels... (Ctrl+L)

Now slowly drag the white input level downwards until the whole sky turns white, and slowly drag the black input level upwards until most of the trunk and inner tree segments appear black.

These settings work well in this case:

Input Levels: 30 / 1 / 145

It may take a number of tries to find the perfect balance.

Be mindful of what is happening throughout the image as the levels are adjusted. If the white levels get too low, thin parts of the branches will disappear into the sky. And if the black levels get too high, the sky will glow through the edges of the tree.

i) Invert the Tree Alpha channel (Ctrl+I) so that the tree appears white, and the sky black.

Applying the Mask

a) Drag the Tree Alpha channel onto the Load channel as selection ( ) button.

b) Go to the Layers palette, and make a copy of the Background layer, by dragging it onto the New layer ( ) button.

Double click on the new layer Background copy, and rename it Tree.

c) Click Add layer mask ( ) on the Layers palette to add the selection as a mask to the Tree layer.

d) Hide or Delete the original Background layer.

e) Click the little black and white thumbnail on the Tree layer to paint on the layer mask.

Isolate the trunk from the background hillside.

Use the Brush tool to make the hillside transparent by painting black on the layer mask or

Use the Polygon Lasso tool to select chunks of the hillside, and then fill them with black. Make the foreground color black ( Ctrl+D ), and then to fill ( Alt+Backspace ) the selection with the foreground color.

Tweaking the Edges

a) Make a new layer ( ), drag it beneath the Tree layer, then paste a scene from the project it will composited into.

Or else fill the background with an average color of the scene, or make a gradient behind the tree layer, to test the mask over a range of colors.

If the tree is moving from a light sky to something relatively darker, there may be a noticeable outline surrounding it, as if it were glowing. There are a number of ways to correct this.

Adjust Mask Levels

a) Click the mask thumbnail on the Tree layer.

Image > Adjustments > Levels... (Ctrl+L)

Now bring the shadow levels way up to minimize the glowing edge, while watching the integrity of the smaller branches.

Input Levels: 120 / 1.2 / 255

A lot of the smaller branches will disappear with this technique. Although it can be effective with other images, do not use it in this case.

Darken the Edges

A better way to fix the glowing edges of this tree is to darken the edges just inside the mask.

a) Hold down Ctrl and click on the Tree layer mask thumbnail to select the mask's transparency.

b) Contract the selection by 1 pixel. Select > Modify > Contract... 1 pixel

c) Invert the selection. Ctrl+Shift+I

d) Create a Curves adjustment layer,

by clicking on the Layers palette, and selecting Curves...

Drag the lower part of the Curve down. Press OK.

e) Holding down Alt, move the mouse cursor between the Curves and Tree layers, until the cursor changes to two overlapping circles, and then click.

Doing this creates a clipping group, so that the Curves adjustment will only effect the Tree layer.

In a clipping group, the bottom layer acts as a mask for the upper layers.

Curving the edges so dark may create a noticeable black line around the hard edges of the tree. This may be softened by Gaussian Blurring the Curves layer mask by one or two pixels.

Though blurring the Tree layer mask may produce undesirably trippy results.

Exporting

The next thing to do is either copy and paste the tree into another Photoshop image, or export it as a transparent .png file so that it can be imported into other programs such as Flash or 3Ds max.

a) Hide the background layer, in this case named Scene, by clicking on the little eye icon to the left of the layer so that it disappears.

To export .png

b) Press C for the Crop tool, and select a box around the tree, leaving a bit of margin around the edges.

Double click in part of the selection, or press Enter to apply the crop.

c) Export the File with Save for Web.

File > Save for Web... ( Alt+Shft+Ctrl+S )

Select the Settings: PNG-24

Enable Transparency.

To copy and paste

b) Select > All ( Ctrl+A )

c) Edit > Copy Merged  ( Shft+Ctrl+C )

d) Switch to another Photoshop document

e) Edit > Paste ( Ctrl+V )