Hey, welcome to my first tutorial (hopefully not the last). I'm really happy with the creation of this new forum, finally I have a place that combines the 3 loves (basketball, photoshop, gaming) for me to call home. I'm really optimistic about learning with and teaching you guys.
I'm separating this tutorial into 2 parts, theory and practical. The theory side contains the explanations of the various concepts involved in what I consider to be professional masking (and hopefully isn't too long winded). I've tried to organise it in such a way that if you do have a particular question that you're able to find it easily without getting too bogged down in stuff you may already know. The practical aspect of the tutorial is where I'll give a detailed example and run through of the theory that I've covered. If you're familiar with the theory already or you feel happier to learn by example feel free to skip ahead to the practical section. I'm providing the theory part just in case you wanted to have a little more in depth understanding of the principles at work in these moderately advanced techniques.
If you have any questions about masking or you want to request another tutorial, feel free to private message me or post in this thread. I hope you pick up something that you can use. Enjoy!
::THEORY::
What is masking?
masking is the process of designating transparent (or semi-transparent) sections of bitmap layers. Masking is commonly used to make selections and remove unwanted backgrounds. There are 2 forms of masking (vector and alpha) which I consider to have the capacity to deliver professional results without fail.
What is professional masking?
From what I've gathered in bits and pieces it seems that the majority of people using PS here are using quick masking, the polygonal lasso, or the magnetic lasso. Those 3 tools all have their place in on the run, simplistic to intermediate detail selections. Where they fall short of meeting professional standards is in the ability to edit and manage those selections after the fact. Most of the time you'll be lucky (or smart) and have the transparency preserved by the layer itself and have the original background still intact in a hidden layer, but if you wanted to then change the boundries of that selection you'd have to make selections from the original, copy them, and merge them with the desired layer. Alpha and Vector masking allow full editing of the boundries of selection at any stage and that's where the whole process of selection becomes professional. Gone are the days of holding alt and shift to add and subtract from a selection. This is getting exactly what you want every time with the movement of a vertex or the stroke of a brush. The results are profound.
What are vector images?
Vector images are graphics that are created and edited by the manipulation of points, which when connected to each other form lines or areas of color. Vector points contain mathmatical information relating to the position, direction, and curve of the lines created when points are joined together. As such vector graphics are (virtually) infinitely scalable and are always expressed (in rasterized form) in the highest detail permitted by the display conditions. I could use the same vector graphic to print an area of a coin as I could the area of a bill board.
What are raster images?
Raster images are essentially the opposite of vector images. The graphical information is stored in pixels, where each pixel contains a set color and those pixels when displayed together adjacently form raster images that usually have inherent meaning (whether it's your signature or a picture of boobies). All images you see on the web (other than flash animations) are rasterized images. By their very nature raster images only contain a set amount of information, meaning that if they're scaled up they only contain the amount of detail expressed in the smallest rendering of that image. Likewise graphical information is lost when you scale raster images down.
What is vertor masking?
Vector masking is the creation of transparent areas of a layer using the pen tool to establish the boundries of those transparent regions. Vectors, as opposed rasterized images, are composed of free moving points (known as vertices or points) that contain information relating to both position and direction. This information, unlike it's rasterized brethren, can be scaled without loss of information and is always rendered in the highest possible detail as determined by the display area and screen resolution (hence doesn't experience pixelation). Basically what this means is that you can create a smooth vector boundry at high levels of magnification and photoshop will automatically anti-alias the smoothest selection given the pixel information contained in the raster image.
Where this all comes into it's own as a professional means of selection is that the designated transparent areas of the vector mask can be rolled back and displayed at any time by moving the vertices and altering their directional handles. In all my years with photoshop (since '98

Pros/Cons of Vector Masking
+ produces the best solid edge smooth curve masking possible bar none
+ easy to edit and maintain
+ excellent for selecting large areas and groups of areas at a time
- not good for creating transparent maps of highly detailed areas (such as hair)
- not good for selecting areas with varying or graded levels of opacity/transparency
- takes a bit of getting used to (but that passes quickly.. you'll be using the pen tool like a pro soon enough)
What is alpha masking?
Alpha masking is a technique that allows you designate transparent areas of a layer by filling in the areas you wish to become transparent with any of the tools that is conventually used to create solid graphics (brush, pen, paint bucket, etc). Probably the best way to describe it is that it's like drawing in reverse where you start with an image and paint away the parts you don't want. It's virtually the same as using the eraser, but with the notable exception that you're never deleting any of the graphical information contained within the unmasked image. By using the eraser whilst alpha masking you actually undo the "erasing" that you've already done. It's very similar in principle and practice to quick masking, except that you can see the results in real time (which is a lot of fun btw), and that in the sense that it's not a conventional selection it doesn't just dissapear when you start doing something else.
Pros/Cons of Alpha Masking
+ excellent for quickly adding masks for sections with fine detail
+ more free form and generally less tedious to use than vector masking
+ the only real way to achieve variation in opacity/transparency in masking (outside of quick masking, but alpha masking is basically quick masking with added managability)
- is tedious when using for large selections (as you either have to paint or fill all sections)
- much harder to produce the same solid boundry smooth line results as vector masking
When should I use vector/alpha masking?
By themselves vector and alpha masking are tremendously powerful, fun, and open up a world of possibilities in and of themselves, but it's when you team these 2 together that flood gates really burst open. Both techniques have their inherent strengths and weaknesses, but luckily for us those strengths and weaknesses line up so that when used together, vector and alpha masking complement each other perfectly.
For anything with solid lines, curves, and that require a crisp result, you should immediately think vector masking. Just thinking of the effort I used to expend aiming for some small semblance of the results made possible by vector masking is enough to make me cry.
For anything with fuzzy edges, varying levels of transparency, smudging or blurring, immediately think alpha masking.
The best part of it all is how easy it is to use these two techniques in conjunction to mask the same layer, as you'll quickly find out in the practical section of this tutorial. Thus concludes the theory component of this tutorial (for now).
::PRACTICAL::
Okay.. Beginning the Practical component at 12:30 PM (GMT + 10). I imagine this is going to take the vast majority of time on the documentation side of things but rest assured that however long I've taken overall that only 10 to 20 mins has been used on performing the vector and alpha masking, which should probably tell you just how slow a writer I am. I'm going to update as I go, so if nething goes majorly wrong I lose very little. Here we go.
1. Getting Started
I always like to start with a bit of house keeping, making things organised from the get go even though we'll probably only be using the 2 layers. Out of habit I whenever I'm editing a layer I always keep the original in a seperate layer, for our purposes you'll find that having the duplicate layer will come in handy should things get fiddly as they can when you have subtle variation between foreground and background. For the duration of this practical component I'm going to write anything I do in photoshop (no matter how slight, unless repeated) in colored text here. So to kick things off.

a. Right click on Stevey directly above and select copy
b. Switch to PhotoShop and press CTRL + N to create a new document
The dimensions of the image on the clipboard will always be adopted as the dimensions for the new document in photoshop.
c. Name the document "nash_masked", make sure that the Background Contents option is set to "Transparent" and click ok
d. Press CTRL + V twice to paste 2 layers of the nash image onto the canvas
e. Select the layers tab (by pressing F7) and rename Layer 1 to "original" and Layer 2 to "masked"
2. The Pen Tool
In step 2 we'll be creating a vector outline of nash using the pen tool. I'll do my best explain all of the tools relating to vectors in PS and go through their uses, but if you're able to do things right in the first go round you'll overwhelmingly just be using the pen tool and the direct selection tool (for minor adjustments). Like I say though I'll try and work the others in there just for the sake of completeness of this tutorial. It just occured to me that up until now I haven't mentioned that in PS vector drawings are known as paths. Paths simply refer to any vector information currently in existence on the canvas. There is a paths tab that keeps track of all the paths (

a. Select the masked layer from the layers window, then select the pen tool from the tool box (or by pressing P)
Even though paths exist independently of layers, I selected the masked layer purely to maintain a sense of mental organisation.
b. Open the Paths window by selecting it from the Window menu
If you don't already have the paths window mounted I suggest you do it at this stage, once again for the sake of organisation and good form.
c. In the paths window create a new path by clicking the little universal new icon in the bottom right corner of the window next to the trash icon and name the new path "nash mask"
When you begin using the pen tool it will automatically add any paths you make to "Work Path", but once you use that for selection or delete it work path is emptied and you lose that path forever. When you use the path to create a vector mask the path information will be stored in a vector mask path so you can get away with never using the features in paths window if you're confident. I'm trying to encourage good technique here, and I can tell you from experience that you can get burned being lax on these fine details that really take only a matter of seconds.
d. With the pen tool selected plot the first point (or vertex) at the base of Nash's right shoe (our left). I'm doing this at 400% magnification (zoom in and out by holding ALT + MOUSEWHEEL UP/DOWN or CTRL + -/+)
fig 2d

As far as I know there's no right or wrong place to start just so long as you start at the midpoint of a curve or a corner. If you start in the middle of a straight line then you've basically created an unnecessary vertex. Not a big deal by any means, but worthy of mentioning none the less.
Time Out: T1. Adding curve to paths using Directional Handlebars
Before we plot the next point in our path I'd like to explain how we create curved lines with paths. If you're familiar with the pen tool and all of it's subtleties, then feel free to skip ahead. Bear in mind, however, that the pen tool is moderately advanced and I certaintly didn't grasp all of it's intricacies immediately, so even if you have experience with the pen tool you may still learn something that could significantly improve the efficiency with which you use it.
There are 4 basic types of line: straight lines, symmetrically curved lines, asymmetrically curved lines, and hybrid lines.
The basic underlying principle behind the creation of all types of lines is if you create a point by clicking once you define no incoming or outgoing curve. However, if you click and drag then release the left mouse button that point will then effect the incoming and outgoing curve of the lines it creates by joining with other points.
fig t1_a

straight lines
straight lines are achieved by creating one point (clicking the pen tool once without dragging to define a point with no incoming or outgoing curve) and then creating another point by clicking the pen tool once again (elsewhere). The resulting line created between the two point will be straight.
fig t1_b (animated)

A straight line will always be created between 2 points that don't define incoming or outgoing curves, but that doesn't mean that the adjacent connected lines won't be curved.
fig t1_c
