Maskless lithography by direct writing (DW) is expensive because it is very slow, but then, there are no photomask costs. Assuming identical capital investment ($10 million) and running costs ($1 million dollar per year) for both optical and direct write lithography systems and 100 WPH (Wafers per hour) for optical lithography and 1 WPH for DW on 300 mm wafers,

What would be the cumulative number of wafers where DW becomes competitive with optical lithography if the mask set is assumed to cost $500,000?

This is often called break-even analysis. A common example is when do you make enough parts in volume to justify injection molding (very high initial cost to make mold; low per unit cost) vs. machining a part (low initial investment; low throughput/high unit cost). I would assume both systems have a 5 year operating life, before they are obsolete, for purposes of calculating depreciation of your capital investment. You may have to do some thinking and make some assumptions about how to represent these costs. I dont think you need to assume the systems are captive to a particular task. Each system could be used for other jobs, and never needs to be idle. In other words, assign a usage cost ($/minute) for each system based on its annual operating cost and depreciation. Obviously, the DW wafer will spend more time at the DW system, but the DW does not incur mask costs. The text states that masks are used for about 5000 wafers. (Masks do wear out!).

cost per wafer=depreciation+operating cost+mask cost (photolith only)

I realize we have to make some assumptions and educated guesses here to arrive at an answer, but anyway, make some kind of reasonable analysis based on the above and info in the text.

As an aside, you may be shocked to learn mask sets would sell for $500,000. This is actually an underestimate of the value of a mask, because it is merely for making the masks; it does not include all the thousands of hours of design (much of it now automated by computer) and proprietary knowledge thats contained in a mask set. In the 1970s, some technicians would surreptitiously copy masks (put a PR-coated blank mask in the mask aligner and copy the existing mask), and then put them in their pockets when they left the clean room. They would sell these masks on the black market to other companies for tens of thousands of dollars. The FBI would try to infiltrate and break up these mask theft rings. Now companies routinely reverse engineer their competitors chips and inspect the circuits with electron microscopes to see if any of their masks (or CAD files) have been stolen by others. Because masks did not fall under the traditional classifications of copyrights and patents, Congress created a special section of copyright/patent law to protect intellectual property of lithography masks. See http://en.wikipedia.org/wiki/Integrated_circuit_layout_design_protection.