Features

Theophano Mitsa ’91 (PhD) walked into her advisor’s office one morning in the early 1990s. Mitsa was there to talk about finishing her doctoral dissertation, a highly technical project on something called a “blue noise mask,” whose aim was to improve images in printers, fax machines, and computer screens.

“He said, ‘You know what, Theo? You have to start thinking about philanthropy,’ ” Mitsa recalls. “I said to myself, ‘What is [he] talking about? I’m hardly scraping by. I need philanthropy.’ ”

Sitting across from Mitsa that day was Kevin Parker, a junior professor of electrical and computer engineering at Rochester who had long been intrigued by the intersection of computer technology, medical research, and imaging.

“I guess he had an inkling this was going to be big from the beginning,” Mitsa says. “I never thought that I had anything big. Never in my wildest dreams.”

Judging from that day’s evidence, Parker, who would go on to become dean of the engineering school for several years, was also adept at fortune telling.

In the midst of Mitsa’s reams of equations and lines of computer code, Parker saw what would turn out to be one of the most lucrative patents in the University’s history. The blue noise mask—a deceptively elegant technological process that transformed how electronic devices in the early days of the digital revolution rendered images—turned out to be the kind of paradigm-changing discovery that academics live for.

Since that day in the early 1990s, the invention of the blue noise mask has generated some $30 million in royalties for the University, one of the most lucrative in Rochester’s history. It’s surpassed only by a Medical Center team’s work that led to vaccines against human papillomavirus (HPV), now marketed as Gardasil by pharmaceutical giant Merck and as Cervarix by GlaxoSmithKline, and by work by Porter Anderson, professor emeritus of pediatrics, in the lab of David Smith ’58, former chair of the Department of Pediatrics, that led to a vaccine against pneumococcal meningitis, sold as Prevnar by Wyeth (Pfizer).

And while the blue noise mask was invisible to consumers and became a standard part of printer drivers and other software, the story of the invention played out as an often grueling, up-close education in patent law involving some of the biggest players in the computer industry of the 1990s and 2000s.

But as the patents for the technology expire within the next year and as the power of personal computer technology has advanced beyond the often bandwidth-deprived days of the 1990s, the story has shifted to one of legacy for the University and its inventors.

The dollars have allowed the institution to invest in new faculty, programs, and buildings, kick-starting what Parker describes as a “virtuous cycle” in which invention begets funding which—once reinvested in teaching and research—begets more invention. That, in turn, brings in more royalties, to the benefit of the institution, its faculty, and students.

“If you can generate good tech transfer outcomes, that generates new things back at the university,” Parker says. “That allows the teaching and research mission to get bigger and better, and generates another round of things. That’s the ultimate goal.”

Rob Clark, the dean of the Hajim School of Engineering & Applied Sciences and senior vice president for research, says that, in addition to the financial benefits that have followed from the invention, the blue noise mask illustrates that inventions—even ones narrowly focused to solve a specific problem—can have broad impact. Its success, he says, sends an encouraging message to faculty about the benefit of working to translate basic discoveries beyond the lab.

A Quirk in Human Vision

In many ways, the blue noise mask was the perfect invention at the perfect time. By the early 1990s, desktop computers had a firm hold in homes and offices. The dot matrix printers that accompanied them in earlier years could only print in black and white. But those printers were giving way to ink-jet and laser printers, able to render shades of gray in black-and-white images. They could also create a wide range of colors from just three basic colors using a process called halftoning.

Those early devices, however, were slow, consumed gobs of computing power, and turned out images that awed no one. Among the unimpressed was Parker, a medical imaging expert.

For him, the print quality problems went beyond mere aesthetics. Parker saw CT scans and ultrasound images so poor that there was the chance something important could be missed.

“The medical images that we’d put out were awful, really awful, because of halftoning,” Parker says. “So I was very interested in that aspect of it. Everyone had an ink-jet or laser printer by then and yet the renditions were terrible.”

Mitsa arrived on Rochester’s campus as a Fulbright fellow from Greece in fall 1986. Interested in imaging, she nonetheless agreed to do her early graduate work with Edward Titlebaum, a professor of electrical and computer engineering who was working on computer coding related to the Star Wars defense program, Mitsa says.

After receiving her master’s degree a year later, Mitsa looked at her situation from a practical standpoint. If she received her doctorate based on defense work, her most likely employer would be the U.S. government or a defense contractor. As a foreign citizen, she realized, she was unlikely to get hired for such sensitive work.

So Mitsa talked with Parker, an affable, thoughtful professor and a careful researcher, who was full of ideas. One colleague described him as “the kind of guy you hope to find at a university.”

Parker was interested in the problems of quality and speed plaguing halftone images, and set Mitsa to the task of improving them.

“All of the halftone techniques of the time left a pattern, what we would call an artifact, a mistake, a characteristic flaw,” Parker says. “You’re adding a weird pattern onto what should be a natural medical image, whether a CT scan or an ultrasound. If the artifact goes right through a small tumor, you may never see that it’s a small tumor.”

The process of halftoning creates images by using a limited palette to create an array of tiny dots whose composition and density render different shades and colors. In black-and-white images, for example, the printer prints only black dots, but creates shades of gray by altering the location and density of the dots. Fewer dots and more white space results in lighter shades, allowing the creation of complex images.

Similarly, color images can be made using a halftone process that changes the dot composition and density of just three basic colors: cyan, yellow, and magenta.

Parker learned that the quality problems might be solved by adding “blue noise” to the image, a technique that would shift otherwise distracting errors in the printing process to higher wavelengths where, because of a quirk of the human eye, they would be less visible. But the existing blue noise algorithms were slow and cumbersome to use for the computers of that time, and were not free of unwanted patterns. Parker suspected that a fixed array of numbers could be constructed in advance that would contain all the required blue noise patterns. The result, in theory at least, would be images that were more pleasing, more accurate and, potentially, could be rendered more quickly in a “paint-by-numbers” scheme that any computer could rapidly use.

“Blue noise is high-frequency noise,” Mitsa says. “You want blue noise in halftoning because it takes all the noise in the halftoning process and moves it to the high frequencies where your eye cannot see it. So your eye gets the big picture, the shapes . . . it’s not disturbed by the distribution of dots in high frequencies.”

In fall 1987, Mitsa set to work, reading everything she could get her hands on about halftoning and then experimenting, making run after run on Rochester’s mainframe—desktop computers of the time had nowhere near the required computing power.

“Generating a good blue noise mask is a really, really tough problem that requires lots of computer time,” Parker says. “Once you make a good blue noise mask, once you crack that nut, you have it. It’s a set of numbers for the computer . . . it can paint-by-numbers your image really quickly and better than other halftone techniques.”

Mitsa input data in the morning, letting programs run all afternoon while she went over the previous day’s results, and then headed back to the computer lab in the evening to collect each day’s output.

“There was just tons of experimentation, tons of it,” she says. “We knew we wanted blue noise and, at some point, we tried a lot of different things with blue noise.”

The breakthrough happened one day in late 1988, when Parker walked into Mitsa’s office brimming with excitement for an idea that he had the night before.

“Kevin walked in one morning, very excited, and said, ‘I had an idea last night,’ ” Mitsa says. “He started drawing on the board in my office, and that pretty much was the birth of the blue noise mask because it captured the main idea.”

Neither was sure the idea would work, but eventually, they developed a set of instructions contained in a 256-by-256 grid of numbers that would direct printer drivers and other software how to handle blue noise.

Because the mask was a fixed set of numbers, the computer didn’t have to constantly recalculate as it created each image. That allowed images to be produced between 5 and 45 times faster than existing techniques. And, at a time when phone companies levied charges on long-distance calls and the office fax machine was an important way to send images and documents, the blue noise mask could cut long-distance fax costs by 75 percent, according to one estimate.

“If I’m to describe the blue noise mask, I’d use two adjectives: it’s beautiful and it’s powerful,” Mitsa says. “It’s beautiful because it’s simple, and it’s powerful because, although it’s fixed, although you don’t have to tweak anything in the mask for any image in the world, the blue noise mask . . . guarantees you’ll get a very high-quality image.”

After developing the first mask, the two continued to tweak it until they had it right. Mitsa also worked on the theory behind the mask, publishing it as part of her dissertation in 1991.

“To put it in a nutshell, my PhD is actually an array of numbers, which is kind of unique and rare,” Mitsa says. “It is a theory, but it’s also a physical thing, an array of numbers, a fixed array.” In 1991, she and Parker published their first article outlining the work, “Digital Halftoning Using a Blue Noise Mask,” in SPIE Electronic Imaging Conference (Vol. 1452). In 1992, the U.S. Patent Office issued patent number 5111310, “Method and Apparatus for Halftone Rendering of a Gray Scale Image Using a Blue Noise Mask,” for the invention.

To Market . . . and to Court

Though Mitsa’s main focus was earning her PhD, Parker understood that the advance had potentially broad commercial applications and began meeting with Research Corporation Technologies (RCT), an outside consultant that, in the early 1990s, many universities used to bring discoveries to market.

RCT Chairman and Chief Executive Officer Gary Munsinger says the advance was so technical that, at first, RCT officials didn’t understand the value it might have. After the inventors explained it to them, they set to work finding licensees, a process that took a couple of years to bear fruit.

“It’s only real when you start getting customers . . . happy customers, repeat customers,” Parker says. “Then you really know, OK, this is actually working.”

As encouraging as getting those initial customers was, it became apparent that, instead of paying licensing fees, some players in the industry went ahead and developed their own versions. RCT filed lawsuits against Epson, Lexmark, Hewlett Packard, and Microsoft, two of which went to trial and all of which were eventually settled in RCT’s—and the University’s—favor.

But before settlement, Parker, Mitsa, and RCT’s legal team had to go through years of court action.

“We had a saying, that decades come and decades go, because things just took forever,” says Timothy Reckart, then an attorney for RCT and now a partner at Arizona-based Rusing, Lopez & Lizardi.

Though they’re pleased at the ultimate result of the suits, Reckart says the results weren’t easily achieved. Part of the challenge, he says, was translating the technology into terms judges could understand and finding experts who could testify.

While the attorneys were in their element, both Parker and Mitsa say the experience was grueling. Parker described their opponents’ legal strategy as first denying any violation took place. Losing that, they would argue that the technology they used was different in important ways. Then, when RCT prevailed on those points, they’d argue that the patent was invalid, sometimes attacking the inventors’ integrity to do so.

“It was just awful, quite frankly,” Parker says. “The years of the infringement legal issues were not only very costly and time consuming to all parties involved, but also they were very distressing, very stressful.”

But in the end, also very successful. HP settled with RCT in 1999, taking out a license to use the mask. Epson and Lexmark followed in 2001, the same year RCT filed suit against Microsoft. That case went to trial, with Microsoft prevailing in federal district court in 2005, but RCT winning on appeal in 2008 and 2010.

Whether licensed initially or only after lengthy court battles, the blue noise mask spread widely. Bobby Hunt, a retired University of Arizona engineering professor and imaging expert, says its impact has been extensive. It was in printer drivers and other software, and was important in the operation of the Internet because it cut the amount of data that had to flow over early, slow connections.

“The impact was immediate in the printer business,” Hunt says. “I think virtually all computer printers use blue noise masks or something derivative from that. But there was an impact in the software business, as can be seen in the fact that Microsoft was motivated to embed blue noise masks in the generic printer driver software used for the cross-platform support of printing in all the different printers that may use Windows.

“There was also impact in the Internet. To make color images requires the transmission of color bytes over a channel. Reducing bits in transmission is important to minimizing user irritation in waiting for an image to load, as well as making more bandwidth available for data shared through the bottleneck of Internet pipes. This is particularly true for images on a cellphone, given the huge growth of smartphones that invite everyone to consume more data.”

Kicking Off the ‘Virtuous Cycle’

Blue noise mask licensing ultimately generated millions of dollars in royalties from licensees. After legal fees, the remainder

was split between the University and RCT, with Rochester’s share further split with the inventors.

Scott Catlin, associate vice president of technology ventures, says the blue noise mask was unusually successful for a faculty invention. The University gets more than 100 disclosures of discoveries from faculty each year, only a few of which ever get to market. He describes the blue noise mask as a once-in-a-decade—or more—discovery.

In an effort to encourage the next blue noise mask, the University is beefing up its technology transfer office, and has created an $800,000 fund to take discoveries through the initial steps of product development, something companies are increasingly reluctant to do, Catlin says.

“They [the companies] don’t want to start from ground zero, they want you to prove that it, at some level, will be a useful product,” Catlin says. “Blue noise mask is hugely valuable and important. It has produced not just money but also recognition for the University. We’d love to have more, and that’s certainly the plan: to maximize opportunities.”

Parker is doing his part, working to ensure the proceeds boost his envisioned “virtuous cycle.” Currently the William F. May Professor in Engineering at Rochester, Parker says he became both department chair and dean in part to ensure the proceeds from the blue noise mask were used in ways that would have a positive and enduring impact on research and teaching there.

Blue noise mask revenues endowed five distinguished professorships—three in electrical and computer engineering and two in biomedical engineering. Richard Waugh, chair of the Department of Biomedical Engineering, says the money has also allowed the department to hire several junior faculty members, almost doubling the department’s faculty to 15. During the same period,

biomedical engineering, with more than 300 undergraduate majors, has grown to be the largest undergraduate department in the Hajim School.

“It came at a critical time,” Waugh says. “It enabled us to reach a critical mass of faculty and grow into the department we are now.”

Mark Bocko, Distinguished Professor of Electrical and Computer Engineering, says the blue noise mask funds allowed the department to expand in similar ways, from 15 to 20 faculty members, and also endowed a research initiation fund of about $50,000 per year. That money can be used to help get a project off the ground and running, to the point where it can attract government or foundation funding.

The funds were also used to match a faculty development grant from the Whitaker Foundation, which also awarded Rochester a special grant to help build the Robert B. Goergen Hall for Biomedical Engineering and Optics. The building, which is home to biomedical engineering, was named in recognition of University Trustee Robert Goergen ’60, who committed $10 million for its construction.

Parker himself continued to work on the blue noise mask, guiding graduate students through aspects of research. The license funding also supported work in medical imaging based on biomarkers, which created a spin-off company, Rochester-based VirtualScopics Inc.

While Parker got his virtuous cycle moving, Mitsa just got moving. She took an assistant professorship at the University of Iowa, where she shifted her focus from halftoning to data analysis in medical imaging.

That led to a stint at GE Medical Imaging in Milwaukee and then to Boston, where she was on the faculty of the University of Massachusetts at Dartmouth for a time and a consultant at Harvard Medical School. Along the way, she became interested in a type of data analysis called temporal data mining and wrote the only book in the field. Today, she consults, blogs, and is at work on a second book from her home in Massachusetts.

As the blue noise mask patents expire, the peak of halftoning research has passed, Parker says. While some work continues, the enormous processing power and high resolution of current printers make the speed and quality gains from the blue noise mask less important.

“In one sense, it’s like a chess game that lasted 20 years, a very complicated chess game,” Parker says. “I couldn’t be sure what the outcome would be, but I knew if we did everything right, we would be able to make very significant advances. Fortunately, we didn’t spill the pieces on the way.”

---

Alvin Powell is a freelance writer based in Cambridge, Mass.