Snapfish Lab: An Open Online Imaging Community - IEEE Xplore

2 downloads 105 Views 309KB Size Report
performance imaging application hosting, rich user feedback collection and satisfactory end user experience. The key val
Snapfish Lab: An Open Online Imaging Community Peng Wu, Eamonn O’Brien-Strain, Jerry Liu Hewlett-Packard Laboratories, Hewlett-Packard Company 1501 Page Mill Road, Palo Alto, California, USA [email protected] [email protected] [email protected] Abstract— With the increasing amount of digital media being stored, shared and consumed on the Internet, more imaging applications are deployed online. However, the usability and use model of those applications are often less studied. In this paper, we present Snapfish Lab, an open online imaging community web site, to enable both imaging technologists and enthusiasts participate actively in exploring the usability and use model of the exploratory imaging applications. The main topic addressed in this paper is the challenges and solutions of building a production quality web site that supports open and high performance imaging application hosting, rich user feedback collection and satisfactory end user experience. The key value offered by Snapfish Lab is to bridge imaging research community and end users so that user’s demand becomes part of innovation cycle to drive web based imaging research and development.

I. INTRODUCTION The proliferation of digital capture devices and Internet media services prompt more and more digital image and video content being stored, shared and consumed on the Internet platform. Many imaging functionalities that are traditionally deployed as bundles of digital imaging devices (cameras, printers, scanners, etc.) and stand alone desktop applications are now available on the Internet. Mature image processing technologies, such as contrast/brightness adjustment, red-eye removal, image cropping and sharpening, can be found in many photo hosting and sharing web sites, including Snapfish [4], Flickr [11], and Shutterfly [3]. Adobe published the online version of PhotoShop in February 2007 [2]. Some ad-hoc imaging technologies recently studied in research community, such as image in-painting [13], interactive image segmentation [14], automatic image cropping based on saliency detection [16] are also made available online (see [1] for one example). With the increasing amount of effort from both imaging research community and industries dedicated to prompt online imaging technologies and applications, we consider the following two issues critical in delivering their values to users: • Usability: How to know whether an imaging technology performs well enough to be released to end users and to which user segment, a certain imaging technology or application should be offered? One unique characteristic of the Internet user is the diversity of user’s profile and each user segment often has its own demand on technologies and capability in comprehending technologies. In [15], photo takers are classified into 5

978-1-4244-2295-1/08/$25.00 © 2008 IEEE

17

categories, from snap shot takers to professional photographers. And each category possesses very different preference on technologies. • Use model: How to grow from point technologies into complete applications with proper design of interaction model and workflow? Internet server/client platform provides rich means to support web page layout and user interaction. An imaging application, built from these means and core imaging technologies, can appear to a user in many different ways with different workflow and interaction model, which eventually lead to different levels of user’s satisfaction. To deliver the value of imaging technologies to end users, the application needs to be deployed with a proper and pleasing use model. Historically, the above two issues are often addressed by running focus group studies and marketing research on a small selected group of users. However, leveraging the prevalence of Internet and the collective intelligence from imaging technologists and imaging enthusiasts, we believe these issues can be better addressed in a larger scope with the participation from research community and users world wide. In this paper, we describe our work in building an open online imaging community, named Snapfish Lab (http://www.snapfishlab.com), to address usability and use model of imaging technologies and applications. The rest of the paper is organized as follows: Section 2 provides an overview of Sanpfish Lab and some related work; Section 3 introduces the Snapfish Lab platform, the backbone that enables large scale deployment of imaging technology and application on the web; In Section 4, we illustrate the benefit and experience of running this online community and conclude this paper with a discussion on future work in Section 5. II. SNAPFISH LAB OVERVIEW As an online imaging community website, Snapfish Lab offers comprehensive capabilities to help imaging technologists and enthusiasts to engage with each other, as shown in Figure 1. For imaging technologists, the following capabilities are offered: • Transform imaging algorithms into online imaging applications that are available 24x7. • Enable imaging application linking and hosting on the Snapfish Lab website.

MMSP 2008

Open online application hosting and presentation

Rich means for user feedback: voting, comment and discussion

Free Snapfish account to allow easy and world wide user access

Figure 1 A snapshot of http://www.snapfishlab.com to illustrate the main features of the site



Track and report user feedback on success/failure cases of applications, dis/satisfaction on application workflow and comments in general. For imaging enthusiasts, Snapfish Lab provides the following values: • A production quality web service that offers the state-ofthe-art imaging applications. • Interaction with imaging applications using user’s personal content or public content. • Opportunities to express satisfaction and demand on imaging applications and participate in driving the technology and application’s advancement. Snapfish Lab online imaging community is essentially a Java-based production quality web service to allow imaging technologists to present their work on the Internet to end users. It also offers rich means to collect and report user feedback to technologists to discover user’s demand, advance technology, and perfect application. In this way, Snapfish Lab builds an imaging community to addresses the usability and use model issues of imaging technologies and applications. Presenting explorative imaging technologies on the Internet is a common practice by many research institutions. For example, [5] has been well known as the place to look for the state-of-the-art computer vision technologies. Many similar sites have algorithm source code, demo prototypes, publication references, etc., made available to the audience. However, Snapfish Lab differentiates itself from those sites in the following perspectives: • On Snapfish Lab, every technology appears as an online application that end user can test with his/her personal content at anytime from anywhere.

18



Snapfish Lab not only presents online imaging applications, also collects user feedback and ground truth data on technologies to help research community to advance technologies. There are other websites also presenting exploratory applications on the Internet, such as Google Lab [6], Yahoo! Research [7], Microsoft Live Labs [10] and Adobe Labs [8]. In the field of imaging, Kodak has published the site Sandbox [9] to showcase their imaging technologies and applications. Comparing with those sites, Snapfish Lab is mainly unique in that Snapfish Lab platform is designed so that it can accept a wide variety of imaging algorithms and applications submitted by any party and transform them into online applications. In another word, the openness of the Snapfish Lab imaging platform makes the site truly a community oriented environment in which both research community and end users can actively participate. In the next section, we describe the Snapfish Lab imaging platform in detail. III. SNAPFISH LAB IMAGING PLATFORM The core challenges of Snapfish Lab imaging platform are to fullfill the following requirements: • Openness in hosting imaging applications. Given the wide variety of imaging technologies and applications, we look for an architecture design that can be open to enable different applications with minimum engineering effort.

Poster Creator

Fisheye view

Clear Note

Common JNI-C Interface:

Auto Level

Note Capture

Snapfish Lab Application Logic (Java)

Global Tool

Album Tool

int foo(Image *inputs[ ], int ni, Image **outputs[ ], int *no);

Algorithm 2

Algorithm 1

Dynamic-link Library

Photo Tool

Snapfish Lab Application Logic (Java)

(a)

(b) Figure 2 (a) support open algorithm integration; (b) support open application logic development



One unique situation of integrating imaging core algorithms is that they are contributed by researchers and researchers have different ways to code the algorithm, even though they mostly use C/C++. On one side, it is not realistic to enforce researchers to change their coding style to fit into Snapfish Lab’s requirement; On the other hand, it is not scalable for Snapfish Lab to integrate individual algorithms one by one. This situation motivates us to come up a solution that can achieve a good compromise of both sides, as illustrated in Figure 2(a). Essentially, the solution is to define a common JNI-C function interface that needs to be implemented by every algorithm. Each individual algorithm, as long as it implements this interface, can be integrated through JNI calling. As seen in Figure 2(a), the defined function interface takes one or multiple image objects as inputs and produce zero, one or multiple image objects as output. For each image object, as depicted in more detail in Figure 3, contains two major parts, one is a pointer to the raw image data. The other is a metadata object. Each metadata object is composed of an array of objects of type OBJECT and an OBJECT struct contains an array of pixel points to describe the shape of the object and also a type ID to describe the nature of the OBJECT.

Performance in supporting large scale user access: As the web portal is exposed to world wide users, it is important to assure the high availability of the site and responsive interaction on the site. In the following, we describe how those challenges are addressed in the design and implementation of Snapfish Lab imaging platform. A. Platform openness Imaging technologies and applications can appear in many forms of functionalities, such as contrast enhancement, redeye removal, face detection, poster creation, and implementations, such as Matlab, C, C++, Java, .NET, Flash, etc. The variations are almost innumerable. The goal of Snapfish Lab platform is to maximize the possible applications that can be automated into online applications. To be specific, the platform is open to the following ways to submit applications to Snapfish Lab: • Imaging algorithm submission: any algorithm written in C/C++ and built to a shared library or executable can be turned into an online application, with the requirement of using a pre-defined function interface provided by Snapfish Lab. • Flash/ActiveX object submission: In this case, the submitted application is treated as black box and Snapfish Lab mainly offers the capabilities to embed and host the application. • External linking through iFrame: The approach provides the maximum freedom to the application developers, and Snapfish Lab is mainly to provide a link and place holder on Snapfish Lab web site. In the above, the last two approaches are common industrial practice to support pluggable infrastructure. For example, using iFrame is also adopted by Facebook [12] as part of its open API to support third part application plug-in. Our main effort lies on enabling the first approach, which is relevant most to imaging research community, in particular, academia.

Image

Metadata

OBJECT

imgData: char *

objs: OBJECT *

pts: PixPt *

metadata: Metadata

classID: int *

type: ObjType

Figure 3 Data structure to support algorithm integration

A broad range of imaging algorithms, such as image enhancement, image classification, image cropping, can fit their function definition into this interface. For example, a

19

drive the imaging innovation with a solid understanding of user’s interest and demand. For this purpose, Snapfish Lab web site enables several means to collect user feedback: 1) discussion forum and news feed: a user can express opinions on imaging applications on discussion forum that is linked to Snapfish Lab web portal. The discussion forum also sends new feed to subscribers to provide update on the site; 2) user behavior logging: a rich set of information can be logged for each application. Through the information logging and analysis, we can derive rich knowledge on the applications, such as which application is the most popular one, and for a certain application, what type of images are more likely to be experimented on.

face clustering algorithm takes multiple input images and produces new metadata including face locations using the OBJECT struct and associated object type ID. In this case, no output image should be created. The new metadata should be added to metadata of each input image. Once an algorithm is integrated into the platform, the platform will automatically supplement application logics, including image display, interaction model and feedback model to produce a complete online imaging application. This automation is made possible by consolidating various applications into categories so that we can properly model them in an object-oriented way. Through clustering applications to be hosted on Snapfish Lab, we define three abstract types of applications, “Photo Tool”, “Album Tool” and “Global Tool”, which correspond to applications that process and manipulate individual image object, a collection of image object in one album and all albums/images as a whole, as shown in Figure 2(b). This separation allows us to further specify behaviours that are associated with each type of application.

2) User experience improvement: There are many factors contributing to a satisfactory user experience. Other than the performance issue discussed in Section III, a significant amount of effort is also spent on the UI design and workflow design. On UI design, we worked with web designers to provide graphics for the web portal. The two pictures shown in Figure 4 illustrates the UI design before and after we adopted desgin graphics. For the workflow design, we work with HP Labs’ researchers specializing in user experience design to improve application’s workflow. One successful example is with the improved experience design and flash based implementation, the Poster Creator application offered on Snapfish Lab is widely perceived as a more attractive application than similar applications currently available on some commercial sites, even though they both utilize the similar imaging algorithm in the backend.

B. Platform performance Platform performance involves many factors, including the scaling of server load, distributing the computation process, pre-cashing and so on. For general server load scaling that operates on multiple Snapfish Lab web servers, we use software load balancing provided by Apache. In the context of improving performance on a single web server, we mainly perform pre-cashing of photo albums and multi-thread calling of imaging algorithms, as those two steps cause the bottleneck of fast response to user’s request. When a user has many albums and many pictures in the albums, finding the pictures and downloading them into the memory can halt other user’s access to the web site if there is only one thread running to handle the downloading in one application instance. Our solution is to concurrently downloading album/picture using multi-threading. A thread is invoked right after a user has provided correct user credential to the web portal to start downloading albums and pictures, before the user actually makes a request to retrieve any album or picture. In such way, we can greatly shorten the waiting time when a user actually requesting albums and pictures. Another cause of slow response to user’s request is when the request involves calling a certain imaging algorithm on a picture or a set of pictures. When multiple requests are made to call the same algorithm, one has to wait for another to finish calling the imaging algorithm, which will produce delay on the response. As it is not guaranteed that the imaging algorithm contributed by researchers can always be thread safe, we develop a solution that makes multiple copies of the dynamic-link library and use Semaphore mechanism to manage the availability of each copy of the libraries.

Figure 4 UI design with (right) and without design graphics

IV. RUNNING SNAPFISH LAB Snapfish Lab web site made its debut to public in March 2008. Currently, it hosts 7 imaging applications, ranging from algorithms operating on single image, such as auto cropping, to the visualization of an image album (fisheye view) and sophisticated poster creation application. Without serious online promotion, the site has attracted hundreds of unique visits daily (see Figure 5). Through the user feedback logging and tracking, some interesting observations are noticed. For example, the “Automatic Horizon Alignment” application is embraced by many amateur photographers. And the “Note Capture” application, which was targeted for business use, is found useful by home users in creative ways, such as recording restaurant menus and recipes.

C. Miscellaneous platform issues 1) Rich feedback collection: As mentioned in Section II, one major value Snapfish Lab offers to research community is to collect real user feedback on imaging applications and thus

20

[15] Infortrends, “Digital Imaging Lifestyles: A Multi-Client Study Digital Imaging in an Era of Excess,” May 2007. [16] Vidya Setlur, Tom Lechner, Marc Nienhaus, and Bruce Gooch, “Retargeting Images and Video for Preserving Information Saliency,” IEEE Computer Graphics and Applications, Issue Date: September 2007, pp. 80-88.

Figure 5 Statistics of daily unique visits to Snapfish Lab

V. CONCLUSION Snapfish Lab, an open online imaging community site, presents an exciting opportunity to bring imaging technologist and enthusiasts together to advance imaging technologies and applications. Empowered by the Snapfish Lab imaging platform, the site provides an open environment to let technologist present the state-of-the-art applications on the Internet and enable users to experience them and provide valuable feedback, which eventually improves the usability and use model of imaging technologies and applications. The current site mainly focuses on photo-centric applications. We are actively working on expanding the functionalities of the platform to support video and audio handling, so that Snapfish Lab can grow to a multimedia community site. ACKNOWLEDGMENT We thank the great contribution made by the engineering team led by Lu-Jin Zhang and Hua Zhang of HP Global Delivery China Center in developing and maintaining Snapfish Lab. We also express our gratitude to Jian Fan, Brian Atkins, Jun Xiao and Phil Cheatle for working with us to present their technologies on Snapfish Lab. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

http://fotoflexer.com/ https://www.photoshop.com/express/landing.html/ http://www.shutterfly.com/ http://www.snapfish.com/ Computer Vision Homepage, http://www.cs.cmu.edu/~cil/vision.html http://labs.google.com http://research.yahoo.com/ http://labs.adobe.com http://www.sandbox.kodak.com http://labs.live.com/ http://flickr.com/ http://www.facebook.com/ M. Bertalmío, G. Sapiro, V. Caselles and C. Ballester, “Image Inpainting,” Proc. of SIGGRAPH 2000, pp. 417-424, New Orleans, USA, July 2000. [14] A. Blake, C. Rother, M. Brown, P. Perez, and P. Torr, “Interactive image segmentation using an adaptive GMMRF model,” Proc. Eur. Conf. on Computer Vision, ECCV (2004).

21