This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
An Approach to Mobile Software Robots for the WWW
July/August 1999 (vol. 11 no. 4)
pp. 526-548

Abstract—This paper describes a framework for developing mobile software robots by using the Planet mobile object system, which is characterized by language-neutral layered architecture, the native code execution of mobile objects, and asynchronous object passing. We propose an approach to implementing mobile Web search robots that takes full advantage of these characteristics, and we base our discussion of its effectiveness on experiments conducted in the Internet environment. The results show that the Planet approach to mobile Web search robots significantly reduces the amount of data transferred via the Internet and that it enables the robots to work more efficiently than the robots in the conventional stationary scheme whenever nontrivial amounts of HTML files are processed.

[1] L.F. Bic, M. Fukuda, and M.B. Dillencourt, “Distributed Computing Using Autonomous Objects,” Computer, Aug. 1996.
[2] C.M. Bowman, P.B. Danzig, D.R. Hardy, U. Manber, and M.F. Schwartz, “The Harvest Information Discovery and Access System,” Proc. Second Int'l World Wide Web Conf., pp. 763–771, Oct. 1994.
[3] L. Cardelli, “A Language with Distributed Scope,” Computing Systems, vol. 8, no. 1, pp. 27–59, Jan. 1995.
[4] F.C. Cheong, Internet Agent, New Riders Publishing, 1996.
[5] S. Chiba, “A Metaobject Protocol for C++,” Proc. OOPSLA '95, pp. 285–299, ACM, 1995.
[6] National Center for Science Information Systems,http:/www. sinet.ad.jp/.
[7] G. Glass, “ObjectSpace Voyager: The Agent ORB for Java,” Proc. Second Int'l Conf. Worldwide Computing and Its Applications, Lecture Notes in Computer Science, vol. 1,368, Springer-Verlag, 1998.
[8] E. Jul, H. Levy, N. Hutchinson, and A. Black, “Fine-Grained Mobility in the Emerald System,” ACM Trans. Computer Systems, vol. 6, no. 1, pp. 109–133, Feb. 1988.
[9] K. Kato, “Safe and Secure Execution Mechanisms of Mobile Objects,” J. Vitek and C. Tschudin, eds., Mobile Object Systems, pp. 201–211, Lecture Notes in Computer Science, vol. 1,222, Springer-Verlag, 1997.
[10] K. Kato and T. Masuda, “Persistent Caching: An Implementation Technique for Complex Objects with Object Identity,” IEEE Trans. Software Eng., vol. 18, no. 7, pp. 631-645, July 1992.
[11] K. Kato, K. Matsubara, K. Toumura, S. Aikawa, and Y. Someya, “Object Passing and Interaction Mechanism of the PlanetMobile Object System,” Proc. France-Japan Workshop Object-Based Parallel and Distributed Computation, pp. 153–162, 1997.
[12] K. Kato, A. Narita, S. Inohara, and T. Masuda,“Distributed Shared Repository: A Unified Approach to Distribution and Persistency,” Proc. 13th IEEE Int'l Conf. Distributed Computing Systems, pp. 20–29, May 1993.
[13] K. Kono, K. Kato, and T. Masuda, “An Implementation Method of Migratable Distributed Objects Using an RPC Technique Integrated with Virtual Memory,” Proc. 10th European Conf. Object-Oriented Programming, pp. 295–315, July 1996.
[14] M. Koster, “Guide for Robot Writers,” http://wsw.nexor.co.uk/mak/doc/robotsguidelines.html , 1994.
[15] M. Koster, “A Standard for Robot Exclusion,” http://www. nexor. co.uk/mak/doc/robotsnorobots.html , 1994.
[16] M. Koster, “Robots in the Web: Threat or Treat?” ConneXions, vol. 9, no. 4, Apr. 1995; also available athttp://info.webcrawler.com/mak/projects/ robotsguidelines.html.
[17] D.B. Lange and M. Oshima, “Aglets: Programming Mobile Agents in Java,” Proc. Worldwide Computing and Its Applications, pp. 253–266, Lecture Notes in Computer Science, vol. 1,274, Springer-Verlag, Mar. 1997.
[18] K. Matsubara and K. Kato, “Implementation of the Mobile Object System Providing Both Distribution and Persistency in a Uniform Way,” Trans. Information Processing Soc. of Japan, vol. 39, no. 8, pp. 2,494–2,508, Aug. 1998 (in Japanese).
[19] M.L. Mauldin, “Measuring the Web With Lycos,” Proc. Third Int'l World Wide Web Conf., Apr. 1995.
[20] O. McBryan, “GENVL and WWWW: Tools for Taming the Web,” R. Cailliau, O. Nierstrasz, and M. Ruggier, eds., Proc. First Int'l World Wide Web Conf., 1994.
[21] Sun Microsystems,http://www.sun.com/solaris/javaindex. html .
[22] Sun Microsystems,http://www.sun.com/solaris/jreindex.html .
[23] Inc. ObjectSpace,http:/www.objectspace.com/.
[24] C.P. Pfleeger, Security in Computing, Prentice Hall, second ed., 1997.
[25] B. Pinkerton,“Finding What People Want: Experiences with the WebCrawler,” Proc. Second Int'l World Wide Web Conf., 1994; http://webcrawler.com/WebCrawlerWWW94.html .
[26] R. Seltzer, E.J. Ray, and D.S. Ray, The AltaVista Search Revolution, McGraw-Hill, 1997.
[27] A. Silberschatz and P. Galvin, Operating System Concepts. Addison-Wesley, fifth ed., pp. 130-133, 1997.
[28] A. Tanenbaum, Distributed Operating Systems. Prentice-Hall, 1995.
[29] B. Welch and J.K. Ousterhout, “Prefix Tables: A Simple Mechanism for Locating Files in a Distributed System,” Proc. IEEE Int'l Conf. Distributed Computing Systems, pp. 184–189, 1986.
[30] J.E. White,“Mobile Agents,” J. Bradshaw, ed., Software Agents, MIT Press, 1996.
[31] B. Yuwono and D.L. Lee, “Wise: A World Wide Web Resource Database System,” IEEE Trans. Knowledge and Data Eng., vol. 8, no. 4, pp 548-554, Aug. 1996.
[32] S. Zhou, M. Stumm, K. Li, and D. Wortman, "Heterogeneous Distributed Shared Memory," IEEE Trans. Parallel and Distributed Systems, vol. 3, no. 5, pp. 540-554, Sept. 1992.

Index Terms:
Internet, WWW, Web, mobile object, mobile agent, search robot, indexing robot.
Citation:
Kazuhiko Kato, Yuuichi Someya, Katsuya Matsubara, Kunihiko Toumura, Hirotake Abe, "An Approach to Mobile Software Robots for the WWW," IEEE Transactions on Knowledge and Data Engineering, vol. 11, no. 4, pp. 526-548, July-Aug. 1999, doi:10.1109/69.790801
Usage of this product signifies your acceptance of the Terms of Use.