# Doom and Gloom... (was:Re: MTV and Panix .plan file ...)

Peter Deutsch (peterd@bunyip.com)
Wed, 13 Oct 1993 18:48:40 -0400

- Messages sorted by: [ date ][ thread ][ subject ][ author ]
- Next message: Marc Andreessen: "Re: (LONG) Detailed notes on URL6.TXT"
- Previous message: Steven J. Richardson: "Re: Help with Dr. Seuss's book of URIs"

    Message-Id: <9310132248.AA11549@expresso.bunyip.com>

    From: Peter Deutsch <peterd@bunyip.com>

    Date: Wed, 13 Oct 1993 18:48:40 -0400

    In-Reply-To: "Rob Raisch, The Internet Company"'s message as of Oct  6,  7:21

    To: "Rob Raisch, The Internet Company" <raisch@internet.com>

    Subject: Doom and Gloom... (was:Re: MTV and Panix .plan file ...)

Hi All,

Sorry for coming into this debate late. The gang at Bunyip
Central have been on the road and tied up finalizing a new
service which we will be announcing in the next few weeks.
I'm now back and wading through a ton of mail and found
this rather pessimistic note from Rob Raisch which quotes
me, so I thought it would be a suitable place to dive in....

[ Rob wrote: ]

> I would like to suggest that this problem goes away by deploying a
> scalable, distributable mechanism to share the load. And even further
> that an incredible amount of data flowing across our wires is there
> because of ill considered or redundant retrievals.
>
> Peter Deutsch tells the tale that shortly after he unleashed Archie, it
> was responsible for some 90% of all traffic coming in and going out of
> Canada. While this is marvelous praise for the usefulness of Archie, I
> think that it is a severe condemnation for single site services.

I endorse the general thrust of Rob's posting to this
point, but do want to correct one error, since his numbers
are liable to be quoted again by others. At one point, the
archie service was actually responsible for 50 (not 90)
percent of all net traffic into Eastern Canada. More
accurately, 50 percent of all packets on the link into
Montreal went to the host "quiche.cs.mcgill.ca", which was
then the only archie host.

Since then we have done several things to lessen the impact
of what we are doing with this service. There are now some
30 archie servers in various parts of the world (some
private, most public) with another five or six coming
soon. I personally think lots of light-weight, redundent
servers, each serving a smaller local community, is a nice
model for the Internet. Each virtual community can be
responsible for finding the appropriate funding model for
its needs and can better respond to changing conditions.

We have also put a _lot_ of work into making the archie
software into a scalable, net-friendly service. Things
we've done so far include adding code to allow multiple
archies to share information amongst themselves (Thus, for
example, usually only one archie server will visit a
particular archive. It then floods the results to the
other 30-odd sites in a compressed, pre-digested format).

The UDP-based Prospero interface also helps a lot, since
it uses far less resources than the old telnet interface
did. We converted the telnet frontend to also use
Prospero, and saw a measurable increase in throughput.

The biggest win we saw was in distributing the load out
onto multiple servers, and for this we all have to give
credit to the many sites which have had the foresight to
license and operate the software. We obviously need to eat
and we appreciate the efforts of the many people who now
operate an archie. This is one funding model that appears
to be working in this particular case.

At the same time, I think there is also place for
commercial services in which the users pay directly for
the services they use. Our new service will use this
model, with us building useful collections of information
and licensing it out on a subscription basis. We _will_ be
site-licensing these, and again expect service providers
to play an important role in getting this material to
their end users.

> We need some mechanism in place -now- to allow a user to make an informed
> retrieval decision. And in the best case, to make this decision for
> the user. Some of the questions we need answered are:
. . .
> (We were making a stab at this problem in the URI working group, but got
> seriously sidetracked when it became clear that various people had
> different fish to fry. And this problem is vastly larger than most are
> willing to admit.)

As I think someone (John Curran?) said in another reply,
it's not so much that we got sidetracked as that we have
been working on some of the basic infrastructure issues.

URLs seem to be advancing, although there is some wailing
and gnashing of teeth about the particular format chosen
(and yes, I've been one of the wailers and gnashers).
There are also tools that use this format in use today
(check out mosaic).

There also seems to be a rough consensus on URNs, and
hopefully there will be progress on URCs at the meeting in
Houston in a couple of weeks. Together, these seem to
offer much of what Rob asks for in his note. The coming of
the first WHOIS++ server implementations seems to offer a
system for serving these things, as well.

All in all, I think we're actually in pretty good shape.
I'd certainly like to see things advancing faster, and if
anyone wants to send me a cheque I'd be pleased to hire a
couple of more programmers and get things done a lot more
quickly than is currently happening, but I still see
measurable progress all around. We certainly are farther
along than we were a year or two ago.

> This meta-information of and about information and its repositories is
> just another call to arms for the Directory Services people.
>
> Unfortunately, I fear that we are really quite far from ubiquitious
> deployment of this most needed technology. We cannot tell who people are.
> We cannot tell what they are interested in. We cannot find information,
> and we cannot make the proper choices to retrieve it for the lack of a
> generalized method of sharing the meta-information we require.
>
> Whois++ is far from deployment. The protocol is still being defined and
> the central concept of the Centroid (the bit which shares meta-information
> regarding content up the tree) has yet to be proven scalable. There are no
> testable, full implementations.

At the same time, there is measurable progress on this
front, too. There are now several separate WHOIS++
implementations, and as far as I can see the only real
open question at the moment in this system is that I need
to repost the core document with the needed words to
specify how we will use MIME to get graphics and so on in
the output format. Yes, there is still work to be done on
Centroids, but I believe we have a useable, working system
even without these and thus think we're closer to being
done than you seem to believe.

> X.500 is far from deployment. While the protocol is defined (some say
> overly so), there is no reference implementation which is trivial to test,
> and the only work that seems to be going on is in the commercial sector.
> "Sure", we are told, "you can have directory services! Did you bring your
> check book?"

It's probably best if I leave this one alone, since my
dislike and disillusionment with X.500 is legend. Suffice
to say that I think that there is a functional alternative
in sight...

. . .
> I'd like to suggest that this is a VERY SERIOUS problem and one deserving
> of an awful lot of attention right now. We are on the verge of seeing
> millions of new users of the Global Internet, through the efforts of
> alliances like PSI/Cable and the influx of PDAs. In some ways this is a
> very good thing, but I fear we are looking the oft predicted "end of the
> network as we know it" squarely in the face, and we don't as of yet
> recognize it. (Funny, I never thought I would be one of the "Imminent
> Death of the Network Predicted" Cassandras. ;)

I certainly don't see that we're facing the "end of the
net", although perhaps "end of carrying on in this
particular fashion". Sure, we need to worry about
scalability. Sure, we need to arrange for resource
discovery tools, directory services and so on. Sure we
need to worry about how we're going to pay for all of
this.

At the same time, I have every faith in the net's inherent
ability to respond to the demands placed upon it.
Cheer up, Rob!

- peterd

    --

- Next message: Marc Andreessen: "Re: (LONG) Detailed notes on URL6.TXT"
- Previous message: Steven J. Richardson: "Re: Help with Dr. Seuss's book of URIs"