Date: Tue, 6 Jul 93 19:04:22 -0700
From: jak@violet.berkeley.edu (John A. Kunze)
Message-Id: <9307070204.AA19824@violet.berkeley.edu>
To: uri@bunyip.com
Subject: where are we now with UR*?
In trying to figure out where we are now with UR*'s, I'd guess most
of us feel that keeping up with the volume and quality of the UR*
discussion these past few months has been like running a marathon.
I'd like to know what people think is a productive way to proceed from
here. Would it be useful to submit to the IETF working group meeting
in Amsterdam some overview papers that collect, distill, and organize
important parts of the recent discussion (whether the authors can
attend or not)? I know this is a bit late to start, but I think it
will be very challenging to conduct the working group without them.
As our discussions sometimes tend to come back to places we've been
before, I'm enclosing an overview paper that I've submitted before.
I believe its organizing principles are as valid now as they were then.
In order to prevent undue prejudice towards the by now vexed concepts of
UR*'s, the paper still (or again) refers to them as things called ER*'s.
If they can be discussed rationally at this safe distance, there is
nothing to prevent their being renamed later on.
-John
Resource Citations for Electronic Discovery and
Retrieval
John A. Kunze-
University of California at Berkeley
Draft 1.1
26 March 1993
This document suggests a few concepts that may be useful in
establishing conventions to assist discovery of and access to
electronic information resources by a wide range of software sys-
tems. This is a revised draft of a document distributed at the
CNI UR/MARC meeting in Denver, 22 January 1993, many concepts of
which implicitly refer to or build on ideas laid out in Clifford
Lynch's discussion paper, "A Framework for Identifying, Locating
and Describing Networked Information Resources".
Prior discussion has introduced terms such as Uniform Resource
Locators (URLs), Unique Resource Serial Numbers (URSNs), and
Universal Resource Identifiers (URIs), with which the reader is
expected to be familiar.
Some of the sections below begin cryptically with a few formal
grammar production rules for the entity in question, followed by
some operations that will be applied to the entity. Please note
that the grammar rules are intended to convey only existence and
number of elements within an entity; they are not intended to
convey either low-level syntax or even element order. Elements
in []'s are optional, + means one or more, * means zero or more,
and | separates a choice branch. The entity operations are
listed as pseudo-code procedure calls and return values simply to
give a rough idea of how the entity will be used. Where exactly
the operation is performed, whether locally or remotely, is not
usually indicated.
1. Terminology: Replace Uniform, Unique, and Universal with
``Electronic''
The terminology could use some work. The ubiquitous and ulti-
mately useless use of `U' in the acronyms tends to signify dif-
ferent marketing oriented words that are no longer necessary.
All the different uses of `U' promotes confusion.
Let's appropriate the word ``Electronic'' for locators, identif-
iers, etc. Technically inaccurate, this word, whether or not we
pick it up, will likely be chosen in spite of a recommendation to
the contrary, and we might as well appear to be in charge.
The word ``Network'' -- as in ``Network Information Resource
Identifier'' -- instead of ``Electronic'' is somewhat more res-
trictive, unless we're sure we we don't need to include or be
backward compatible with non-network resource identifiers, such
as ISBN, ISSN, and call numbers.
2. Electronic Resource Identifier (ERI)
ERI ::= IdAuthority IdDesignator
IdDesignator ::= IdString [ VariantSpecifier ] [ Hints ]
Hints ::= { byteCount | LicenseCode }+
Operations:
ERItoERLlist(ERI) --> (ERL1, ERL2, ...)
ERIequal(ERI1, ERI2) --> { true | false }
ERIroot(ERI) --> { ERI | ERIwithoutVariantSpec }
ERIhints(ERI) --> { none | hints }
An ERI is a pair consisting of an identifying authority (IdAu-
thority) and an identifier string specific to that authority,
plus optional flags indicating the size and/or whether usage
rights apply. This is a coded pointer to an organization that
assigns identifiers (e.g., a publisher), a specific *designator*
for the resource, and hints about sizes and usage restrictions.
Resource designators have the following properties:
(a) two resources have identical designators if and only if the
IdAuthority deems them the same in content (even if another
IdAuthority would call them different),
(b) the IdAuthority guarantees not to re-use designators,
(c) while they are in general opaque to everyone but the IdAu-
thority, the IdAuthority may choose to assign designators
using a system that conveys extra information about the
resource (e.g., subject area derivable from call number),
and
(d) they may contain an optional *variant* specifier.
ERIs were conceived to fulfill the need for a small amount of |
per-item information that can be used to derive the location of |
the rest of the per-item information. This helps systems keep |
down the overhead by returning only a modest amount with each |
item in the initial search results. Rough sizes and usage res- |
trictions, if available from the IdAuthority, are important |
selection criteria; their presence in initial search results can |
help users quickly eliminate expensive or restricted candidates, |
saving unnecessary round trips to distant servers. |
The guarantee never to re-use designators is critical to support |
search and retrieval far into the future. Because an ERI itself |
is also unique, it can be used by client systems to help remove |
duplicate items from the pooled results of searching several dif- |
ferent servers.=
A *variant* specifier is an optional subpart of a resource desig-
nator that preserves all of the above semantics while allowing an
IdAuthority, as an added service, to encode the designator for-
mally to convey specific extra information to external systems
(different from (c) above). By this means, for example, a
software system comparing two designators could tell when the
IdAuthority deemed two documents to be the same except for cer-
tain dimensions along which similar documents often vary. Of
course, no IdAuthority is obliged to reveal such distinctions
even if it knows them.
The variant dimensions, each one optional, are: (a) format, (b)
encoding or archive scheme, and (c) version number (1, 2, 3, etc.
with -1 meaning the most recent, -2 the next most recent, etc.).
By using the variant dimensions, an IdAuthority can continue to
inform others what it considers different resources (as before)
while assisting users in making up their own minds.
An important property of a designator containing a variant
specifier is that the IdAuthority assigning it implicitly agrees
to recognize the "root" designator obtained by stripping off the
variant specifier. Because it is a logically distinct subpart,
user systems can remove the variant specifier to derive the root
designator, which can then be used to query the IdAuthority about
the root document, or about other variants that might be avail-
able. This is particularly important when an ERI designates a
variant that is unusable by the client (e.g., an unsupported for-
mat), because it provides a backward pointer through which more
suitable forms may be discovered.
3. Electronic Resource Locator (ERL)
ERL ::= ANCHOR Locator [ AccessFlags ]
Operations:
NetLocOpen(ERL) --> ioport
ERLflags(ERL) --> flags
NetLocClose(ioport) --> 0
NetLocRead(ioport, byterange) --> { OK | undefined } bytes
NetLocSearch(ioport, query) --> { OK | undefined } count
....
An ERL is a string describing how to access a specific instance
of a resource variant, together with optional flags indicating
what access, cost, and licensing restrictions apply, if any.
Unlike the ERI, it does not provide a backward pointer to a
"root" variant since that is a concept associated with an IdAu-
thority and not with the location of an instance of the resource;
variant discovery may still be undertaken through an ERI that is
associated with the ERL (e.g., bundled together with it inside an
ERR (below)). The initial ANCHOR token is a literal that distin- |
guishes this kind of full-form locator from its lightweight |
cousin, the relative resource locator (see RRL below). It is |
similar in function to the initial `/' that flags a UNIX pathname |
as absolute.
The restrictions flags communicate important information to the
user who is considering whether to establish access. This flag
indicates whether a license is required, that it costs money, or
that provider's (not IdAuthority's, as for the ERI) usage rights
apply. In the absence of these flags, everyone has access, no
charging applies to access, and no license is required. Presence
of a flag is useful as a gross selection hook, because specific
information such as actual charges, copyright statements, etc.
should not be part of the flag. The precise definition of these
flags is not understood yet (e.g., should approximate charges be
included?).
4. Electronic Resource Fragment (ERF)
ERF ::= Range [ Units ]
Operations:
Fragment(ERI, ERF) --> { OK | undefined } bytes
An ERF is a string that, applied to an ERI, defines a fragment of
a resource. Here *fragment* refers to a logical component of a
resource, such as pages of text or frames of video (as opposed to
simple block style segmentation of a resource) which, if you are
lucky, is supported by the underlying access protocol. For some
resource types the concept of fragment demands creative interpre-
tation (see the telnet discussion below).
Typically, an ERF accompanies an instance access request (as
directed by an ERL) and is interpreted by a server system as best
it can. Applying the ERF to a resource may be thought of as a
procedure call, Fragment(ERI, ERF), with several possible out-
comes. A single ERF syntax is desirable for the sake of simpli-
city, but due to the diversity of variants in the ERI universe
and the grab bag of fragment types (words, lines, sentences,
paragraphs, pages, chapters, frames, minutes of audio, etc.), an
important and not uncommon virtual procedure call outcome will be
one of several error conditions.
One such error occurs when the fragment request makes no sense
(e.g., asking for pages 10-20 of a video clip, or, in some cases,
for the third paragraph of a PostScript document). Another com-
mon error occurs when the particular server providing the
resource instance does not support the fragment request, even
when the request would make sense on a fuller function server
(e.g., asking for pages 10-20 of a document makes sense in prin-
ciple, but internal server maintenance considerations may make it
infeasible).
In a more perfect world, some way of querying a resource instance
for all the fragment types it supports would assist orderly
retrieval, but the dependency of fragment type on variant type
and variant instance is not well understood (by me at any rate).
For this reason, it may make sense to rely on the error condi-
tions to support a more interactive style of fragment type "nego-
tiation".
The delivery of fragments presents other complications not
addressed here. One problem is that a requested document format
does not necessarily match the type of the fragment request
accompanying it; for example, a troff document may contain a
preamble of macro definitions crucial to the overall format, but
when you request paragraphs 3-5 you may not get a piece of text
of the same type as the whole document (i.e., with the preamble),
and the fragment may not even be a valid sequence of troff.
Another issue to be looked at is whether fragment boundaries are
preserved on delivery; in the above example, the server may or
may not know the paragraph boundaries, the retrieval protocol may
or may not support preservation of those boundaries during
transmission, and the client may or may not care about boun-
daries, or may be able to discover them heuristically. Admit-
tedly, this headache goes away when you restrict fragment
retrieval to only one unit at a time (e.g., paragraph 3, then 4,
then 5), but perhaps that's too limiting for real systems.
In the short term, it may get us off the ground to consider the
fragment request as inherently messy, and accept some help from
the client, the user, and the error conditions returned to guess
at fragment types.
5. Electronic Resource Reference (ERR)
ERR ::= ERI [ ERF ] { ERL }*
Operations:
NetRefOpen(ERR) --> ioport
NetRefClose(ioport) --> 0
NetRefRead(ioport, byterange) --> { OK | undefined } bytes
NetRefSearch(ioport, query) --> { OK | undefined } count
....
An ERR is an ERI, optionally accompanied by an ERF, and by zero
or more ERLs. Each ERL gives a location where an instance of the
resource might be accessed. If none of the ERLs is valid or none
are given, the ERI can be presented to a resolution authority
(e.g., the IdAuthority) to return a current list of ERLs, other-
wise the accompanying ERLs function as an informal cache (without
any cache consistency checks) of known or probable alternate
locations.
6. Electronic Resource Citation (ERC)
ERC ::= { HumanString }+ { ERR }+
Operations:
Sync(ERC) --> ERCupdated
NetCitOpen(ERC) --> ioport
NetCitClose(ioport) --> 0
NetCitRead(ioport, byterange) --> { OK | undefined } bytes
NetCitSearch(ioport, query) --> { OK | undefined } count
....
An ERC consists of a set of human readable descriptive fields
(e.g., title, author, and date), a resource type (e.g., data,
query, or interaction), and one or more ERRs. The ERRs list a
number of references for a resource described by the descriptive
fields. These references should be equivalent in the sense that
each ERI's IdAuthority-supplied descriptive fields would appear
to a human being to identify resources roughly equivalent in con-
tent. In fact, short of identical ERIs, two ERRs with different
ERIs can (as before) identify identical resources (with two dif-
ferent IdAuthorities) or two variant forms of a resource.
The human readable fields help users to identify resources,
select them from menus, and search for them in databases. Some
selection criteria, such as size, cost, access restrictions, and
licensing requirements, are not actually carried at the top level
of the ERC, but within the ERIs and ERLs to which they pertain.
This presents a problem in designing searches if the granularity
of database records is such that one record covers one ERC,
because if that ERC covers many variants, the search process will
be unable to weed out many unwanted variants. At the other
extreme, if one ERC covers only one variant, a straightforward
author-subject search may end up flooded by many unwanted vari-
ants. I suspect that selection criteria such as cost and size,
if useful in searching at all, are usually add-on searches (i.e.,
AND'd restrictions to primary search criteria), and that typical
variant set sizes will be low enough with the former scheme to
make variant selection subsequent to a search viable.
One special need introduced by the human readable fields is that,
because they are actually data (in a sense), they may need to be
updatable. For example, a very dynamic resource may well change
its modification date without changing ERI, and that changing
modification date is of interest to users of the ERC. In this
case, it would be useful for the IdAuthority to respond to a
"synchronize" request given an ERI, which causes it to return
descriptive and location information related to it that is both
authoritative and current; this would give providers of databases
and hypertexts tremendous maintenance relief.
The resource type indicates what paradigm best models resource
access, and is one of data, search, or interaction. The para- |
digms for a resource are given by the collection of operations |
defined for that resource. In the data paradigm, the resource |
supports a I read operation, which retrieves a contiguous |
sequence of bytes representing either the entire resource or a |
range of bytes within the resource. In the search paradigm, the |
resource may designate any one of several specific subtypes: a |
query (to be evaluated), a database, a predefined result set of |
search records, a record in a result set, or an element of a |
record in a result set; operations supported allow for either |
search or retrieval against the appropriate subtypes [XXX]. The |
interaction paradigm allows for resources of three subtypes: tel- |
net, tn3270, or X windows; the ERL for such a resource encodes |
the necessary connection information.
Previous proposals [Berners-Lee, Lynch] have suggested that this
is the job of a *service* identifier, which just happens to dou-
ble quite often as a protocol specifier. As protocols become
more multi-function and get used in non-traditional ways (e.g.,
using telnet for non-interactive data retrieval), however, the
paradigm is no longer deducible from the protocol. For example,
the Z39.50 protocol provides a search model, a data retrieval
model independent of prior search, and an interaction model (sort
of, with Scan). User interfaces need to reflect the paradigm to
the user (e.g., with different icons) upon presenting resource
choices because users with strong preferences, say, against tel-
net interactions, may not like to be surprised by one (e.g., when
they double-click on an undistinguished icon). |
7. Relative Resource Locator (RRL) |
RRL ::= Locator [ AccessFlags ] |
Operations |
ParentOf(RRL) --(remote-evaluation)--> ERL |
NetLocOpen(ERL, RRL) --> ioport |
ERLflags(RRL) --> flags |
Previously referred to as partial locators, relative resource |
locators come up in the context of hypermedia. Crudely, an RRL |
is to an ERL what a UNIX filesystem relative pathname is to a |
full pathname. Superficially, an RRL looks like an ERL minus the |
initial ANCHOR token. |
A resource access using an RRL must be chaperoned by an ERL for |
the referencing resource, which relieves the server from having |
to maintain this state information. Well-behaved servers always |
return an ERL along with any data for a document or fragment that |
might contain RRL hyperlinks, because if for some reason the |
client never had the ERL or lost it, the links would be inacces- |
sible to it. |
An imporant difference between RRLs and partial locators is that |
the client cannot deduce the antecedent (the referring) document |
given an ERL or RRL, which is opaque to it. If it wants this |
information, it can only get it from its own memory (in case it |
once descended through the antecedent) or by explicitly querying |
the server. Advantages are that the client need not know how to |
parse the server-defined locator string and the server has more |
freedom in determining antecedents (e.g., in previous proposals, |
the antecedent of /foo/bar had to be /foo).
8. Factoring out Common Elements of Resource Locators
Previous proposals for ERLs [Berners-Lee, Lynch] have recognized
the difficulty in defining multi-service/protocol locators that
share syntactic elements, and so they have suggested a scheme in
which the overall format is a pair consisting of a service iden-
tifier and a parameter string that is opaque except to that ser-
vice.
While this gets us off the ground, it also promotes duplication
of effort and development of features and requirements that are
incompatible with other service identifiers, even those that con-
form to the same access paradigm. Part of the resistance is due
to inherent limitations of one service/protocol, and that should
not limit a similar service.
I propose that we look carefully for common elements that can be
factored out of the opaque protocol-dependent locator and put
into a common protocol-independent locator syntax. This is a
requirement for the resource paradigms above.
9. How to Fragment an Interactive Resource
A telnet service is an interactive resource that it does not make
immediate sense to talk about fragmenting. Still if fragments
are to apply to resources, it would be nice to map the concept to
interactions with, say telnet or an X Windows client.
Considering that most resources have retrievable data as their
primary outcome one could step back and think of a telnet service
as a resource that has retrievable data as a secondary outcome
(e.g., via screen capture) and has an *experience* of data as the
primary outcome. Consider now a series of commands that causes a
series of screens to be displayed, starting with the first entry
screen and ending with the screen from which you terminate your
session.
Such a series of commands is a telnet fragment specifier, and the
experience of the series of screens it evokes is a fragment of a
telnet resource. The series of screens it produces, recorded,
say, in a session log, is the retrievable data associated with
the fragment. One detail needed to completely describe this
fragment specifier is time sequencing data. This is more
apparent in generalizing from a telnet interactive resource to an
arbitrary interaction (e.g., with an X Windows client); in gen-
eral, a fragment specifier for an interaction is an event stream.
This actually meshes well with the current practice of managing
part of the interaction on behalf of the user, usually the ini-
tial part. It is not uncommon for client software to call tools
such as "expect" with an event stream script to manage the telnet
start up, host connection, login, password, etc. before turning
control over to the user. A natural extension is to manage the
entire session for the user from start to finish.
_________________________
- Supported in part by the Coalition for Networked In-
formation.
= De-duplication is generally hard because IdAuthori-
ties may assign different numbers to the same work; for
example, the hardcover and softcover editions of a book
may have two different ISBNs upon printing, but receive
only one call number upon entering a library collec-
tion.
DRAFT 26 March 1993 Resource Citations