Folks, for me it looks like we are talking about two different
possible developments:
1. HTTP compliance tests.
2. Connect-a-thone.
A while ago I was in NFS development and connect-a-thon was a very
important part of testing. But the way it worked had nearly nothing to
do with the protocol compliance testing. Different vendors came together
with equipment and software, point clients to other people’s
servers and run a group of standardized tests. If our new client has
problem with the current version of let's say HP-UX - it does not matter
who is in compliance, it just means that WE do have a problem.
If such problem is found when both versions are not yet released – you
talk to other party, and also look how this specific test works with
other people implementations.
The primary goal of such tests is inter-vendor compatibility and
"current industry practices" definitely dominate over puristic protocol
compliance issues.
What kind of possible even/alliance are we trying to create?
Both have the right to exist.
The first one, compliance tests, is kind of more straightforward,
one can just write a set of tests with limited interaction with
cache/service vendors. Still to be really useful such tests
should take into account current practices, the way HTTP is
practically implemented, not just seat in the clean
room with nothing but the set of RFC's.
I agree with Alex that there is no need for all of us to come
together for such testing. Tests can be made available as
a software package - or alternately run remotely by configuring
the test lab and system under test to point to one another. The
latter form creates more controllable environment and may prevent
some problems Alex is talking about.
The form of organization Duane is proposing is best suited exactly
for this type of testing.
Connect-a-thone may be also a good thing to have. It is especially
useful for cache/layerX switch/router vendors - current environment
becomes extremely complicated. Too many different devices available,
and understanding the source of the problem when interacting with
specific device often requires a help of vendor engineers - not just
sales support. Service vendors may be interested in both testing
possible elements of infrastructure and preventing problems for
service customers.
The connect-a-thone event may be opened to non-members for some
registration fee.
My two cents.
- Oskar
>-----Original Message-----
>From: Alex Rousskov [mailto:rousskov@ircache.net]
>Sent: Monday, June 05, 2000 9:42 AM
>To: Mark C Nottingham
>Cc: wrec@cs.utk.edu
>Subject: Re: HTTP compliance
>
>
>On Sun, 4 Jun 2000, Mark C Nottingham wrote:
>
>> Others will have more intelligent things to say about the organisation of
>> funding, membership, etc., but it would be nice if vendors who aren't
>> directly participating weren't cut out (e.g., six-month code delay), as
>> they're generally the people who need the most help.
>
>We would be more than happy to cut nobody out. However, there are two
>problems that are hard to solve if you allow free timely access to the
>tool:
>
> - a not-for-marketing license would be nearly impossible
> to enforce; a company or a magazine are likely to violate
> the licensing terms (knowingly or not) and publish claims
> of [non]compliance.
>
> - there is not much incentive for vendors to pay for the
> development if they can get the tool anyway (just look
> at the number of Polygraph sponsors versus the number of
> caching vendors who use the tool)
>
>> How is the connect-a-thon organised in this respect?
>
>It will probably up to the participants to decide. IMO, connect-a-thon
>will not be very useful for these kinds of tests because the tool will
>be testing mostly compliance with standards and current practice rather
>than compatibility with similar software. I suspect that most of the
>tests can be done in isolation so connect-a-thon will not be required.
>
>> * What are the target devices for such tests?
>> Describing server and client as well as proxy/surrogate tests is
>useful, but
>> much more work, and it may turn out quite a bit harder (and more
>disruptive)
>> to pull the vendors on board.
>>
>> I'd recommend starting with proxy/surrogate devices, and eventually
>> expanding into clients and servers if there's enough interest.
>
>I would make it demand driven. If there is enough interest/support from
>the server/client community, the tool can include server/client tests.
>
>> * What kinds of tests?
>> The most obvious goal is to prove full 1.1 compliance, according
>to stated
>> RFC2119 requirements. I think most people realise that doing so
>is actually
>> a very difficult task, because of all of the possible
>permutations of HTTP
>> mechanisms, etc that present themselves.
>
>Full HTTP/1.1 compliance is a nice goal, but probably not very
>practical. Again, since the tool is designed for product developers, I
>would leave the scope for them to decide on. Initial goals should
>probably be more modest to get some visible progress sooner.
>
>> As I said in Lisbon, my interest is primarily in assuring that caches
>> operate in a predictable way to content providers; while we could test to
>> make sure that each message is well-formed, etc., IMHO that won't be
>> necessary, as it's already in the best interests of the vendor to do so.
>>
>> It may be more realistic to define a relatively small group of relevant
>> tests to start with, making it easier to get a first go at it
>out quickly,
>> to gauge response and hopefully have some positive effects ASAP.
>
>Agree.
>
>> Finally, I'd be very surprised if any of the commercial vendors
>who showed
>> interest didn't have a pre-existing compliance engine that they use
>> interally. If one could be convinced/arm-twisted that it would
>be in their
>> best interests to donate it (or a portion), we'd save a lot of effort.
>
>I am less optimistic after talking to several vendors :). However, we
>definitely should try to use the results of past developments if any.
>
>Thanks,
>
>Alex.
>
This archive was generated by hypermail 2b29 : Thu Nov 18 2004 - 11:21:28 MST