Workshop Budva
From SWS Challenge Wiki
This page is updated with the latest information regarding the second SWS Challenge Workshop in Budva, Montenegro [1]
Date June 15-16, 2006
The workshop was co-located with ESWC and the Knowledge Web Plenary)
Contents |
Participants
- Anupriya Ankolekar - AIFB Karlsruhe
- Jos de Bruijn - DERI Innsbruck
- Dario Cerizza - CEFRIEL, Semantic Web Activities group
- Federico Facca - Politecnico di Milano, WebML Group
- Christian Kubczak - University of Dortmund
- Ulrich Küster - University of Jena, Endowed Heinz Nixdorf Chair
- Holger Lausen - DERI Innsbruck
- Charles Petrie - DERI Stanford, Stanford Logic Group
- Tomas Vitvar - DERI Galway
- Michal Zaremba - DERI Innsbruck
- Maciej Zaremba - DERI Galway
- Thomas Haselwanter - DERI Innsbruck
- Axel Polleres - Universidad Rey Juan Carlos
Draft Agenda
All SWS Challenge meetings in the Tramontana Congress Hall of the Maestro Hotel.
THU 15 June
- 09:00 - 10:40 Knowledge Web Plenary (for interested people)
- 10:40 - 11:00 Coffee - after the Knowledge Web Plenary in the Banquet Hall (9am)
- 11:00 - 11:30 Introduction - Charles Petrie
- 11:30 - 12:00 Organization and Status of the Challenge - Holger Lausen and Michal Zaremba
- 12:00 - 12:45 DERI Galway - Tomas Vitvar and Maciej Zaremba (paper, presentation, Implementations: [2] [3])
- 12:45 - 14:00 Lunch
- 14:00 - 14:45 Politecnico di Milano & CEFRIEL - Federico Facca and Dario Cerizza (paper, presentation, Implementations: [4] [5], Demo [6])
- 14:45 - 15:30 U. Jena - Ulrich Küster (paper, presentation, Implementations: [7])
- 15:30 - 16:00 Coffee
- 16:00 - 16:45 University of Dortmund - Christian Kubczak (paper, , Implementation: [8])
- 16:45 - 17:15 Report on preliminary evaluation - Lausen
- 17:15 - 17:45 Discussion of future organization of the challenge
- 18:00 Adjourn
- 20:00 Plenary KW Dinner
FRI 16 June
- 09:00 - 09:15 Organization into evaluation reviewing groups
- 09:15 - 11:00 Group Reviewing Work
- 11:00 - 11:30 Coffee
- 11:30 - 13:00 Plenary: preliminary reports and reconciliation
- 13:00 - 14:00 Lunch
- 14:00 - 14:30 Group preparation of final reports
- 14:30 - 15:30 Plenary discussion of next steps
- 15:30 - 16:00 Coffee
- 16:00 - 18:00 KW Plenary session in Banquet Hall - SWS Challenge summary presentation
- 18:00 Finish
Evaluation
Evaluation criteria are reported at: http://sws-challenge.org/wiki/index.php/SWS_Challenge_Levels
| Problem Level | PoliMi - Cefriel | DERI AT & DERI IE | FSU Jena | University of Dortmund |
|---|---|---|---|---|
| 0: static mediation | √ | √ | √ | √ |
| 1a: changes data mediation | 2 | 11 | - | - |
| 1b: changes process mediation | 22 | 22 | - | - |
| 2a: discovery based on location | √3 | - | √ | - |
| 2b: discovery with arithmetic price and weight computations | √3 | - | √5 | - |
| 2c: discovery with temporal semantics | √34 | - | - | - |
| 2d: discovery with conversion of measurement units | - | - | - | - |
| 3a: discovery including request for quote | - | - | √ | - |
1Only Adapters Changed 2different adresses on line item level have not been adressed correctly 3no invocation 4current date entered manually 5arithmetic calculation performed by external Web services
Comments on Budva Results
There were four teams present with implementations and papers and there were a lot of asterisks on the evaluations. There are many interesting details hidden in these asterisks.
One of the specifications for the change in the mediation scenario was that line items might have different addresses. But the Moon company did not support such POs. Therefore different POs would have to be generated. No one drew this inference. In at least one case, it was because the implementers gave the UML diagram more weight than the text specifications. I (Petrie) personally conclude that this is a great instance of the business intention not being sufficiently captured by the programmers, and an example of why we need more formal expressions of business policies.
That said, the more important point is that this was a preliminary evaluation. We learned a lot by doing it. We expect the next evaluation not to have so many footnotes. And we also learned the importance of making the implementations public - because the evaluations are just an approximation. Anyone who wants to really know, should look at the implementations to see what was really done.
The evaluations did indeed involve code examinations and discussions over what counted as a code or data change. We also made some important changes to the evaluation protocol. First, we eliminated the evaluation criterion of only adding and not deleting any data as ultimately not making any sense. It is difficult enough to argue the difference between code and data, especially in the case of software engineering approaches that merely move arrows around in a GUI, which we decided were data declarations, at least in this case.
One of the benefits of this evaluation is that we now have trained evaluation people, whom I hope will serve as team leaders in the next phase of the 2006 workshop. Which we will hold in November in Athens, Georgia, with Amit Seth as our host.
Finally, all of the four teams did amazing, and informative work. The most complete coverage of problems was achieved by the team of the Politecnico di Milano and CEFRIEL. They accomplished this coverage by teaming up approaches best suited for each part of the challenge: a tactic worth emulating.
Each of these four implementation teams is now authorized to use the SWSC logo on their website and point to their preliminary certifications. Congratulations to the participants on a job well done.!
General Workshop Organization
- Participants are highly encouraged to submit their initial solution by April 28th, however later submission will be accepted.
- please send a mail to holger.lausen'at'deri.org to indicate that you want to submit to obtain the access data for ftp.
- solutions are available at http://sws-challenge.org/2006/submission/
- the PC will do an review of the initial submission (discussed during the workshop)
- Format of submission:
- include a file 'declerative.txt' that describes what parts of your submission you consider to be descriptive and not part of your execution environment (in case you also upload parts of that)
- please also include information about where the invokable service (endpoint is)
- 1 week before the workshop participants shall freeze and upload their final solutions together with a max. 8 pages paper
- the peer review of the paper and the related changes in the implementation will be done during the workshop
We intent to publish the papers after the workshop (publisher to be decided). The intent of the publication is to explicitly link / refer to the implemented solutions in the context of the challenge.
