how to somewhat certify matsim simulations

Description

This is just a decision item that we need to start debating IMO. Probably even not dev mtg but somewhere else.

My preference would be (#)

(1) people fork matsim-example-project

(2) they use some deterministic matsim version as a maven dependency

(3) they keep their own code short

(4) everything is available in github or similar

(5) there are regression tests around their studies

In reality, I see (II)

(a) people clone some random snapshot of matsim

(b) they implement their own changes directly into that code

(c) there are many changes all over the place

(d) the code is not available and often not even backed by a repo

(e) there are no regression tests.

My expectation is that simulation science in general, and matsim in particular, will go under if they cannot make this more systematic. Question is what we, as a community, should ask for, in particular when refereeing.

(I) One thing that I can see coming for myself is that I will refuse to referee dissertation work if it does not follow (#).

(II) Maybe the same for journal papers?

(III) A minimum scientific requirement possibly would be: Code that has run a certain study is available in github, with commit hash. Unfortunately, that still makes it possible to start with an arbitrary matsim version, possibly not even passing tests, and do shotgun changes all over the original code.

Environment

None

Activity

Show:
Thibaut Dubernet
September 18, 2019, 10:14 AM

In principle a good idea, but I have difficulty to see how to pull out (II) and (III): we have no guarantee to be reviewers of all papers/grant proposals involving MATSim.

However, this echoes the research I am currently conduction about open source software foundations (OSSF) in an interesting way: a few OSSFs give the possibility for supporting members to display a badge “certified XYZ member” on their material. This is for instance the case of the Drupal Association or the KDE e.V.. Others make it necessary to be a foundation member to use trademarked material, such as logos or names. This is for instance the case of the Eclipse foundation (although the idea is slightly different, as this is an “umbrella foundation” supporting projects).

My thinking is this: can we somehow translate that to our case? Basically one of:

  • allow higher membership tiers only to projects or institutions that commit to following those rules or demonstrated following those rules in the past

  • Make this part of a loosely enforced “charter” for foundation members

  • make that a requirement to being on the “gallery of projects” or “list of supporting members” on matsim.org

  • make it a requirement to be allowed to display a “MATSim foundation member” badge on communication material.

I would see it fitting with the aim of the foundation to shape and organize the MATSim community in a positive way.

Of course, that would work only if any of the “benefits” listed are something members actually care about, which might not be the case…

Assignee

Unassigned

Reporter

Kai Nagel

Labels

None

Priority

Major
Configure