how to somewhat certify matsim simulations


This is just a decision item that we need to start debating IMO. Probably even not dev mtg but somewhere else.

My preference would be (#)

(1) people fork matsim-example-project

(2) they use some deterministic matsim version as a maven dependency

(3) they keep their own code short

(4) everything is available in github or similar

(5) there are regression tests around their studies

In reality, I see (II)

(a) people clone some random snapshot of matsim

(b) they implement their own changes directly into that code

(c) there are many changes all over the place

(d) the code is not available and often not even backed by a repo

(e) there are no regression tests.

My expectation is that simulation science in general, and matsim in particular, will go under if they cannot make this more systematic. Question is what we, as a community, should ask for, in particular when refereeing.

(I) One thing that I can see coming for myself is that I will refuse to referee dissertation work if it does not follow (#).

(II) Maybe the same for journal papers?

(III) A minimum scientific requirement possibly would be: Code that has run a certain study is available in github, with commit hash. Unfortunately, that still makes it possible to start with an arbitrary matsim version, possibly not even passing tests, and do shotgun changes all over the original code.






Kai Nagel