Very nice writeup! I’d love to see this happening, and to find a way to contribute.
I do not think offline work (and later, deferred synchronization) is required; networks are becoming more and more ubiquituous.
I disagree on this point. There are, and will be, lots of situations without network connection, e.g.:
Deliberately separated networks inside companies
Specific Domains: Construction, Infrastructure, everything outdoor-y, WiFi-free (medical, research)
Travelling (airplane or Deutsche Bahn)
Every time $MY_OS or $MY_NETWORK_PROVIDER decides to do strange network things
Do you have some ideas how to build this incrementally and collaboratively?
I doubt we’d manage to pull this off if we aimed for “the whole nine yards” at once.
And we won’t get all of it right the first time.
Hi all… thanks Markus for putting forward this proposal. That kind of “cloud-scale” modeling infrastructure is something that I have been thinking about for some time as well.
Since there seems there are many complex technical issues to build something like that, I’d propose we start by focusing on interfaces based on the initial requirements we know about, and back them up with naive/cheap but functional implementations, that even if they cannot operate at scale, they complied with the contract correctly (if we ignore the more demanding non-functional requirements). Maybe even a hackathon would be enough to take us to an initial toy-like implementation that addresses the first few critical use cases adequately and displays a sensible architecture.
That would allow us to start implementing clients for this repository, and allow us to find time/resources to replace the naive implementations with much improved ones that can actually provide the latency/reliability/throughput characteristics that would be really required for this kind of infrastructure later.
Another upside of this interface-centric approach is that it promotes a model that makes it easier to combine open-source and proprietary/commercial components.
Finally, of course, such interfaces will need to be revisited as time goes by, and we discover technical requirements that challenge the choices made so far, or even require things that used to be separated to be implemented together, or vice-versa.
I would agree with @rafael, however from the discussion of today I had the impression that @voelter has much more experience than most with incremental transformation engines. My understanding was that he was saying that it is a difficult problem to tackle and that in order to have decent performance one has to couple the way tranformations are executed with the way data is stored. So this would leak into the design of the API, if I understood correctly. I personally feel that I do not know enough about these incremental transformations, so it is difficult for me to comment.
I wonder if there is something I can read to learn more about these incremental transformations
For sure, @ftomassetti, the more the persons defining the interfaces understand the basic requirements, the closer they will match what would be needed for the final interface.
But we don’t need to get things totally right at first, if we are prepared to evolve as required. Also, we should only define interfaces where it is clear they are needed, and not separate things “just because”, in order not to make it harder for implementations to benefit from coupling. At a minimum, there must be at least one interface for clients to put/find/take things from the repository.
Example: User edits model, change notification is sent to the typesystem service.
The typesystem does its thing. Now the editor service wants to update its error markers.
How does this work? I can imagine different approaches:
There is a separate channel between typesystem and editor, the active repository doesn’t care.
The active repository only offers event propagation, without any incrementality.
The typesystem delivers a derived model to the active repository. The derived model is processed the same way as the “real” model, but not persisted.
The typesystem delivers amendments to the real model, which are not conceptually different from the “real” model. The active repository treats them the same way as any other model change (including persistence).
My gut feeling says 3), but I cannot overlook the consequences.