Offering parsing API

[ftomassetti]

One recurring idea we are considering is to offer parsing API for a bunch of languages (e.g., COBOL, PL/SQL, Python, etc.)

Why?

Because in this way we can write our parser once and it can be used by many host-languages, because it could be integrated in other services (e.g., LSP servers) and because it would be easier to try (we offer parsers under a commercial license and just trying the parser is not as trivial as call some API).

What do you think?

[ftomassetti]

Also, we would offer a coherent APIs to process multiple languages, with some similarities in ASTs (e.g., you could recognize comments, statements, and expressions in all languages )

[pjmolina]

Interesting technical idea Federico!
This raises three questions for me:
Let me play the advocate’s devil role for a moment.

1. Is there a market for this? Are there are customers willing to pay for an AST as a service? Without further research, it looks to me too technical with a narrow niche market.
2. How do you plan to price that service? By number of lines, size of files (bytes), calls to API, CPU time, monthly subscription, etc.?
3. How to overcome the problems with latency? For real-time editors you need less than end-to-end 100-200 ms of feedback after every any keystroke. Calling and using cloud services is complicated to get good latencies without using CDNs or placing servers close to your customers (ideally: same region-country). It is true that for other use cases like batch processing or compiling, real-time and thus latency is not a concern.

Nevermind: if you have people willing to pay for this and can handle the latency problem, go for it!

Pedro J.

[damilare]

What if the AST is represented in existing popular data format like JSON, YML or Protobuf?

This way, implementation languages only need to convert the syntax tree into data structures.

However, we may also need to consider network latency in delivering the AST to the client. JSON RPC, Websockets or GRPC?

Perhaps, what we need are dynamic parser generators.

[ftomassetti]

Hi Pedro,
very good questions, I wished I knew how to answer

@pjmolina

Is there a market for this? Are there are customers willing to pay for an AST as a service ? Without further research, it looks to me too technical with a narrow niche market.

I do not know. We have a client and some requests for our Cobol parser. Normally things stagnate during the discussion on how they could test the parser: if there was a way to simply use it with a free account to test the idea out, maybe it could help

@pjmolina

How do you plan to price that service? By number of lines, size of files (bytes), calls to API, CPU time, monthly subscription, etc.?

I did not think about that: initially I would offer it for free

@pjmolina

How to overcome the problems with latency ? For real-time editors you need less than end-to-end 100-200 ms of feedback after every any keystroke. Calling and using cloud services is complicated to get good latencies without using CDNs or placing servers close to your customers (ideally: same region-country). It is true that for other use cases like batch processing or compiling, real-time and thus latency is not a concern.

Yes, I was thinking that for code-analysis, diagram generation and transpilation it should not be a problem. I would not suggest using it for syntax highlighting.

For me in general it is very difficult to reason on the business-aspects. I wish I had books to read to acquire the right mentality to understand these issues

[ftomassetti]

@damilare

What if the AST is represented in existing popular data format like JSON, YML or Protobuf?

This is already the case: we support JSON and XML, but we could support more formats

@damilare

However, we may also need to consider network latency in delivering the AST to the client. JSON RPC, Websockets or GRPC?

True

@damilare

Perhaps, what we need are dynamic parser generators.

What do you mean by that?

[damilare]

Re: Dynamic Parser Generator

Instead of providing AST over a network, you can have a :

1. Parser implementation that compiles to target language at runtime
2. A parser generator tool that generates a parser for the target language at runtime

Generated parser could be a Directed Acyclic Graph that can mutate according to the DSL that is being loaded at runtime.

The problem with these options would be how to monetize the solution

[meinte.boersma]

I think it’s always good to think about ways to offer functionality in a non-coupled/separated/composable way, e.g. through making an API for it.

Do you know the Software Improvement Group (SIG), based in the Netherlands? I could see a partnership between them and you. They have a high, recurring volume in the form of code bases being assessed on quality time and time again.

[ftomassetti]

@meinte.boersma

Do you know the Software Improvement Group (SIG), based in the Netherlands?

I know they exist, but I never worked with them or know anyone working there

@meinte.boersma

They have a high, recurring volume in the form of code bases being assessed on quality time and time again.

This is interesting. Do you have any suggestion regarding how could I contact them?

Thank you for your comments!

[meinte.boersma]

I don’t have personal contacts there, but you could drop a message on their site. Sorry I can’t be of more help

[ftomassetti]

This is already quite helpful, thanks!