Kotlinx-llvm and Beagle roadmap

1

Federico asked me to post about this so I am. I am working on a kotlin multiplatform library for llvm. Right now, this only works (as an internal DSL) with globals, but I have successfully been able to expand on that by allowing the creation of functions and local code blocks (including local variables). But there is much to improve. Unfortunately JNI isn’t so good with native exceptions so I have fish them out myself and design the library’s architecture around not throwing native exceptions. Cause then we would be presented with blatantly unreadable native stack traces, memory addresses, etc all because of something simple. I’m suffering so you don’t have to lol!

I just integrated Kolasu with the test language which is going to be a toy language (perhaps there will be multiple toy langs to test different paradigms and styles) that is using ANTLR to generate the lexer and grammar, which I am then feeding into Kolasu, which will then feed into llvm. I was able to successfully generate a simple Kolasu parse tree of this.

I’ll be expanding on the toy language as I develop the library. For now it’s on JVM until I get a feel for the architecture, then I’ll move on to design the overall architecture. I’m not sure I’ll do UML models or flowcharts but I’ll definitely have a wiki for it at some point.

This is a vital part of Beagle’s development lifecycle, as I am actually considering forking Kolasu for multiplatforming, since Beagle is on Kotlin/Native, I would love to have Kolasu on K/N before I get kotlinx-llvm to be multiplatform. I would also love to see antlr-kotlin to be multiplatform too!!

There are several other miscellaneous modules I have to make the Beagle compiler, including creating an external debugging mechanism, which I will write about. It takes the idea of stepping through a debugger out of a program’s runtime by generating debug data on the disk in a simple byte format, and allowing external programs to deserialize them. I hope to make a neat visual program that presents you with graphical analysis of your debug data. It’ll be a vital part of Beagle that I will be trying out. It in theory should be much better than stepping through a debugger at runtime. After I get all of these modules done, it should be smooth sailing from here!

I also need to create the pipeline architecture and the exact parsing passes the parser will go through. I have a general idea of how I want it to go down, I will be sure to post here about it once I’ve made up my mind.

I hope to have a working demo of Beagle by the end of the year!

3 Likes
2

My dream vision is similar to yours:

  1. to improve ANTLR-Kotlin, which is already multi-platform
  2. to make Kolasu multi-platform,
  3. then having a sample project using ANTLR-Kotlin, Kolasu and Kotlinx-llvm.

With ANTLR-Kotlin one would get a Kotlin multi-platform parse-tree, with Kolasu one could transform it into a Kotlin multi-platform AST, with features for manipulating and transforming the AST. Finally one could attach Kotlinx-llvm to complete the compiler. For me it would be amazing :smiley:

Now, it seems you are already working on this setup, and that makes me really happy!

I am wondering how can you use JNI in a Kotlin multi-platform project. Are you targeting JVM and Native? What about Javascript?

Regarding Kolasu, the most immediate need we are seeing is to have it usable from C++ projects. We are looking at Kotlin Native but I am not sure it would make possible to use Kolasu in C++ projects. I think one could expose C functions but not C++ classes from Kotlin/Native. Am I mistaken?

By the way, in the meantime Kolasu is improving a lot thanks to @hexagonaal

I would also be interested in hearing about your debugger mechanism!

1 Like
3

My dream vision is similar to yours:

  1. to improve ANTLR-Kotlin, which is already multi-platform
  2. to make Kolasu multi-platform,
  3. then having a sample project using ANTLR-Kotlin, Kolasu and Kotlinx-llvm.

With ANTLR-Kotlin one would get a Kotlin multi-platform parse-tree, with Kolasu one could transform it into a Kotlin multi-platform AST, with features for manipulating and transforming the AST. Finally one could attach Kotlinx-llvm to complete the compiler. For me it would be amazing :smiley:

Now, it seems you are already working on this setup, and that makes me really happy!

Yes, you can see here I’ve already got it hooked up for global variables successfully. I’m just trying to rearrange the project to accommodate more test projects using modules but I’m having a hard time with gradle (imo, I don’t like gradle. Beagle is gonna have a much nicer build system). I’ll be needing to create a kolasu generator soon to make things easier.

I am wondering how can you use JNI in a Kotlin multi-platform project. Are you targeting JVM and Native? What about Javascript?

Right now, it is only targeting JVM but since kotlin relies so heavily on gradle, I feel it wouldn’t be hard to convert it to multiplatform. I’m not sure what you’re asking. JNI should work fine in a multiplatform project if you’re on JVM. For now, it’s on JVM until I get the main infrastructure worked upon, then I’ll design the architecture. Also, JS I’m not so sure about. Looks like this might be our friend in this adventure. Although, I’m quite worried about browser side performance. I don’t really know much about JS so I’ll probably learn more about that when the time comes.

Regarding Kolasu, the most immediate need we are seeing is to have it usable from C++ projects. We are looking at Kotlin Native but I am not sure it would make possible to use Kolasu in C++ projects. I think one could expose C functions but not C++ classes from Kotlin/Native. Am I mistaken?

No you are not mistaken. C++ name mangling can be a real pain for language developers. I remember for work I was trying to get Kotlin/Native to work with Snowboy’s C example and I was having to hardest time cause of linker errors, and K/N doesn’t have very helpful error messages at the moment. It took me switching completely back to pure C to figure out what the problem really was (I was missing lapack and cblas from linker options). The whole time I thought it was because of C++, but nope!

By the way, in the meantime Kolasu is improving a lot thanks to @hexagonaal

That’s great!

I would also be interested in hearing about your debugger mechanism!

My debugger mechanism is going to allow each pass of the compiler to be serialized into a byte format of the data it was able to produce, or has produced so far. It will also create a byte formatted snapshot of the actual module itself; what state it was in, what variables are set to what, system details, threading, etc. It will then output this all to a file on disk (one for each instance). I’ll then create an interactive, visual program that will deserialize this byte format in a presentable way that will allow me to traverse through the state stack, call stacks, etc. Each pass will be different. I’m going to make this the way you debug in Beagle. In debug mode, every function call will have a snapshot taken of the program’s state. Really, it’s going to stack local states together with global states, and that will produce the overall program’s state when a function was called. You can then interactively investigate with your program without needing to be in runtime. This also means that people can share debug data with each other very easily while preserving the debug data with the problem itself still intact. Debugging efficiency should skyrocket.

1 Like