So.. hey, it’s done. Basically. tinypy is a 64k implementation of a subset of python. It can bootstrap itself into a single executable that can compile python files to bytecode and run them on a VM. Thanks to everyone who gave feedback thus far on this project. Double thanks to allefant who listened to me blab about it endlessly on irc for the last month 🙂

I found all the stuff people told me about for parsing was a huge help. This article http://javascript.crockford.com/tdop/tdop.html was what I ended basing it on. It’s almost like magic, but it makes for a really simple easy to follow parser. The VM is based on stuff I read about the lua VM.

So what’s next? I need to let it sit around for a week and then I’ll do a “release” I guess. I’ve gotta pick a license or something for it (probably MIT? I’m open for suggestions.) I’m also mulling over possible names. Maybe “tinypy” .. or “wedge” .. or “cupcake” .. or “garter”. Hmmn.

Anyway – I’m sure I’ll be tweaking it a bit over time, but I’m pretty happy to have it to this point now. I probably won’t do much with it until I try making a game with it. Right now it depends on libgc for garbage collection. If someone clever out there can implement a garbage collector for it that works in like 2-4k, that’d be better. My brain is pretty spent.

For the brave: svn://www.imitationpickles.org/tinypy/trunk or tinypy.zip. The following is only tested under linux, but I bet it would work in any bash environment. Maybe.

$ python boot.py

Will run the 3-phase testing + bootstrapping process. It will first use python to generate the .tpc files for the compiler. Second phase uses the VM to generate those same files. Third phase uses the bootstrapped tinypy executable to “re-bootstrap” tinypy to get the final version. The -nopos option strips out debug info from the .tpc files.

$ ./tinypy julia.py

Run the julia demo without dependence on *anything* but the tinypy executable.

$ ./tinypy your_own_code.py

Will do something! Probably print out a pretty traceback about how you tried to use a python feature / module that tinypy doesn’t support 🙂 “batteries not included”

I’ve been out of town for the last few days, so I haven’t done much to tinypy. However, I have realized that my current “dumbparse” module is pretty dumb. Two things are wrong with it:

• It’s slow. Noticeably so with 8k modules, etc.
• It’s dumb. It doesn’t give “useful” parsing errors.

I’ve read up a bit on LR, LL, top-down, and a whole bunch of kinds of parsers and it all sounds a bit magical to me. I’d be glad for some suggestions or ideas on where I could go from here. My requirements are:

• must be pretty compact (I want my parser to be < 12k in size)
• must not get exponentially slower on larger files (but it doesn’t have to be blindingly fast, just decent)
• must be able to notice where a syntax error is (indicated by token – I’ve got a good tokenizer already)
• must parse “basic” python and be written in that as well (see my current files for examples)

Thanks! svn://www.imitationpickles.org/tinypy/trunk – for the brave

Hey all you open source folks out there .. I’m looking for an electronic document / data / business intelligence tool. Something where I can set up a handful of forms, collect the data, search the data, and generate nice reports.

I’m open for any language, any platform, whatever (though, PHP and python are preferred). I’ve done some searching but so far most things seem too “heavyweight” I want something “lightweight” that I can actually get into and use without dedicating the rest of my life to figuring out someone’s junk.

Thanks!

Well, I got two of my favorite python features added in – list comprehensions [x*x for x in range(1,5)] and fancy arguments test(1,2,a=3,*c,**d).

Adding list comprehensions was painless, took only a few minutes. The arguments change was rather difficult because I had to add more rules to the parser, change how the bytecode was outputted, and rework all the internal calling stuff in the VM. Now every function takes a single argument of type ‘params’ which can contain all the details of a call.

I’ve also added in better error handling. Errors print out a backtrace of the function calls and lines of code. This is making debugging everything much easier.

I’m also finding that the more features I add, well, the slower things get. I’m also coming up against my 64k limit, so I’ve moved my tests out of the core code into a separate tests.py file to ensure that I don’t “trim down the tests” to give myself more space 🙂 The tests I’ve made have been invaluable in making it possible to do big changes fairly quickly.

Next on the agenda – either bootstrapping, or making a game with it 🙂

For the brave: svn://www.imitationpickles.org/tinypy/trunk ; python tests.py ; ./run_julia_o3 (not very fast)

I’ve managed to build a simple VM into tinypy – modeled after the lua VM. It’s register based and “stackless” in the “it doesn’t use the C-stack” sense of the word. (Not in the, it does anything fancy like “stackless python” does sense.)

‘ve just reached the 64k mark, so that means anything I add into the code will require me to clean up other code to save space. I’ve already done a bit of that with good results. The one rule I follow in shortening code is that the code must retain readability, if not improve it.

Garbage collection is rather complicated, so I think I’ve decided to continue to leave that to libgc. I read some literature on the matter, and it sounds like I could write one, but it probably wouldn’t be very good. I’ve put it at the bottom of my TODO.txt list in the section labeled “Probably not going to happen”.

At present tinypy supports basic python code with functions and loops and lists and dicts and classes. At least, in some basic form. I’m looking towards adding in list comprehensions and *args **nargs to tinypy as those are two of my favorite python features. After that I’ve got a handful of functions I want to write and then some packaging work.

For the curious svn://www.imitationpickles.org/tinypy/trunk – ./run_julia_o3 to see it all happen. It depends on python to compile the bytecode (we’re not bootstrapped yet), libgc, and SDL.

Well, things continue to come along with tinypy. I’ve been experimenting with a lot of different things with this. Some things I’ve tried:

– Assumed all infix operations were for numbers – was able to get quite a bit of speed at the sacrifice of things being “obvious”. I’ve disabled this for now, since I figure if you really want speed, you’d be using C anyways, not “tinypy”.

– Tried switching from passing around 16 byte structures to pointers to those structures. This didn’t work so well, because given my simple implementation it meant that any time any operation happened, another 16 bytes had to be allocated to store my floats. Not so hot for rendering fractals 😉 This caused a 10x slow-down. I reverted that out, but I copied that stuff to a branch for reference.

– I learned more about how libgc works. Basically, you need to use the GC_MALLOC and GC_REALLOC functions instead of the libc ones. libgc takes care of the rest. Also, if you integrate with libsdl (or some such library), you’ll need to do the memory management for it. There is hope – gc.h includes a function called GC_REGISTER_FINALIZER which makes it possible to trigger finalizers when an object is about to be freed. I’ve added some code to do this to tp.c, but haven’t tested it yet. That’ll get more testing when I start using libsdl for a game.

At present I’m debating between two options — leaving tinypy as-is and just cleaning it up, thus using it as a C code generator. Or changing it into a bytecode generator and writing a simple VM to run the bytecode. Advantages and disadvantages as I see them:

C code generator:

• Quite simple and reasonably fast. Optimizations and goodies come from the compiler.
• Really swell “FFI” (all functions are accessed directly and available as functions in C. Really, this means, no notable FFI at all.)
• No need to do much more than tweaking stuff, etc.
• No “eval” unless I use tinycc. Even then, no “safety”.

Bytecode + VM:

• I’d be able to implement exceptions and tracebacks and stuff
• I’d be able to build a “safe” environment for running scripts “eval”
• Things would be a bit slower, but not much
• FFI would be more complex (but I could probably work around that by auto-generating some goodies for people)

I’ll have to think about it for a bit .. I might just try making a fun little game with tinypy as-is, and work to keep the FFI really clean. And if I feel like upgrading tinypy, try to do it in such a way as to not break my “clean” FFI.

If you care to take a peak: svn://www.imitationpickles.org/tinypy/trunk – ./run_julia to see the julia demo. Must have libsdl and libgc installed. It’s sort of slow, but if you want to see it “fast” set JIT_DISABLE = False at the top of dumpout.py