If I were going to OSCON, I’d certainly go to this talk. But I’m not planning on it, so …

… So I’ve got that tinypy project sort of taking off. It’s got a SOC student (thanks google & PSF) and a bit of interest. So far most of the open source projects I’ve done haven’t generated more than a couple patches — in their lifetime. With that in mind, I’m not entirely sure what is a good way to accept patches and run a project, since I don’t have loads of experience in it. I’m a “solo” dev in my day job as well. I want to do a decent job of managing the tinypy community without having to work too hard.

Do I take whatever people give me and patch it as-is and just figure, well, someone else will fix it?
Obviously not. Especially not for tinypy where “good” and “concise code” is of such large value to the project. And with the SOC coming up, “secure code” is also very important, as well as testing.

Do I be really strict and keep rejecting a patch until it’s “perfect”?
Maybe? I remember submitting a patch to one project ages ago, and it got rejected without much good explanation. I’ve also submitted others successfully. Not quite sure what the balance is.

Do I take the patch and apply it and then fix things up myself?
Maybe? If it is easy to fix … but then again, why should I have to do all the work? I don’t really have that much spare time (all evidence to the contrary.)

How do I grow the project (in general) without it turning into a monster?
Stick to my guns on the 64k code limit*? Maybe not, since things like VC support, security, etc, are going to take up *some* bytes. Not to mention, tinypy could use a few batteries … At the same time, I’ve gotta draw the line somewhere.

Obviously some common sense is needed here, but sometimes I run a bit short on that. Advice / links to good articles would be swell at this point. I could google this sort of stuff, but I think having the context of responses from a community I know would be much more valuable here.

-Phil

* on that point, I think having the 1.0 source always in the “featured downloads” might be a good idea. Even if the project gets bigger, people can still check out the “classic version.”

For the sake of this post, I’m going to pretend to know what metaprogramming* is.  Yeah, so tinypy** totally has that.  At least, since the parser and compiler in tinypy is written in tinypy, you are able to modify those modules on-the-fly and add new features into the tinypy language.  (Not that you’d want to, but certain other languages get so uppity about being able to do that, I figured I’d plug for tinypy here.)

For example, (at present) tinypy doesn’t have support for decorators.  I’ve always liked decorators, so I made this code (a zip of main.py, deco.py***, and test.py) so that if you have a main.py:

import deco
import test

When the deco module is loaded, it cleanly**** adds decorator support into the tokenize, parse, and encode modules of tinypy.  Then when the test module is loaded, it is able to use decorator syntax.  Yay!  This mostly thanks to the top down operator precedence implementation in tinypy.

So now, if say, you have some crazy idea for how the $ operator should be used in bigpy, you can go ahead and use metaprogramming to add it into tinypy and show all your friends how awful your new syntax looks and have a working proof-of-concept!  Yay!

* feel free to enlighten me
** it’s got a mailing list now, join in on all the fun!!
*** only 611 bytes :)  They were pretty simple to implement, since they really just mean: “given ‘@a \n def b …’ do ‘def b … \n b=a(b)'”
**** Since all the language features are stored in dictionaries, it’s “pretty easy” to add new symbols / operators.  (Or remove features, or whatever!)

Yay!  Go to tinypy.org and spend the rest of your day basking in the glow of the new tinypy website!  Yipee!  Thanks to everyone who told me not to re-invent the wheel on this one.  I spent a large bit of the week thinking about doing that, but eventually I gave up the idea because I was feeling kind of worn out.  So I just threw together a google code and google group and slapped a website in front of it all.

If you’re interested in tinypy in any way, be sure to go to the site then navigate to the google group from there and join in the fun.  I suppose I could have direct links from this post, but then you’d miss out seeing my new swell site 🙂

Share and enjoy!
-Phil

So I’m getting the feeling I should host tinypy somewhere greater than just in my blog. What a pity, the blog suited me so well. I dunno, I guess some people have unreasonable demands. (If I were feeling clever, I’d link all the words in that last sentence to other people’s blogs who’ve bugged me about this. But I don’t think there are actually that many.)

Anyway, my options are as follows:

• use the project management software I’ve written for my health-care company. Pros: I wrote it, so it must be better than anything else. Cons: I’ll have to maintain it when I find out otherwise.
• use google code. Pros: People know how to use it, and it’s pretty clean looking and they host it for me. Cons: I don’t get to host it. But I guess I could set up the first few pages of the tinypy site on my own server to make myself feel good.
• use trac or something. Pros: I’d get to host my whole project myself. Cons: I’d have to host the whole project myself and learn some new software and maybe set up some stuff.

So .. all my aimless thoughts aside, since this isn’t really about me. (If it were, I’d just blog about it every now and again and leave it at that.) It’s about you, since there is interest in the project it needs to be easier for people to poke at. What setup do you think* would be the best?

* it probably doesn’t matter what you think, because after writing this post, I think I’ve already made up my mind. Writing down stuff like this helps me think things through. But since you’ve bothered to read this far, you might as well put in your 2c.

If you’re *not* in school, interested in tinypy, and want to help out, you can become a mentor!  I’m working via the PSF to get tinypy some SOC help.  Since I announced this a few days ago I’ve already got four students who want to do SOC projects on tinypy!

Reasons to become a SOC-python-tinypy mentor:

• you want to help out tinypy – if multiple projects get accepted, it would be great to have some people helping me out!  I don’t think I could mentor them all.
• you want to help out python – the more mentors python has, the more projects will be sponsored :)  This will increase the odds that tinypy gets some slots.
• you want to help out students – this is the Summer of Code, after all!  It’s a great opportunity to help new coders get some great experience.
• you want to help out open source – the SOC gets students involved in Open Source :)  Several of the students who have contacted me say this is the first time they’ve been interested in Open Source.

Qualifications for mentors:

• you aren’t a student.
• you know python and C reasonably well.
• you want to help out!

How to join:

• contact me, so I know who wants to be a tinypy mentor.
• join the python mentor’s mailing list.
• join the Google SOC mentors as part of the PSF.

Thanks!
-Phil

If you’re a student who’s been following the tinypy project and you’re interested in working on it for the Google SOC, the PSF is willing to consider tinypy projects for submissions under their SOC mentor group.

I’ve put together some ideas for projects here.  You would probably want to combine a few of those ideas together for your project.  Please contact me to talk a bit about your idea.  Thanks and good luck!

I’m going to pycon next week, yah!  I’m planning on sprinting on PyGame on Flash.  However, if someone wants to sprint on tinypy, just find me and let’s talk!  There are loads of cool things that could be done:

• JIT
• improved bytecode generation
• LLVM or luavm backend
• add your favorite module
• that module could be pygame 🙂
• speed improvement
• general code review
• bug fixes / tests
• documentation
• ???

Hope to see you at pycon!  I’ll be wearing Galcon t-shirts all week, so I shouldn’t be too hard to find.  🙂

Its been a long two months getting this project to the 1.0 state. Here it is:

tinypy1.zip | tinypy.tgz | svn://www.imitationpickles.org/tinypy/tags/1.0.0

$ python build.py
$ ./tinypy-sdl julia.py *
$ ./tinypy your-program-goes-here.py

The win32.zip includes the tinypy.exe and tinypy-sdl.exe binaries, so you can skip the building process.

Features include the ability to parse and compile to bytecode a pretty decent subset of python code. Its got a pretty simple API, incremental garbage collection, and a handful of builtin functions. The vm supports strings, dicts, lists, numbers, functions, methods, and custom data types. tinypy even has exotic features like list comprehensions, variable and named function arguments, inheritance, and exceptions with tracebacks. All this in just 64k** of code!!

I also had some fun this past weekend testing tinypy “in the field” making an OpenGL tetris knock-off. My apologies to Alexey Pajitnov for my misguided contribution of tinypy to the open source stack. win32.zip | svn://www.imitationpickles.org/ld105/trunk

I hope you enjoy checking out tinypy – I’ve had a pretty wild ride building it.

*julia.py included is a low-res “realtime” jula fractal demo. I felt like rendering a mandelbrot for this article.

**See README.txt for my definition of 64k. It’s close enough.

Updates – I’ve got it all fitting in 64k*. It’s amazing how many functions that don’t do anything you can come across if you look around long enough. Not to mention how many little things you can trim out that don’t actually do anything. I have no idea where all this cruft comes from, but having a nice suite of tests sure is helpful for re-working stuff. I also cut out a bunch of stupid features nobody would ever use**. I was able to reduce the number of native types from 9 to 7.

One of the challenges I faced was trying to fix up the incremental garbage collection. My initial implementation was rather inefficient and caused some odd problems with how I wanted to code things. I was using a dict to store all the “white” items, which caused loads of dict hash lookups.

So in my mind I crafted a grand vision on how to accomplish this goal. I would adjust all objects (sans numbers) to contain a pointer to some data which would have some header data for the GC to do some bookkeeping in. Great! However, when I implemented this, I found that a number of problems presented themselves: I had to perform a malloc for each and every string that I used, which killed performance, actually making things 2-3x slower. I also noticed that the weird struct I defined was maybe a bit less standards compliant. This attempt was a wash.

So I re-crafted my grand vision. This time I would do the same thing. Brilliant aren’t I? Anyway, the results were basically the same. Who’d’ve thunk? It was slower again, this time I was quite confused by it, since I had worked around some of the string issues. I also found that the API for creating new strings wasn’t quite as “clean” as my original simple one. This caused some issues in the exception handling mechanism. I had to toss this try as well.

At this point, having re-mangled the code twice and having poor results, I suspected something else might be wrong. My brain was turning into mush. Each time I had completely edited my “tp.h” with all my struct changes in one go. I decided to make a final attempt at reworking tinypy, this time *one* data type at a time. After each data type I added I was able to see if my changes caused any performance issues. I found that my function data type was the culprit. My hashing function (borrowed from lua) wasn’t getting enough entropy and was generating massive collision cases! A few tweaks later, this was resolved. I was able to also craft the string interface to be backwards compatible with the original string interface while also working with the new garbage collection. This “step by step” approach got me to my goal. All said and done, with a bit more tweaking, I was able to *double* the speed of tinypy 🙂

Lesson learned – even if it’s only 64k, it’s better to do changes step by step instead of in one big go. valgrind and callgrind are your friends. (Although I found that tinypy doesn’t entirely agree with callgrind … ideas anyone?)

To wrap up this excessively long post about me trying to get code to work — this weekend I’m hosting a Ludum Dare warmup compo. I’m going to give tinypy a run in the “real world”. Here’s to hoping! Next week I plan on releasing the 1.0 version of tinypy.

I’m also thinking about renaming some of my files. And although pylang, dumbparse, and dump2vm have a certain rustic charm, I wonder if I’d do better with names like goat, gorilla, and sausage. Or maybe more descriptive names like tokenator, parsalizer, and bytecodatron.

svn://www.imitationpickles.org/tinypy/trunk for the brave. If you want a zip or an exe, check back in a week. I’ll have all those and more (a game!) Note that I’ve split the SDL dependency out of the main tinypy code. tinypy-sdl.c lets you run my julia.py example. The bootstrapping process also has a final step of compiling with -O3, which I think might not work for everyone. It gives pretty good speed gains on my system, so if it works for you, great!

*python mk64k.py will do a bit of search-n-replace to cut it down to size. I’ve resisted doing anything really ghastly, the code is still indented and readable. See README.txt for more disgusting details on how I cheat to pretend this is 64k.

**Okay, I’ve used some of those features. But hey, this is a 64k implementation, I’ve got to trim the fat.

Yay – I got incremental garbage collection added into tinypy. It took a good deal of troubleshooting, but thanks to valgrind (and thanks to the people who pointed me to it) after some work I got all the memory leaks and crashes worked out. This is a nice step forward, since it enabled me to eliminate the libgc dependency.

tinypy is slightly larger than 64k, but I have vague hopes that will be remedied* at some point. In the meantime, I’m going to take a bit of a break from it. On Feb. 23rd, the Ludum Dare community is having a “warm-up” compo, and I shall give tinypy a real try at that point. After that I will probably put out the 1.0 release.

All that said, its been quite a learning experience for me. I learned how to tokenize, parse, generate byte code, build a virtual machine, and do garbage collection. I’m hoping this will make looking at the innards of other languages seem less intimidating in the future.

* if you’re a clever C / python coder and feel like taking a look at the tinypy code and give me ideas on how to compact the code a bit, I’d sure appreciate it. It’s 4k too big right now, and I’d really like to fix that. I’m cool with any suggestions though I won’t implement anything unless it leaves the code just as (if not more) readable. Please examine the README.txt to find out how I calculate the code size first.

svn://www.imitationpickles.org/tinypy/trunk or tinypy.zip for the brave. I’ve managed to compile it under mingw32 (Minimalist GCC for windows) as well as using GCC under MacOSX. Perhaps for the 1.0 release I’ll include binaries 🙂