The Coder's Buffet

A Dropbox Replacement

2012-02-07T11:58:00.000-08:00

So, I've just had an idea, and am not having much success finding it out there already. If you know of it, please let me know. What am I thinking of having it do?

This will need to do several things. First and foremost, it needs to support the same ideas as Dropbox: It syncs files across Windows, Linux, Mac, Android, and iPhone. It will allow people to share files with each other. It will also allow public sharing of files (such as photos).

The remaining pieces are where it gets interesting. I'd like the system to use a system like TrueCrypt. Using this, it will keep the entire dataset encrypted. It will also use many small files (on the order of 10M-100M), so that only an encrypted chunk has to be transferred, instead of the whole encrypted dataset. Doesn't sound like a big deal, until you start factoring in the ability to store photos. Imagine having to upload the entire dataset as one big chunk just because you rotated that picture of Uncle Jim where you were holding the camera sideways. 10G (or more) being uploaded, just for that, kinda sucks.

The backend storage is also different: Instead of uploading those chunks of data to a specific server somewhere, it uploads to a service utilizing Amazon's S3 protocol (obviously, Amazon itself is a viable choice for this).

As an added bonus feature, it should be able to commingle chunks from different S3 backends, and use that as a way to provide steganographic features. Using that, it would even be possible to have multiple layers of hidden volumes.

Another bonus feature: A destruct password. One that, when entered, will render the entire volume, and any hidden volumes, unusable. Considering the direction the government is going, this could be a very good thing to have.

An even better bonus feature would be having the TrueCrypt container dynamically resize as you add data, so that your space only goes as large as you need it to. With the idea of hidden volumes, this might not be feasible, but it would be nice.

Now, put all of this together, and you wind up with a service that is actually very easy to manage. It gives the security of TrueCrypt, the convenience of DropBox, and the mobility to use all of your data where you want and how you want.

Is this out there? Does anybody already have it available? Or is this something to be written at some point?

Wikis Suck

2012-01-01T18:11:00.000-08:00

There, I said it. They suck. And it's not just wikis; pretty much every documentation system I've seen sucks. I'm not going to outline a solution here, because I don't have one. I can only say that these things are pretty awful, and until we can find something that actually solves these problems, we won't have a truly great documentation system out there.

I apologize for the nature of this post. It's a problem I've seen for years, and been trying to find an answer to. I just don't have one, and the problem is only getting worse. Maybe somebody out there can help to refine this problem into something that an answer can happen. In no particular order, here's the problems I've got with wikis:

Organization. I have yet to see something really handle this well. A straight up hierarchy doesn't work. Too often, a given document belongs in two places (or more). Anything that enforces a hierarchy must require a choice of where to place something. On the flip side, when discussing a wiki, very few of them impose any sort of hierarchy. In fact, some of them don't even have a hierarchy at all, placing every item in the same "folder". This makes for a mess when it comes time to search for something. Tags can help in this regard, but they're still far from ideal. How do we make sure that a given article is sufficiently tagged? How do we make sure that the searcher knows which tag will provide the desired results?

Authoritativeness. Sometimes, you need documentation that you know the source. You need to know it came from the right person or team. Documentation that is crowd-sourced simply doesn't always cut it. On the other hand, sometimes you need to have areas where the crowd can update things. This is a balancing act that is very difficult to get right, and I'm not convinced that the open model of the wiki is the right solution. Simply locking a contentious page doesn't actually solve the problem of determining accuracy, and the lack of accuracy in those cases undermines authoritativeness.

Access Control. When dealing with something open to the public, you're going to get spammers and defacers who don't care about what you have to offer. They're just going to come in and update every page they can with their garbage, and you have to clean up the mess. Some wiki systems simply go with "allow the user to revert to a previous version of the page", which is not actually useful at prevention.

Cross-Referencing. This seems like I'm talking about the primary strength of wiki: It cross-references so easily it's amazing. I can reference something anywhere (within the wiki, outside the wiki) without much effort at all. And yet, it's still a major weakness. Cross-references are entirely manual across all of these systems. If I'm reading a wiki, and decide to make a new page for a given phrase, I can find myself surprised that the page already exists. I have no way of knowing it unless someone else does the work of making the link.

Supported Formats. They support one specific type of media: Specialized markup text using the markup style of their choice. Why can I not upload a PDF, or DOC, or ODT? Hell, any other format. And I don't mean as an attachment: I mean as a first class piece of data to be reviewed, searched, and edited.

wikis are impressive, and do a difficult job. I'm not convinced they do it well, though, and I'm sure I'll have other issues to discuss about them later. I just wish I had a solution, instead of a rant.

The closest description I can have is that the ideal documentation system would be automatically cross-referencing, supporting a variety of data formats, with an intelligent search that would help me find what I'm looking for. It sounds like I'm asking for some sort of local internet with a Google search. Maybe I am, but I don't think so.

I just don't know. Does anybody else have any thoughts?

Announcing TurboGears 2.1.4 Release

2011-12-12T20:17:00.000-08:00

The TurboGears team is proud to announce the release of TurboGears 2.1.4!

This release has many new features and bugfixes, all of them listed below. The most important one, though, is that this is the final 2.1.x release, and the final release that will support Python 2.4.

Please make sure to update your code to work on Python 2.5, as that will be the next supported Python version.

I would like to take a moment to thank Alessandro Molina especially. His work made this release possible, and as large as it is. He's been a great asset to the team, and I'm glad to have him on board.

Deprecated Python 2.4 support. In 2.2, Python 2.4 support will be removed, and only 2.5-2.7 will be supported
TurboGears extension architecture enhanced. Better support for hooks before and after configuration
Performance enhancements in tg module and in default templates
Enhancements to lazy_url
WebOb version locked in. The change of virtualenv to "distribute" by default has broken dependency_links and allow_hosts in the config files, and this works around that issue.
Jinja2 filters are now automatically loaded
Work arounds for best_match, which was not producing the expected behavior with IE7 and IE8
Add "auto_reload_template" as an .ini option
Performance tuning the default size of the Genshi cache
Added Genshi advanced i18n support
Better compatibility with SQLAlchemy 0.7
Changed default quickstart options to help ensure that some model is always available
Documentation enhancements
Nested RestControllers now work as expected (no longer required to implement "_custom_actions")

Worst Law Ever

2011-11-14T18:47:00.000-08:00

What would it allow to happen? If you make a site that allows a user to post *anything*, no matter how small, and some user decides to post a link to a random youtube video which includes copyrighted music in the background, *your* site could be shut down.

This could even shut down Facebook, Google+, and Twitter. Possibly going so far as to shut down LinkedIn, too. Forums, Blogs, anything where any user is allowed to post content, can be completely shut down without viable recourse for the person being shut down.

How about Google groups, too? Those can be killed. Once even a link to copyrighted content appears, a copyright holder can demand the shut down of the site that has the link.

I am astounded. This might be the single worst piece of legislation I've ever even heard as regards copyright and the internet.

Here's a link to a better writeup, and includes some action points. Please, people, take them. Otherwise, the internet as we know it will be dead in this country, and the economic issues that we have right now are going to be called the good old days within a year.

Request For Ideas for Hiring Pond

2011-10-18T19:44:00.000-07:00

So, a favor to ask of people: I'm working on a web application to help people manage their resumes. As I've gotten further in, I've realized I don't have an actual todo list for it. So, since I'm making this to be used by others, I'll ask everybody here:

What would you want to see? Currently, I've added enough code to allow the program to output something very close to my online resume ( http://www.icelus.org/ ). Next up, I have the following features on my todo list already:

add docutils to output strings where appropriate (everywhere?)
add login
add ability to edit own resume
add ability to compile new custom resumes
add ability to output PDF
add ability to output ODT
add ability to output DOC
add generic logo
add generic photo / head (shadow) shot
add ability to login using Google, Yahoo, Facebook, etc
add ability to send resumes directly from Hiring Pond
add ability to track where resumes have gone
add ability to track job hunt progress
add ability to display current employment status (employed, unemployed, contracting)
add ability to display current employment seeking status (not at all, active, passive)
add logo to qrcode
change colors of qrcode
change color scheme for all of hiring pond to sunset and water
add fisherman silhouette as hiring pond logo

What else would you all want to see in order to make you want to use this?

"What Do You Do, Anyway?"

2011-10-01T09:04:00.000-07:00

I get asked this quite a lot. What do I do? What value do I bring to a company? When asked, my answer is frequently "The impossible." I say this with a smile, and then explain "I do the stuff that everybody else says is too hard to do, can't be done, etc." and then go on to list some examples. If you want to see them, go check out my resume, and look at the projects at the bottom.

This answer, though, is very unsatisfying. It sounds like I'm bragging. It sounds like I'm saying I'm better than everybody else. It sounds, well ... it sounds arrogant. Since I'm on the job hunt now, I really shouldn't say this, but it needs to be said: I'm not that great, nor am I that special. I'm just a guy who happens to be passionate about his career, and doesn't shy away from some aspect simply because it doesn't fit with what I normally do.

This results in me doing such a variety of things that simply calling me by any one label is quite unrealistic.

Am I a system administrator? Yes, absolutely. I take care of systems. I maintain servers. I upgrade them, install software on them, troubleshoot problems (both hardware and software), and keep them running smoothly so that people don't get the chance to realize that problems happen. Yes, I'm a system administrator.

Am I a developer? Again, absolutely, yes. I write software. I design software programs on both large and small scale. I debug software. I profile and optimize. I analyze the libraries that are in use, and recommend upgrades and changes when appropriate. I perform testing of the software that I write. I work in whatever programming language (C, Java, Perl, Python, Javascript, etc) that is needed for the job. I work in whatever programming paradigm (procedural, object oriented, functional (my weakest one)) that is needed to solve the problem. Yes, I'm a developer.

How about a software tester, am I one of those? I devise plans for testing the entire program. Sometimes those can be automated, and I do so. Other times, they cannot, so I devise a manual plan for checking to make sure everything is working. When the software is being prepared for release, I execute those plans and verify that everything is functioning as expected. Yes, I'm a software tester.

The list goes on and on. I do some network administration, some configuration management, some release management, some build management, and frequently find myself solving problems that cross the boundaries between two of those functions.

This will seem irrelevant, but I promise it all ties together: this year, I finally truly understood the phrase "The buck stops here". We've all heard it. It's famous, and we can thank Truman for that. What we've missed, though, is what it means. If you adopt that phrase as your personal motto, it means no more passing the buck. Whatever comes across your desk is now your problem. It doesn't belong to anybody else: You must solve it. This year, when I took on my new job, that phrase came to mind, and I realized that I was entering a position where the buck stops with me. It's terrifying and invigorating at the same time.

Now, in recent times, I've begun identifying myself as an "IT Generalist". I do so many different things, and wear so many different hats, that labeling myself by any one title felt wrong. As the IT Generalist, the problems belong to me. My job is to turn problems into solved things that people say "Remember when that was so badly broken? Yeah, I'm glad we don't do it that way anymore." The buck stops here, with me.

And then we come to today, and the bit of reading I did that sparked this post. Browsing slashdot (yes, I read it, I still find it useful), I came across a post which mentioned the "Cult of DevOps". Digging in a bit, I read about it on Wikipedia, and found something that resonated with me. I understand the ideas, because I'm already living them.

It also seems too "insular", though. The reading I've been able to do so far focuses on the idea of internal software development for a company, whereas my personal experience also looks at the outside vendors for a company. When a problem occurs, I identify what's going on. Sometimes, that means reporting it to an outside vendor so they do a proper resolution while I find a workaround that allows my company to finish the work they need to do.

I think I'm going to stick with the "IT Generalist" moniker for a bit longer, but I think I can see myself joining the Devops movement. It strongly resonates with me and how I view my career.

I'm not just a system administrator. I'm not just a developer. I'm not just a software tester, nor just anything else. I'm all of these. I'm a problem solver.

Announcement: TurboGears2 2.1.3 Released!

2011-09-28T20:22:00.000-07:00

We are pleased to announce the latest release of TurboGears, 2.1.3!

This release adds support for the Mongo object database and the Ming ORM for it, in addition to bringing in some small bugfixes. The full changelog is below

The number of changes is, admittedly, small, but don't underestimate the importance: NoSQL databases are an important tool for the web, and adding the support for Mongo brings TurboGears into the group of frameworks that not only supports it, but supports it well.

We're already looking forward to expanding support and closing even more issues for 2.1.4, so we'll see you next month for that release!

* Added support for MongoDB into the quickstart template
* Added lurl (lazy_url) support for late evaluation
* Adding tests module back into source distribution. Solves ticket 115
* Template caching is now manageable by TG applications
* Clean up output for flash() in default quickstart
* Performance enhancements
* Fixed missing dependencies in TG quickstart on Python 2.4

TurboGears and Ming/Mongo

2011-09-18T20:28:00.000-07:00

I know that my followers are few, and my posts are too infrequent. I'm looking for ways to change my posting frequency.

But, even still, I have an important request: TurboGears 2.1.3 is getting ready to be released. We're adding Ming/Mongo support in this release. Anybody who can, please review the changes (core, devtools, and docs), run tests, etc. We need to make sure we've got this as correct as possible for everybody.

Thank you for your time!

Announcement: TurboGears2 2.1.2 Released!

2011-08-24T19:58:00.000-07:00

We are pleased to announce the latest release of TurboGears, 2.1.2!

This is a maintenance/bugfix release for the 2.1 series which also adds some new features. The full changelog is below.

Highlights for this release include the beginnings of our new book "TurboGears2 Web Applications: Batteries Included", Python 2.7 support, improved Jinja2 support, and quite a few bug fixes.
You can see the book as it stands right now at http://www.turbogears.org/book/ . We're working towards getting more details and more of the older documentation updated to go into the book.

We took longer to reach this release milestone than we had planned, but Real Life(tm) takes its toll on everybody during the summer months, it seems.

Even still, I can speak for the whole team when I say that we look forward to continuing to serve the community with quality code, and we look forward to your continued support!

Full Change Log Here:

For TG2.x Core:

Improved content-type handling ( https://sourceforge.net/p/turbogears2/tickets/73/ )
Fixed dotted filename support for Genshi and Chameleon.Genshi
Removed lxml dependency; now preparation of virtualenv is much easier on 64 bit systems
Upgraded Chameleon.Genshi support to Chameleon1.3rc
Set up render_genshi on demand only, improves Google App Engine support
Support for Python 2.7
Improved Jinja2 and Jinja2 filters support ( https://sourceforge.net/p/turbogears2/tickets/107/ )
Added option to prevent auto-saving sessions ( https://sourceforge.net/p/turbogears2/tickets/83/ )
Fixed crash where multiple templates were registered and user called override_template ( https://sourceforge.net/p/turbogears2/tickets/82/ )
Fixed problem with ":" in error messages ( https://sourceforge.net/p/turbogears2/tickets/86/ )
Improved i18n support
Improved pagination support ( https://sourceforge.net/p/turbogears2/tickets/99/ )
Fixed warning resets when in debug mode ( https://sourceforge.net/p/turbogears2/tickets/84/ )

For TG2.x Devtools:

Improved Genshi support in quickstart template
Improved content-type handling ( https://sourceforge.net/p/turbogears2/tickets/73/ )
Improved bootstrap if database is under version control
Changed default password hashing to SHA256 ( https://sourceforge.net/p/turbogears2/tickets/94/ )
New "--minimal" quickstart template
Separated password hashing mechanism to make it more easily usable ( https://sourceforge.net/p/turbogears2/tickets/67/ )
Fixed Unicode return type in auth.py ( https://sourceforge.net/p/turbogears2/tickets/89/ )
Added missing dependencies for Python 2.4 and the quickstart app ( https://sourceforge.net/p/turbogears2/tickets/100/ )

For TG2.x Docs:

Introduced new book beginnings
Rewrote 20 Minute Wiki, brought it forward to working with TG2.1.2
Rewrote installation instructions, much simplified
Began description of new application for tutorial/book
Improved documentation on how to contribute
Made all standard modules autodocumented in the appendices
"ignore_parameters" documented ( https://sourceforge.net/p/turbogears2/tickets/15/ )
"tg_avoid_touch" (method to avoid saving sessions) documented ( https://sourceforge.net/p/turbogears2/tickets/83/ )
Updated references regarding _default and _lookup ( https://sourceforge.net/p/turbogears2/tickets/5/ )
Documented custom filters for Jinja2
"migrate" command is now documented ( https://sourceforge.net/p/turbogears2/tickets/106/ )
Documented pagination parameters
Documented "--minimal" quickstart option ( https://sourceforge.net/p/turbogears2/tickets/102/ )
Changed references to "aptitude" to "apt-get" ( https://sourceforge.net/p/turbogears2/tickets/104/ )

Announcement: TurboGears 2.1.1 Released!

2011-06-15T20:21:00.000-07:00

We are pleased to announce the latest release of TurboGears, 2.1.1!

This is a maintenance/bugfix release for the 2.1 series. The full changelog is below.

This release has been a long time in the making, and I would like to take a few minutes to thank the people who made it happen.

Mark Ramm, Chris Perkins, and Kevin Dangoor, for supporting me in beginning this whole process back in January.

Alessandro Molina, Christoph Zwerschke, and Diez Roggisch, for contributing much of their precious free time to writing code, making suggestions, and keeping me on track.

Florent Aide and Jon Schemoul, for providing us with a server and tgext.pages to make the site work.

And to everybody in the community that I did not name: Without you, I don't think I would have kept myself working. The results are here, and they belong to you as much as any of the people I named above.

Thank you, everybody. Your support and help mean a great deal to me. I only hope I've lived up to some of your expectations.

I can speak for the whole team when I say that we look forward to continuing to serve the community with quality code, and we look forward to your continued support!

And I know I can say it most strongly for myself. We're back on the move. And I'm liking the results.

Full Change Log Here:

TG2 Core:

Fixed dependencies for Python 2.4. Now any packages that are needed are automatically installed.
Updated package requirements as high as possible.
Verified nested RestControllers work as expected
Added/fixed Kajiki support
Ignore repoze.who_testutil when running nosetests
Fixed import order for pylons.middleware
Fixed crash when PYTHONOPTIMIZE is enabled
Report a warning about ErrorMiddleware is disabled
Fixed concurrency issues with use_custom_format
Fixed 404 errors if a controller uses only custom formats
Verified that user object is available inside of the error controller/template
Fixed expansion of arguments on before/after calls
Fixed wrong header response for 405 error
Fixed WebOb version requirment. Newer version required
Added test case to check for replace_header when called from WSGIApp
Fixed issues with Content-Type header appearing multiples times on 204/205 responses
Removed redundant hasattr checks on override_template
Improved support for pylons 1.0 strict_c
Fixed post traceback, now reports to Pylons correctly
Added test case to check for spurious content type removal on empty content
Fixed crash when content type header is missing
Fixed crash when response Content-Type is set to None
Fixed support for etags. Pylons 1.0 changes slightly, we support the correct version now
Added dependency_links and setup.cfg allow_hosts: easy_install TurboGears2 now works
Fixed DecoratedController. should not call super(), 2.6 revealed a problem
Fixed Genshi output method. Use XHTML if none specified, instead of XML

TG2 Devtools:

Fixed Python 2.4 compatibility issues. Dependencies are now automatically specified
Updated package version requirements as high as possible
Fixed about.html instructions about where the logo is found
Set "zip_safe=False" by default in the templates now
Tests fixed, now pass
Added support for sqlalchemy-migrate
Added option to choose config file
Added archive_tw_resources command for projects
Fixed deprecated redirect calls
Set Genshi templating method by default to XHTML
Adding dependency_links: easy_install tg.devtools now works

TG2 Docs:

New book doc project started.
Added note about how/why to set zip_safe to False
Fixed import statement in some examples
Fixed spelling errors
Fixed wiki tutorial errors about bootstrapping, also the missing setup.py
Removed dead references in modwsgi deployment documentation
Fixed warnings. None remain right now
Implemented new theme donated by Christoph Zwerschke
Added documentation on how to extract ToscaWidgets resources
Added tutorials for DataGrid, pagination, sorting, and custom columns

Announcing TurboGears 2.0.4

2011-06-04T20:55:00.001-07:00

We are pleased to announce the latest release in the 2.0 series, TurboGears 2.0.4.

This is a maintenance/bugfix release for the 2.0 series. The full changelog is below.

Note that, unless security issues are found, this will be the final release of the 2.0.x line. As 2.1 is fully backwards compatible, and release of 2.1.1 is imminent, we strongly encourage everybody to upgrade as soon as possible.

We look forward to continuing to serve the community with quality code, and we look forward to your continued support!

The following changes were added for 2.0.4:

Python 2.4 compatibility was fixed.
Nested RestControllers had a bug preventing the proper
resolution of the "delete" method. This has been fixed.
When all @expose'd methods of a controller had a
CUSTOM_CONTENT_TYPE, a 404 would be returned. This has been fixed.
Problem existed with getting tuples into controllers. Patch from
bug 37 applied to fix this.
Brought the current TG2 private PyPI current. Getting installation
to work is now as easy as "easy_install TurboGears2==2.0.4"
Got automated testing working consistently.
Added tests for mixing positional args, keyword args, and pagination
dependency_links is now in place to make it easier to get the initial
setup done
Several other smaller bugs were fixed.

On Problem Solvers and Problem Throwers

2011-04-22T13:04:00.000-07:00

People have problems; some of them very large, some of them very small, but everybody has problems that have to be solved. In many ways, this is the very foundation of business. When somebody has a problem, and (for whatever reason) he is unable to solve it himself, he can find someone else to solve it for him, for a price. Along with that comes the needs of the problem solvers, too.

When a business is formed to solve a problem, new problems get found to be solved. For instance, a retail store solves the problem that many people have: "How do I get the stuff that I want?" That store, though, has to deal with problems of "How do we get the stuff that people want to buy?" and "So, we're using computers. How do we keep them running properly?"

That last question is where I make my living. My career is helping people get the most out of their computers. I take this very seriously, much more so than I should, maybe. To that end, I feel I have earned the title of "Problem Solver." I work with people to understand the problem, and fix what I can. If I can't fix it, I find ways to help someone else fix it, and make sure the original problem gets solved.

I know other problem solvers. They do the same thing. Together, we fix problems, large and small. Sometimes, we fix problems that are not strictly what we were hired to do, but need to be fixed. As one example, I was not hired to move furniture. When the time came to move office buildings, I was there, ready to assemble desks and chairs if need be.

Now, to some degree, we are all problem throwers, too. If ever you've said something like "I can't" or "Not right now" or "Joe over there is better able to help you", you've thrown a problem at someone else to resolve. The difference is really in the way you handle problems.

If you view problems as belonging to you until someone else takes over, you're more of a problem solver. If, on the other hand, you work to prove that the problems should be handled by someone else, then you're a problem thrower.

What I have been finding more and more is that problem throwers are problems themselves. For instance, one of my coworkers has shown himself to be a problem thrower. We began experiencing a problem today with a specific machine connecting to our mail system. If we changed its IP address, it would work fine. When we tried the problem IP address on a different machine, the problem moved to the new machine.

All of that put together indicates either a network device between the workstation and the server (since both workstations we gave that address to are attached to the same switch), or a problem on the server. I asked the network engineer to double-check the network devices, while asking the sysadmin to double-check the server and verify no problems.

The replies I got amounted to "The problem can't be on my equipment, since we never set it up that way." No verification, no checking, just "Nope, not my problem." Problem throwers.

I have neither time nor patience for problem throwers. Most people don't have the time for it, either. I hope none of you are one, and if you are, please reconsider. We're all trying to ensure that the people we're taking care of are actually taken care of. When you throw problems out there for someone else, all you do is make more problems for everybody else.

A Git Script for Python

2011-03-09T07:27:00.000-08:00

Working on TurboGears, I found myself trying to run all the tests we provide. And then I found myself getting a failure even though I didn't change anything. I had simply switched to a different branch.

Turns out the problem was left over .pyc files. Also turns out that the fix is extremely easy. In your git repository, create a file named "hooks/post-checkout". In that file, put the following lines, and make the script executable. All .pyc files will be scrubbed every time you switch, preventing the problem entirely.

#!/bin/bash

find . -type f -print | grep -v ^.git | grep '\.pyc$' | xargs rm

Eclipse: Great or Overrated?

2010-12-14T10:26:00.000-08:00

I've been working on integrating Openfire and vBulletin, to allow a vBulletin based community to have a full fledged instant messaging service. It's been challenging at times, but mostly fairly easy. I'd already done some work learning how to authenticate with a vBulletin community. My biggest problem has been, surprisingly, Eclipse.

I've been actively coding on this since, roughly, the beginning of November. I followed the documentation from the Ignite Realtime site to configure my Eclipse workspace. This was my first time using Eclipse, so I had nothing else on my machine to conflict with it.

In the time that I used it, I found my workspace corrupted so badly that I had to destroy the entire workspace, along with my Eclipse configuration files, and reconfigure from scratch, just so that Openfire would run again.

Now, it's easy to believe that I was misusing the tool. I may have been. So far as I know, though, I was not. I loaded files. I edited the files. I saved the files. I compiled the project. I ran the result. I never used the class explorer, never figured out why the autocomplete would show results when I didn't care (and not help when I did), and never ran the debugger.

Yet, my workspace got corrupted. So, I rebuilt from scratch. The same thing happened again, and I rebuilt from scratch again.

The third time, though, was the killer. Openfire has a web interface. I could no longer access it. I kept getting NullPointerException errors. I rebuilt the whole workspace from scratch. Same problem. I deinstalled Eclipse entirely, reinstalled, and built from scratch again. Same problem.

I gave up, and went to Emacs. After a day, I had a simple configuration file that allowed me to build and run Openfire successfuly. It even worked across multiple machines with minimal effort.

I've long known that people love Eclipse, and consider it to be the default way to build Java based applications. I'm not one of those people though, and am not going to bother anymore. It's just not worth it.

For me, at least, Eclipse is very overrated.

Making an MP4 Video

2010-11-24T11:12:00.000-08:00

I've been working on switching any/all of my older videos from their original format (.mov, .avi, etc) into an .mp4 container. This will let my PS3 stream the videos, allow my MythTV setup to stream them, and even allow my phone to do the same. It's pretty nice, actually.

Of course, getting to that point was more than a little painful. It seems that the whole process is spottily documented (at best). In the end, I did manage to make it work on my system. While this won't work for everybody (since it's written for Ubuntu 10.10), hopefully it will help out somewhat.

Steps:

Install bc and mpeg4ip-server (for the command mp4creator). These packages come from Ubuntu's standard repositories.
Install mplayer from Medibuntu
Download this script (syntax highlighted version here) into a file named /usr/local/bin/androidmp4 and make that script executable

Now, you can simply run "androidmp4 video_file_name", and wait. You will be given a file that can be played on at least your PS3 and the Samsung Galaxy S series of phones.

Announcing gogo, The Bash Project Switcher

2010-09-14T18:32:00.000-07:00

If you're a developer, then you've faced this problem: When you switch between multiple projects, you have to switch your shell, your editor, and possibly a lot more. If it's time to start a new project, then there's all the busy work of making a new project.

Those problems are now ending.

gogo is a set of bash functions that will enable you to more easily switch things around. Since I wrote it (first) for me, it supports my most common needs up front: Python, Mercurial, Emacs, and TurboGears. The only piece that is not trivially swappable is Mercurial. I'm sorry, that's on my todo list, but I wanted to let people see it and maybe start getting some value from it.

I highly recommend checking out the README file. It includes instructions on getting set up, along with instructions on how to use the tool. For now, a quick summary of usage:

Make a new project: gogo -n myproject
Activate the new project (after having closed your current shell): gogo myproject
Make a new Python project: gogo -n -l python pyproject
Make a new TurboGears project: gogo -n -l python -t tg21 tgproject

Switching between projects is downright trivial. gogo even supports tab completion of project names. It's already helping me, and hopefully it can help you.

You can download the v0.4 release, or clone the latest repository (or even download a .tar.gz file of it).

Steal This Idea: Writer's Tools for Web and Android

2010-08-09T22:41:00.000-07:00

Okay, yes, I have an Android phone, and I like it quite a bit. This idea would allow a story author to maintain notes on the phone/tablet and synchronize them with a web site. More than that, though, it would include features to help writers keep their ideas organized. A good example of what I have in mind is something like the Writer's Cafe, but written for the web and Android, instead of being only a local app. Even though Writer's Cafe is very good (and it really is) being only a local app does make it harder to use than I would want it to be, personally. It also doesn't support all of the things I would want to see supported, which (in particular) includes the support of audio and video notes/clips. Using Android and Web should make these items considerably easier.

Python, Soap, and Stub Libraries

2010-08-06T12:57:00.000-07:00

Last week, at work, I came across a new problem for me. Specifically, I was working out how to submit information to a set of SOAP servers from a Python program. This was supposed to be the easiest part of the whole project. It turned out to be the hardest. Why?

As it turns out, SOAP is not very popular amongst the Python community. We have fantastic support for XMLRPC, but SOAP support is surprisingly difficult. At best, it's incomplete. At worst, you have to pick your SOAP library just for compatibility.

After spending a fair bit of time Googling, I found a reference to a SOAP library called "Suds". This looked to be the simplest way to get myself speaking to the SOAP servers. Suds works, and it does things properly. Unfortunately, the SOAP servers I needed to speak to do not do things properly.

SOAP uses a WSDL file to tell clients what procedures are available and how to call them. The servers I needed to speak with did not provide a WSDL file. Instead, we have a static WSDL file, and from that we are supposed to be able to make a stub library, and using that we can submit jobs. This mechanism did not sit well with Suds at all.

After digging, I did find the WSDL file on all the SOAP servers. I thought I was set. I began submitting information, only to get way too many errors back. Digging more, I found out why: I was only submitting to one SOAP server, not all of them. The WSDL file on our servers hard codes the URL, so that it points to the first server in the group. I had to make a custom WSDL file for each of the servers.

Fortunately, that turned out to be relatively easy. I just did a simple string replacement in the incoming WSDL file (so that the URL always pointed to the correct SOAP server), saved it to disk, and pointed Suds at the server-appropriate WSDL file url ('file:///tmp/my/wsdl/file'). Suds parsed it correctly, and sent the information to the appropriate SOAP server.

If I had the time, I'd like to help take Suds to feature completion. It looks like a nice library. I don't, though. I will, instead, be grateful that they've done the hardest work, and that I could use their library with my problem. It did solve it, and fairly easily, too (especially in comparison to having to write my own code).

Steal This Idea: Smart Phone Becomes Universal Remote Control

2010-08-02T08:08:00.000-07:00

This one will take a bit more work than the other ideas, mainly because this one will require some hardware work, too. I want to see my Android based phone able to act as a remote control for all of the devices in my entertainment system. This means Bluetooth, UPnP, DLNA, and IR. The first three are pretty easy, but the last will require the creation of some sort of IR transmitter than can accept commands from my phone, and turn them into IR commands to control my other devices. It should, of course, be fully programmable, learning, etc. Noticeably harder to accomplish, but I think well worth the effort.

Steal This Idea: Electronic Emergency Preparedness Kit

2010-07-30T21:51:00.000-07:00

When Hurricane Katrina hit New Orleans, I remember reading somewhere about a person who recommended a full electronic version of important documents, stashed them on a USB key, and stashed the USB key into his emergency kit. I'm looking at my Android based smart phone and saying "Wait, why not use this?" I'd want any program to have top-notch encryption and allow me to store any images I choose. This would allow me to store mine (and my family members') driver licenses, marriage certificate, home ownership papers, birth certificate, social security cards, everything. Of course, all of these would need to be encrypted. This would help to ensure that, in the event of an emergency where I lose all of those originals, I could use the information on my phone to at least find out the things I need to get them replaced, and restore my life.

Announcing the Availability of tgext.menu v1.0b3

2010-07-26T20:21:00.000-07:00

tgext.menu v1.0b3 is available for immediate download. It can be found on PyPI in an easy_install'able form. The source may be found at BitBucket. This release now allows specifying the icon for a menu entry in the decorators and functions, fixes an issue where an id could be duplicated, and allows for replacement of the default template.

Steal This Idea: Shared SmartPhone Shopping List

2010-07-23T13:03:00.000-07:00

I've been using some form of PDA type device for a long time now. I used to use a PalmOS based device until last summer, when I switched to Android. On PalmOS, I had HandyShopper, and it was a shopping list done right. I would love to see HandyShopper done for Android, but with one additional tweak: It should have the ability to synchronize with a Google Docs spreadsheet. This would provide a nicely open way for someone to maintain a shared shopping list with someone else on a completely different phone. GroceryIQ comes very close, but it suffers from some speed issues.

In Praise of MythTV Developers

2010-07-21T22:16:00.000-07:00

I use MythTV, and have done so for a very long time. I think that, right now, we're looking at about 7 1/2 years. I have no intention of giving it up, it's just too useful.

Just two weeks ago, I got my new phone, a Samsung Galaxy S. Beautiful phone. It supports using DLNA to stream video. DLNA, it turns out, is a subset of UPnP. MythTV, it turns out, has had a UPnP server built into it for a long time.

So, in theory, I should be able to stream from MythTV to my phone. The reality wasn't so great. I tried, sproadically, for about a week. Then, on Monday night, I decided to get serious. I did all the groundwork to find out what was going on, and found out that MythTV had a problem.

I submitted a bug report, and included all of the details that I knew, including purpose, what I had done, everything. Tuesday night? By the end of the night, we had a working patch for MythTV that will be included in their next release. That same patch works against the version that ships with Ubuntu.

I can stream from MythTV to my phone without a problem. And the MythTV folks fixed my problem within 24 hours. That's seriously amazing work guys.

Thank you. May you continue to enjoy the success you have so far, and may ever more of it come your way.

Steal This Idea: CarPC with Android and DLNA

2010-07-19T15:03:00.000-07:00

Lots of folks have their own Car PC built and doing things already. This, though, is slightly different. Use an Android tablet as the base. Give it some extra storage somehow, so it can hold videos, music, and the like. Hook it up so that it can read OBDII from the car. If the tablet can handle its own internet connection via a SIM card or some such, do so, or else have it use a tethered phone of some sort for on the road. Hook the tablet into environmental controls for the car. Do some hardware hacking with antennas, and make it so that the tablet can positively identify the location of all phones in the car. allowing people to customize their environment as much as possible. Ensure DLNA support, and get DLNA support on all the phones (Android, iPhone, etc) to allow people to share their media wirelessly with the main tablet and with each other.

My Issues With Perl, Redux

2010-07-16T11:08:00.000-07:00

I recently had a conversation with a colleague about Perl, and I kind of screwed up my explanation of why Perl drives me nuts. He started talking about Perl's object model, and I tried to explain why Perl doesn't really have one, and things went downhill. I've been thinking about it since then, and my single biggest issue is not with the object model, nor the ugly syntax. No, my issue is with what happens when a given function/method gets called by something.

To explain why this bothers me, we have to compare some code snippets from a couple of languages.

In C: void foo(int bar, int baz, char* hello, char* world, char** argv) {};

In Python: def foo(bar, baz, hello, world, argv): pass

In Perl: sub foo { my ($bar, $baz, $hello, $world, @argv) = @_; }

At first glance, all of these segments appear to be equal to the developer. After all, each of them accepts the same parameters, in the same order, and the same type of parameters being passed in. Neither Perl nor Python have the strong typing that C does, so neither of them actually needs to specify what data type is being passed in at what position. So, all of these look the same.

Take another look at Perl, though. In C and Python, the parameters are given names by the function prototype. I have no code in the body of the function at all. With perl, though, I have to write a line of code to explicitly name these parameters. If I were to rewrite the Perl version to do the exact same function body as the C and Python versions, I would get this line of code:

sub foo {}

Now the difference becomes more apparent. C and Python actually have an explicit list of parameters being sent to the function. Perl doesn't have anything being explicitly passed into it. So, what's the single line of code doing? Why does it matter?

my ($bar, $baz, $hello, $world, @argv) = @_;

In Perl, when a function is called, all of the parameters are pushed into a single list. That list is then made available to function in a specific variable: @_. Now, this can (and does) work well enough most of the time. It does have some gotchas in it, though, and these are where my major issues come from.

Perl will only pass a list of scalars to any given function or method. Nothing else. In Perl, a scalar is a specific, small, value. A number is a scalar. A string is a scalar. A pointer (called references in Perl) is a scalar.

So, what happens if you try to pass a list to a function? All of the elements of that list are appended to the list of parameters. To continue with our code examples above, the following code segments produce the exact same effect in Perl:

foo(10, 20, "hello", "world", (1, 2, 3));

@vals=(10, 20, "hello", "world", (1, 2, 3)); foo(@vals);

And, even more confusing will be this example:

sub foo { my ($hello, $world, @vals)=@_; } foo("hello", "world", (1,2,3), (4,5,6));

The value of @vals in that example is (1,2,3,4,5,6). A single list, containing all of the elements of both lists at the end of the call. If you are using a hash (associative array), it gets even more fun:

sub foo { my ($hello, $world, @vals)=@_; } foo("hello", "world", ('1' => '2', '3' => '4'));

@vals is now just a list instead of the hash that was passed in. Depending on the order Perl adds to the stack, the list could be either (1,2,3,4) or (3,4,1,2).

Okay, one last example to show how messed up this is:

sub foo { my (%vals)=@_; } foo("hello", "world", ('1', '2', '3', '4'));

Now, all of the parameters that were passed in have been munged into a single hash. The keys of that hash are "hello", "1", and "3".

Yes, Perl can be made to work. People have done so, and have even done so with very large projects. That does not mean it is a good programming language. That does not mean it is easy. Nor does it mean it is an easily maintained programming language. With the issues that I have shown in passing parameters around to functions/subroutines/methods, I can think of only one other programming language that makes it so difficult to be consistent in how things get done, and that's assembly.

I found myself tempted to add a derogatory remark, but I don't think that's really necessary. When even such a (usually) disliked language as C manages to get something this basic correct, and Perl doesn't, I think enough has been said without adding the snark.

TurboGears, SQLAlchemy, and Parent/Child Relations on the Same Model

2010-07-15T10:00:00.000-07:00

Best thing to do is to read this post. I'm just posting it here to try to help it get more exposure to those who might need it.

http://www.5dollarwhitebox.org/drupal/node/110

Steal This Idea: White Board

2010-07-15T09:17:00.000-07:00

I want a simple whiteboard. The problem with most paint applications of any sort is that they attempt to grow into something truly grandiose. The problem for me is that, sometimes, I need something that is extremely simple, and a whiteboard is about as simple as it gets. Using a real world one, I can pick a marker color, or an eraser. That's it. I want a virtual one that provides me just as simple a set of options. I want it on the web, I want it on the local desktop, and I want it as an Android application. The only add-in features should be an ability to change the background color, maybe some sort of lines (like ruled paper, or grid paper, or hexagons), and some way to save and search for them. Anything more than that, and it becomes way too complicated for rapid fire drawing of an idea, and that is where the whiteboard will always win.

Announcing the Availability of tgext.menu v1.0b2

2010-07-12T21:59:00.001-07:00

This is a minor update. It contains new installation instructions due to testing failures right after installation in a new quickstart, along with a fix for a menu display bug with jdMenu, and some new API enhancements from bitbucket user scottawilliams. It is already available on PyPI, and you may upgrade as soon as you are ready.

Steal This Idea: Media Library

2010-07-12T11:26:00.000-07:00

I have a lot of different media in my life that I want to track. I have document files of all sorts (word docs, text files, pdf files, epubs, etc), photos, videos, and music. I want to be able to organize them all in one place, tag them all, search them all, and display them all. I want it to be a web interface, and I want to be able to set up my own server hosting it. I want the files to be something that can be read by way of a plugin library, so that I can easily add support for unknown formats in future. I want it to support external sites using it (so that I can tell youtube to pull from it, or tell my ebook reader to pull from it, etc). It should support OPDS for textual information (epub, html, mobi, pdf, doc, etc), and Gallery 2's remote protocol for visual media (videos, photos). Since I'm a Python and TurboGears fan, I'd love it to be written in TurboGears.

Unit Testing Redux

2010-07-11T19:48:00.000-07:00

I've mentioned unit testing before. I've extolled its virtues before. I'm not going to go over those items today. Instead, I'm going to go over how it saved my project at work.

Back in March, I proposed a new idea at work. My idea, when put into production, would actually make money for the company. I don't mean it would save money, either. I mean that we would be able to actually properly bill customers when they go over their allocations/what they're paying for. Right now, this is, at best, a difficult process. As a result, it is frequently ignored. With this project I'm building, it would now be able to become how we do business. My project will actually make money for the company, not just save it.

Unfortunately, I screwed up in my early testing and coding. I created a situation where an M:N join table was not properly being joined, because I missed a test case. The results were looking correct for the test cases I had, so I assumed things were okay. As it turns out, they weren't.

The system is in QA right now, and someone found a bug and reported it to me. After digging in, I found the problem, and realized I had missed this test case. My initial reaction was bad. How could I ever check this? How could I possibly prove that this bug was fixed, the fix didn't introduce new ones, and that the system was in good shape?

The answer comes from my already extensive test suite. I added some more test data to it, and exposed the bug. Test cases were now failing, the way they should have. I then fixed the join condition, and ran my test cases. I had some new failures, but that was because I failed to account for the new test data in my reporting test cases. I accounted for the new data in those test cases, and now everything came up green. Everything was working as expected.

Now, how bad was this problem? Without those test cases, I would be unable to be sure that my fix actually fixed the problems. I would be unable to be sure that bugs didn't exist elsewhere, waiting to be detected. I would not know that the problem was resolved. I would believe it, but I would not know it, not for sure. And with that level of uncertainty, my management might have either killed the project, or made me start over.

With those test cases, it became a line item in a status report: Bug in join condition reported, found, fixed. Tests now look for this problem, and all tests now pass.

I don't think I'm exaggerating when I say that unit tests saved my project. People laugh when I tell them I spend as much time on my tests and documentation as I do on my code. Now, I'll just point them to this story. I am so glad I found out about unit tests right now.

Steal This Idea: RPG Assistant

2010-07-10T22:00:00.000-07:00

This is just the first post of many I will make like this. I've long had in mind an RPG Assistant. It should allow storage of all the information about a given campaign world, including maps, creatures, characters, plot hooks, current events, everything. It should be platform agnostic (using something like wxWidgets, wxPython, Qt, Python-Qt, whatever, and you will maintain easy cross-platform compatibility with Windows, Linux, and Mac). A really nice addition to it would be to have an Android version compatible with the other versions. It should also be game system agnostic, so that any version of D&D can be put in, along with GURPS, TMNT, Rifts, Torg, Fudge, etc. I hope to be able to write it up someday, I really do. But, if not, I'll just be happy seeing it actually happen.

TurboGears and Amazon EC2 Benchmarking, Part 4: Conclusions

2010-07-10T21:55:00.000-07:00

Reviewing my methods, and the results I got, I found out quite a bit about TurboGears. I learned about how it will perform in real world scenarios, and I learned how to figure out the level of hardware I will need to accommodate a community. So, what does this mean for you?

For TurboGears, this is about answering the question: "How well does this scale".

The results of that question cover a very limited set of variables: How much hardware do you need to scale? What version of TG provides the best performance? Is Genshi or Mako the better performing template engine?

That's it. No more, no less. By and large, the numbers speak for themselves. The only gotcha in this is that those numbers are for all the dynamic pages. They do not cover the static pages, and that is done deliberately. I configured Apache to handle serving static CSS and JavaScript files. You get the full speed of Apache when the files are static, and then you get the full benefits of TurboGears for all your dynamic pages.

If you need to deploy and scale your TurboGears application, look at those numbers, use the definition of an Amazon EC2 Compute Unit (along with how many each instance provides), and then comes the hard part: You must estimate how many dynamic pages are going to be viewed at a time. I would then leave an additional 50% capacity, just as a buffer.

So, if you expect your site to attract, at peak, 10 dynamic page views per second, and you're on TG2.1b2 with Genshi, then you can get away with the smallest instance. The requirements sound like you need something huge, but you really don't. Remember what I said in yesterday's post: 864,000 dynamic page views per day on something that small. That's a lot of page views, and will support a very active community.

I'm glad to be working with TurboGears. Lots of power, lots of scalability, and it's all built in. I just have to use it.

Now, as I close out this series, I'd just like to say thank you to the developers who work so long and hard with TurboGears. They make things like these posts possible.

TurboGears and Amazon EC2 Benchmarking, Part 3: What I was Testing

2010-07-09T08:43:00.000-07:00

I actually learned a few things, some of which were surprising, some of which were not. One thing that I don't think I've communicated very well to my readers is that I most emphatically am not comparing TurboGears to any of the other frameworks out there. I'm comparing TurboGears to itself, and finding out what I need to do to scale it up.

I could compare to Ruby on Rails, CakeWalk, CodeIgniter, WebCore, Django, and a host of others. The problem with doing so is that doing that sort of comparison is nearly impossible. Why? Each of these frameworks has their own particular pluses and minuses. PHP versus Python vs Ruby. Pure unadulterated speed vs size of community vs how much scaffolding you need to build vs how locked in you are after you start using it.

For every developer, these are tradeoffs. Some will be okay with scaffolding. Some demand PHP. Others want the framework to put no demands on them. Comparing all of these frameworks in an unbiased fashion just is not feasible. That's why I didn't do it.

All of these, from very very slow to very very fast, will serve the needs of a great many web sites. Even at a "mere" 10 requests per second, that's 864,000 dynamic page views per day. I happen to participate in a forum which transfers around 3G of data every day, and it only gets 380,000 URL requests per day (both dynamic and static).

This is part of why I've stressed that I'm testing the whole application stack so very much in these past few posts. Any of these frameworks are more than capable of serving a rather large and thriving community on very simple hardware.

So, think about those numbers long and hard before you say that a "mere" 14 requests per second is not good enough. If you actually analyze your needs, you're likely to find that you're overly focused on a number that won't matter nearly as much as you think it does.

FreeBSD 8, Apache 2.2.14, Python 2.6.4, mod_wsgi 3.2 compile failure

2010-06-15T22:34:00.000-07:00

This is a keyword laden post. I spent far too long trying to figure this out, and Google could not help me. Turns out it's really simple, but getting there was challenging, to put it mildly. Here goes:

/usr/local/include/pth/pthread.h:537:1: warning: this is the location of the previous definition
mod_wsgi.c: In function 'wsgi_discard_output':
mod_wsgi.c:11599: error: 'apr_bucket_type_t' has no member named 'pth_read'
mod_wsgi.c: In function 'wsgi_execute_remote':
mod_wsgi.c:12112: error: 'apr_bucket_type_t' has no member named 'pth_read'
apxs:Error: Command failed with rc=65536

If you are using the platform above, and get an error message that looks like that when building mod_wsgi 3.2, the problem is due to a configure option in your Python installation. Take the following steps:

cd /usr/ports/lang/python26
make config
make deinstall distclean
make install

Update the configuration, and disable GNU Pthreads. I'd also turn on SEM and THREADS, so that you can get the benefit of the multiprocessing module in Python 2.6.

Now, time to fix up mod_wsgi

cd /usr/ports/www/mod_wsgi3
make deinstall distclean
make install

And that will do it. You should now get everything working cleanly.

Announcing tgext.xmlrpc v0.8

2010-06-15T19:11:00.000-07:00

This is an important update, and also one that is 100% backwards compatible.

This update allows you to have nice Python method signatures, as opposed to the hacky "*p, **kw" trick that v0.6 required on all methods.

To clarify, an example. v0.6 code required this:

class MyXmlRpc(XmlRpcController):
    @xmlrpc([['int', 'int', 'int']])
    def sumthem(self, *p):
        return p[0] + p[1]

v0.8 finally corrects this ugliness. This now works:

class MyXmlRpc(XmlRpcController):
    @xmlrpc([['int', 'int', 'int']])
    def sumthem(self, i1, i2):
        return i1+i2

Note that any code written for v0.6 will work just fine with v0.8. This is a purely cosmetic change, though a welcome one, I think.

It is available via PyPI and BitBucket (with documentation also on BitBucket).

Note that I did not forget to finish my Amazon EC2 benchmarking series. The next post in the series has been bugging me. I might not be William Shakespeare, but I do have some standards for my writing, and I've not been able to write up anything decent yet. I think I've finally figured out why I hated what I was writing, so will finish that series this week.

TurboGears and Amazon EC2 Benchmarking, Part 2: Requests per Second

2010-06-09T19:18:00.000-07:00

That all important question has been asked, and is now answered: How fast is TurboGears? I started this series yesterday when I described my methodology. I conclude soon with my analysis of what I did, and what I found.

The numbers below are not what you might expect to see. Many people, when doing benchmarks, do the equivalent of writing "hello world", and say "That's how fast my web framework is." No, it's not. It's considerably slower than that. I've seen far too many benchmarks like that. I'll not link to any of them, as I don't wish to encourage more like that. I want to get an idea of how a full featured application will perform, not some dinky "hello world".

When looking at these numbers today, keep one thing in mind: These are not "hello world" requests. These requests include templating, caching, sessions, database requests and transactions, authentication, authorization, and all of the other little pieces that make an actual web application into an application versus something that manages to simply print "hello world" on the screen.

I'll be back tomorrow with some analysis, both on the numbers, and my own biases in the whole process. For now, look these over. I think you'll find them interesting.

In the below table, the columns have the following meanings:

EC2 Instance Name: This is the marketing name of the instance as visible on Amazon's page about EC2.
EC2 Instance Type: This is the instance name that you will see when you create a new instance on EC2.
TG Version: The version of TurboGears that produced these results
Genshi/Mako: When running these tests, I varied the template engine between these two where possible. This column indicates the template engine that produced these results.
Index or Database: The default index page for a quickstarted application does not hit the database. This column indicates whether I was forcing a database hit, or just hitting the index page.
Req/Second: How many requests per second were being processed. I truncated the results (so nothing after a decimal).

TurboGears on EC2 Results
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.0.3	Genshi	Index	24
Small Instance	m1.small	2.0.3	Genshi	Database	13
High-CPU Medium	c1.medium	2.0.3	Genshi	Index	100
High-CPU Medium	c1.medium	2.0.3	Genshi	Database	64
Large	m1.large	2.0.3	Genshi	Index	86
Large	m1.large	2.0.3	Genshi	Database	44
Extra Large	m1.xlarge	2.0.3	Genshi	Index	148
Extra Large	m1.xlarge	2.0.3	Genshi	Database	80
High-Memory Extra Large	m2.xlarge	2.0.3	Genshi	Index	171
High-Memory Extra Large	m2.xlarge	2.0.3	Genshi	Database	94
High-Memory Double Extra Large	m2.2xlarge	2.0.3	Genshi	Index	284
High-Memory Double Extra Large	m2.2xlarge	2.0.3	Genshi	Database	171
High-Memory Quadruple Extra Large	m2.4xlarge	2.0.3	Genshi	Index	388
High-Memory Quadruple Extra Large	m2.4xlarge	2.0.3	Genshi	Database	229
High-CPU Extra Large	x1.xlarge	2.0.3	Genshi	Index	217
High-CPU Extra Large	x1.xlarge	2.0.3	Genshi	Database	123
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.1b2	Genshi	Index	28
Small Instance	m1.small	2.1b2	Genshi	Database	14
High-CPU Medium	c1.medium	2.1b2	Genshi	Index	124
High-CPU Medium	c1.medium	2.1b2	Genshi	Database	58
Large	m1.large	2.1b2	Genshi	Index	96
Large	m1.large	2.1b2	Genshi	Database	41
Extra Large	m1.xlarge	2.1b2	Genshi	Index	168
Extra Large	m1.xlarge	2.1b2	Genshi	Database	70
High-Memory Extra Large	m2.xlarge	2.1b2	Genshi	Index	196
High-Memory Extra Large	m2.xlarge	2.1b2	Genshi	Database	86
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Genshi	Index	349
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Genshi	Database	150
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Genshi	Index	468
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Genshi	Database	194
High-CPU Extra Large	x1.xlarge	2.1b2	Genshi	Index	283
High-CPU Extra Large	x1.xlarge	2.1b2	Genshi	Database	126
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.1b2	Mako	Index	115
Small Instance	m1.small	2.1b2	Mako	Database	22
High-CPU Medium	c1.medium	2.1b2	Mako	Index	480
High-CPU Medium	c1.medium	2.1b2	Mako	Database	93
Large	m1.large	2.1b2	Mako	Index	282
Large	m1.large	2.1b2	Mako	Database	63
Extra Large	m1.xlarge	2.1b2	Mako	Index	415
Extra Large	m1.xlarge	2.1b2	Mako	Database	94
High-Memory Extra Large	m2.xlarge	2.1b2	Mako	Index	630
High-Memory Extra Large	m2.xlarge	2.1b2	Mako	Database	131
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Mako	Index	976
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Mako	Database	220
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Mako	Index	1354
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Mako	Database	274
High-CPU Extra Large	x1.xlarge	2.1b2	Mako	Index	776
High-CPU Extra Large	x1.xlarge	2.1b2	Mako	Database	181

TurboGears and Amazon EC2 Benchmarking, Part 1: Methodology

2010-06-08T21:25:00.000-07:00

This is going to be a multi-part series. As of right now, I've got at least three parts, and I may be adding another one in. I've just gathered a lot of data from Amazon, and the TurboGears EC2 images I made. Enough of it, in fact, that I'm not sure how to present all of it. For now, I'll explain how I gathered the data.

First, I signed up for Amazon's Relational Database Service (RDS). This allowed me to move the database off of my EC2 instances. It did subject me to the limits of their setup, and their network, but this provides two advantages:

Web servers and database servers are typically separated in real world deployments. Using RDS allows me to have a similar configuration.
This allows me to show the speed of TurboGears. It's a fairly safe bet that Amazon has tuned their RDS service very well. Therefore, any speed issues that occur are likely to belong entirely to TurboGears.

From there, I used the EC2 Amazon Machine Images I announced last week. I would start up an instance, create a quickstarted application, and then test against both the index page (which produces no database hits) and against the index page with authentication cookies, which did produce database hits.

That's the summary.

For 2.0.3, I did the following:

Start up the instance of the TG 2.0.3 AMI I created. Use the correct one to get an instance size I've not yet measured.
ssh into the machine
Become the "turbogears" user
Create a quickstart app named "tgtestapp"
Run "python setup.py develop" to register the application
Copy development.ini to tgapp.ini, as expected by the AMI
Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
Use "paster setup-app tgapp.ini" to initialize the database
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.

For 2.1b2, I did the following (very similar to 2.0.3):

Start up the instance of the TG 2.1b2 AMI I created. Use the correct one to get an instance size I've not yet measured.
ssh into the machine
Become the "turbogears" user
Create a quickstart app named "tgtestapp" using genshi templates
Run "python setup.py develop" to register the application
Copy development.ini to tgapp.ini, as expected by the AMI
Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
Use "paster setup-app tgapp.ini" to initialize the database
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.
Destroy the current quickstarted app (rm -rf)
Make a new one, using mako templates
Run "python setup.py develop" to register the application
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.

I repeated all of the steps (for both 2.0.3 and 2.1b2) for all Amazon EC2 instance sizes.

The commands I used for running the actual benchmarks are as follows:

ab -c 10 -n 400 http://localhost/
ab -c 10 -n 1000 http://localhost/
ab -c 10 -n 1000 -C 'tgtestapp=06c4a524f1a6c3196b1c491e95517c61c1770ef7efeaaacdcc29437c70cc8737438fc02d' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' http://localhost/

I have noticed a sometimes significant delay in mod_wsgi getting started serving the site. The first command overcomes that delay, and gets everything flowing as smoothly as possible. The second command focuses on raw rendering speed. The third looks at speed when the database server is in the mix, as it will generate hits.

Many people insist on multiple runs when doing benchmarks. To a degree, they are right. They are looking to know the absolute values, and benchmarking is inherently non-absolute. I did not do multiple runs, because I was looking for approximations of real world performance. Any one run will provide that.

Even those times when I said "Wait, that can't be right", and re-ran the numbers, I found the results were always within a few percentage points of each other. I quickly figured out that, for the needs of this article, anything more than one run would be superfluous.

You will note that the pages being retrieved are not simple "hello world" pages. The entire TurboGears stack gets tried out here. This is a full, real world example of what you can expect to see from TurboGears on Amazon's EC2 service. As such, multiple runs are of questionable value in this case (at best). If enough people are bothered by this, I'll do re-runs, and average out the results (or get the median, whichever), but I really do not feel it is necessary for this particular test.

Preparing the results for viewing on blogspot is actually taking a bit of time. I'll get them up in the next two days, so that everybody can see the numbers that I see.

Next: Requests per Second

Announcing TurboGears EC2 Images

2010-05-31T18:53:00.000-07:00

Amazon has long offered a service whereby the community can purchase time on their clusters. Today, TurboGears is able to take advantage of this in a new way. It is now easier than ever before to get a TurboGears instance up, running, and viewable by you, allowing you to find out how that TurboGears based application really looks and functions.

The whole process is actually fairly simple. You will need to know the AMI ID of the version of TurboGears you wish to test. Here are the AMI IDs we have available now. This post will be updated when new versions of the AMIs are made available.

TurboGears 2.0.3, 32 Bit - AMI ID: ami-c5648dac
TurboGears 2.0.3, 64 Bit - AMI ID: ami-a17e97c8
TurboGears 2.1b2, 32 Bit - AMI ID: ami-67678e0e
TurboGears 2.1b2, 64 Bit - AMI ID: ami-7b719812

Installing your application and getting it live is very simple. If you don't know how to work with Amazon's EC2 service, please read their documentation for it. Here's the process:

Launch an instance of the AMI you have selected.
Log in to the instance:
ssh ubuntu@public.dns.name
sudo su - turbogears
Install the application following whatever directions are provided by the application, but do not run "paster setup-app" yet.
Instead of making "production.ini" or "deployment.ini", you will make "/home/turbogears/app.ini". This is absolutely identical to the other files, it's just the name that is special. Yes, you can modify configuration to use one of the other names, but you do not have to.

When you do make this file, don't forget to set "debug=false" in it. This is meant to be a production environment, and will break if "debug=true" is set.
Run the following two commands:
cd $HOME
paster setup-app tgapp.ini
If your configuration uses a SQLite database, don't forget to fix the permissions. Either make www-data the owner, or set the permissions to 0777.
A couple of files need to be made, but will be made automatically for you. When you installed the application in step 3, a Python egg was installed. You will need the name of that egg (but can find it fairly easily). A command has been written to make this whole part of the process painless. Do this:

tglinker eggname eggversion

If you are unsure of the eggname, simply type "tglinker", followed by a space, and then press the tab key twice. You will be given a list of all installed eggs, and can start typing the name in from that list. Once you get enough to be unique, press tab again, and the name will be completed.

Once you have that, the eggversion will be even easier: Pressing tab twice will likely fill in the full version number. If not, simply type it in again usaing the tab completion as above.

Press enter, and the required files will be generated.
Finally, logout of the turbogears user account (either by using the "exit" command, or the "logout" command), and restart apache using this command:

sudo /etc/init.d/apache2 restart

You're done, and by visiting "http://public.dns.name" in your browser, you will now see your web site function.

A word of caution: If you are a new user of the EC2 service, you might well terminate any instance that you create. All data on the local drive will be destroyed. Their documentation covers how to preserve data. I recommend you read it.

Announcing tgext.xmlrpc

2010-05-24T09:43:00.000-07:00

One of the questions that we in the TurboGears community get asked is "How can I have XMLRPC in my TurboGears application?" Until now, we didn't have a useful answer beyond "Write it up". Now we have tgext.xmlrpc.

tgext.xmlrpc allows you to have an XMLRPC hierarchy. Your code becomes a case of simply setting up your controller and working with it like any other controller. So, if you need to have XMLRPC in your TurboGears application, check it out (and don't forget to read the docs!). I think you'll find it quite helpful.

What To Do When It's Time To Find A New Maintainer

2010-04-07T21:36:00.000-07:00

This post is going to ramble a bit, and I apologize. Almost two years ago, I got an Asus EEE 901 netbook. This is a great little device, letting me have a highly portable laptop style computer. I then got the idea to use it for GeoCaching, and wrote Cache 901. The problem I face now is that it's time for me to hand Cache 901 over to somebody else.

When I wrote that program, GeoCaching was something that would be done every couple of months. We'd go out, find a couple caches, have some fun while doing so, record them, and get back on with our lives. Since then, something's changed, and we haven't done it for over a year by now at all. Since we're not GeoCaching, and it doesn't look like it will happen again anytime soon, I've stopped working on Cache 901.

Now, Cache 901 is a good program, and fairly popular (as of now, it's been downloaded nearly 900 times so far this year). I don't want the program to die. I also don't want to be responsible for taking it to the next level. I want to hand it off to somebody else. I want it to become active again. The worst part is that I don't know how to find someone to take it over.

If you have any suggestions, please let me know.

Emacs As 3Way Diff Tool, Use With Mercurial

2010-03-26T21:16:00.000-07:00

So, I've switched recently to EMacs. Mostly doing okay with the switch, but today I wanted to do something I'd not seen mentioned elsewhere: I wanted to use it as a 3 way diff tool, and integrate this with Mercurial. Turns out it's not too hard to do at all.

First, you'll need to make the following into a script. I named it "ediff3":

#!/bin/bash
if [ $# -ne 3 ]; then
echo Usage: $0 local base other
exit 1
fi
emacs --eval '(ediff-merge-with-ancestor "'$1'" "'$2'" "'$3'")'

Don't forget to set that script executable!

Next up, you'll need to tell Mercurial how to call your new ediff3 script. Edit $HOME/.hgrc, and add the following lines:

[ui]

merge = ediff3

[merge-tools]

ediff3.executable = ediff3

ediff3.args = $local $other $base

You're now done with configuration. This blog post over here told me how to actually use ediff3. All in all, I'm pretty pleased. It was quick, easy to learn, easy to use, and did what I needed. If only all the tools I ever needed were like that.

JSOn vs XML vs YAML, and Python Parsing

2010-03-05T21:00:00.000-08:00

I've begun working on translating a board game into a computer game. In order to do so, I need to be able to represent the state of the game and its many (many) tokens. In addition, I need to be able to store and load that state quickly. Finally, I need that state to be something that I can represent with a single string.

EDIT: I added YAML testing data to the results. I'm not posting my original files, as I don't have a license to be converting this game, and don't want to trip any legal issues. Suffice to say that the names in the file give it away.

EDIT: I added ElementTree testing data to the results.

EDIT: I added marshal data to the results.

EDIT: I added cPickle data to the results.

Storing the state presents a simple hierarchical structure: Multiple decks of cards are shuffled, and the shuffled state must be saved. This allows for a web front end to the game. The cards are just one aspect: I have to store information about what cards the player has, what state the cards are in, where the player's token is, and what feels like about 500 pieces of information (no joke: I scanned 494 images for this game).

So, how do I represent this data? How do I make it so that a web server can load and save the state quickly? XML is good for hierarchical data, as are JSON and YAML. But which one is right this time?

I needed to test it out. So, I wrote three equivalent files, one in XML, one in JSON, one in YAML. I then ran the timeit module against those files, using the following commands:

python -m timeit -n 100 -s 'from xml.dom.minidom import parse' 'd=parse("allboard.xml")'
python -m timeit -n 100 -s 'from xml.etree.ElementTree import parse' 'd=parse("allboard.xml")'
python -m timeit -n 100 -s 'import simplejson' 'd=simplejson.load(open("allboard.json"))'
python -m timeit -n 100 -s 'import yaml; from yaml import CLoader as Loader' ' d=yaml.load(open("allboard.yaml"), Loader=Loader)'
python -m timeit -n 1000 -s 'from lxml import etree' 'd=etree.parse(open("allboard.xml"))'
python -m timeit -n 100 -s 'import marshal' 'd=marshal.load(open("allboard.marshal"))'
python -m timeit -n 100 -s 'from xml.etree.cElementTree import parse' 'd=parse("allboard.xml")'
python -m timeit -n 1000 -s 'from cPickle import load' 'd=load(open("allboard.pickle"))'

The results were simply staggeringly different.

XML (minidom) took 15.3ms per run. XML (ElementTree) took 19.6ms per run. XML(lxml) took 925usec per run. XML (cElementTree) took 1.35ms per run.JSON took 250usec per run. YAML took 10.1ms per run when using the CLoader. Without the CLoader, it took 115ms per run. Marshal took 1.01ms per run. cPickle took 2.5ms per run.

JSON wins this one hands down.

Is it possible I chose poorly? Of course. There are many XML parsers for Python, just as there are many JSON parsers. I wanted pure Python here, so as to maximize portability. I chose these modules since they should be completely standard. If I'm wrong, tell me, and I'll revisit this.

Announcing the Availability of tgext.menu v1.0b1

2010-03-01T11:14:00.000-08:00

It's been a lot longer in coming than it should have been, maybe, but it's here finally: The menuing extension for TurboGears. This extension gives TurboGears developers the ability to stop looking for where their controllers are set, how to integrate new controllers into their navigation menus, etc.

All of that goes away. Now you can just use simple functions and decorators to generate your menus for you. As it also uses jdMenu, you get dynamic menus that are very suitable for a web application.

If you're a TurboGears developer, you owe it to yourself to check it out today.

Python: Shuffling A Deck Of Cards

2010-02-15T08:11:00.000-08:00

Python continues to amaze me on a regular basis.

I love programming in Python. It's simple to read, the code is clean, it's testable, it's just plain a pleasant experience. And, quite frequently, something happens that just makes me go "wow".

If you've ever written code that needs to shuffle a deck of cards, you know that it can be annoying. It's not a difficult problem,. but it is one that you have to deal with on occasion.

You have a list of things, and you need them to now appear in a random order. One of the simplest ways to do it is to run a loop something like this (using Python):

unshuffled_list = [1,2,3,4,5,6,7,8,9,10]
shuffled_list = []
while len(unshuffled_list) > 0:
   r = random.randint(0,len(unshuffled_list)-1)
   shuffled_list.append(unshuffled_list[r])
   del unshuffled_list[r]

shuffled_list will now contain a randomly ordered list consisting of the original contents of unshuffled_list.

Python, though, makes it much easier, and removes the possibility that you will introduce some small bug in your code. The way you should do it in Python?

shuffle_list = [1,2,3,4,5,6,7,8,9,10]
random.shuffle(shuffle_list)

shuffle_list now contains the randomly ordered list. You're done.

It's a small thing. It's a simple thing. Some would even call it a silly thing. And yet, it makes life easier.

Thanks Python!

On Using The Wrong Tool

2010-02-10T18:42:00.000-08:00

Subversion, as a version control system (VCS), is a mostly decent version control system. You check code in, you look at revisions, you look at logs, you can restore the state of your code at any time. It does the job it was built to do. This is not to say that it doesn't have problems, though. In fact, some of these problems have bitten me more than once. Using Subversion when other tools exist that actually correct these problems seems, to me, to be madness.

A typical setup for svn will have a repository layout that looks as follows:

repository

-branches

-tags

-trunk

I've also seen where there will be a special branch named "qa" in there. The workflow works something like this: Minor changes will checked directly into trunk. Major changes will see a copy of trunk made as a branch, and then when work is done, the branch will be merged back to trunk. If the "qa" area is in there, then the trunk will be copied to qa when a new release is ready for qa. Once the new release has vetted, it will then be copied to its own tag, and released to the world.

When a released version has an issue, and it is small enough, then some places will put the fix into that release, while others will put it into a new release. The key fact here is that it is possible for a fix to be put into a release and not copied to trunk or the optional qa branch (and, since we are only mortals, this will happen on occasion). Once the new release is cut, that fix will be lost. If svn is being used to maintain a single website, then the appearance will be that the website had a bug fixed, and then it came back for no apparent reason.

This means that somebody must now take the time to figure out what the differences are between the previous working version and the current version. Time that could be better spent on pretty much anything else.

As it turns out, a different version control system, when setup properly, handles this particular issue wonderfully (along with every other issue I've found with Subversion). The one I've used is Mercurial. It's command set is very similar to Subversion, which makes the transition extremely easy.

When you set up Mercurial to provide a similar structure to Subversion, you will have three separate repositories: devel, qa, and production. Each of them will (at the project start) be a copy of the others. When a new release is ready for qa, a tag is added, and the release is pushed to the qa repository. When that release is vetted, a tag is added, and then pushed to the production repository.

Now, assume the same thing: We have a production repository which has fixes that did not go back to qa or devel. When you try to push to production, you will receive a warning about this, since you are now creating two head revisions in the production repository. You are prompted, before any user sees the issue, to find and fix this. You wind up keeping the bug fixes. The problem goes away, and cannot come back unless you force it to come back.

Or, you can keep on futzing with Subversion, and hoping no one ever makes the mistake I've mentioned above.

And that doesn't even get into the issues of permissions, setting up the web server, dealing with patches from outside that you must now credit in log messages only (as opposed to the outside user being the one that receives the credit for the commit), none of that.

I know which one I choose and prefer.

Announcing Availability of tgext.menu, v0.2

2010-01-19T07:00:00.000-08:00

Long story cut short: I've not been moving on Bit Vizier as fast as I wanted for a number of reasons. One of them is the pain that is managing the navigation bar from a development perspective. If I want to move things around, I have to update my controllers and my navigation bar template. If I need to rename something, I have to do the same. If I want to add a feature, I have to do the same. The navigation bar, the side bar, and any other menus are all easily determined from the code. Why do I have to maintain a separate template?

With tgext.menu, I don't have to do so anymore. Instead of having a separate template, I simply have to provide a list of the menu entries, and the rest just happens. It makes dealing with things much easier.

tgext.menu is available on the Python Package Index (for use with easy_install), and also on bitbucket.org (to get the source code).

I'll apologize now for the shortness of this post. I'm still recovering from ear surgery, so mildly unavailable, but didn't want to hold this back any longer.

Choose The Right Data Structure

2010-01-13T10:44:00.000-08:00

This one is actually almost embarrassing to post. I've been working on an extension for TurboGears called tgext.menu. I'll not get into long explanations of why, or what I hope to accomplish with it (I'll leave that for when the first release happens, hopefully early next week).

I will say this, though: If you don't choose the right data structure, right off the bat, you are condemning yourself to a hellish experience coding.

In my case, I had a list that looked like this:

ExitApp
Foo Spot
Foo Spot || Bar
Foo Spot || Baz
Foo Spot || Foo
Foo Spot || Sub || Bar
Foo Spot || Sub || Baz
Foo Spot || Sub || Foo
Sub || Sub 1
Sub || Sub 1 || Nested 1
TestHome

I wanted that to become a nested unordered list in HTML, looking like this:

ExitApp
Foo Spot
- Bar
- Baz
- Foo
- Sub
  - Bar
  - Baz
  - Foo
Sub
- Sub 1
  - Nested 1
TestHome

Looks really simple, doesn't it? All you have to do is loop over the list, split each item on the || symbols, generate openings, put your entries, and generate closings. Except you have to compare the current base of the tree with the next base of the tree and the previous base of the tree.

I can't count the number of methods I tried, and every one of them had a small, subtle, error that threw off all the output. Sometimes elements would not be closed properly. Sometimes they would not be opened properly. I tried iterating, I tried recursion. All of them failed with errors that I could not find a generic way to correct.

Saying this was frustrating is a major understatement. Last night, I finally changed my data structure that represented the output: Instead of a list, I went with a tree. Using a tree, I was able to write up (in less than a dozen lines) a simple recursive loop that wrote out the entire tree, with appropriate whitespace, CSS class tags, the work.

In one hour, I solved problems that had been plaguing me for two weeks.

Lesson learned: If I'm spending a lot of time trying to solve a specific problem, and my algorithms are always producing errors, I need to look at the underlying data structures. I'll probably save myself a lot of time that way.

Are You So Arrogant ...

2010-01-06T09:08:00.000-08:00

I was insulted last night. Now, there's nothing new about that, nor even anything particularly blog worthy. After all, we all get insulted in some fashion or another on a regular basis. We shrug it off, and we move on. This time, though, I've heard this particular insult too often. I want to address this, in particular: I was called arrogant.

Why? Because I answered a question.

The person who called me arrogant had asked about a particular piece of a program I've written, wondering what would happen if the user did a specific action. I replied that them doing so would be irrelevant, since it could do no harm. The reply came back, and amounted to "Are you sure?"

Suddenly, I wasn't sure. I went back to the code, and re-read it. I checked the actual behavior while running. I verified the contents of the database. Now, I was sure again: It simply didn't matter if the user did this action. It produced an off-by-one sum error for something which already has a much larger margin of error (counting cards that got sent and receieved from around the world). Being off by one in that situation causes no concern whatsoever.

Despite this, I then checked the database to see who was doing it, and found a whopping four people out of fifty three had done it. I spelled all of this out for the original questioner. Since the original questioner is not a technical person, I made sure to use terms that would make sense to her.

And then she asked the question that started with the title of this post: "Are you so arrogant ... " (yes, I've left the rest out, as I'm trying very hard not to make this an attack against her, and quoting her excessively would do just that).

I'm arrogant because I answered her question in terms that I knew she would understand.

After that, she mentioned that, in her experience, computer people (programmers, techs, administrators, etc) come across as if dealing with idiot children when dealing with non-computer people. So, instead of just being arrogant, I'm arrogant and condescending.

This is where the great disconnect occurs, and is the main point of this post: As a developer, I have to know things about the computer that many people do not. Doing web development, I have to know HTML, CSS, and JavaScript. If I'm doing anything with dynamic data, I also have to know whatever programming language I'm using on the server, and some database dialect so I can store, retrieve, and manipulate that data. All of that is the bare minimum required to do that work.

In order to do it well, I need to know some bits about the underlying operating system, the web server software being used, basics about HTTP (and possibly https), and the quirks of the various browsers in use around the world (IE6/7/8, Firefox, Opera, Chrome). Possibly more importantly, I have to have the knowledge that I can make no assumptions about anything. I can't count the number of times I've said "That's not possible to happen!" only to find out later that not only was it possible, it happened due to a bug in my code, and that bug came from an assumption.

In other words, I have to have a huge amount of specific, detailed, technical knowledge rolling around in my head just to do the basics of my job well. When I get somebody who is a self-proclaimed non-technical person asking me for answers that require at least some of that knowledge be handed out, I have to mentally change gears and use terms that I know will work to explain that knowledge. Of course I sound condescending, simply because I try very hard to make no assumptions.

The part that I find most interesting to me, in all of this, is one thing that my experience has taught me: The more understanding people have of their own systems, the less likely they are to call me arrogant when I explain an answer to their question.

I care a great deal about making sure I'm understood. There are times when my doing so comes across as rude, arrogant, or condescending. I do not set out to do so, but there are times when I cannot think of a way to avoid it. I actually don't like doing so. To have the insult lobbed in my direction on top of that is more than I can accept. I had to say something.

My final thought on all of this comes from her final remark: "Not everybody is a computer genius." That's correct, not everybody is. Not even me. If you want to find out about a few of them, allow me to give you some names to look up: Alan Turing, Steve Wozniak, Linus Torvalds, Alan Cox, Theo de Raadt, Brian Kernighan, Dennis Ritchie, Larry Wall, and Guido von Rossum. I am not, and never will be, in their league.

Bit Vizier Design Ideas

2009-10-28T20:36:00.000-07:00

This is going to be a long post, but I'm trying to express my main concepts for Bit Vizier itself, and how I can leverage TurboGears to make Bit Vizier happen.

I am not limiting Bit Vizier to be used for F/OSS projects. I want it to be capable of being used for something as small as a one man project, up to as large as a medium sized corporate help desk. This changes some of the assumptions that would normally be made.

Bit Vizier will be built around three core concepts: Entities, Tags, and Views.

Entities:
An Entity is, simply put, a thing to be stored and retrieved from the system. It can be anything at all. In my white board notes, I used three examples: Tasks, Documentation, and Assets. Allow me to break these down a bit more.

Tasks represent work to be done. These can take on numerous forms, such as scheduled tasks, development tasks, call logs, and trouble tickets. Each of these task types has their own requirements, their own way to be displayed, and their own work flow types.

As an added bit of complexity, a user might not have permission to view a specific task until it enters a given state. For example, a development task might have to go through an approval process before it can be viewed by the development team.

Finally, tasks can spawn other tasks, and even be dependent on them before they can be closed. A classic example of this would be a trouble ticket spawning a development task, and unable to be closed until development is complete.

Documentation actually comes in two flavors: Static and User-Generated.

Static documentation will include such things as doc strings and any other documentation that comes with Bit Vizier. This is intended to allow an installation of Bit Vizier to be complete and stand alone from any other installation.

User-Generated documentation is likely to take the form of a wiki, allowing the users of a given installation of Bit Vizier to enter their own documentation about the topic at hand. For instance, if Bit Vizier is deployed in a F/OSS project, I would expect to see pages about getting the latest development version, a FAQ, how to get involved, and the like. Whereas for a corporate help desk, I would expect to see pages about policies and procedures.

As a possible side note: It would be worth the time investment to ensure the ability to store sensitive data in the wiki, including passwords. With sufficient protections, it could make a very viable way of distributing sensitive data only to those who need it (such as passwords, account numbers, etc).

Assets can be most anything, but are generally going to represent some physical item such as a computer or a cell phone. This would provide a way of tracking who has been given what specific item. This is almost exclusively useful for corporate deployments, and should likely be an entirely separate (and optional) extension.

As a final note, all entity types will support the concept of a changelog, recording the data about who changed what values and when. This is important for accountability purposes.

Tags:
Tags are arbitrary names that can be assigned to Entities using an M:N relationship (any tag can be on any number of entities, and any entity can have any number of tags).

Some specific choices about tags here that make them slightly different than what you might be used to elsewhere have been made. For instance, it should be possible to place tags into tag groups, and make it so that a given entity type must choose one (and only one) from a given tag group. This would replicate the idea of a ticket going to specific department, and only being in that department, even though it may well have other tags that could be interesting.

In addition, entities should be able to specify requirements regarding tags and tag groups. For instance, no tags allowed, one (or multiple) from specific tag groups, or a free for all. And it should also be controllable by the installation administrator.

Tags should also be able to be defined by the installation administrator for system wide tags, or by the user for private tags.

Finally, tags should also have a "shared" attribute, which will serve a dual purpose: On system wide tags, this will indicate that other systems can receive this tag/entity. On user defined tags, this will indicate that other users can see this user defined tag.

Views:
The last significant piece of work (for now) for the system will be views. A view is a collection of tags and tasks displayed to the user.

Similar to tags, views can be defined by the installation administrator, or they can be defined by the user. Also similar to tags, views will have a "shared" attribute, which will perform the same function as the same attribute on tags.

Views will also be the result of a search, which means a simple search will produce a new user defined view that the user then will have the option of saving.

Finally, views will come in three types: Condensed, Expanded, and Detailed.

Condensed Views are meant to provide single row display of summary information about the entity.

Expanded Views are meant to provide the equivalent of Google search results, where you will get a couple of lines of output regarding the entity.

Detailed Views are meant to provide a full screen view of an entity, and will (as a result) show everything: change logs, comment logs, patches, everything.

Where To From Here?
There's a lot of information up there to digest, and that's all from just one white board's worth of notes (and, considering my handwriting, that's pretty impressive).

TurboGears, with Sprox and the tgext.admin extension, are going to provide me with much of the backbone for this. In fact, now that I have this much, I can foresee getting a basic issue tracking system that meets these requirements out the door quite quickly thanks to TurboGears. Since it's now the end of October, I'd expect to see the equivalent of a 0.4 release by the end of December which fulfills all of these basics.

And The Shared Issue Tracking?
After that, it's going to be time to consider getting the shared issue tracking working. I think I have ideas for this now that are worth considering.

Right now, I'm looking at having the installations of Bit Vizier use self-signed SSL certs for validation. Provided I do this carefully, it will be possible to build up a web of trust, much like GnuPG/PGP. This would allow authenticated connections between Bit Vizier installations.

When those options are not available, it will also be possible to use plugins capable of talking to other systems, so that I will have (for example) a BugzillaAutomator that will upload and download from Bugzilla trackers, a similar one for Trac trackers, etc.

In conclusion, I'd like to toss a few words of thanks to the TurboGears team. They're being hugely supportive of these ideas, and their work is making mine much easier.

Thanks guys. I do appreciate it.

Why Not Launchpad?

2009-10-27T18:52:00.000-07:00

samokk asked a very simple question: Why not Launchpad? After all, I get several advantages if I choose to use Launchpad:

It's already fully functional
It's written in Python
It's open source
It also seems to be heading in much the same direction that I'm looking to go with Bit Vizier.

I'd dabbled with the idea here and there, especially after hearing that they were doing some of the same things in Launchpad that I'm looking to do with Bit Vizier. So, I went to look at their instructions for getting a working Launchpad development environment working tonight, just to make sure I wouldn't be wasting my time with Bit Vizier.

My initial reaction was to be somewhat less than impressed. They have a whole setup script that is designed to get the various packages, get them built, and reconfigure your machine in order to support doing development work on Launchpad.

I've installed Request Tracker a couple of times. Anything that starts instructions with something similar to "Run the setup script to download and install all the required dependencies" makes me question if this is a good idea. I've gone through such tools in the past, and do not like finding myself working with them. I decided to try it anyway.

One of the first things I noticed was the script asking for my sudo password. That immediately raised red flags for me. This is supposed to be my development environment. Why do I need sudo access to run this? And then, when I aborted, it tried (and, fortunately, failed) to update my /etc/hosts file.

After finally managing to quit out, I look at the installer script. 526 lines long, little of it commented. It wants to update my ssh config file, my /etc/hosts file, my apt sources list file, and install system wide packages. This is when I finally notice this quote from their instructions:

Note: the changes introduced by the install script may break your current web development setup, so it is advisable to try Launchpad on a separate virtual or physical machine.

Wait, this is for my development environment? And it's going to take over my machine possibly to the point of breaking other web development on my machine?

That was the end of my willingness to look at Launchpad. If they had done things nicely, then they'd already be able to use a virtualenv, and the end result of working on their system would be more disk space usage. They chose to go a route that means that I have to devote my entire development machine to them, and them alone, unless I want to work in a virtual machine.

Add in their reliance on bzr and lack of interest in other distributed version control systems, coupled with my strong preference for Mercurial, and we just get further into the non-starter category (as if we needed to go any further).

No thank you. Their ideas might be great. But their cost is too high. I'll stick with building a TurboGears based system that allows me to let my developers use a virtualenv safely, reducing their risk to their machine.

So, What's The Difference?

2009-10-22T19:29:00.000-07:00

I went to reply to a comment on my previous post about Bit Vizier, and realized that I was actually heading into a whole post in and of itself. The commenter had mentioned Ditz, and a spin off of it named Pitz. They have some very good ideas in there, and I'm likely to borrow heavily. However, they focus on distributed issue tracking, while I'm looking to focus on shared issue tracking.

So, as the title asks: What's the difference?

First, we need to look at what issue tracking means, in and of itself.

Issues tend to have a common life cycle, regardless of company, organization, group, disregarding project size. And the various issue tracking systems out there reflect this. I know, I've just made what amounts to a heretical statement to many people: I've just said that all issue tracking systems more or less implement the same basic workflows.

For proof, consider this: An issue is discovered by someone. It is entered into the tracking system. It is worked on, with a back and forth going on between the reporter and the people working to resolve it. Sooner or later, it is closed with any of a variety of statuses (resolved, won't fix, user error, etc, etc, etc).

Before you tell me how wrong I am about what your preferred issue tracker does, stop and look at your issue tracking system of choice. You'll see that, when it comes to tracking issues, that work flow is at the heart of what's going on. Different systems have different ways of implementing that work flow, and place different requirements on the users of the system, and provide differing capabilities for reporting on what's going on, but that work flow is at the heart of every issue tracker out there.

Now, looking at how that work flow is typically implemented, we can see (broadly speaking) two distinct categories of issue tracking systems.

To help illustrate what I mean below, I'm going to use the following example: TurboGears, as a project, has several dependencies. Each of those dependencies can have their own issues that, while they affect TurboGears, do not actually get resolved by the TurboGears team. For instance I found out about a compatibility issue with Beaker and Python 2.4 (yes, this actually happened). A patch had been submitted, but had not been applied to Beaker until I spoke with one of the maintainers about it on IRC. Note that the patch was submitted to TurboGears, but was applied to Beaker. With the shared issue tracking I mention below, it would have been more visible to the Beaker team.

In that example, the two projects are TurboGears and Beaker. Myself and Ben Bangert are the developers in question.

The most common issue tracking systems all focus on a centralized model, where a single repository of issues exists. This repository can be load balanced, multiple databases, replicating, etc, but it is all under the purview of either one person or a team within the organization. When an issue arises that actually comes from outside the organization, an issue is created in both organizations systems to track the progress of the issue, and each of those issues has a different goal.

Using the example from above, an issue would have been created in both TurboGears' and Beaker's issue tracking systems. TurboGears would have worked independently of Beaker, trying to find a work around, while Beaker's developers would have worked to resolve that compatibility issue. I would have eventually logged into Beaker's issue tracking system, found the patch, spoken to Ben, gotten it applied, and we would all be on our merry way.

The newer type of issue tracking system that is coming about is called a "Distributed Issue Tracking System". This type of system functions much like Mercurial, Git, Darcs, and other distributed version control systems. By allowing the developers of a project to take the issue tracking offline (and, in some cases, even merging the issue tracking with the vcs itself), the developer has complete access to the issue, and can merge his (or her) changes back to the main tree once work is done in the offline state.

To use the TurboGears and Beaker example, the work flow would have looked more like this: Ben would have downloaded a local clone of Beaker's issue tracking system, and I would have downloaded a local clone of TurboGears' issue tracking system. In poring through the TurboGears issue tracker, I would have found out about the Beaker/Python 2.4 compatibility issue, but noted the lack of follow up. I would then seek out Beaker, find out about the patch, speak with Ben, get the patch applied, and then Ben and I could both close out our respective issues. Once done, we merge our local clone's changes back to the main line respoitories elsewhere, and call the work totally done.

Ben and I are connected to our own repositories, and to the central repository that others in our projects share, but that's as far as it goes. Our only commonality is something like Google. We can do better, though. We can implement shared issue tracking, and this is where Bit Vizier comes in.

Keep in mind that much of what I'm about to say is speculative. I don't have concrete code examples, nor even very good diagrams to explain things to everybody yet. But I do believe that shared issue tracking can fundamentally change how we view issue tracking in general.

To use the same TurboGears/Beaker/Me/Ben example, from above, if shared issue tracking were already fully functional, the work flow would look like this:

I go through the TurboGears issue tracker, and find out about the Beaker/Py2.4 incompatibility. Since the issues are already shared amongst the projects, the TurboGears tracker can see the patch that's attached to the Beaker side of the same issue. I now see that I need to ask Ben about it, and he can then apply it. We can each close our half of the issue, with our own different resolutions, and get back to working on the parts that matter.

It doesn't sound much different, I admit. The key difference between this and other issue trackers, though, is that everything happens automatically. When you view an issue in one tracker, you see everybody else who is tracking that same issue. For instance, if five different projects had the same issue, and were each watching that same Beaker issue, then as soon as the update occurred in Beaker, the other five projects would be able to see it without having to speak directly to the Beaker tracker.

Furthermore, because they are all integrated, all of them can easily see each other's shared comments, which can help each of them to develop work arounds for the problem in a more uniform fashion. Perhaps it's the same type of fix in all of them, but the exact place to implement it differs. In that case, once one of the project members finds the fix, all of the projects have access to the work around until the core project is fixed.

Shared issue tracking is, in many ways, the exact opposite of distributed issue tracking. Instead of taking issues offline, it uses the power of the internet to help multiple projects manage the issue resolution process together.

I'm still working out the schema for the database, on a basic level. I know, to some degree, what I want to do, but actually coding it is providing a bit of a brain strain. I'm heading back to the whiteboard in a minute to try to get this out of my brain and into some sort of coherent form that I can turn into working code. Hopefully, tomorrow I'll be able to explain the design ideas in such a way as to be understandable by others.

Announcing the Commencement of Bit Vizier Development

2009-10-21T20:30:00.000-07:00

I was going to announce this about two weeks ago, but work suddenly kicked into high gear, and left me wishing for time to work on anything.

Considering this previous post of mine, seeing me announce something brand new has to seem more than slightly wrong. After all, why make a new item, rather than extend an existing one? How can I justify making Bit Vizier (AKA yet another issue tracker) when I just condemned making yet another framework?

In this case, I have two reasons for doing so. The first is that the other issue trackers out there fall broadly into two categories:

Code I do not wish to extend. I'm a Python developer, so tools like Request Tracker and Redmine do not appeal to me due to their language choice. The products may well be wonderful, but if I hate working in the codebase, I'll never do anything with it.
Systems that I would have to nearly reinvent anyway. Trac is a marvelous tool. But for me to carry my issue tracker to completion (as I see it), I would have to alter so much of the core of Trac that it would no longer be Trac.

Trac in particular makes for a useful comparison. I've heard from a friend who spoke with one of the Trac developers about some of the same ideas I'm putting into Bit Vizier. On hearing those ideas, he was told that it was very unlikely to happen for the following reasons:

The ideas would require significant modification to the Trac core components.
Making those modifications, and getting them accepted, would basically mean you would have to become a member of the core Trac team.
Becoming a member of that team is extremely difficult.
These ideas go against what the Trac team wants to do, and are not likely to be implemented by them any time soon.

I am being deliberately vague about who said what and when because I don't want to cast aspersions, and it's possible I misunderstood what was said to me. In any case, as much as I like Trac, I do not feel that my ideas will be well received by the Trac community as a part of Trac.

On the other hand, the ideas standing on their own have merit. I've at least set up a website for the project (yes, I'm using Trac until this is self-hosting. I believe I mentioned that Trac is excellent, didn't I? They just won't like my ideas), along with a Mercurial VCS that can be cloned. With any luck, I'll even be doing a talk on it at Pycon in Atlanta, GA.

As a very quick overview of my plans: I'm looking to make a shared issue tracker, built on top of TurboGears. The ultimate goal for it will see me able to integrate with a variety of tools, and even use Bit Vizier as the basis for other tools I've been thinking of for a few years. The pieces are finally coming together for me.

For now, take a look at the site, and tell me what you think.

Reflections on 1.0

2009-10-16T20:20:00.000-07:00

I'm working with TurboGears a fair amount lately, so you might be expecting me to discuss TurboGears 1.0. I'm not. I'm talking about the general magic of seeing a program with a version number of "1.0".

If there are any non-developers out there reading this, you're likely to only have some vague idea of what I'm talking about. The rest of you, though, are already familiar with it to some degree.

As you might have noticed, the past two weeks have seen a major slowdown in my posting. It would be easy to see just another short term blog, where the author flat out gave up after a couple weeks. What you haven't seen has been what my job did.

For the past three months, I've been working on a fairly significant project. About two weeks ago, almost all of the development aspect became completed, and now came the install. We had to have a fully hardened system, extremely restrictive access policies, token authentication, the works. During the installation, issues kept cropping up every day with what to do next.

This has been the most stressful project I've ever done, and I've even single-handedly converted a company (granted, a small company of less than 50 employees) from one ERP type system to another. The pressure of "So, we're ready to go in just two more days, right?" has gotten to me tremendously.

Every day, right after waking up, I started focusing on what needed to be done next. I thought about it while showering, driving to the office, during the day, while driving home, while spending time with my wife, and after she went to bed and before I did. This project has dominated every waking moment.

Today, though, marked a huge milestone: I called the code "1.0". It turns out we have a few more minor issues to resolve, but they're getting done, and are all system configuration related.. This code finally has reached a release state.

For the non-developers out there, you might not understand what this sort of moment means to a developer. Developers have too many ideas. I have around 10 projects I'm trying to work on, and that's just personal stuff. I come up with at least three new ideas every day for "Wouldn't it be neat if ... ?" Every couple days, one of those ideas gets a bit of a sketch, a few lines of code, some random notes.

Once in a while, it gets something more: Maybe a whole module or so, maybe even a full on program, or just a hacked up proof of concept. A few times out of those times, the idea will get some real attention, and actually start to go somewhere.

Very few ideas ever manage to make it to 1.0, though. Something always seems to happen. Other obligations come up, other tools obsolete the idea, better ideas come out, ideas get somewhere and prove unworkable after that point, the list is endless. Extremely few ideas can be carried through to something that can be called 1.0.

Today, that happened: A project made it to 1.0. Granted, it was a work project, so I got paid to make that happen, but it still happened. It's a nice feeling. Despite that, I wish I didn't have to go through the last two weeks to have reached this point. I've taken to calling it my "hell project", and that label still fits.

Still, though, today, I accomplished something. I'll be back with more regularly scheduled blogs by Monday. I think I'm going to take the weekend off and enjoy it (well, as much as I can, got some unit tests/functional tests to write before I can completely let this go).

How about you? Any similar experiences out there?

Automated Testing and Why You Should Do It

2009-10-12T12:04:00.000-07:00

I first learned about automated unit testing (in the style of extreme programming) about four years ago, and decided to experiment with it. I quickly found out that code I thought I had thoroughly debugged had errors in it. Since then, I've become quite fond of automated testing. While it's not perfect, it does help prevent some of the worst errors, such as the one that I saw happen today.

Every time I've set up automated tested of my code, I've found and fixed mistakes that would be embarrassing to have other people find. Today, the same happened to a colleague of mine.

He spent the past 6 months working on an internal project. The diff was 6000 lines long. He touched 74 files for this project. In other words, this was a very large project, time consuming, arduous, and draining. Today was the big day to roll it out into production.

Suffice to say that things did not go well. A single debugging line slipped through the cracks. That debugging line wound up providing normal users with access that they should not have. A single line inside of a 6000 line diff caused a huge issue at work for him.

Had the environment been using automated testing, at least one of those tests would have been checking for unauthorized access. Running those tests would have revealed the bug well before it went to production systems.

Instead, a huge ruckus ensued. He now feels embarrassed. He knows I'm writing this post, and feels like he deserves the humiliation. I happen to disagree with him on that point. He worked within the constraints of his environment, and with the tools he was given.

I'm hoping that that environment will change some procedures in the near future, and start to use automated unit testing. It may well slow you down somewhat during the development phase, but you will feel more confident in your results, and your code will have fewer bugs.

For your own sake, learn how to use the tools from your language of choice. Find out how to do automated testing, and start doing it. You will only benefit from knowing your code works as designed. Some links that can help you get started:

Test Driven Development, Perl, Python, Java C++, Selenium

Also, some empirical research into the benefits: Test Driven Development reduces defects by 40% to 90%. Finally, a link to several papers about the benefits of test driven development.

Go, read those links, and learn how test driven development can save you from significant embarrassment.

PyFlakes, Wingware, And PEP8

2009-10-09T18:33:00.000-07:00

So, as Python developers, we all know about PEP8, right? But how easy is it to follow that standard? If you're like me, having a tool to help you check it is a good thing. Also, if you're like me, you use Wingware.

The tool you've been looking for has arrived, and it's called PyFlakesPanel.

This allows you to add a PyFlakes panel to your tool panels. From there, you can run pyflakes on any open file or package. I like it, quite a bit. It's still in an early form, and there's still room to improve, but it makes for a great start.

Once I manage to get through this deployment that we've been doing all week (and part of next week) at work, I'll see about contributing to it to make it everything I want it to be (assuming the author doesn't beat me to it!).

Check it out. I think you're going to like what you see.

Why Not Roll Your Own Web Framework?

2009-10-06T18:42:00.000-07:00

I was actually going to do a long series of "Why TurboGears, Why Not ___?", but have decided not to do so, especially since a couple of paragraphs will summarize my feelings on the others well. After that, though, I'll explain why you should almost never roll your own framework.

So, why not one of the other popular frameworks? Why not Django? Well, I think it can be summarized by reading this article. Seriously, executing code in another module to read the URLs stored in the module? What? If that's not ugly, I don't know what is. And if that's the sort of code to write with Django, I don't want to use it.

As for Ruby on Rails: Plenty of people love it. I've looked at the Ruby syntax, and found that I feel it is unreadable. Add in the idea of scaffolding, which amounts to generated code that I'm not allowed to modify directly (not without potentially having to rebuild the scaffolding later, thereby discarding my changes), and I have to discount Ruby on Rails entirely.

There's plenty of other options, but I'm not going to devote space to them at this time. I'm happy with TurboGears.

For me, the major question for today is simple: Why shouldn't you roll your own web framework?

The answer is simple: Rolling your own web framework may well be interesting, but it is going to take up a large part of your time and effort, and in the end is likely to have fewer features than the other frameworks out there.

Take a look at some of the tasks you'll need to consider:

Authentication and Authorization
Internationalization
Page Caching and Templating
Form Validation and Handling
Code snippet reuse (a la ToscaWidgets)
Java Script library choice and integration
Database integration (possibly using an object relational mapper)

And that's just what I could think of inside of a few minutes. Now, after you've got the basics done, you need to look at code organization, maintenance, and optimization. By the time you complete all of this, you're likely to have reinvented someone else's wheel. Furthermore, their wheel is likely to be better than yours, because they've been working on it longer, honing it more, and having a whole community of people help them do so.

Now, consider the developer who is selecting a framework to use: In order to choose between any given set of toolkits, a developer has to sit down, review docs, review requirements, experiment with code, see what fits his style best, see what can be done, etc. The worst part about this involves the amount of time wasted. If, somehow, your custom rolled web framework manages to make the shortlist of the developer who's looking around, then your framework is very likely to waste a significant amount of that developer's time, since it's unlikely to be chosen over one of the other frameworks which have already gone past what you wrote.

The above ignores a few other facts that very relevant:

If you code for a living, then any clients of yours which are deployed on your framework are bound to you. This is great for your income, but terrible for your clients' futures. If something happens to you, who else can pick up where you've left off?
Any bugs are entirely up to you to resolve. You cannot ask for help, since no one else knows your framework. The only way you manage to change this item is if you somehow make it enough of a success to be noticed. This might be harder to do than writing the code, especially if you're not a marketing expert.
Any new features are also entirely up to you to implement. It's all your framework. Until you reach that critical mass, nobody else will help you. And getting to that critical mass produces the classic chicken and egg problem.

None of this means you can't make your own framework. None of it means you won't come up with something better. But it does mean you should think very carefully about what you're doing.

Are you doing it to better the state of the frameworks out there? Has anybody else done anything similar enough? Can your ideas be implemented in someone else's framework? Will your ideas be accepted in that framework when brought to fruition? Would forking an existing project be a better answer than starting from scratch?

These are all questions you must answer. You might find yourself making a new framework, and have very good reasons for doing so, but make sure. Consider contributing to one of the existing ones if at all possible.

Who knows, I might be lucky enough to find you contributing to TurboGears itself. I hope to see you on the IRC channel sometime soon.

Why TurboGears Instead of Zope / Plone?

2009-10-05T13:03:00.000-07:00

Zope and Plone are the most recent ones I tried to work with. I'm discussing them together because Plone is built on top of Zope, so it inherits any/all good (and bad) things from Zope.

So, why did I eventually abandon Zope and Plone?

Zope uses an object database. This is neither good nor bad, it is simply a different way of storing the data. However, it does wind up dictating many things to you as a developer that you might not like very much.

For instance, in order to query the database and retrieve data stored in it, you must have a compatible class for the object in your sys.path. Without that, you will not be able to access the data stored in the database. To provide a concrete example, I was using Plone 2.5.3, and needed a wiki. ZWiki managed to provide what I needed, and so it got used. When it came time to upgrade to Plone 3.0, I could not access the database on another machine without doing careful version matching of everything. Anytime I tried, I would get very long tracebacks in my browser, resulting in me having to unwind and translate everything to find the specific issue.

The worst part of this was that, instead of affecting only the wiki, it affected the entire site, as the main page referred to the wiki area. When that section of the page was rendered, it would ask the ZWiki object for information (titles, for example). Since the ZWiki class(es) could not be loaded from sys.path, the main page of the site was broken, and I was unable to fix it.

As another side effect, the database will grow without limit, since Zope stores all versions of objects until you tell it to purge them. You can schedule this, but that will be an external process.

Finally, we come to zc.buildout. After spending several weeks, I was unable to find any documentation (that made sense) for how to use buildout. As a result, I could only do weak copy and paste type operations, leaving me at the mercy of whatever recipes I was using.

I like Zope. I think it looks like an amazing piece of technology. I also think it's too opaque for me. Every so often, I felt like I had managed to make some progress, but in the end, I could not gain sufficient understanding of Zope to make it work for me.

And then comes a discussion about Plone. Plone has documentation on documentation. It's possible to start reading about Plone, and keep on reading for months. In the end, though, it suffers from the same issues that plague Zope: Opacity. For me, Plone is completely opaque.

I bought the books. I read the tutorials. I worked through everything. And, in the end, all I could get Plone to do was to let me install ZWiki and have a custom theme (that I never did understand well enough to make it look right).

And, for all of that effort, I still had to run it on a fairly powerful machine: dual processor, dual core opterons with 4G of RAM. And the page loads were still noticeably sluggish.

I feel like I was able to learn next to nothing about Zope and Plone after several months of trying. The system was always outside of my control, and relied on the maintainers of the products I needed to be stable. If ever they made an incompatible change, I would be the one to suffer. And I was powerless to fix it.

When I finally twigged on to TurboGears, though, I found something wholly different: A system that made sense to me. It's not completely mastered by me, not yet. But I can see that I have the tools available to me to master it. The code is there, it's understandable, and it's extensible.

They use the more traditional SQL databases by default, which is also more understandable and easier to extract the data. This, in turn, means that it's easier to manage third party products without rendering the whole site useless.

For me, at least, TurboGears made sense where Zope and Plone do not. Make no mistake: For other people, those two products are the greatest thing ever. I even concede their overall quality. I just couldn't gain control over them, and couldn't figure out what I needed to learn to gain that control.

I'll stick with TurboGears. I hope you'll come on that journey with me, since I'm definitely enjoying it.

So, Why TurboGears, Why Not ___ ?

2009-10-01T20:02:00.000-07:00

I've decided to skip out on the "Why Linux?" question. If you have to ask, you won't agree with my answer. If you already agree with my answer, there's no point in preaching to the choir.

Instead, I'd like to address a simpler question, and one that is in some ways more important than the choice of OS, or choice of editor, or even choice of IDE. That question is the choice of which web framework to use.

I'm not sure how many posts this will become since there are so many things that can be considered, and so many comparisons that can be made. Should I choose C, Java, Python, Perl, Ruby, PHP? Struts? HTML::Mason? Django? Ruby on Rails? Smarty? Plone?

I could probably fill several screenfuls with nothing but lists of choices. When it comes to frameworks, we can find one that suits your taste every time. If, by some miracle, we can't, you can always roll your own, too. That's just the nature of the beast.

Since all of that would be boring, I'll instead focus this series of posts on why I chose TurboGears, and why I think you might like it as well. For today, I'm just going to focus on a very high level view of why. Tomorrow I'll start into nuts and bolts comparisons to show why TurboGears works best for me.

It's no secret that I'm a fan of Python. I feel that the syntax is clean and readable. The community promotes the idea of doing documentation and unit testing, both of which are items I strive for in my own code. Using those, the code tends to be more complete and produce fewer bugs.

Since I enjoy working with Python, that meant I should find a web framework that would let me use Python. I went through a couple of them before TurboGears was suggested to me.

I worked through the 20 minute wiki tutorial, and I was hooked. I knew that, even though I didn't know everything there was to know about TurboGears, I could understand the framework. It clicked for me. I can work with this.

As a set of feature highlights, consider this:

It uses SQLAlchemy to manage the database. This provides the ability to use (a)raw SQL, (b)SQL constructs (without using raw SQL, but giving nearly the same flexibility), and (c)an Object Relational Mapper. I'll not dig into the advantages of this, not yet, but I will point you to a blog posting from Chris Perkins on it.
It uses Object Dispatch. This lets you register a class instance as handling a segment of your URL space. The exposed methods on the class instance become your URLs. It's delightfully simple to manage what your URLs will map to because of this. Again, Chris Perkins did a great job with more in-depth discussion of Object Dispatch.
Flexible choice in templating language. If you prefer validating XHTML, you can use Genshi. If you prefer raw speed, you can use Mako. You can add your own renderer, too.
Flexible choices in JavaScript library. If you see one you like, you can arrange to have it available on every page of your application easily.

The list goes on and on, really. You get features on features that you never knew could be useful mainly because you never thought of them. And yet, here they are, making your life as a web developer easier.

Since I found TurboGears, I've become hooked, and have started working on their docs, trying to help them be more complete, and letting the core devs work on the core code even more. We all know how much docs suck, but at least I can do something that will help them not have to focus on them so much.

Now, if only I could somehow talk them into making a Firebug equivalent for Internet Explorer, that would make all the web development so much nicer. And, while I'm dreaming, I'd like tomorrow's winning lottery numbers, please.

The Oldest Flame War

2009-09-30T18:34:00.000-07:00

At least, if it's not, it's certainly the most well known: vi vs emacs. I'm not here to say one is better than the other, only to say why I finally wound up switching.

I've been a ViM user for about a decade, and been very happy with it for the most part. I preferred gvim, since I liked having the GUI window. I wrote up a fairly complex .vimrc, pulled in vim tips, read books, and overall did quite a bit with it. I even tossed in a few of the old nuggets at emacs: Good OS, needs a good text editor; must have 12 fingers to work, etc.

The only thing that ever presented a problem was because I don't much care for screen, and frequently find myself editing the same file from two different locations. The end result was dealing .swp files, sometimes lost edits due to them, etc. It was frustrating, but it was my own fault, and there was nothing I could do, especially not while I addicted to gvim.

Then, I read an article on Slashdot: emacs 23 had just been released, and with that release came daemon mode. daemon mode is so much more than just a simple screen session, and I'm worried I'll not do the explanation justice, but I'll try.

With daemon mode, there is an actual daemon running in the background that holds all of your buffers. The client connects to that daemon, and displays your buffers. The end result is that you can have GUI mode, text mode, and even both modes in different connections. For example, I have the GUI mode running at my workstation at work. I come home, VPN into work, and use the text mode to edit the self-same buffers. I can't have .swp files in the way, since I'm editing the same pieces of memory as I would at my desk at work.

It's an amazing thing. I put items into my todo list (maintained via orgmode), leave work, edit the list at home, etc. And my edits cross the boundaries between GUI and text mode without me actually having to do anything.

I've discovered the joys of tramp: I can use sudo to edit files, ssh to edit files, ftp to edit files, and I get tab completion all the way.

I participate in IRC, and use ERC, the Emacs IRC client. My IRC windows stay open, as do my editing buffers. I get a shell, so I don't even have as much need of a regular command line.

Daemon mode opened my eyes, and with each day ViM would have to do more to get me to switch back. It's not impossible, but I don't believe it will happen.

I've defected from ViM, and gone over to Emacs. At least I like my new OS.

On The Choice of IDE For Python

2009-09-29T20:45:00.000-07:00

Everybody has their own preferred environment for writing code. Some prefer Eclipse, others Emacs, and still others just plain old vi or notepad.

For each of us, we find ourselves looking at our tools from time to time, wondering how we can use them more effectively. When I made the switch to Python as my primary choice of programming language, the question came up for me: Is there a better way to write and debug my code?

Taking a look at the choices out there made me less than happy. Eclipse is very heavily Java based and oriented. Every Java developer I've ever spoken with swears by Eclipse. If you're writing Java, and not using Eclipse, chances are you're not getting the most out of your tools.

As good as Eclipse is for Java, it's support for other languages has never felt right. Every time I've tried, I've found myself frustrated by what it didn't do. Other languages work, but not nearly as well. Without even trying, I am 100% convinced that Eclipse is a bad choice for doing Python development, just because of how bad it was for C, C++, and PHP when I tried them.

I've been a ViM user for about 10 years. But ViM never quite made it to a full IDE for me. Add in the dynamic nature of Python, and ViM couldn't keep up. It had no way of knowing what a given object was, so I couldn't get features like autocomplete working. I've loved ViM for years, but ViM as Python IDE just didn't do it.

Emacs is a whole different beast. It might well be great for Python, I don't know yet. I've just switched to being an Emacs user (and I'll discuss why I switched tomorrow), and am still learning everything. As such, I'll not comment on what it can and cannot do.

Before I made that switch, though, by over a year, I had been looking for a quality Python IDE. I found a thread over on Slashdot speaking about Python IDEs, and took a look through them.

IDLE was too simplistic. PIDA looks good, embeds ViM, but didn't run everywhere I wanted it to (On occasion, I work on Windows, and need my IDE able to run there when I do). Eric is another good looking one, and is my second choice. My only issue with Eric is what feels like extreme clutter on the screen at startup.

I knew that Eric could be the right choice, but I wanted to see the other choice I'd found in that thread first, before I started working with Eric. I'm very glad that I did. The Wingware IDE might be the best money I have ever spent on software (considering that I rarely use proprietary software, favoring OSS even when the proprietary might well be better, that's saying something).

The interface starts out clean and simple. The initial opening screen walks you through a tutorial, teaching you the basics of using it. And then it sets you loose. You get all the features you would even think to ask for and then some.

Of course, you've got the standard debug mode. You also are able to change variables using the debug probe. Conditional breakpoints, normal breakpoints, exception catching (allowing you to look through the different points of the stack trace to see what the various values are), a Python shell that features tab completion, a project search, version control system integration, test running, docstring display, and tab completion of method and variable names. Yes, that last feature works, though sometimes you have to give a hint to Wing IDE by specifying an isinstance() call.

All of those are common features, though. If you're using an IDE that doesn't have that, you're probably using an IDE that is still in beta. Some uncommon features include remote debugging (which is very useful when debugging a WSGI application), debugging of various server type apps (Django, TurboGears, Zope, Plone, Google App Engine), support for multiple GUI libraries (wxPython, PyGTK, PyQT), support for other framework (Twisted, Blender).

And, lastly, a very uncommon feature: Wingware is cross platform. You can get it for Windows, Linux, and Mac. All of them work.

And that's just the application. The company itself maintains a mailing list in which they actively participate. Every question I've seen come across the list in the past several months always included at least one Wingware support person replying. They listen to their customers, and use the mailing list to get ideas for what to put into the next release.

If you go for the Pro version, you get access to their source code. You can write your own plugins. They'll even provide help by answering your questions on the mailing list for dealing with issues in writing that plugin.

All in all, this is the best money I've ever spent on any software program. I can't recommend them enough.

Mercurial, Subversion, and virtualenv

2009-09-28T11:10:00.000-07:00

I do enjoy using Mercurial. My only complaint has been when using it inside of a virtualenv, and using the hgsubversion extension. Every single command I ran, I would get this message:

*** failed to import extension hgext.subversion from ~/dvcs/hgsubversion/hgsubversion: No module named svn

That gets annoying, especially since if I ran it outside of a virtualenv, the command would work without problem. The fundamental issue comes down to this: I did not want to use the system site-packages in my virtualenvs, and that's where the svn SWIG bindings get installed. The idea of rebuilding svn for every virtualenv holds extremely little appeal as well.

Fortunately, there is a better way, and it is in part due to the work of Doug Hellman on the virtualenvwrapper tool. These steps could be done without that tool, but why bother? The tool makes life so much easier for it.

Before doing anything else, get the svn module working, such that you can do this from a python prompt, and get it to work. My examples are for Ubuntu 9.04, but simply changing the paths appropriately should work for you.

From the python prompt:

   >>> import svn

   >>> print svn.__file__

Once that works, you will get a path that looks like this: "/var/lib/python-support/python2.6/svn/__init__.pyc"

This means that the path for the module I need to symlink is "/var/lib/python-support/python2.6/svn". The question becomes where to place the link, so that I will only bring in the svn module, without all the other packages in the system site-packages?

I happen to store my extensions in a directory named $HOME/dvcs , so the best place for this is there. YMMV. With that in mind, I do the following:

   $ mkdir $HOME/dvcs/svnpkg

   $ ln -s /var/lib/python-support/python2.6/svn $HOME/dvcs/svnpkg/svn

Now, thanks to the virtualenvwrapper I mentioned above, I can do this, and get rid of that error message once and for all:

   $ workon tg21

   (tg21) $ add2virtualenv $HOME/dvcs/svnpkg

And that's it. hgsubversion will now work cleanly inside of my virtualenvironment.

Cleaning Up After The Doc Sprint

2009-09-27T21:42:00.000-07:00

So, we had our doc sprint over the weekend, and I feel I should post my thoughts on the process.

Before anything else, allow me to say this: We had some amazing success. After speaking with Chris Perkins, we had success beyond his wildest dreams. And I'm blown away by how well we did. So, even though the next couple paragraphs might sound like I'm down, I'm not.

Now, let me confess that this was my first ever sprint of any sort. Somehow, I was also (to some degree) the person that was supposed to lead the sprint. It felt kind of weird to have been entrusted with that, especially considering my relative newness to the project.

As to how I did in filling that role, I'd only give myself a marginal passing grade. I feel like I didn't do enough preparatory organization, and didn't do enough maintaining of the actual focus on the task at hand. In my defense, I didn't really know what I needed to do to help make this doc sprint successful. I've learned, though, and will do better at these things next time.

My feelings over the whole sprint did quite a bit of up and down, as well. At some points, it felt like we had mired ourselves down, and were getting nothing done. At others, it felt like we were making amazing progress, and nothing could possibly slow us down. It's only now, after we've pretty well wrapped up, that I can see things a bit more clearly, and realize that we made amazing progress on this.

Before we started, on Friday afternoon, we had 192 items on our todo list, and they ranged from "define this term" to "write this whitepaper". So, we had a wide variety of items to do, and even wound up identifying previously unidentified items during the sprint. We're still closing out the sprint, but as of right now, we have 137 items on our todo list.

When you consider that we also found ourselves adding more todo items as the sprint progressed, we are certain that we closed out 55 todo items, and quite possibly we closed out 65 of them.

I don't care how you look at it, that's impressive. Nearly 1/3 of the original todo list was completed over the past two days. The people who participated in the sprint did a lot of work, and I'm proud of them. In alphabetical order (using hg log | sort), they are as follows:

Christoph Zwerschke / cito
Lee McFadden / splee
Mike C. Fletcher / mcfletch
Chris Perkins / percious
Sanjiv / sanjiv
Seth Davis / (seedifferently)

I'd also like to thank Jorge Vargas / elpargo. He did work on documenting custom content types, but in so doing he found an issue that needs to be resolved. Unfortunately, I'm not involved in the code side of TurboGears enough to really have understood, the issue, but it sounds like he's found something that will be of great help when he's cleaned it up.

I'd like to pay special attention to Mike Fletcher and Chris Perkins, though. Mike's work on cleaning up the tutorials improved them so much I can't begin to express it. And Chris's work on getting the middleware configuration documented so thoroughly is only going to make the 2.1 release of TurboGears that much better. Thanks go out to both of you for all your hard work this weekend.

Something else that might be of interest if you've read this far would be the stats on what we did. Well, here it goes: 86 commits, 51 files changed, 2568 insertions(+), 629 deletions(-). Think about that for a while, and remember that that's just the work from one weekend.

I'm still amazed, and likely will be for some time to come. This community really managed to come through, and clean up our documentation. It's a good feeling to see this happen.

wxPython in a virtualenv

2009-09-25T14:42:00.000-07:00

virtualenv is an absolutely wonderful tool. I use it every day in so many ways, and enjoy the freedom it gives to experiment with anything. Found a new tool, and want to try it out, but don't want to screw up your system's Python libraries? Try it out in a virtualenv!

Unfortunately, getting wxPython in a virtualenv is a painful process, and definitely takes some work. Here's the steps to do so:

Download this patch, and save it as $HOME/wxpatch.txt.Without this patch, wxPython2.8.10.1 will fail to compile properly. Hopefully, this will be fixed in the next release.
Make your virtualenv. If you're like me, you're going to be using virtualenvwrapper, so you'll run this command:
```
mkvirtualenv --no-site-packages wxpython
```
Activate your virtualenv. If you are using the virtualenv wrapper I linked above, you would do "workon wxpython". Otherwise, it will be "source $PATH_TO_VENV/bin/activate".
Download the wxPython source and save it as $HOME/wxPython-src-2.8.10.1.tar.bz2

Now comes the messy command. Yes, all of these are genuinely necessary in order to make this work properly. Make sure to substitue "$PATH_TO_VENV" with the appropriate real path to the root of the virtual environment you made in step 2. Warning: The next steps will take a long time.

cd $HOME
tar -xjf wxPython-src-2.8.10.1.tar.bz2
cd wxPython-src-2.8.10.1
patch -p0 < $HOME/wxpatch.txt
./configure --prefix=$PATH_TO_VENV --with-gtk=2 --enable-unicode --with-opengl
make install
cd contrib/src/stc
make install
cd ../gizmos
make install
cd ../../../wxPython
python setup.py install

Finally, you will need to ensure that your LD_LIBRARY_PATH variable is configured. If you are using the virtualenv wrappers mentioned above, create a script named $PATH_TO_VENV/bin/postactivate that has the following contents (no substitutions of any sort here at all!):
```
export LD_LIBRARY_PATH=$VIRTUAL_ENV/lib:$LD_LIBRARY_PATH
```
Now, set the script to be executable.
If you're not using the virtualenv wrappers mentioned above, edit $PATH_TO_VENV/bin/activate and add that same line at the end of the file.

With all of the above done, you only have to activate the virtualenv, and you can then import and use wx inside of it without problem.

Version Control Systems

2009-09-25T10:45:00.000-07:00

As developers, we all love and hate the various version control systems out there. RCS, CVS, Subversion, Perforce, CMVC, Visual Source Safe, Clear Case Mercurial, Git, Darcs, Bazaar, the list goes on and on.

Each of them has their own set of strengths and weaknesses. I won't go over all of them here, as that topic can easily become a book, and whatever weaknesses I point out will have someone else point out that I obviously just don't know how to use the tool properly. I will discuss one of them, though, and that's Mercurial.

About 6 months ago, I started my first ever TurboGears project. Digging in, learning what I could, participating in the IRC channel, I kept seeing mention of Mercurial, and how wonderful it was. I finally decided to take the plunge, and try it out.

My version control has not been the same since. If I can at all avoid it, I do not use anything but Mercurial. I even use hg-git and hgsubversion to allow me to access Git and Subversion repositories from within Mercurial. The end result is a tool that works for me, and makes my life easier.

Basic usage, to get a set of files into version control, couldn't be simpler:

cd $directory; hg init ; hg commit -A -m "Message"

That's it. The files are now under version control. When you make a change, simply do "hg commit". When you add a file, "hg add", and remove is "hg remove". I've just taught you about 99% of daily usage commands.

Mercurial supports extensions, too. You can get an extension that let you maintain a patch queue, similar to quilt. Another extension presents a graphical view of the branching history of your source. Still other functionality will let you self-host (on your laptop) a repository and allow others to pull from it.

With Mercurial, I get a tool that's easily managed, has a full plugin architecture that allows me to bridge version control systems transparently, and is something even I can extend (and I have).

I've barely scratched the surface of what Mercurial can do. I'd suggest taking a look both at the online documentation on their website, and also at the Mercurial book.

Before I close out this post, I want to mention Git, since I know some of you will feel the need to point out how wonderful Git is. I don't dispute that Git has some great features. It also has some amazing complexity (last time I looked, over 140 executables got installed). I have no interest in managing that much complexity. Mercurial suits me very well. I doubt I'll be changing any time soon.

Updates On The TurboGears Doc Sprint

2009-09-23T20:10:00.000-07:00

These are some pretty minor updates, but I want to take a few minutes to convey these pieces to everybody.

I've already done the triage, and categorized everything into Easy, Medium, or Hard difficulty. Difficulty, in this case, is rated as time it is expected to take a person to finish.

Easy should be less than 15 minutes (37 items), Medium should be less than an hour (140 items), and Hard is a completely unknown time (15 items).

Check out the todo list, updated daily, and decide what you would most like to accomplish.

I'd especially ask people to focus on the medium difficulty items. We've got too many of them, and I would really like to see those go way down. Perfection would be that all of these items get closed during the sprint. Reality will be far from that, though, and I'd be thrilled to see 30 of those medium items gone.

The current plan is this: I'll begin working come Friday night (Sept 25), around 10pm Eastern, on closing up some of those items. I'll be active on IRC (channel: #turbogears, nick: mpedersen) until I fall over, ready to coordinate updates from everybody. The "in progress" docs will be updated every 15 minutes.

When you want to work on an item (or items), update the "claimed todo list" wiki page. Once your name is attached, update the file(s), send me a pull request, and I'll update the docs. Easy enough, I hope :)

But Why Python? Why Not Perl?

2009-09-22T12:24:00.000-07:00

I enjoy using Python. It's clean, it's simple, it's readable. Even truly bizarre stuff manages to maintain some level of readability, at least for me. This doesn't change the fact that so many people, on hearing me say that I enjoy Python, make the statement "But why? I mean... it's Python, for god's sake!"

You're right, it is. And that's kind of the point, really. Python is not perfect. In fact, one thing always jumps out at me as being glaringly annoying: The white space. I hate it. I've always liked having a clearly delineated begin/end pair (such as {} in C, Java, etc). I know, it's silly, but it is a personal preference. No coherent reason for hating it, I just do.

The rest of the language, though, is wonderful. The syntax is consistent, the introspection capabilities allow you to do things easily that would be difficult (or impossible) in others, unit testing is easily done, and using language features you can make large scale API changes rather easily and safely.

I'm going to go with just a short example of why I prefer Python over Perl: Method signatures. Consider this method signature in Perl:

sub edit_form_data {
my ($name, $email, $url, @addresslines, $city, $state, $zip, $extradata) = @_;
}

Experienced Perl people will point out the error in there: @addresslines will take on the remainder of the values, so that $city, $state, $zip, and $extradata will all be marked as undef. If you're a relative newbie to Perl, that will bite you as you struggle to figure out why your remaining form fields are always blank.

With Python, you would do something like this:

def edit_form_data(name, email, url, addresslines, city, state, zip, extradata):
pass

Now, your data will always match up. Each of the items being passed in is an object, and those objects are self-contained enough not to suck up other objects.

A feature I left out of that signature, but that is expected by other Python authors, is the presence of a docstring for your method. It's actually a built in expectation of the language, since there exists a function named "help" that extracts that docstring and displays it to the user. Perl has no similar facility as a built-in. Yes, perldoc exists, but it is an add-on, and one that is all too frequently ignored.

Also, for that method, in Perl, what exactly are you passing in? Those $ variables could be scalars (numbers and strings), or references. If they are references, they could be references to scalars, lists, or associative arrays. Getting at the actual object for each of those is different, and the syntax is hideous.

Now, here's where the fun comes in: Notice the lack of documentation. If you're trying to call this method from outside, you have to know what the method needs (as is true with any language). But, with Perl, you have to stop, read the code, analyze the syntax, and determine the exact type based on what the syntax needs. And, if you get it wrong, you're going to pass in the wrong type of reference, and the called method is going to blow up. You might not even get a clear reason why, leaving you to debug a mess.

Suffice to say that, with Perl, knowing what you need to do when using an API is harder. I would go so far as to say that one of the only languages which is harder is assembly. Even C is easier, since you get some information from the function prototype.

Perl has frequently been called a write-only language. After trying to maintain Perl programs that have been poorly documented and have little to no unit tests, it is easy to see why.

Yes, Python can be written badly, same as Perl. But, for some reason, it usually is not. With Python, the code is normally documented, frequently has tests, and is quite readable even without the docs and tests. I don't know why that is.

But I do know that the Perl community needs to find a way to change that. Until something even better comes along, I'll stick with Python.

TurboGears Doc Sprint

2009-09-21T19:47:00.000-07:00

One item I've not spoken about before now (mainly because I've not had the time, the blog is still new) is that I use TurboGears, and enjoy it. It makes web development into something that is actually a pleasant task.

I'm working on the documentation for the next release, and we have a lot to do. As of last count, 194 items on our todo list, ranging from defining terms all the way up to writing a whole new tutorial from scratch about something new that's going into the next release.

As such, we need help to get the list done. This weekend, Sept 25 through Sept 27, we are holding a doc sprint. We're gathering up everybody we can to make these todo items go away.

Stop by on IRC, check out the current docs, check out the steps for getting everything set up (which gets rid of every warning, even) and lend a hand. We could use every extra paragraph!

Introductions

2009-09-17T11:23:00.000-07:00

That's right, another blog from another developer. This particular developer happens to be a Pythonista working on the TurboGears project (amongst far too many others).

I've dealt with (in varying degrees), CVS, Subversion, Mercurial, Make, C, C++, Perl, PHP, Python, Java, HTML, CSS... the list goes on and on, as it does for so many of us.

We each have our own views on all of these technologies, how to use them, what makes them wonderful, and what makes them awful.

Welcome to mine.