Turns out the problem was left over .pyc files. Also turns out that the fix is extremely easy. In your git repository, create a file named "hooks/post-checkout". In that file, put the following lines, and make the script executable. All .pyc files will be scrubbed every time you switch, preventing the problem entirely.
find . -type f -print | grep -v ^.git | grep '\.pyc$' | xargs rm