Document Actions

Hoss's ToDo List

by David Hostetler [modified 20110410:16:02 (Sun)] [posted 20071111:12:05 (Sun)]

stuff I need to do (in no particular order)

NOTE: These items accumulated prior to my use of a mantis tracker, and all of the uncompleted items will eventually migrate there. The 'Completed' section of this page remains useful as a log of technical solutions -- and will likely migrate to the tracker as well, for archival purposes.

Technical

General

decide about new PC purchase (dmraid HOWTO for nforce 590)
choose an svn web interface (SVN-WEB, websvn, svnbrowser, subportal)
decide whether or not to use ezmlm-web
reinstall winxp on game box
reinstall gentoo on game box
reinstall xp on laptop (blocked by DVD boot issue)
install gentoo on laptop (blocked by DVD boot issue)
decide about full svn backups (hot backup)
implement postgresql backup
upgrade network picture
automate evms reporting (to get raid status)
setup the UPS monitoring software on the server
investigate how to broadcast UPS events to to other hosts on the LAN (for coordinated shutdowns)
implement svn sync to backup proxy
implement zope sync to backup proxy
implement qmail sync to backup proxy
implement ldap sync to backup proxy
implement mysql/postgresql sync to backup proxy
implement proxy rollover process
integrate backup logs into logrotate

Frivolous

tinker with GKrellM
tinker with XPlanet
tinker with Celestia
tinker with fluxbox themes

Zope/Plone

install external editor
get different versions of IE running concurrently (for compatibility testing) (see linux instructions)
figure out how to get zope backup (repozo) to retire old backup data
investigate and implement work-around for IE problems with CSS in plone skin
attempt a kupu feature customization
fix zoom for the kupu edit window
fix kupu style issues (dark on dark)

Games

write jak review
write dw2 review
write gow review
write TA review
create rfactor comp mod
create morrowind comp mod
create oblivion comp mod
create thief comp mod
create UT2k4 comp mod
create UT comp mod

Completed (yay!)

investigate switching svn repos from BDB to FSFS (might allow repos to live directly on the RAID) See the FAQ entry on FSFS conversion (20070801)

done -- had to do an svndump from the old and svnload into a newly created FSFS repo.

migrate to netqmail (mini howto, full howto, forum post) - qmail is now masked by gentoo in favor of netqmail (200706??)

this was fairly straightfoward -- the howto's were sufficient

implement mysql/postgresql backup (20070428)

mysqldump run for each DB
full DB stored as compressed SQL file

implement ldap backup (20070428)

slapcat run through sudo
full DB saved as a compressed ldif file

investigate issue trackers (20070401)

chose mantis, despite the fact that it's PHP-based
readily customizable
good email support
supports attachments and good permissions hierarchy

investigate exposing ezmlm archives (FAQ, ezmlm home, ezmlm manual) (20070313)

chose ezmlm-www, because of its search capability
see the howto

decide how best to sync the test zope with the live zope, and vice versa (20070307)

there are 3 pieces, the zodb (Data.fs), the FSS repo, and the Products folders
the zodb and FSS need to go from live to test, the Products need to go from test to live
zodb is accomplished by using restore-zope-db.py (use the latest zodb backup and point at the test zope var/Data.fs)
FSS is accomplished by using rsync locally: rsync -avz --inplace <live FSS>/ <test FSS>
Products/NegativeSum* folders are already managed by svn (debating putting the whole Products hierarchy under svn)
the etc/zope.conf files are easy and different enough at the moment to just leave alone

config firefox to use gvim for 'view source' (20070301)

edit (create if necessary) <profile>/chrome/user.js and add the following:

user_pref("view_source.editor.external", true);
user_pref("view_source.editor.path", gvim);

restart firefox
note: support for external editor is wickedly busted in firefox 2.0.* (it doesn't escape spaces in filenames when it invokes the external app), so I resorted to using ViewSourceWith add-on, which works wonderfully
resources:

mozille kb page on config variables

implement automated 'emerge --pretend world' reporting (20070210)

wrote a nice python script to replace the simple 'emerge --sync' shell script that runs in cron.daily
script does both the sync and the pretend and reports results of both via email, with a slew command line options
email subject looks like its from portage, so mail filters that work on ELOG output will grab these reports also

get email working from shelll (and python) scripts (20070209)

the /bin/mail program comes from 'emerge mail-client/mailx'
there might be a native python module that accomplishes emailing, but /bin/mail is wicked easy to use and it seems reasonable to expect that it's always available (though I was surprised that I had to emerge an ebuild to get it)

get rid of plone roles left over from the PloneSVN product (20070208)

instead of being able to just delete the roles from acl_users/portal_role_manager, as you'd expect, you have to use the Security tab from the root plone folder, which gives a full presentation of all the roles in the site, and a 'Delete Roles' option at the bottom of the page. Simple once you know where to look.

fix kupu button problems with non-admin users (20070208)

had to explicitly enable a kupu-related security permission for Authenticated users
the default permission settings should have been sufficient, but for some quirky reason aren't
the permission in question was: 'Kupu: Query libraries', which is found with the rest of kupu's permissions under http://<plone>/kupu_library_tool/<'Security' tab>
resources:

plone users news thread
my follow-up question (not available yet)

allow <object> to be used in plone page content (20070131)

necessary to reconfigure plone's HTML filters to allow the <object> tag (which is dangerous for publicly editable sites)
edit Products/CMFDefault/utils.py and Products/PortalTransforms/libtransforms/utils.py:

remove the "object" entry from NASTY_TAGS, and add it to VALID_TAGS

edit portal_transforms/safe_html (in plone's web management interface -- i.e. TTW)

remove the "object" entry from NASTY_TAGS, and add it to VALID_TAGS

restart zope
this is an interim fix, since it allows <object> site-wide. A better solution is to create a new plone product (can just be a subclass of ATDocument) but have its default_output_type be specified as text/html instead of text/x-html-safe (thus bypassing the safe_html filter). This new type can then have permissions that make it only addable in a particular folder, or by a particular group.
resources:

fix direct link of PDF files into svn repo from html/apache (shows up as raw, not PDF) (20070130)

this was easy: just set the mime-type for the file in the repo:

svn propset svn:mime-type "application/pdf" <file>

subversion doesn't automatically set mime-types for binary files, so if there something specific, you have to do it individually (you could probably get really fancy and have some clever hook-script that does the propset automatically for recognized types)
resources:

'svn help propset'
'svn help propget'

upgrade BIOS on laptop (20070108)

used the cdrom suite from FreeDOS to make a bootable DOS cdrom image (after adding in the new AMI BIOS files)
upgraded to v670 - which didn't seem to have any effect on the winxp install problems
Averatec 2100 series forum
forum post: useful 2100 series links

securely expose apache webroot through samba, now that only the main server is running smbd (20061106)

this was an issue because winXP has the following quirk: once you've authenticated to a smb server as a particular user, windows won't connect to any other shares on that same server as a different user, despite pretending to offer precisely that capability in the 'mount network drive' wizard. Previously, I had samba running on two servers, each mapping to the same nfs mounted drive.
I wanted to have access to apache's webroot from my win32 box, but I already map to the smb server as myself ('hoss') to get my home drive. I needed to figure out how to get correct access to the webroot under these conditions.
there were a couple of possible solutions:

I could have enabled some netbios aliases in samba, and exploited the fact that if you specify a different netbios name when mapping the network drive, winXP doesn't know that it's actually the same server that you've already mapped to for a different share.
Instead, I opted to properly define a specific share in smb.conf for the webroot, and rely on proper group membership to allow me r/w permssions to the folder. So, since I was already a member of the 'apache' group, I just used the following in smb.conf to securly expose the folder over samba. The key parts are 'force user', 'force group', and 'valid users', which enable samba to restrict access, and to ensure that all files created in the folder are of the correct ownership.
[web]
comment = Apache web root
path = /home/apache
public = yes
writable = no
write list = @apache
create mask = 0644
directory mask = 0775
force user = apache
force group = users
valid users = @apache

resources:

'man smb.conf'

find a 4 port dual DVI KVM with USB2.0 and audio, and if it exists, decide if it costs too much (20061105)

apparently, that's too much to hope for. Either dual DVI, or USB2.0 and audio. Not both. I settled on the Avocent SwitchView DVI. 4 port, high resolution DVI, independent switching of KVM and USB2.0 ports AND audio. Since I'm using a triple-head setup, I don't really care if I only get the middle DVI display switched (as opposed to the middle and either left or right). USB2.0 was the big thing. My current KVM switch is USB1.1 and winXP complains everytime I switch. Very annoying.

map 'music.negativesum.net:9000' to 'music.negativesum.net' via apache (20061102)

Added a VirtualHost block to /etc/apache/vhosts.d/00_default_vhosts.conf with the following key elements:

ServerName music.negativesum.net
ProxyPassReverse / http://music.negativesum.net:9000/
RewriteRule ^/(.*)$ http://music.negativesum.net:9000/$1 [P,L]

That pretty much does it.
Something like mod_proxy_html might be in order for a more complete solution if there are ever any explicit :9000 references in the html handed back by SlimServer.
Note - I had to set CSRF security to 'none' in SlimServer to prevent it from noticing that the referer header isn't the same as the requested URL. Not a big deal since I'm still restricting SlimServer access to just the LAN.
resources:

get post-commit emails from svn commits (20061027)

subversion has a script called mailer.py (configured via mailer.conf) that you can put in <repo>/hooks
take the post-commit.tmpl and create an executable script 'post-commit.sh', and edit it to call mailer.py, with the appropriate arguments
I created a dedicated email list for each repo that I want to have svn emails, and use that address in the mailer.conf
Since the svn commit email has to have a 'from address' that is allowed to post to the email list, I added 'svn@negativesum.net' to the list, and then created a qmail alias entry (via /var/vpopmail/domains/negativesum.net/.qmail-svn) that just throws away any incoming mail

figure out how to discover the python.exe path from a windows shell batch script -- this will significantly simplify the F-Sum rFactor mod sync/install process (20061025)

turns out there's a shell cmd 'reg' which lets you do stuff to the registry, so I was able to use it to query for the installation path of python
windows shell (i.e. batch) scripting is an insult to scripting
resources:

here's the actual batch snippet that does the work (note, I don't think it's actually got the correct parsing to handle multiple installations of Python -- that is, I know it works for python25, but not python24 or python23)

for /F %%A in ('reg query "hklm\SOFTWARE\Python" /s ^| find "InstallPath" ^| find /v "InstallGroup"') do call set pykey=%%A
echo using python registry key: %pykey%
for /F "Tokens=3*" %%A in ('reg query "%pykey%"') do call set pypath=%%B
echo using python installed at: %pypath%
"%pypath%"\python.exe your_script.py

install and configure new RAID server (20061025)
wipe and reconfigure 2nd proxy server (leaf) (20061025)
decide how to get images in plone as file-system-stored image objects (20061025)

created a custom product that uses FileSystemStorage

figure out why PloneSVNAccess isn't working (20061025)

well, for one, it seems terribly buggy.
for two, it's incompatible with plone 2.5 anyway.
I'll keep an eye on it, waiting for an updated version

investigate why 'fullscreen mode' doesn't do what I think it should do (it leaves some of the page slots visible) (20060915)

simple matter of identifying which pieces of page layout needed to be customized with a .fullscreen section in the stylesheet
note - fullscreen mode uses a cookie, so it's sticky
mostly, it sets some page elemects to 'display: none', and a few elements need their width/height adjusted

investigate unified authentication scheme (LDAP?, NIS?) (20060914)

using openldap, nss_ldap, pam, and samba to unify authentication on both linux and windows.
Wow, this was quirky.
HOWTO forthcoming.

fix qmail config so that outgoing mail to comcast.net addrs get delivered (20060910)

had to establish a route for qmail smtp that relays emails for @comcast.net to smtp.comast.net
add the following to /var/qmail/control/smtproutes: comcast.net:smtp.comcast.net
see 'man qmail-remote'

investigate file size limit for ATManagedFile (either fix or switch to something else - plone FSS) (20060828)

FSS seems to be the solution. It has similar storage strategies to ATManageFile, without an inherent size-limit
I had to create a subclass of ATFile, using the new storage mechanism (FSS) instead of the default storage (AnnotationStorage - i.e. ZODB). The new type - NSATFile - is the first of several core types I'll create that use FSS, and mirror the capabilities of core AT types.
I created a new product package (independent of my skins package), though I may ultimately merge them if I end up doing some custom skin/view templates for the types.
tested upload of a large (~700MB) file. Worked find over lan. Moving (reparenting) the file in plone does what you'd expect on the backend in the file repository heirarchy: i.e. it correctly moves the content and associated folder(s).
So far so good! Yay FSS!

implement zope backup (see the Plone HowTo) (20060826)

created a custom python script that calls the main function in repozo, which comes with zope
moved the rdiff-backup commands into a more general backup script, which is still in backup's crontab
zope backup occurs before the raid rdiff mirror
zope backup destination is the raid: /home/backup/...

upgrade plone on gentoo and document process (20060824)

stop the zope/plone instance:

/etc/init.d/zope-<instance> stop

get the new plone source tarball, and unzip it somewhere
cd to the new plone source dir (something like 'Plone-2.5')
clear out the existing product folders:

ls -1 | xargs -L 1 -I '{}' rm -rf /var/lib/zope/zope-<instance>/Products/'{}'

install the new plone source into the instance:

ls -1 | xargs -L 1 -I '{}' cp -R '{}' /var/lib/zope/zope-<instance>/Products/

ensure ownership of the new plone source is correct:

chown -R apache:apache /var/lib/zope/zope-<instance>/Products

start the new zope/plone instance:

/etc/init.d/zope-<instance> start

http://domain.com:8080/manage

note that you might not be using port 8080, see /var/lib/zope/zope-<instance>/etc/zope.conf

navigate to your plone instance in zope and perform the migration action with the following objects (they'll have red exclamation marks next to them indicating that the version discrepancy has been detected):

portal_migration
portal_atct

note: you can make a backup copy of your Products directory and var/Data.fs prior to the migration, just in case
resources:

upgrade zope on gentoo and document process (20060824)

build the new zope version:

emerge --update --deep --ask zope
note that this doesn't interfere with any current or past zope versions installed

stop the current zope and remove it from the default runlevel:

/etc/init.d/zope-<instance> stop
rc-update del zope-<instance> default

create a zope instance of the new version:

zope-config
select the new version number
specify the same user:group settings as your old zope instance
edit the new /var/lib/zope/zope-<instance_new>/etc/zope.conf to match settings from your old zope.conf

copy the Data.fs from the old instance to the new one:

cp -p /var/lib/zope/zope-<instance_old>/var/Data.fs /var/lib/zope/zope-<instance_new>/var
verify the ownership is kosher

copy all of the installed products and extensions into the new instance:

cp -Rp /var/lib/zope/zope-<instance_old>/Products /var/lib/zope/zope-<instance_new>/
cp -Rp /var/lib/zope/zope-<instance_old>/Extensions /var/lib/zope/zope-<instance_new>/

start the new zope instance and add it to the default runlevel:

/etc/init.d/zope-<instance_new> start
rc-update add zope-<instance_new> default

note that the old instance is still viable and can even be active simultaneously with the new one (ensure you use a different port in its zope.conf, though)

figure out how to get wireless out to the other tivo (20060821)

the antenna on the USB wireless nic just wasn't strong enough
the dlink dwl-2100ap can be configured to 'client' mode, wherein it acts essentially like an external wireless nic, so I used one of my 3 units in that mode for the tivo.
also upgraded the firmware on all of my wireless hardware
NOTE: the new dlink firmware uses cookies with the http interface, so I thought I'd screwed up the firmware upgrade when in fact I just had to clear session data and enable cookies for the device's IP
now using the USB wireless nic with the upstairs tivo (~65-70% signal strength with the downstairs AP)

find a solution for how to manage/package/distribute the individual comp mods for games (20060818)

using rsync on win32, with a UTF8 patch

cwrsync is a win32 packaging of rsync, cygwin dll, and some other win32 ports of unix tools (ssh related)
unfortunately, the cygwin guys can't get their act together and implement UTF8 support, which is essential for this to work, as there are unicode characters sprinkled in among the filenames for the various game mods. Without UTF8 support, downloads from the rsync server to a win32 box will bugger the filenames, making them potentially inconsistent with the same mod content installed natively.
fortunately, a japanese group has implemented UTF8 support for cygwin and provided it as OSS: UTF8-Cygwin
unfortunately, the cygwin guys are being pricks about the patch and refusing to integrate it, pissing off everyone who wants to use cygwin-enabled tools to work with the rest of the universe that has moved into the 21st century and is properly supporting unicode
so I bundled the UTF8 enabled cygwin dll from okisoft with the cwrsync package and am treating it as the front end for my compilation mods

I'm using svn as the back-end for the comp mods, letting me easily maintain the content and version control it. svn is efficient at storing binary files.
I sync each mod repo to a directory on the server, and then expose that directory as an rsync location. I can custom configure each rsync location in /etc/rsyncd.conf.
clients can use the UTF8/cwrsync client to grab any particular mod comp easily, incrementally, and anonymously.
disadvantages:

having to open the rsync port to the WAN
rsync isn't particularly conducive to managing actions on the repo besides additions, since I can't use the 'delete' command because the mods are typically overlays on a game installation

fix the UTF-8 problem with svn syncs on the server (20060817)

it was a locale problem (big surprise) on the client end of svn (I was doing syncs to the repo)
had to add the following to /etc/env.d/02locale:

LC_ALL="en_US.utf8"
LANG="en_US.utf8"

then to make it stick, I did:

locale-gen
env-update && source /etc/profile

the locale-gen command uses the contents of /etc/locale.gen, which should contain at least the following:

en_US ISO-8859-1
en_US.UTF-8 UTF-8

references:

http://forums.gentoo.org/viewtopic-t-486439.html
http://www.gentoo.org/doc/en/utf-8.xml
http://www.gentoo.org/doc/en/guide-localization.xml
the 'Localization' section of ch.7 (Advanced Topics) of the Subversion book: "Subversion's use of locales"
http://svn.haxx.se/users/ (the svn users mailing list archives)

investigate how to eliminate the SSL cert warning when logging into plone (20060806)

regenerated SSL key and server cert (problem was that apache was using the auto-generated cert from the gentoo install, which had 'localhost' as the CommonName
used cacert.org to sign the crt, which is a little better than a self-signed cert at the moment, and hopefully as good as a commercial cert once the CACert root certificate gets included in browsers, etc..
cert and key files are stored in /etc/ssl/private
they are referenced in /etc/apache2/modules.d/41_mod_ssl.default-vhost.conf
the server certificate will expire 6 months after it was signed by cacert.org
looks like I can just login to cacert and request a 'renew' for the cert, when needed
tips:

use the following to show the contents of a crt file:

openssl x509 -text -in <filename>

references:

verify rdiff-backup script is working with cron (20060730)

Sections

Personal tools