The Code Cave

I’m in the process of gathering a list of things to check in your plugins directory to see if your site will survive the 2.1 upgrade without blowing up. I might turn this into a “Ready to Upgrade Plugin” I’m sure it won’t ever be able to catch everything, but it just might make things go more smoothly for some. With 2.1 possibily coming out on Jan. 22, I don’t know how complete this will be, but it might be worth a shot…

The replacement of the Table* variables is a good place to start. Here’s an email I sent two days ago on the subject. A person had written for help to the hacker list. He had just upraded from WordPress 2.0.5 to 2.1 and suddenly was getting an error in an sql statement.

The SQL statement looked something like this: “select blah, blah, blah from , where“

After a from statement, there is normally a table name. In this case the comma indicated there were two table names. He wanted to know if this could have been caused by the upgrade.

Here is how I responded:

Yes, there were separate variables that held the table names. They were depreciated in version 1.5 and the developers tried to get the word out for this. In the comments, the 1.5 code said that the variables would be around for a few more months:

As you can see the preferred way to access this information is through the $wpdb->tablename method that was added to the (then) version 1.3 source code in the summer of 2004.

As it is now 2007, the 2.1 release is FINALLY removing these variables as the comments indicated would be done two years ago.

Please contact the author of those plugins and send them to this link: http://wordpress.org/development/2006/12/naughty-or-nice/

That will help get them back on trac with the WordPress code.

And btw, the pun in the last line was intended, but no on seemed to catch it!

[edit]
Hilarious - Because I put “Summer of 2004″ and “release” in the post, adsense thinks I’m writing all about movies and is recommending DVDs when you view only this post…
[/edit]

Well I can’t call it the 5 second upgrade script any more… Since adding full file backups, and compressed database backups, from Windows desktop, through manual log in and launch of the update script, it took me ~15 seconds to update an active blog with a couple dozen posts and log all of the results to a html log. I’m fairly certain I could type my password faster and shave off a few seconds. The script itself, which now shows start and stop times, only took 2 seconds to do its work. The rest was connect and login time. Sometimes it took a whole 30 seconds for the process to complete, web and server usage being what it is, but either way, wow. I should say that I used this last night with all of the options turned on, creating file backups AND gzipped backups AND database backups AND an HTML log file AND adding extra verbosity AND updating my 6 WordPress blogs and it took a full 8.5 minutes. I had to actually minimize the window to get it out of my way…. Between that and typing in the script name at the shell prompt, I was exhausted!

When I think of how long it used to take me to update just my wife’s blog, I just have to shake my head. Each release was a many-night, if not many-week process till I had the spare time to concentrate on doing the whole thing right. And I had to look up the instructions on the database backup every time… I’m just so glad this script is done.

The basic functionality is now complete and I am calling this a RC 1 release. It still needs further testing (especially the MySQLDump stuff. Does everyone HAVE MySQLDump? Should I disable this feature by default?), but it is fairly stable now.

Here’s the basic functionality

#  You can use this program in several ways:  
#    * In the default mode to download the latest and greatest update,
#      make an uncompressed copy of your files, makes a compressed backup
#      of your database(s), distributes the file to any number of 
#      directories, and performs the web steps
#    * Configure it to make a online copy of your files you use for easy 
#      recovery AND a compressed copy that you can download.
#    * Add custom directories and backup MORE than just WP. 
#    * Configure it update from a local file each night and start with
#      a clean blog every morning.  
#    * Use it as a nightly backup script by disabling all other steps

I’ve made about a gazzilion improvements and took the advice of a dozen or more reviewers out there. I think the new script is much improved. I’m really pleased with how well the database backup stuff works. I change into each blog directory, read all of the connection information from the wp-config.php file and use that to connect to the database. (I’m fairly certain this will work well for most servers, but I’m a little worried that *nix gurus will not be using TCP to connect to their databases and I have that hard coded. You gurus should let me know if this is an issue in the comments for this post!) I also then query the tables names from the specified database using the prefix specified for the blog. This means that this process will work for ALL versions of WP and will not grab non-WP stuff like vBulletin tables. It also means that it works just as well if you have 1 blog per DB or ALL blogs in 1 DB. It doesn’t matter. The DB backup for Blog1 has ONLY the data for Blog1. That is better for security, size, time and opens a neat avenue for testers who want to restore their blog to a different database/databasename and then test a major upgrade running their full blog out of a different directory. I’ve structured the tarball backups to make this easy too. *SORRY* There I go into tech speak again, but it is neat stuff, that is normally totally hidden from view.

You can peruse the text version, EasyWPUpdate.txt, here: link

Like the new name? I think it is better. I put TCC in front of all of my plugins, but really there is no need here. And yes, the sample version has grown to 851 lines. That’s not ALL code of course. It is heavily documented and includes some HTML that will give you a nice webpage log for you to peruse after the process is done. You can see a sample log here: link.

Now, I had deliberately made that last post very intimidating. I wanted people to be wary of the script. Now, I have much more confidence in its ability and quality. I’ve learned a lot in the last week. From using procedures, to the trap function, to sed and MySQLDump, to basic shell coding practices. It was all fun and you get the benifit. Especially because there are three versions of WP in the pipeline: 2.0.7 (Very, Very Soon), 2.0.8 (in the works), 2.1 (Very Soon).

So, I’ve made this post easier to read and the script easier to configure. I’ll do a full document later, but here are the basic steps to install this script:
1. Use Telnet or Putty to connect to your website and log into the shell
2. Type the following line:
wget http://www.thecodecave.com/downloads/EasyWPUpdate
3. Type the following line:
chmod +x EasyWPUpdate
4. Use an editor to change the values in Step 1 and save it again.
5. Run the script by typing:
EasyWPUpdate

That’s it. You will have just made backups of the files and database and updated all of your blogs. When 2.0.8 comes out, the process will be:
1. Log in
2. Type
EasyWPUpdate

And you are done.

Now, step 1 looks like this:

# ##################################################################
# Step 1. Tell the script where to find the blogs
# ##################################################################
# List all of your WordPress directories and urls here.
#
# Each Blog should have a BlogDir and a BlugURL.
# Each Blog should have its own number [1],[2],[3] etc
# Delete the ones you don’t need.
#
BlogDir[1]=’site1dir’
BlogURL[1]=’www.example.com’

BlogDir[2]=’site2/news’
BlogURL[2]=’www.site2.com/news’

BlogDir[3]=’wordpress’
BlogURL[3]=’blog.site3.com’

That isn’t that hard to change is it? Even in VI.

Some quick tips on editing the script
1. type
vi EasyWPUpdate
2. Hit i
3. Make your changes
4. Hit ESCAPE COLON W to save your changes (or skip this step to lose changes)
5. Hit ESCAPE COLON !Q to immediately quit

Also, if your root directory is accessible from the web, you might want to change the name of the script
mv EasyWPUpdate SomeSneakyName
to prevent unauthorized access.

If you ran the alpha 3 version of the script, you can copy and paste configuration over BUT!!!! you have to make the following changes:
The BlogDirs[] array has been renamed to BlogDir[]. Drop the “s” from all of those variables.

You should not need to copy the Common*Prefix variables over, but if you do, make sure to remove the trailing slash from the CommonRootPrefix variable.

I think that’s all you need to be aware of.

If you are a guru, please read through all 6 setup steps (and the rest too) there may be things you want to change.

I’ve also updated my Did That Help page and added a forum specficly for this script.

That might make discussions a little bit easier.

Well that’s about it. Let me know how it works. I’d like to get some good testing in before 2.0.7 comes out. I’ll also do some testing with updating to 2.1 so I am certain that will work well. I also need write instructions for the database recovery steps. The script has built in instructions if it blows up in the middle of updating the files. So, that is handled.

I’ll leave you with the change history and credits section from the script. Enjoy!

# History:
#    01/AUG/2006 - BL - Created
#    21/DEC/2006 - BL - Added multiple blog arrays 
#                       Added options at the top of the script
#    04/JAN/2007 - BL - Added File Backup routines
#                       Added web update
#                       Added tmp directory usage
#                       Added local source ”freshen” option
#    11/JAN/2007 - BL - Added ”steps” and further comments
#                       Added quotes around many vars to protect against spaces
#                       Changed TMPDIR-/tmp to TMPDIR:-/tmp
#                       Changed `pwd` != ”$tmp” to `$pwd` != ”$tmp”
#                       Added further error trapping around cd and cp routines
#                       Fixed file backup procedure, was adding extra layers
#                       Added ability to backup to tarball
#                       Added SQL backup procedure
#                       Fixed local file backup procedure
#                       Removed “Verbose” from cp to make messages clearer
#                       Added log to webpage for Joe.
#                       Fixed inconsistent use of trailing / in path variables
#                       Added status messages throughout
#                       Added recovery instructions in case of failure mid backup
#                       Added a list of directories to backup
#                       Added Credits section
#
# Credits - I want to thank all of the readers of TheCodeCave.com, for
#   their testing of this script. I especially appreciated Michael, Maciek, 
#   Aaron and Joe for all of the helpful suggestions.  
#   A very special thanks goes out to goldfish on the FreeNode #sed channel                     
#   who will be PayPaled a Cafe Voltaire tomorrow.  I would have spent days
#   figuring out the RegEx for the SED commands.  Prec, also from #sed gave 
#   provided me with a working cr/lf stripper.  For bash, lhunath, jp-_ and the                     
#   whole crew at #bash on FreeNode gave great line by line suggestions.
#   They basically gave it a full code review!  None of this would have been 
#   possible without Advanced Bash-Scripting Guide. 20 days ago I didn’t know
#   what bash was.  Now I’ve written a powerful script with features I’ve not
#   seen anywhere else.  If you have any questions about the code in this 
#   script, you’ll find the answers here:&nbsphttp://www.tldp.org/LDP/abs/html/
#

Before I release the next version of the easy WordPress upgrade script, I would like to have some solid testing done specifically on the Database backup section.

Anyone familiar with backup up with MySQLDump and restoring from that backup is invited to try this script.

I suggest you have a known good backup of the DB in case any thing goes wrong.

This file is meant to be run from your blog’s directory and will get all of the information it needs from the wp-config file. That’s really the part I want to test. This script will display your database and password information on the screen. Please verify that it matches what you have in wp-config.php. It then uses that information to get a dynamic list of the tables in your database having the prefix you specified in wp-config. Then MySQLDump is used to export ONLY those files from the database.

So, you have a targeted database backup that is agnostic to the version of WP that you are using. It will work on WP 2.1 or 1.2 (assuming there were no major wp-config format changes). It will even withstand minor changes to the way the info is stored in wp-config.php.

So I hope it is resiliant.

The easy way to test is to:
1. Telnet to your system (having verified you have a good backup of the DB already)
2. Cd to your root WP directory containing wp-config
3. Execute the following wget:
wget http://www.thecodecave.com/downloads/wpbackup4testing
4. Give yourself execution permission
chmod +x wpbackup4testing
5. Call the script
wpbackup4testing

Then just look for the file named SQLDump.gz

IMPORTANT!!!! DELETE THE DUMP FILES WHEN YOU ARE DONE!!!!
Nothing in the public areas of your folder is safe. Google will find it. Then your passwords and everything else are theirs for the download. Do not keep DB backups online.

The code looks like this:

#!/bin/bash
#
# WordPress Backup script from wwwTheCodeCave.com
#
# This is the incomplete testing version. The final version will be better.
#
# Go find it from http://www.TheCodeCave.com
#
# Brian Layman
# Jan 11 2007
#
RANDOM=$$$(date +%N)
Dumpfile=”SQLDump$RANDOM.gz”
echo “DFName: $Dumpfile”
DB_NAME=$(sed -n “/define(’DB_NAME’, ‘/s/.*, ‘\([^']*\).*/\1/p” wp-config.php)
DB_USER=$(sed -n “/define(’DB_USER’, ‘/s/.*, ‘\([^']*\).*/\1/p” wp-config.php)
DB_PASSWORD=$(sed -n “/define(’DB_PASSWORD’, ‘/s/.*, ‘\([^']*\).*/\1/p” wp-config.php)
DB_HOST=$(sed -n “/define(’DB_HOST’, ‘/s/.*, ‘\([^']*\).*/\1/p” wp-config.php)
TABLE_PREFIX=$(sed -n “/\$table_prefix = ‘/s/.*= ‘\([^']*\).*/\1/p” wp-config.php)

echo “Gathering information:”
echo “DB_NAME: $DB_NAME”
echo “DB_PASSWORD: $DB_PASSWORD”
echo “DB_USER: $DB_USER”
echo “DB_HOST: $DB_HOST”
echo “TABLE_PREFIX: $TABLE_PREFIX”
mysql $DB_NAME –port=3306 –protocol=TCP –host=$DB_HOST –user=$DB_USER –password=$DB_PASSWORD -e”show tables like ‘$TABLE_PREFIX%’”>TableNamesForSQLDump.txt
WPTables=$(sed -e “/^\($TABLE_PREFIX[^ ]*\).*/!d;s//\1/” -e :b -e “N;s/\n/ /;bb” TableNamesForSQLDump.txt)
echo “WPTables: $WPTables”
echo
echo “Performing backup”
(mysqldump –opt –port=3306 –protocol=TCP –host=$DB_HOST –user=$DB_USER –password=$DB_PASSWORD $DB_NAME $WPTables | gzip > $Dumpfile)||{
echo “SQL Dump Failed”
exit 999
}
echo “Backup successful. Please examine file $Dumpfile”

THIS ARTICLE IS OUT DATED. Please see: http://www.TheCodeCave.com/EasyWPUpdate for the current release.

Well I’ve gotten some good feed back from all of you. And I want to say Thanks!

I do have a new version of the script that has some improvements.

• Fixed extra directory levels in the backup


• Improved use of quotes


• Local file location did not untar the file


• Removed “verbose” from cp to make messages clearer


• $CommonRootPrefix no longer requires trailing ‘/’


• Added comment “DO NOT PUT A SLASH AFTER ANY OF THESE VARIABLES”


• Added a list of directories to backup


• Changed long backup section into a loop of the directories


• Changed to tmp directory before performing wgets


• Added a few more update steps for forward compatiblity


• Added SQL backup code


• Added Zip of backups

I’m doing local testing on these changes and will probably release this more publicly Wednesday or Thursday.

In the mean time, I would like to ask for your help.

What is the most optimized version of this line that we can come up with:

grep “define(’DB_NAME’, ‘” wp-config.php|sed -e ’s/define(.DB_NAME….//g’|sed -e ’s/.); .. The name of the database//g’

That line, run from any blog directory, returns the DB_Name in a way I can pass it to MYSQLDump. It is neither, pretty, nor optimized, nor resiliant.

What is the most optimized way YOU can think of to write that bash statement?

EDIT:

SED problem is solved. Thanks to GoldFish on the #SED channel of FreeNode.

I stumbled across this one weeks ago, and saw it again yesterday. I though I would share:

Warning:
This could bother very young kids even with an explanation of what’s happened…* It certainly bothered the peole in the park that day… but my 10, 8 & 5 year olds watched it. It’s not gruesome.

I give you, The MindFreak
(more…)

I’ve taken an idea from Lloyd D Budd’s email to me “Well Met Cave Coder” and created a “People of WordPress” link category. Never mind the fact that during his rearrange, I seem to have dropped off the end of his list… But we won’t go there ;P (I’ll still put you on my list Lloyd…) You can find it on the right hand side of my blog.

I made this list by taking a look at my WP- email folder history back to an arbitrary date. I then grabbed the names of people with 100+ contributions and by adding a few more names that stood out to me - probably because I’ve talked with them in the past or their name is unique enough that it just stuck in my brain. This, by far, is not the list of everyone that’s made significant WordPress contributions. But, if you want to dive into the mindset of people that have an influence on the future if WordPress, this is a good place to start.

In the random order my site just chose, I give you my initial People of WordPress list:

People of WordPress

• Podz
• Andy Skelton (skeltoac)
• Peter Westwood (Westi)
• David Chait
• Alex King
• Donncha O'Caoimh
• Craig Hartel
• Michael Adams-mdawaffe
• Matt Mullenweg
• Ryan Boren
• Dougal's Geek Ramblings
• Robert Deaton
• Justin
• Michel Valdrighi
• Owen Winkler
• Mark Jaquith
• Mike Little
• Lloyd D Budd
• Roy Schevstowitz
• Doug Stewart
• libertus
• Viper007Bond
• io_error
• Rob Miller
• Computer Guru
• Michael / miklb
• spencerp

When working toward updating my upgrade script to read the usernames and passowords from the wp-config files, I came across this helpful list:

————————————————————————-
USEFUL ONE-LINE SCRIPTS FOR SED (Unix stream editor) Dec. 29, 2005
Compiled by Eric Pement - pemente[at]northpark[dot]edu version 5.5

Latest version of this file (in English) is usually at:
http://sed.sourceforge.net/sed1line.txt
http://www.pement.org/sed/sed1line.txt

This file will also available in other languages:
Chinese - http://sed.sourceforge.net/sed1line_zh-CN.html
Czech - http://sed.sourceforge.net/sed1line_cz.html
Dutch - http://sed.sourceforge.net/sed1line_nl.html
French - http://sed.sourceforge.net/sed1line_fr.html
German - http://sed.sourceforge.net/sed1line_de.html
Italian - (pending)
Portuguese - http://sed.sourceforge.net/sed1line_pt-BR.html
Spanish - (pending)

FILE SPACING:

# double space a file
sed G

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed ‘/^$/d;G’

# triple space a file
sed ‘G;G’

# undo double-spacing (assumes even-numbered lines are always blank)
sed ‘n;d’

# insert a blank line above every line which matches “regex”
sed ‘/regex/{x;p;x;}’

# insert a blank line below every line which matches “regex”
sed ‘/regex/G’

# insert a blank line above and below every line which matches “regex”
sed ‘/regex/{x;p;x;G;}’

NUMBERING:
(more…)

THIS ARTICLE IS OUT DATED. Please see: http://www.TheCodeCave.com/EasyWPUpdate for the current release.

Well I’ve upgraded the 35 Second upgrade script significantly for this release.

Here’s a summary of the new features:

1. Customizable options at the top of the script
2. Works for unlimited numbers of blogs with just updating the header
3. Optionally performs Web update steps as well
4. Optionally performs backups of all WP related files
5. DISABLED: Performs a Database Backup
6. Backups are to a directory of your choosing suffixed with “today’s” date
7. Can now upgrade blogs in a WordPress directory or any other
8. Respects the tmp directory on your server
9. Can be modified to perform nightly refresh of all blogs from a local tarball
10. Can be customized to retrieve beta releases
11. Can be customized to retrieve from WP.org’s archive folder

This hasn’t had a lot of testing yet outside of my blogs, but it has worked BEAUTIFULLY for me and as you can see I am running Version 2.0.6 thanks to spending 5 seconds this morning to update all of my blogs.

Here is the text to download: (link)

Perhaps the easiest way to get the file is to telnet into your account and run this line:
wget http://www.thecodecave.com/downloads/tcc_wp_upgrade

Then give yourself execute permissions on the script, choose your own or run this:
chmod +x tcc_wp_upgrade

If you get errors, remember that you might be dealing with passing the script through Windows and may have tacked on an extra CR at then end of each line… You should be able to fix that by using an SED command something like this
tr -d ‘\r’ tcc_wp_upgrade > xx && mv -f xx tcc_wp_upgrade
(Thanks to Prec on FreeNode #SED)

Then edit the file to your specifications using VI or whatever. I use NetDrive to make my website Drive X on all of my machines and then use Notepad2.exe to edit it and enforce the Unix line endings.

Right now I REALLY would like to have only proficient people run this script. It has only been tested on my sites.
This script messes with your files. If something really horrible goes wrong, you could lose information. I’ve done my best to ensure this doesn’t happen, but literally everything I know about bash I learned in the last month writing this script. A good programmer can go from any language to any language with relative ease (and bash IS a language), but we can’t know everything about how it works on all systems. That only comes with experience.

When you run this script, please come back here and report your results.

Thanks!

Here’s the source for those that want to see it online:
(more…)

WordPress 2.0.6 has been officially released (link) and you definately need to get this one. 2.0.6 was originally just a bunch of minor updates, but now includes several critical security patches. In fact, one of the core “make this information safe” filters that protects you from malicious abuse through the address used to reach your site has been replaced. More detail follows.

I’ve not yet finalized my list of 2.0.6 features, but for now you can review the 2.0.6 RC1 features listed here: (link)

Also, I’ve upgraded my “Update WP in 35 seconds” script. It was 6 lines, it’s now well over 300. And instead up upgrading in 35 seconds, I was able to upgrade 3 WP blogs in <5 seconds without a single browser seeing the site down. The script also will do a full file backup of your WP site into a dated directory (though not in 5 seconds). Pretty nifty. You can see it here: Upgrade your WordPress sites in 5 Seconds(link)

I often see an rr.com site in my logs and wonder where it is from. When trying to figure out if WOH.RR.com was Western OHio, I came across this list (link).

Into cold storage it goes… (It really isn’t very interesting to read. So, I’ll hide the rest…)
(more…)