dl

David Lundgren

Web Developer & Systems Administrator

Leaving the LPi Development Team

It’s never easy leaving a great team. The last three years have been full of growth for me, both mentally and professionally. In those years I have been given the chance to integrate Phing as our deployment automation tool, learn more about coffee (thanks Eric!) and begin the path of mentoring without being completely condescending. I’ve worked on projects that handled geo-spatial searching in MySQL [FYI: you want to use this UDF for distance calculations as it is order of magnitudes faster], to those that help manage church websites. Those who know me, would have asked why I chose to work at LPi given it’s religious affiliations, it’s about the code, and the chance to work on stuff that a large group of people will actually use!

I will miss the team, but by embracing the changes in our life we learn to move forward. IT is full of change anyway, staying static means certain death. Some LPi competitors are learning this the hard way, and I still look forward to seeing them either being acquired or leaving LPi’s market.

SPL FileObject & LimitIterator

Over that last couple of weeks I’ve come to use the SPL far more than I have in the past. The SplFileObject for reading CSV files is far more convenient than the fgetcsv() function, and associated code needed for a CSV file. Using the LimitIterator allowed me to easily bypass the first row of the CSV, as they were headers and I knew the format of those headers.

$csv = new \SplFileObject("some-file.csv");
$csv->setFlags(\SplFileObject::READ_CSV | \SplFileObject::READ_AHEAD |
               \SplFileObject::SKIP_EMPTY | \SplFileObject::DROP_NEW_LINE);
foreach (new \LimitIterator($csv, 1) as $row) {
    /** process the rows **/
}

That was fast for iterating the CSV, the previous code I had for doing this was more verbose about processing for the header row, making it a little more error prone. Take a look at the setCsvControl() method on SplFileObject for more fine grained control over the delimiter, enclosure and escape parsing.

The SPL LimitIterator has also come in handy in another project. This time though I was using a SQL union in a query to merge two separate datasets and I only wanted to list the first 12 out of a potentially larger number of objects.

$sql = '(SELECT id, somestuff FROM somedb.sometable)
        UNION
        (SELECT id, otherstuff FROM otherdb.othertable) ORDER BY `date` DESC';
$data = [];

// $db is a ZF1 adapter object
foreach ($db->query($sql) as $row) {
    $data[$row['id'][] = $row;
}

foreach ($data as $id => $d1) {
    foreach (new \LimitIterator(new \ArrayIterator($data), 0, 12) as $id => $row) {
        /** process the rows **/
    }
}

I didn’t use the following code in the end, but it did work. I suspected, but didn’t confirm, that MySQL’s GROUP BY MAY introduce unknown bugs to the data imported, so I used the above method instead. Here is an example of how the SPL MultipleIterator could be used:

$sql = 'SELECT id, GROUP_CONCAT(`type`) `type`,
               GROUP_CONCAT(`date`) `date`,
               GROUP_CONCAT(`path`) `path`,
               GROUP_CONCAT(`synced`) `synced`
        FROM somedb.sometable
        GROUP BY id, `date` DESC';
foreach ($db->query($sql) as $row) {
    $id = $row['id'];
    // build our array iterators
    $typeArray = new \ArrayIterator(explode(',', $row['type']));
    $dateArray = new \ArrayIterator(explode(',', $row['date']));
    $pathArray = new \ArrayIterator(explode(',', $row['path']));
    $syncArray = new \ArrayIterator(explode(',', $row['synced']));

    // build the main iterator
    $iterator = new \MultipleIterator(\MultipleIterator::MIT_KEYS_ASSOC);

    // make the array iterators combinable into a single array entry
    $iterator->attachIterator($typeArray, 'type');
    $iterator->attachIterator($dateArray, 'date');
    $iterator->attachIterator($pathArray, 'path');
    $iterator->attachIterator($syncArray, 'synced');

    // The array keys of $data will be as set in the attachIterator()
    // method of the MultipleIterator
    foreach(new \LimitIterator($iterator, 0, 12) as $data) {
        /** process the data **/
        if ($data['synced'] == (int)1) {
            continue;
        }
        $data['type']; 
        $data['date'];
        $data['path'];
    }
}

Take a look at the Standard PHP Library for ideas on using the other iterators, data structures, and objects that bring potential for more readable code into your life. They’ve simplified my life a bit because now I don’t have to keep track of variables just to handle counts and other things for state.

When does Dependency Injection become an anti-pattern?

During my tenure as a seasoned, and tenderized, PHP developer I have used many design patterns: adapters, factories, data mappers, facades, etc. The most recent one that I have been working with is Dependency Injection. Inversion of Control is not a new idea, at least not in the programming world, but in the PHP world it seems to have taken us by storm in recent years. Every framework will often have a Dependency Injector built in, or offer it as a component for use. It is the hip thing to do in PHP, but I believe we need to take a step back and evaluate it for what we are really trying to achieve. That is reducing the tight coupling that our objects may have. I view it as removing the new-able’s from our objects code, and handing the object creation over to something else to deal with. Read further on how I believe DI can become an anti-pattern.

Continue reading

BlockList.de IP lists with CIDR notation

Several weeks ago one of the servers I manage ended up with a lot of comment spam. After several hours of searching through the logs and correlating the IP’s, I found out that a majority of the hosts were listed in the BlockList.de’s bots.txt file. I didn’t want to import ALL the IP’s on the list as I noticed that quite a few could be pushed into fewer lines using CIDR notation.

I set out to compress this and ended up with this Gist that can do exactly that.

<?php
require_once "/path/to/BlocklistDe.php";

$botList = new BlocklistDe('bots.txt');
$botList->writeToFile('converted-bots.txt');

This particular server was using Ubuntu, so I pushed the 13k addresses into the iptables configuration. Spam has been cut quite dramatically.

ZfRest — RESTful-ness for Zend Framework 1

For both a client and my work we are working on creating a RESTful API. The problem is that all of the current solutions for Zend Framework are either one off (Resauce, ZfApiVersioning) or otherwise do not play well with a pre-built Zend Framework 1 application. The solution I was looking needed the following

  • handle API versions in the route (I know this isn’t in the spirit of RESTful)
  • handle OAuth
  • Plays well with ZF1 Module architecture (using a Module Bootstrap)

REST modules

In the end most of the solutions out there handle one thing or another but do not address the playing well with ZF1. Resauce deletes the default routes and sets it’s own root url (‘/’), this is not acceptable when you are trying to integrate it with an existing application. ZfApiVersioning suffered the same thing. The only available solution that I found was to write my own REST library for ZF1, so I am introducing ZfRest. It handles the requirements above, and more. The OAuth part is actually loaded as a plugin, requires PECL Oauth, and provides a basic way to handle using Oauth.

OAuth

I also needed an OAuth provider or server. Out of the few solutions I found for ZF1 (link: Zend_Oauth_Provider [incorrect naming], link: Oauth_Server [too complex]), I opted to just use the PECL OAuth module. I quickly ran into a problem where the Exceptions from an internal function were not being passed to my script when you pass in a path such as “/api/users”. Once I figured this out, everything was just working.

$oauthProvider = new OauthProvider();
$oauthProvider->consumerHandler('lookupConsumer');
$oauthProvider->timestampNonceHandler('lookupTimestampNonce');
try {
    $oauthProvider->checkOAuthRequest($uri, $method);
}
catch (Exception $e) {
    // Set HTTP response code to 403
    echo json_encode(array('statusCode' => 403, 'message' => 'Access denied to this resource'));
}

I’ve had several updates to problems I’ve found while using it, but overall I’ve found it works rather well.

MySQL Transactions and Altering Schema

While reviewing the log files for a project I am working on I came across a MySQL error that dealt with mismatched character sets. Specifically the table was set to latin1 and the connection was set to UTF8. This error was rare but happened at the right time for me to catch it, as I was reviewing the log files for an unrelated problem.

Further investigation revealed that the schema and some tables were set to latin1. If we weren’t setting the connection to UTF8 this wouldn’t be a problem. So I set to work on creating a database update to include the alter database and necessary alter table statements.

The first statement among the list of statements that needs to run:

ALTER DATABASE project CHARACTER SET UTF8.

Response from MySQL: Error 1192: “Can’t execute the given command because you have active locked tables or an active transaction”

I was stuck, as this had run fine in the development, QA, and staging environments. Why would it fail in production? The tables didn’t have any locks, and there were no transactions that I was aware of. My Google-Fu led me to the MySQL 5.1 Reference section 13.3.3.┬áStatements That Cause an Implicit Commit. Armed with the knowledge that create, alter, delete table statements cause implicit commits, I was still stuck where I began.

I began reviewing our database update mechanism (based on Zend Framework Database Table Adapters) and found that it was starting a transaction before running the statements. I set forth to update the tool, but some updates needed transactions for their modifications, so I ended up running the query manually.

Sometimes there is no easy answer and you need to just Ops up and get your DBA on.

Creating a DHCP Server with vboxapi

While working on a project for work recently I stumbled across the need to create a network and associated DHCP server. The API documentation however only gave me the following reference to start the DHCP server.

void start (in wstring networkName, in wstring trunkName, in wstring trunkType)

trunkName and trunkType are not really documented in the IDHCPServer API page. Here are their documentation as I have discovered.

  • networkName: This is the IHostOnlyInterface.networkName property
  • trunkName: This should really be adapterName, as putting the IHostOnlyInterface.name property is correct
  • trunkType: This is one of the following TRUNKTYPE_* constants
    • These constants don’t exist in the VirtualBox_constants.py, but can be found in the NetworkServiceRunner.h
      • TRUNKTYPE_WHATEVER = ‘whatever’
      • TRUNKTYPE_NETFLT = ‘netflt’
      • TRUNKTYPE_NETADP = ‘netadp’
      • TRUNKTYPE_SRVNAT = ‘srvnat’

Putting this all together I ended up with code similar to the following

from vboxapi import VirtualBoxManager

# configuration
addresss     = '192.168.8.8'
netmask      = '255.255.255.0'
dhcp_from_ip = '192.168.8.100'
dhcp_to_ip   = '192.168.8.150'

## Configure a HostOnly Interface and the associated DHCP Server
vbm = VirtualBoxManager(None, None)
vb  = vbm.vbox

# create the network on the host
(progress, interface) = vb.host.createHostOnlyNetworkInterface()
interface.enableStaticIPConfig(address, netmask)

# create the dhcp server for this network
dhcpServer = vb.createDHCPServer(interface.networkName)
dhcpServer.setConfiguration(address, netmask, dhcp_from_ip, dhcp_to_ip)
dhcpServer.enabled = True
dhcpServer.start(interface.networkName, interface.name, 'netadp')

FreeBSD and sudo defaults

Several weeks ago I started transitioning some Ubuntu VM’s to FreeBSD VM’s . On previous VM’s I was able to use the following command line without any problems

sudo phing code-update

After switching to FreeBSD I found that sudo, or its “sudo -E” variant, was having problems when running in sub shells. Phing svn tasks were asking for passwords that were previously setup to use svn+ssh. Using “sudo svn list svn+ssh://svn.example.com/svn/project” worked but not when phing ran. It turns out there are two environment variables that Ubuntu’s sudo package was preserving: HOME & MAIL. NOTE: Ubuntu 14.04LTS’ sudo package appears to only preserve HOME.

I created /usr/local/etc/sudoers.d/svnusers

Defaults env_reset
Defaults env_keep+="HOME"

This made FreeBSD’s sudo work as it had on Ubuntu. A day’s worth of investigation to solve the riddle but it works as I would expect it to.

Unix::ServiceConfig released

In 2004 I started working with multiple FreeBSD servers for multiple clients, that needed to be administered by non-admin users. I know you are saying “you idiot” and “why would a non-admin user need to administer the server?” I was hired as a consultant and they wanted to be able to add users, web hosts, databases and dns entries more easily than remembering all the little things that were needed. I didn’t trust WebMin at the time due to being hacked several times prior. In response I wrote Unix::ServiceConfig which I hooked up to a perl script as a way to help me with allowing the non-admin users to more easily manage the server.

It worked really well at it’s job, and the users were happy. I haven’t updated the code since 2008, and it is primarily FreeBSD centric. But I figured it is better to release it now, than to never release it. It is under the MIT license.

Next Page »