dl

David Lundgren

Web Developer & Systems Administrator

Ansible “Authentication or permission failure.”

I recently upgraded some servers, and on reboot I ran into the peculiar condition where I received the following warning:

fatal: [user] => Authentication or permission failure. In some cases, you
 may have been able to authenticate and did not have permissions on the remote
 directory. Consider changing the remote temp path in ansible.cfg to a path
 rooted in "/tmp". Failed command was: mkdir -p
 $HOME/.ansible/tmp/ansible-tmp-1401973086.25-185293296215162 && echo
 $HOME/.ansible/tmp/ansible-tmp-1401973086.25-185293296215162, exited with
 result 1

I followed the instructions I found on Changing Ansible Temporary Directory, as it has worked for many others. I had to turn on verbose logging but I still couldn’t see the issue. After running the command manually I got the following error

mkdir: cannot create directory '.ansible': Disk quota exceeded`

Basically, when I restarted my servers the grpquota and usrquota commands in /etc/fstab took effect. I’m not sure why they were on as we have restarted these servers on other occasions and they were not there. While I have these servers scheduled for a restart, to remove the quotas, and add noatime, I’ve simply turned off the quotas using quotaoff /

Cleaning up Ansible task formatting

I’ve been using Ansible for the last several years, and I’ve used YAML just as long. Yet a lot of playbooks and tasks for Ansible are often horribly formatted. This causes anger within me, so I want to let others know, that there is a better way.

“Use the YAML, Ansible Writers”

YAML may not be as expressive as other formats, however, as authors of roles and tasks for Ansible we can do better at formatting so that they are readable.

Example of hard to read

---
- name: download file
  get_url: url=https://raw.githubusercontent.com/some/path/some.file dest=/usr/local/share/some.file

- name: update permissions
  file: path=/usr/local/share/some.file mode="0644"

Example of a more readable version of the above.

---
- name: download file
  get_url:
    url: https://raw.githubusercontent.com/some/path/some.file
    dest: /usr/local/share/some.file

- name: update permissions
  file:
    path: /usr/local/share/some.file
    mode: "0644"

It’s easier to change change /usr/local/share/some.file into "{{ download_file_dest }}" because you don’t have parse the entire line and scroll over in your HEAD!

Could not generate DH keypair on “Add composer dependency” in PhpStorm

If you are like me and you run an older version of OS X, Mavericks in my case, it means that you may run into the problem with PhpStorm that you can’t use composer. It seems like you are stuck in this rut. Well, I’m going to put this here mainly for my own memory, and in the hopes that it brings someone else luck in getting PhpStorm working correctly.

https://youtrack.jetbrains.com/issue/WI-27354#comment=27-999854

pure-authd external script failing

I have a PHP script handling authentication for pure-authd, and the problem is that it fails. After a reboot it is still failing. If I log in, restart it manually and monitor the log files, it works. As soon as I log out it fails!

I’ve come to the conclusion that pure-authd doesn’t properly daemonize, and detach from the controlling terminal. PHP was actually using fsockopen to another server and it stopped being able to use DNS because I logged out of the shell.

I ended up switching to the IP of the server. While this is not the ideal solution, it works because internally the IP is a reserved DHCP address for the server, and we shouldn’t be updating the server any time soon.

Updating your shell paths on OS X

By David Lundgren | Reply | Tags: ,

Because I forget that ~/bin isn’t on my path, I am placing this here because I also search the internet all the time. Hopefully I’ll remember I put this here:

echo "export PATH=$PATH:~/bin" >> ~/.profile

Leaving the LPi Development Team

It’s never easy leaving a great team. The last three years have been full of growth for me, both mentally and professionally. In those years I have been given the chance to integrate Phing as our deployment automation tool, learn more about coffee (thanks Eric!) and begin the path of mentoring without being completely condescending. I’ve worked on projects that handled geo-spatial searching in MySQL [FYI: you want to use this UDF for distance calculations as it is order of magnitudes faster], to those that help manage church websites. Those who know me, would have asked why I chose to work at LPi given it’s religious affiliations, it’s about the code, and the chance to work on stuff that a large group of people will actually use!

I will miss the team, but by embracing the changes in our life we learn to move forward. IT is full of change anyway, staying static means certain death. Some LPi competitors are learning this the hard way, and I still look forward to seeing them either being acquired or leaving LPi’s market.

SPL FileObject & LimitIterator

Over that last couple of weeks I’ve come to use the SPL far more than I have in the past. The SplFileObject for reading CSV files is far more convenient than the fgetcsv() function, and associated code needed for a CSV file. Using the LimitIterator allowed me to easily bypass the first row of the CSV, as they were headers and I knew the format of those headers.

$csv = new \SplFileObject("some-file.csv");
$csv->setFlags(\SplFileObject::READ_CSV | \SplFileObject::READ_AHEAD |
               \SplFileObject::SKIP_EMPTY | \SplFileObject::DROP_NEW_LINE);
foreach (new \LimitIterator($csv, 1) as $row) {
    /** process the rows **/
}

That was fast for iterating the CSV, the previous code I had for doing this was more verbose about processing for the header row, making it a little more error prone. Take a look at the setCsvControl() method on SplFileObject for more fine grained control over the delimiter, enclosure and escape parsing.

The SPL LimitIterator has also come in handy in another project. This time though I was using a SQL union in a query to merge two separate datasets and I only wanted to list the first 12 out of a potentially larger number of objects.

$sql = '(SELECT id, somestuff FROM somedb.sometable)
        UNION
        (SELECT id, otherstuff FROM otherdb.othertable) ORDER BY `date` DESC';
$data = [];

// $db is a ZF1 adapter object
foreach ($db->query($sql) as $row) {
    $data[$row['id'][] = $row;
}

foreach ($data as $id => $d1) {
    foreach (new \LimitIterator(new \ArrayIterator($data), 0, 12) as $id => $row) {
        /** process the rows **/
    }
}

I didn’t use the following code in the end, but it did work. I suspected, but didn’t confirm, that MySQL’s GROUP BY MAY introduce unknown bugs to the data imported, so I used the above method instead. Here is an example of how the SPL MultipleIterator could be used:

$sql = 'SELECT id, GROUP_CONCAT(`type`) `type`,
               GROUP_CONCAT(`date`) `date`,
               GROUP_CONCAT(`path`) `path`,
               GROUP_CONCAT(`synced`) `synced`
        FROM somedb.sometable
        GROUP BY id, `date` DESC';
foreach ($db->query($sql) as $row) {
    $id = $row['id'];
    // build our array iterators
    $typeArray = new \ArrayIterator(explode(',', $row['type']));
    $dateArray = new \ArrayIterator(explode(',', $row['date']));
    $pathArray = new \ArrayIterator(explode(',', $row['path']));
    $syncArray = new \ArrayIterator(explode(',', $row['synced']));

    // build the main iterator
    $iterator = new \MultipleIterator(\MultipleIterator::MIT_KEYS_ASSOC);

    // make the array iterators combinable into a single array entry
    $iterator->attachIterator($typeArray, 'type');
    $iterator->attachIterator($dateArray, 'date');
    $iterator->attachIterator($pathArray, 'path');
    $iterator->attachIterator($syncArray, 'synced');

    // The array keys of $data will be as set in the attachIterator()
    // method of the MultipleIterator
    foreach(new \LimitIterator($iterator, 0, 12) as $data) {
        /** process the data **/
        if ($data['synced'] == (int)1) {
            continue;
        }
        $data['type']; 
        $data['date'];
        $data['path'];
    }
}

Take a look at the Standard PHP Library for ideas on using the other iterators, data structures, and objects that bring potential for more readable code into your life. They’ve simplified my life a bit because now I don’t have to keep track of variables just to handle counts and other things for state.

When does Dependency Injection become an anti-pattern?

During my tenure as a seasoned, and tenderized, PHP developer I have used many design patterns: adapters, factories, data mappers, facades, etc. The most recent one that I have been working with is Dependency Injection. Inversion of Control is not a new idea, at least not in the programming world, but in the PHP world it seems to have taken us by storm in recent years. Every framework will often have a Dependency Injector built in, or offer it as a component for use. It is the hip thing to do in PHP, but I believe we need to take a step back and evaluate it for what we are really trying to achieve. That is reducing the tight coupling that our objects may have. I view it as removing the new-able’s from our objects code, and handing the object creation over to something else to deal with. Read further on how I believe DI can become an anti-pattern.

Continue reading

BlockList.de IP lists with CIDR notation

Several weeks ago one of the servers I manage ended up with a lot of comment spam. After several hours of searching through the logs and correlating the IP’s, I found out that a majority of the hosts were listed in the BlockList.de’s bots.txt file. I didn’t want to import ALL the IP’s on the list as I noticed that quite a few could be pushed into fewer lines using CIDR notation.

I set out to compress this and ended up with this Gist that can do exactly that.

<?php
require_once "/path/to/BlocklistDe.php";

$botList = new BlocklistDe('bots.txt');
$botList->writeToFile('converted-bots.txt');

This particular server was using Ubuntu, so I pushed the 13k addresses into the iptables configuration. Spam has been cut quite dramatically.

Next Page »