dl

David Lundgren

Web Developer & Systems Administrator

Large file upload size issues

Lately I’ve been working on a project to help share my virtualbox vagrant’s privately. This is a very simple clone of the vagrant-cloud API that interacts with Packer & Vagrant. As part of this process, Packer uploads the artifacts it creates, and this became a problem in my dev environment. Between Nginx & Apache there are a few differences in how file uploads are configured, both defaults and in general.

Continue reading

The allure of static proxies

Several weeks ago I started playing with Laravel. Primarily because several colleagues are using it, and have suggested that I take a look at it. During my time reviewing how to build a view template I came across references to Html, Form, View and other static calls. Initially I was not impressed due to the use of so many static calls. I have come to an understanding about how static calls in certain circumstances can actually enhance code readability.

Continue reading

Ansible “Authentication or permission failure.”

I recently upgraded some servers, and on reboot I ran into the peculiar condition where I received the following warning:

fatal: [user] => Authentication or permission failure. In some cases, you
 may have been able to authenticate and did not have permissions on the remote
 directory. Consider changing the remote temp path in ansible.cfg to a path
 rooted in "/tmp". Failed command was: mkdir -p
 $HOME/.ansible/tmp/ansible-tmp-1401973086.25-185293296215162 && echo
 $HOME/.ansible/tmp/ansible-tmp-1401973086.25-185293296215162, exited with
 result 1

I followed the instructions I found on Changing Ansible Temporary Directory, as it has worked for many others. I had to turn on verbose logging but I still couldn’t see the issue. After running the command manually I got the following error

mkdir: cannot create directory '.ansible': Disk quota exceeded`

Basically, when I restarted my servers the grpquota and usrquota commands in /etc/fstab took effect. I’m not sure why they were on as we have restarted these servers on other occasions and they were not there. While I have these servers scheduled for a restart, to remove the quotas, and add noatime, I’ve simply turned off the quotas using quotaoff /

Cleaning up Ansible task formatting

I’ve been using Ansible for the last several years, and I’ve used YAML just as long. Yet a lot of playbooks and tasks for Ansible are often horribly formatted. This causes anger within me, so I want to let others know, that there is a better way.

“Use the YAML, Ansible Writers”

YAML may not be as expressive as other formats, however, as authors of roles and tasks for Ansible we can do better at formatting so that they are readable.

Example of hard to read

---
- name: download file
  get_url: url=https://raw.githubusercontent.com/some/path/some.file dest=/usr/local/share/some.file

- name: update permissions
  file: path=/usr/local/share/some.file mode="0644"

Example of a more readable version of the above.

---
- name: download file
  get_url:
    url: https://raw.githubusercontent.com/some/path/some.file
    dest: /usr/local/share/some.file

- name: update permissions
  file:
    path: /usr/local/share/some.file
    mode: "0644"

It’s easier to change change /usr/local/share/some.file into "{{ download_file_dest }}" because you don’t have parse the entire line and scroll over in your HEAD!

Could not generate DH keypair on “Add composer dependency” in PhpStorm

If you are like me and you run an older version of OS X, Mavericks in my case, it means that you may run into the problem with PhpStorm that you can’t use composer. It seems like you are stuck in this rut. Well, I’m going to put this here mainly for my own memory, and in the hopes that it brings someone else luck in getting PhpStorm working correctly.

https://youtrack.jetbrains.com/issue/WI-27354#comment=27-999854

pure-authd external script failing

I have a PHP script handling authentication for pure-authd, and the problem is that it fails. After a reboot it is still failing. If I log in, restart it manually and monitor the log files, it works. As soon as I log out it fails!

I’ve come to the conclusion that pure-authd doesn’t properly daemonize, and detach from the controlling terminal. PHP was actually using fsockopen to another server and it stopped being able to use DNS because I logged out of the shell.

I ended up switching to the IP of the server. While this is not the ideal solution, it works because internally the IP is a reserved DHCP address for the server, and we shouldn’t be updating the server any time soon.

Updating your shell paths on OS X

By David Lundgren | Reply | Tags: ,

Because I forget that ~/bin isn’t on my path, I am placing this here because I also search the internet all the time. Hopefully I’ll remember I put this here:

echo "export PATH=$PATH:~/bin" >> ~/.profile

Leaving the LPi Development Team

It’s never easy leaving a great team. The last three years have been full of growth for me, both mentally and professionally. In those years I have been given the chance to integrate Phing as our deployment automation tool, learn more about coffee (thanks Eric!) and begin the path of mentoring without being completely condescending. I’ve worked on projects that handled geo-spatial searching in MySQL [FYI: you want to use this UDF for distance calculations as it is order of magnitudes faster], to those that help manage church websites. Those who know me, would have asked why I chose to work at LPi given it’s religious affiliations, it’s about the code, and the chance to work on stuff that a large group of people will actually use!

I will miss the team, but by embracing the changes in our life we learn to move forward. IT is full of change anyway, staying static means certain death. Some LPi competitors are learning this the hard way, and I still look forward to seeing them either being acquired or leaving LPi’s market.

SPL FileObject & LimitIterator

Over that last couple of weeks I’ve come to use the SPL far more than I have in the past. The SplFileObject for reading CSV files is far more convenient than the fgetcsv() function, and associated code needed for a CSV file. Using the LimitIterator allowed me to easily bypass the first row of the CSV, as they were headers and I knew the format of those headers.

$csv = new \SplFileObject("some-file.csv");
$csv->setFlags(\SplFileObject::READ_CSV | \SplFileObject::READ_AHEAD |
               \SplFileObject::SKIP_EMPTY | \SplFileObject::DROP_NEW_LINE);
foreach (new \LimitIterator($csv, 1) as $row) {
    /** process the rows **/
}

That was fast for iterating the CSV, the previous code I had for doing this was more verbose about processing for the header row, making it a little more error prone. Take a look at the setCsvControl() method on SplFileObject for more fine grained control over the delimiter, enclosure and escape parsing.

The SPL LimitIterator has also come in handy in another project. This time though I was using a SQL union in a query to merge two separate datasets and I only wanted to list the first 12 out of a potentially larger number of objects.

$sql = '(SELECT id, somestuff FROM somedb.sometable)
        UNION
        (SELECT id, otherstuff FROM otherdb.othertable) ORDER BY `date` DESC';
$data = [];

// $db is a ZF1 adapter object
foreach ($db->query($sql) as $row) {
    $data[$row['id'][] = $row;
}

foreach ($data as $id => $d1) {
    foreach (new \LimitIterator(new \ArrayIterator($data), 0, 12) as $id => $row) {
        /** process the rows **/
    }
}

I didn’t use the following code in the end, but it did work. I suspected, but didn’t confirm, that MySQL’s GROUP BY MAY introduce unknown bugs to the data imported, so I used the above method instead. Here is an example of how the SPL MultipleIterator could be used:

$sql = 'SELECT id, GROUP_CONCAT(`type`) `type`,
               GROUP_CONCAT(`date`) `date`,
               GROUP_CONCAT(`path`) `path`,
               GROUP_CONCAT(`synced`) `synced`
        FROM somedb.sometable
        GROUP BY id, `date` DESC';
foreach ($db->query($sql) as $row) {
    $id = $row['id'];
    // build our array iterators
    $typeArray = new \ArrayIterator(explode(',', $row['type']));
    $dateArray = new \ArrayIterator(explode(',', $row['date']));
    $pathArray = new \ArrayIterator(explode(',', $row['path']));
    $syncArray = new \ArrayIterator(explode(',', $row['synced']));

    // build the main iterator
    $iterator = new \MultipleIterator(\MultipleIterator::MIT_KEYS_ASSOC);

    // make the array iterators combinable into a single array entry
    $iterator->attachIterator($typeArray, 'type');
    $iterator->attachIterator($dateArray, 'date');
    $iterator->attachIterator($pathArray, 'path');
    $iterator->attachIterator($syncArray, 'synced');

    // The array keys of $data will be as set in the attachIterator()
    // method of the MultipleIterator
    foreach(new \LimitIterator($iterator, 0, 12) as $data) {
        /** process the data **/
        if ($data['synced'] == (int)1) {
            continue;
        }
        $data['type']; 
        $data['date'];
        $data['path'];
    }
}

Take a look at the Standard PHP Library for ideas on using the other iterators, data structures, and objects that bring potential for more readable code into your life. They’ve simplified my life a bit because now I don’t have to keep track of variables just to handle counts and other things for state.

Next Page »