https://vaneyckt.io/ Recent content on vaneyckt.io Hugo -- gohugo.io en-us Tom Van Eyck Sun, 04 Nov 2018 11:12:56 +0000 https://vaneyckt.io/posts/using_random_number_generators_to_solve_problems/ Sun, 04 Nov 2018 11:12:56 +0000 https://vaneyckt.io/posts/using_random_number_generators_to_solve_problems/ I recently came across the following blog post detailing a very nice Ruby solution to the following problem: you are constantly creating robots, each of which must be given a unique name this name must follow the pattern letter letter number number number names need to be assigned at random, e.g. you can’t just call your first robot AA000, your second robot AA001, … there needs to be a way to reset this name generator The author came up with a very beautiful Ruby solution that makes clever use of ranges. https://vaneyckt.io/posts/ruby_concurrency_building_a_timeout_queue/ Mon, 12 Feb 2018 10:21:34 +0000 https://vaneyckt.io/posts/ruby_concurrency_building_a_timeout_queue/ Ruby’s built-in Queue class is an incredibly handy data structure for all kinds of problems. Its versatility can be attributed to its pop method, which can be used in both a blocking and a non-blocking manner. However, when using a blocking pop, you may not always want to run the risk of this pop potentially blocking indefinitely. Instead, you might wish for your pop to block only until a given timeout interval has expired, after which it’ll throw an exception. https://vaneyckt.io/posts/ruby_concurrency_in_praise_of_condition_variables/ Mon, 17 Jul 2017 10:21:34 +0000 https://vaneyckt.io/posts/ruby_concurrency_in_praise_of_condition_variables/ In a previous post, we talked about the benefits conferred by Ruby mutexes. While a programmer’s familiarity with mutexes is likely to depend on what kind of programs she usually writes, most developers tend to be at least somewhat familiar with these particular synchronization primitives. This article, however, is going to focus on a much lesser-known synchronization construct: the condition variable. Condition variables are used for putting threads to sleep and waking them back up once a certain condition is met. https://vaneyckt.io/posts/using_dtrace_to_measure_mutex_contention_in_ruby/ Mon, 31 Oct 2016 20:24:12 +0000 https://vaneyckt.io/posts/using_dtrace_to_measure_mutex_contention_in_ruby/ I recently found myself working on Ruby code containing a sizable number of threads and mutexes. It wasn’t before long that I started wishing for some tool that could tell me which particular mutexes were the most heavily contended. After all, this type of information can be worth its weight in gold when trying to diagnose why your threaded program is running slower than expected. This is where DTrace enters the picture. https://vaneyckt.io/posts/ruby_concurrency_in_praise_of_the_mutex/ Thu, 17 Mar 2016 19:12:21 +0000 https://vaneyckt.io/posts/ruby_concurrency_in_praise_of_the_mutex/ When reading about Ruby you will inevitably be introduced to the Global Interpreter Lock. This mechanism tends to come up in explanations of why Ruby threads run concurrently on a single core, rather than being scheduled across multiple cores in true parallel fashion. This single core scheduling approach also explains why adding threads to a Ruby program does not necessarily result in faster execution times. This post will start by explaining some of the details behind the GIL. https://vaneyckt.io/posts/how_to_write_your_own_rspec_retry_mechanism/ Sun, 17 Jan 2016 20:17:35 +0000 https://vaneyckt.io/posts/how_to_write_your_own_rspec_retry_mechanism/ Imagine you have an rspec test suite filled with system tests. Each system test simulates how a real user would interact with your app by opening a browser session through which it fills out text fields, clicks on buttons, and sends data to public endpoints. Unfortunately, browser drivers are not without bugs and sometimes your tests will fail because of these. Wouldn’t it be nice if we could automatically retry these failed tests? https://vaneyckt.io/posts/the_disaster_that_is_rubys_timeout_method/ Sat, 19 Dec 2015 19:20:03 +0000 https://vaneyckt.io/posts/the_disaster_that_is_rubys_timeout_method/ On paper, Ruby’s timeout method looks like an incredibly useful piece of code. Ever had a network request occasionally slow down your entire program because it just wouldn’t finish? That’s where timeout comes in. It provides a hard guarantee that a block of code will be finished within a specified amount of time. require 'timeout' timeout(5) do # block of code that should be interrupted if it takes more than 5 seconds end There’s one thing the documentation doesn’t tell you though. https://vaneyckt.io/posts/a_javascript_closures_recap/ Sat, 26 Sep 2015 17:54:23 +0000 https://vaneyckt.io/posts/a_javascript_closures_recap/ Javascript closures have always been one those things that I used to navigate by intuition. Recently however, upon stumbling across some code that I did not quite grok, it became clear I should try and obtain a more formal understanding. This post is mainly intended as a quick recap for my future self. It won’t go into all the details about closures; instead it will focus on the bits that I found most helpful. https://vaneyckt.io/posts/understanding_iostat/ Mon, 24 Aug 2015 19:47:55 +0000 https://vaneyckt.io/posts/understanding_iostat/ I’ve been spending a lot of time lately looking at I/O performance and reading up about the iostat command. While this command provides a wealth of I/O performance data, the sheer amount of it all can make it hard to see the forest for the trees. In this post, we’ll talk about interpreting this data. Before we continue, I would first like to thank the authors of the blog posts mentioned below, as each of these has helped me understand iostat and its many complexities just a little bit better. https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/ Mon, 16 Mar 2015 19:43:34 +0000 https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/ Often times developers go about writing bash scripts the same as writing code in a higher-level language. This is a big mistake as higher-level languages offer safeguards that are not present in bash scripts by default. For example, a Ruby script will throw an error when trying to read from an uninitialized variable, whereas a bash script won’t. In this article, we’ll look at how we can improve on this. https://vaneyckt.io/posts/an_introduction_to_javascript_promises/ Sat, 07 Feb 2015 18:34:09 +0000 https://vaneyckt.io/posts/an_introduction_to_javascript_promises/ I recently had to write some javascript code that required the sequential execution of half a dozen asynchronous requests. I figured this was the perfect time to learn a bit more about javascript promises. This post is a recap of what I read in these three amazing write-ups. What are promises? A Promise object represents a value that may not be available yet, but will be resolved at some point in future. https://vaneyckt.io/posts/unwanted_spot_instance_termination_in_multi_az_asg/ Sat, 24 Jan 2015 19:17:53 +0000 https://vaneyckt.io/posts/unwanted_spot_instance_termination_in_multi_az_asg/ An auto scaling group is an AWS abstraction that facilitates increasing or decreasing the number of EC2 instances within your application’s architecture. Spot instances are unused AWS servers that are auctioned off for little money. The combination of these two allows for large auto scaling groups at low costs. However, you can lose your spot instances at a moment’s notice as soon as someone out there wants to pay more than you do. https://vaneyckt.io/posts/creating_an_ec2_instance_in_a_vpc_with_the_aws_cli/ Wed, 29 Oct 2014 17:36:12 +0000 https://vaneyckt.io/posts/creating_an_ec2_instance_in_a_vpc_with_the_aws_cli/ Setting up an EC2 instance on AWS used to be as straightforward as provisioning a machine and SSHing into it. However, this process has become a bit more complicated now that Amazon VPC has become the standard for managing machines in the cloud. So what exactly is a Virtual Private Cloud? Amazon defines a VPC as ‘a logically isolated section of the AWS Cloud’. Instances inside a VPC can by default only communicate with other instances in the same VPC and are therefore invisible to the rest of the internet. https://vaneyckt.io/posts/finding_and_deleting_old_tags_in_a_github_repo/ Fri, 18 Jul 2014 19:32:12 +0000 https://vaneyckt.io/posts/finding_and_deleting_old_tags_in_a_github_repo/ It’s very easy for a Github repository to accumulate lots of tags over time. This onslaught of tags tends to be tolerated until it starts impacting git performance. It is at this point, when you have well in excess of tens of thousands of tags, that a call to action tends to be made. In this article, we’ll look at two approaches to rid yourself of these old tags. https://vaneyckt.io/posts/adding_a_post_execution_hook_to_the_db_migrate_task/ Mon, 09 Jun 2014 16:31:22 +0000 https://vaneyckt.io/posts/adding_a_post_execution_hook_to_the_db_migrate_task/ A few days ago we discovered that our MySQL database’s default character set and collation had been changed to the wrong values. Worse yet, it looked like this change had happened many months ago; something which we had been completely unaware of until now! In order to make sure this didn’t happen again, we looked into adding a post-execution hook to the rails db:migrate task. Our first attempt is shown below. https://vaneyckt.io/posts/installing_chromedriver/ Wed, 14 May 2014 20:14:48 +0000 https://vaneyckt.io/posts/installing_chromedriver/ Some time ago I needed to install chromedriver on a ubuntu machine. While this wasn’t too hard, I was nevertheless surprised by the number of open StackOverflow questions on this topic. So I decided to leave some notes for my future self. First of all, let’s install chromedriver. $ LATEST_RELEASE=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE) $ wget http://chromedriver.storage.googleapis.com/$LATEST_RELEASE/chromedriver_linux64.zip $ unzip chromedriver_linux64.zip $ rm chromedriver_linux64.zip $ sudo mv chromedriver /usr/local/bin Let’s see what happens when we try and run it. https://vaneyckt.io/posts/programmatically_rotating_the_android_screen/ Thu, 20 Mar 2014 20:08:17 +0000 https://vaneyckt.io/posts/programmatically_rotating_the_android_screen/ A lot of digital ink has been spilled on this subject, so I figured it might be worth to briefly talk about this. You can either change the orientation through ADB or through an app. While the ADB approach is the easiest, it might not work on all devices or on all Android versions. For example, the dumpsys output of a Kindle Fire is different than that of a Samsung Galaxy S4, so you might need to tweak the grepping of the output. https://vaneyckt.io/posts/programmatically_creating_android_touch_events/ Tue, 04 Mar 2014 20:40:56 +0000 https://vaneyckt.io/posts/programmatically_creating_android_touch_events/ Recent versions of Android have the adb shell input touch functionality to simulate touch events on an Android device or simulator. However, older Android versions (like 2.3) do not support this command. Luckily it is possible to recreate this functionality by running adb shell getevent to capture events as they are being generated. These events can then later be replayed using the adb shell sendevent command. Running adb shell getevent when touching the screen might get you something like shown below. https://vaneyckt.io/posts/some_lesser_known_github_api_functionality/ Sat, 08 Feb 2014 18:05:12 +0000 https://vaneyckt.io/posts/some_lesser_known_github_api_functionality/ One of our automation tools occasionally needs to interact with our Github repositories. Unfortunately, the current implementation of this tool leaves something to be desired as it requires cloning these repositories to local disk. Changes against these local repositories are then made on local branches, after which these branches get pushed to Github. However, in order to save on disk space this tool will only ever create a single local copy of each repository. https://vaneyckt.io/posts/the_amazing_bitwise_xor_operator/ Sun, 12 Jan 2014 19:35:25 +0000 https://vaneyckt.io/posts/the_amazing_bitwise_xor_operator/ One of my colleagues recently mentioned this interview question to me. Imagine there is an array which contains 2n+1 elements, n of which have exactly one duplicate. Can you find the one unique element in this array? This seemed simple enough and I quickly came up with the Ruby solution below. > array = [3, 5, 4, 5, 3] # => [3, 5, 4, 5, 3] > count = array. https://vaneyckt.io/posts/a_visual_explanation_of_sql_joins/ Sun, 17 Nov 2013 20:22:17 +0000 https://vaneyckt.io/posts/a_visual_explanation_of_sql_joins/ I admit that I find myself going to this article every time I need to write some joins. Hopefully putting it here will save me from always having to google it. https://vaneyckt.io/posts/check_the_order_of_your_rescue_from_handlers/ Mon, 11 Nov 2013 21:34:16 +0000 https://vaneyckt.io/posts/check_the_order_of_your_rescue_from_handlers/ Our rescue_from handlers used to be defined like shown below. This might look okay to you. At first glance everything looks fine, right? class WidgetsController < ActionController::Base rescue_from ActionController::RoutingError, :with => :render_404 rescue_from Exception, :with => :render_500 end Turns out it’s not okay at all. Handlers are searched from bottom to top. This means that they should always be defined in order of most generic to most specific. Or in other words, the above code is exactly the wrong thing to do. https://vaneyckt.io/posts/the_javascript_event_loop/ Sun, 10 Nov 2013 20:50:04 +0000 https://vaneyckt.io/posts/the_javascript_event_loop/ Sometimes you come across an article that is so well written you can’t do anything but link to it. So if you’ve ever wondered why the javascript runtime is so good at asynchronous operations, then you should definitely give this article a read. Some snippets: JavaScript runtimes contain a message queue which stores a list of messages to be processed and their associated callback functions. These messages are queued in response to external events (such as a mouse being clicked or receiving the response to an HTTP request) given a callback function has been provided. https://vaneyckt.io/posts/bug_hunting_with_git_bisect/ Mon, 04 Nov 2013 20:02:14 +0000 https://vaneyckt.io/posts/bug_hunting_with_git_bisect/ Today I was looking into what I thought was going to be a simple bug. The problem seemed straightforward enough, so I did a quick grep of the codebase, found three pieces of code that looked like likely culprits, made some modifications, triggered the bug, and found that absolutely nothing had changed. Half an hour and a lot of additional digging later I was stumped. I had no idea what was going on. https://vaneyckt.io/posts/why_is_mysql_converting_my_nulls_to_blanks/ Fri, 01 Nov 2013 19:17:56 +0000 https://vaneyckt.io/posts/why_is_mysql_converting_my_nulls_to_blanks/ A while ago I ran into an issue where some records were showing a blank value in a given column. This was a bit weird as a blank value had never been written to that column. After a bit of searching we found that we had a bug that had inadvertently been writing the occasional NULL value to that particular column though. So how did those NULLs get turned into blanks? https://vaneyckt.io/posts/using_environment_variables_in_migrations/ Tue, 29 Oct 2013 17:42:28 +0000 https://vaneyckt.io/posts/using_environment_variables_in_migrations/ Recently we had to run a migration that was so slow we couldn’t afford the downtime it would cause. In order to get around this, it was decided to put two code paths in the migration: one that was slow and thorough, and one that was quick but didn’t perform any safety checks. The first path would be run on a recent database dump, whereas the latter would be executed directly on the live database once the first had finished without error. https://vaneyckt.io/posts/getting_connection_information_with_lsof/ Mon, 21 Oct 2013 17:21:52 +0000 https://vaneyckt.io/posts/getting_connection_information_with_lsof/ The lsof command is one of those super useful commands for figuring out what connections are taking place on your machine. While the lsof command technically just lists open files, just about everything in linux (even sockets) is a file! Some useful commands: List all network connections $ lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME Spotify 36908 vaneyckt 53u IPv4 0x2097c8deb175c0dd 0t0 TCP localhost:4381 (LISTEN) Spotify 36908 vaneyckt 54u IPv4 0x2097c8deab18027d 0t0 TCP localhost:4371 (LISTEN) Spotify 36908 vaneyckt 71u IPv4 0x2097c8deba747c1d 0t0 UDP *:57621 Spotify 36908 vaneyckt 72u IPv4 0x2097c8deb18ef4cf 0t0 TCP *:57621 (LISTEN) Spotify 36908 vaneyckt 77u IPv4 0x2097c8deb993b255 0t0 UDP ip-192-168-0-101. https://vaneyckt.io/posts/carefully_converting_your_mysql_database_to_utf8/ Sun, 20 Oct 2013 17:19:31 +0000 https://vaneyckt.io/posts/carefully_converting_your_mysql_database_to_utf8/ Converting all the data in your database can be a nail-biting experience. As you can see from the code below we are doing our best to be super careful. We convert each table separately and before each conversion we store the table’s column types and an MD5 hash of every row in the table (we were lucky enough to not have enormous tables). After converting the table we check that no column types or rows were changed. https://vaneyckt.io/posts/character_set_vs_collation/ Sat, 19 Oct 2013 20:31:43 +0000 https://vaneyckt.io/posts/character_set_vs_collation/ There’s a surprising amount of confusion about the difference between these two terms. The best explanation I’ve found is here. A character set is a subset of all written glyphs. A character encoding specifies how those characters are mapped to numeric values. Some character encodings, like UTF-8 and UTF-16, can encode any character in the Universal Character Set. Others, like US-ASCII or ISO-8859-1 can only encode a small subset, since they use 7 and 8 bits per character, respectively. https://vaneyckt.io/posts/mysql_write_locks_also_prevent_reads/ Thu, 17 Oct 2013 17:20:45 +0000 https://vaneyckt.io/posts/mysql_write_locks_also_prevent_reads/ Locking a table with table_name = 'widgets' ActiveRecord::Base.connection.execute("LOCK TABLES #{table_name} WRITE") ensures that only the current connection can access that table. Other connections cannot even read from this table while it is locked! https://vaneyckt.io/posts/rails_migrations_and_the_dangers_of_implicit_commits/ Wed, 16 Oct 2013 18:45:12 +0000 https://vaneyckt.io/posts/rails_migrations_and_the_dangers_of_implicit_commits/ I recently came across the migration below. At first sight it looks like everything is okay, but there is actually a very dangerous assumption being made here. # migration to convert table to utf8 class ConvertWidgetsTableToUtf8Unicode < ActiveRecord::Migration def up ActiveRecord::Base.transaction do table_name = 'widgets' say "converting #{table_name} table to utf8_unicode_ci" execute("ALTER TABLE #{table_name} CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci") execute("ALTER TABLE #{table_name} DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci") end end end Notice how the utf8 conversion code is wrapped inside a transaction. https://vaneyckt.io/posts/iterating_over_a_hash_containing_arrays/ Tue, 15 Oct 2013 16:46:02 +0000 https://vaneyckt.io/posts/iterating_over_a_hash_containing_arrays/ Last week I was implementing some auditing functionality in a rails app. At some point I was writing a page that would display how the attributes of a given ActiveRecord object had been changed. One of my colleagues spotted this and pointed out the following neat bit of syntactic sugar in Ruby. changes = {:attribute_a => [1, 2], :attribute_b => [3, 4]} changes.each do |attribute, (before, after)| puts "#{attribute}: #{before} - #{after}" end I later learned you can even do things like this. https://vaneyckt.io/posts/uri_js_and_url_manipulation_in_rails/ Sun, 13 Oct 2013 14:55:23 +0000 https://vaneyckt.io/posts/uri_js_and_url_manipulation_in_rails/ Manipulating urls in javascript often ends up being an exercise in string interpolation. This rarely produces good looking code. Recently we’ve started enforcing the use of the URI.js library to combat this. Our new approach has us embed any necessary urls in hidden input fields on the web page in question. Rather than hardcoding these urls, we use the named route functionality offered by rails as this provides more flexibility. When the page gets rendered, these named routes are converted to actual urls through ERB templating. https://vaneyckt.io/posts/the_css_important_keyword/ Sat, 12 Oct 2013 15:03:23 +0000 https://vaneyckt.io/posts/the_css_important_keyword/ Today I learned about the css !important keyword. I was trying to change the way code snippets (gists) were being displayed on a site, but found my css rules being ignored. As it turned out, the javascript snippets used for embedding gists were adding an additional css stylesheet to the page. Since this stylesheet was getting added after my own stylesheet, its rules had priority over my own. The solution was to add ! https://vaneyckt.io/posts/finding_models_from_strings_with_rails/ Fri, 11 Oct 2013 16:27:35 +0000 https://vaneyckt.io/posts/finding_models_from_strings_with_rails/ Imagine you have a Widget model that stores data in a table ‘widgets’. At some point in your rails app you find yourself being given a string ‘Widget’ and are asked to find the Widget model. This can be done like shown here. str = 'Widget' model = str.constantize However, things get a bit harder when you have multiple Widget model subclasses (Widget::A, Widget::B), all of which are stored in the widgets table. https://vaneyckt.io/posts/retrieving_data_in_a_time_range_with_rails/ Thu, 10 Oct 2013 19:22:45 +0000 https://vaneyckt.io/posts/retrieving_data_in_a_time_range_with_rails/ I’m writing this mostly as a reminder to myself, since I keep forgetting this :) Instead of: widgets = Widget.where("? <= created_at AND created_at <= ?", time_from, time_to) do this: widgets = Widget.where(:created_at => time_from .. time_to) https://vaneyckt.io/posts/get_vs_post/ Wed, 09 Oct 2013 16:09:32 +0000 https://vaneyckt.io/posts/get_vs_post/ Today I was looking into why a particular GET request was failing on IE. As it turned out this was due to IE not appreciating long query strings. While going through our nginx logs, we also found nginx had a default query string limit that was being hit sporadically by some other customers as well. The solution in both cases was to move the affected calls from GET to POST. https://vaneyckt.io/posts/the_dig_command/ Tue, 08 Oct 2013 13:24:17 +0000 https://vaneyckt.io/posts/the_dig_command/ Today I learned of the existence of the dig command. A very useful little tool for DNS lookups. Here’s an example of it in action. $ dig www.google.com ; <<>> DiG 9.8.3-P1 <<>> www.google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4868 ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www. https://vaneyckt.io/posts/profiling_rails_assets_precompilation/ Sun, 01 Sep 2013 21:01:02 +0000 https://vaneyckt.io/posts/profiling_rails_assets_precompilation/ Assets precompilation on rails can take a fair bit of time. This is especially annoying in scenarios where you want to deploy your app multiple times a day. Let’s see if we can come up with a way to actually figure out where all this time is being spent. Also, while I will be focusing on rails 3.2 in this post, the general principle should be easy enough to apply to other rails versions. https://vaneyckt.io/posts/regarding_if_statement_scope_in_ruby/ Sat, 31 Aug 2013 20:22:58 +0000 https://vaneyckt.io/posts/regarding_if_statement_scope_in_ruby/ I recently learned that if statements in Ruby do not introduce scope. This means that you can write code like shown below and it’ll work fine. # perfectly valid Ruby code if true foo = 5 end puts foo At first this seemed a bit weird to me. It wasn’t until I read this that I realized Ruby was even more versatile than I had first thought. As it turns out, it is this somewhat unconventional scoping rule that allows us to conditionally replace methods. https://vaneyckt.io/posts/ec2_instance_cost_comparison/ Sun, 11 Aug 2013 21:24:12 +0000 https://vaneyckt.io/posts/ec2_instance_cost_comparison/ Amazon’s pricing scheme for its ec2 instances never struck me as particularly transparent. Until recently some of my DevOps colleagues even estimated cost by cross-referencing instance details with pricing information. While this approach gives reasonable results for finding the cost of a given instance type, it doesn’t lend itself very well to comparing prices across a range of different types. When talking to an ex-colleague of mine about the hardships encountered for such a common task, he pointed me to this absolutely brilliant page.