Jekyll2020-11-03T16:52:49+00:00https://florianthomas.net/Florian Thomasmy personal blogRequest cancellation in Go2020-11-03T00:00:00+00:002020-11-03T00:00:00+00:00https://florianthomas.net/2020/11/03/go-cancel-requests<p><a href="https://blog.golang.org/context">Context</a> is one of my favourite packages in Go. There is
a very high chance you came across it if you’ve written any web application code in Go before.</p>
<p>The description of the <code class="highlighter-rouge">Context</code> interface in the standard library says:</p>
<blockquote>
<p>A Context carries a deadline, a cancellation signal, and other values across
API boundaries.</p>
</blockquote>
<p>This made me curious, how does Go know when a client cancels a HTTP request?</p>
<p>Some research did not provide a sufficient answer so I went and looked through the code of the standard library.
The answer is suprisingly simple:
Whenever your Go server receives a new incoming request it spawns a new background thread which reads from the incoming TCP connection.
As soon as this blocking read receives an error*, it knows that the client is gone and it can cancel the request context.</p>
<p>One caveat to this is that for <code class="highlighter-rouge">POST</code> requests Go will only start this background read once the entire request body has
been read by the server (which Go does not by default do for you).</p>
<p>Especially during an outage when users are often retrying requests it’s critical to stop fulfilling cancelled requests as
early as possible to prevent further request queueing.</p>
<p>You can see in the example below how it can be implemented in practice:</p>
<noscript><pre>import (
"fmt"
"net/http"
"time"
)
func hello(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
fmt.Println("server: hello handler started")
defer fmt.Println("server: hello handler ended")
select {
case <-time.After(10 * time.Second): // simulate server work
fmt.Fprintf(w, "hello\n")
case <-ctx.Done():
err := ctx.Err()
fmt.Println("server:", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
func main() {
http.HandleFunc("/hello", hello)
http.ListenAndServe(":8090", nil)
}</pre></noscript>
<script src="https://gist.github.com/Crunch09/40002d3dc508d5d86190fc55610b21f9.js"> </script>
<p><img src="https://florianthomas.net/assets/gist_example.gif" alt="Cancel server request" /></p>
<p><em>* (Note how if this read receives data instead of an error it means that the client is doing a <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">HTTP 1.1 pipelining request</a> which means executing multiple HTTP requests on the same TCP connection. This is very rarely used
as we now have HTTP 2!)</em></p>Context is one of my favourite packages in Go. There is a very high chance you came across it if you’ve written any web application code in Go before.Xargs2020-03-22T00:00:00+00:002020-03-22T00:00:00+00:00https://florianthomas.net/2020/03/22/xargs<p>It came up at work that I had to find ~700 numerical IDs in a file and delete
every line in which these IDs occur. Doing this by hand would have taken me <em>forever</em> and
probably also wouldn’t be the best use of my time. I had a hunch that tools like <code class="highlighter-rouge">grep</code> and
<code class="highlighter-rouge">sed</code> might be useful for this kind of task so I decided to give it a try and automate it.</p>
<p>This post is just focusing on the <code class="highlighter-rouge">xargs</code> part of the script I eventually used, so here is
the (slightly simplified) command in it’s entirety first:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>list_of_ids.txt | xargs <span class="nt">-I</span> <span class="s1">'{}'</span> <span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">''</span> <span class="s1">'s/{}: true,//g'</span> target_file.txt
</code></pre></div></div>
<p>Let’s split this up:
<code class="highlighter-rouge">cat</code> is a unix command which prints the entire content of a given file. We then use the pipe (<code class="highlighter-rouge">|</code>)
command to use the stdout of the <code class="highlighter-rouge">cat</code> command as the stdin of the next command.</p>
<p>Given that we want to invoke the <code class="highlighter-rouge">sed</code> command for every line of our input file instead of once for
the entire content, we need a command in between <code class="highlighter-rouge">cat</code> and <code class="highlighter-rouge">sed</code>.</p>
<p>That’s where <code class="highlighter-rouge">xargs</code> helps us out.</p>
<p>According to its <a href="http://man7.org/linux/man-pages/man1/xargs.1.html">man page</a> it does the following:</p>
<blockquote>
<p>The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments.</p>
</blockquote>
<p><code class="highlighter-rouge">utility</code> in our case being the <code class="highlighter-rouge">sed</code> command.</p>
<p>We have to be able to use the strings passed from the <code class="highlighter-rouge">xargs</code> command within the <code class="highlighter-rouge">sed</code> command. That’s why we pass
the <code class="highlighter-rouge">-I</code> option: It tells the utility command (in this case <code class="highlighter-rouge">sed</code>) to replace occurrences of <code class="highlighter-rouge">{}</code>
within the arguments of the utility command with each input line of <code class="highlighter-rouge">xargs</code>.</p>
<p>Given a <code class="highlighter-rouge">list_of_ids.txt</code> with:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>123
444
789
</code></pre></div></div>
<p>Behind the scenes this will result in the following <code class="highlighter-rouge">sed</code> invocations:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sed -i '' 's/123: true,//g' target_file.txt
$ sed -i '' 's/444: true,//g' target_file.txt
$ sed -i '' 's/789: true,//g' target_file.txt
</code></pre></div></div>
<p>Which is exactly what we wanted! This script only used a few minutes of my time, it’s reusable and let me get back
to my other tasks much more quickly than doing all this by hand.</p>
<blockquote>
<p>N.B. The <code class="highlighter-rouge">sed</code> command replaced the matched lines with an empty string as it seemed like quite a struggle to remove
the entire line with <code class="highlighter-rouge">sed</code>. So I was pragmatic and removed all the leftover empty lines with just one quick command in my editor afterwards.</p>
</blockquote>It came up at work that I had to find ~700 numerical IDs in a file and delete every line in which these IDs occur. Doing this by hand would have taken me forever and probably also wouldn’t be the best use of my time. I had a hunch that tools like grep and sed might be useful for this kind of task so I decided to give it a try and automate it.How being able to code helped me get tickets for Borussia Dortmund2019-04-15T00:00:00+01:002019-04-15T00:00:00+01:00https://florianthomas.net/2019/04/15/how-being-able-to-code-helped-me-get-tickets-for-borussia-dortmund<p>17.Dec 2018: Champions league draw for the round of 16 and my favourite team,
Borussia Dortmund, is in the pot of the group winners. As I’m living in London
at the moment I rarely see them live so I’m hoping for an english team to be
drawn as our opponent. And it works out: Dortmund will face Tottenham and the
first leg will be played in their temporary home, Wembley Stadium.</p>
<p>16.Jan 2019: Borussia Dortmund starts their pre-sale for away tickets at
Tottenham. UEFA rules dictate that the visitors are allocated 5% of the total
capacity which is about 4,500 tickets at Wembley stadium.
Not all of those tickets get into the pre-sale though. Borussia Dortmund
allocates a non-specified number of tickets to away season ticket holders,
fanclubs, sponsors and VIPs.</p>
<p>The pre-sale of tickets always works in the same way: At exactly 8.30am german
time on the specified day you have to call the ticket hotline, talk to an IVR,
enter your credentials like membership number and hopefully purchase tickets to
the game.</p>
<p>Because of Dortmund’s massive fan base (we’ve got the largest average home
attendance in all of Europe for a few years now) you have to be on the line
right on time.
And by right on time I mean that if you’re a single second too late the hotline
is overloaded and you miss out on tickets.</p>
<p>The pre-sale started at 7.30am and as always I tried to time my call to be
exactly on the line at 07:30:00am. It didn’t work. I called the hotline 101
times that morning only to hear that the game has been sold out.</p>
<p>😞 What now?</p>
<p>Buying overprized tickets on stubhub or ebay was not an option for me. As was
buying a ticket from Tottenham outside of the away sector. I already did this
for our away game at Atletico Madrid, a few months earlier. While it was great
to be in the stadium at all, it was hard to see my friends in the away
sector and not being able to support the team with them.</p>
<p>Then I remembered that sometimes a few tickets suddenly become available on the
hotline again. The reason is not entirely clear to me but I imagine some
supporters don’t have enough money on their bank account or fanclubs need less
tickets than previously allocated.
Whatever the reason is, when tickets do become available again it’s not
announced anywhere. You just have to keep calling and hope that you catch the
right moment.</p>
<p>I knew that I wouldn’t be able to call multiple times per day while being at
work but I can code so I was wondering if I could automate this.</p>
<p>From a project at my current job I knew a bit about Twilio and I was wondering
if I would be able to replicate the call procedure.</p>
<p>The sequence of steps when calling the hotline is always the same.
<em>(Translated and shortened for the sake of this blog post, the
actual announcements are in german and add a bit more information):</em></p>
<blockquote>
<p>IVR: Welcome to the Borussia Dortmund ticket hotline. Do you know this
service already?</p>
</blockquote>
<blockquote>
<p>Me: Yes!</p>
</blockquote>
<blockquote>
<p>IVR: Do you have a membership at Borussia Dortmund?</p>
</blockquote>
<blockquote>
<p>Me: Yes!</p>
</blockquote>
<blockquote>
<p>IVR: Please start by entering 0 on your keyboard <beep></beep></p>
</blockquote>
<blockquote>
<p>Me: *Enters 0*</p>
</blockquote>
<blockquote>
<p>IVR: Please enter your membership number <beep></beep></p>
</blockquote>
<blockquote>
<p>Me: *Enters membership number*</p>
</blockquote>
<blockquote>
<p>IVR: Please enter your zip code <beep></beep></p>
</blockquote>
<blockquote>
<p>Me: *Enters zip code*</p>
</blockquote>
<blockquote>
<p>IVR: Thanks. I can offer you tickets for the following games. When I said the game of your choice please say “STOP” loud and clearly.</p>
</blockquote>
<blockquote>
<p>IVR: *Announces games*</p>
</blockquote>
<p>From this procedure it was clear to me what I needed to be able to automate:</p>
<ul>
<li>Transmit speech at specified point during phone call</li>
<li>Transmit keyboard input at specified point during phone call</li>
<li>Record announced games</li>
<li>Transcribe recording to find out if Tottenham tickets are available</li>
</ul>
<p>I wanted to create the project in Go but as Twilio doesn’t have an SDK for Go
I needed to send HTTP requests to Twilio’s API endpoints instead. Twilio created a set of instructions called <a href="https://www.twilio.com/docs/voice/twiml">TwiML</a> which allows you to specify commands which should be executed on certain
events like making a phone call or receiving an SMS.</p>
<p>A phone call is very easy to initiate, just use the TwiML verb
<a href="https://www.twilio.com/docs/voice/twiml/call"><Call></a> with
a previously bought number. As the number of the hotline is a restricted number
I needed to make sure that I allow calls to the hotline in my Twilio settings.</p>
<p>Once that was done it was important to figure out when to send the specific
commands. The way to find this out was to just call the ticket hotline and
write done the times on when to say what.</p>
<p>After this I could use
<a href="https://www.twilio.com/docs/voice/twiml/pause">TWIML’s <Pause> verb</a> to
simulate a wait time between executing my actual commands.</p>
<p>As described in the transcript above a caller has to say
things and also enter credentials on the keyboard. For these
tasks I used the TwiML’s verb
<a href="https://www.twilio.com/docs/voice/twiml/say"><Say></a>,which fortunately also
allows to specify non-english language, and the TwiML verb
<a href="https://www.twilio.com/docs/voice/twiml/play"><Play></a> to transmit digits.</p>
<p>Once I finally arrived at the game announcement stage I needed to start
recording what was being said so I used TwiML’s
<a href="https://www.twilio.com/docs/voice/twiml/record"><Record></a> for that. It is
important to follow legal requirements when doing this but luckily I was just
recording a bot :)</p>
<p>Given all that the final TwiML looks something like that:</p>
<noscript><pre><?xml version="1.0" encoding="UTF-8"?>
<Response>
<Pause length="21" />
<Say voice="alice" language="de-DE">Ja</Say>
<Pause length="5" />
<Say voice="alice" language="de-DE">Ja</Say>
<Pause length="45" />
<Play digits="0"></Play>
<Pause length="5" />
<Play digits="12345"></Play>
<Pause length="5" />
<Play digits="54321"></Play>
<Pause length="5" />
<Record maxLength="45" timeout="10" playBeep="false" recordingStatusCallback="recording_completed" />
</Response></pre></noscript>
<script src="https://gist.github.com/Crunch09/3fdcb2643a855e316764fb84359bee62.js?file=twiml_bvb.xml"> </script>
<p>At the specified callback route I received the URL to my recording. I needed to do some research to find a service that could transcribe a recording in german.
While Twilio offers transcribing of recordings themselves it’s only available in
english. My next idea was AWS but at the time of my project their service
<a href="https://aws.amazon.com/transcribe/">Amazon Transcribe</a> also didn’t support
German.
So I went to Google Cloud’s offering and luckily their <a href="https://cloud.google.com/speech-to-text/">Google speech</a> service offered transcription support for German.</p>
<p>There are multiple ways to submit a recording to Google but the easiest way
seemed to be to download it from Twilio and submit it as base64 - encoded string to Twilio’s API. An <a href="https://www.youtube.com/watch?v=uIA1s3Rhpv8">episode</a> of the Go youtube channel <a href="https://www.youtube.com/channel/UC_BzFbxG2za3bp5NRRRXJSw">Justforfunc</a> really helped me to figure out
the correct encoding options so Google could understand the bytes correctly.</p>
<noscript><pre>func fetchTranscription(ctx context.Context, audio []byte) (string, error) {
req := map[string]interface{}{
"config": map[string]interface{}{
"languageCode": "de-DE",
"enableWordTimeOffsets": false,
},
"audio": map[string]interface{}{
"content": b64.StdEncoding.EncodeToString(audio),
},
}
payload, err := json.Marshal(req)
if err != nil {
return "", fmt.Errorf("could not encode speech request: %v", err)
}
ctxWithTimeout, cancel := context.WithTimeout(ctx, 40 * time.Second)
defer cancel()
res, err := urlfetch.Client(ctxWithTimeout).Post("https://speech.googleapis.com/v1/speech:recognize?key="+ os.Getenv("GOOGLE_CLOUD_SPEECH_API_KEY"), "application/json", bytes.NewReader(payload))
if err != nil {
return "", fmt.Errorf("could not transcribe: %v", err)
}
// handle the transcribed recording</pre></noscript>
<script src="https://gist.github.com/Crunch09/3fdcb2643a855e316764fb84359bee62.js?file=transcribe_recording.go"> </script>
<p>When the string <code class="highlighter-rouge">Tottenham</code> is included in the transcript I send myself a text
to let me know to call and buy them. Also whenever no games at all were included
in my transcript I send myself a text to update the length of the pauses as the
text of the IVR had changed then and disrupted the execution of my commands.
This was a bit annoying but luckily only happened a couple of times.</p>
<p>As I was already using Google’s Voice API I decided to also host the code on
Google Cloud. This was also a good learning exercise as I previously only used
AWS. I used <a href="https://cloud.google.com/appengine/docs/go/">Google App engine</a> and
it worked quite well.</p>
<p>One of the important things I had to set up was a cron schedule that issues a
call regularly.</p>
<noscript><pre>cron:
- description: "hourly call to hotline"
- url: /call
- schedule: every 60 minutes from 07:00 to 22:00</pre></noscript>
<script src="https://gist.github.com/Crunch09/3fdcb2643a855e316764fb84359bee62.js?file=cron.yaml"> </script>
<p>As I quickly learned the hotline is closed at night so I didn’t have to run my code then.</p>
<p>1.Feb 2019: My bot was running fine for about 10 days now and finally it
happened around 1pm, I got an SMS confirming that tickets were available again!</p>
<p>I couldn’t really believe it until I called myself but tickets were
actually available and I bought two of them in the Dortmund sector! Needless to
say that I was very happy that day :)
To make all those calls and then transcribe them cost me around £30 in addition
to the tickets for the game but it was all so worth it.</p>
<p><img src="https://florianthomas.net/assets/sms.jpg" alt="Best SMS ever" /></p>
<p>Quickly I informed two of my friends who were also still looking for tickets.
While one of them got my message quickly and was also able to buy tickets, the
other one only called a half hour later when tickets were already sold out
again.</p>
<p><img src="https://florianthomas.net/assets/tottenham_ticket.jpg" alt="Tottenhame Ticket" /></p>
<p>13.Feb 2019: Finally the day has come, Dortmund is in town! Of course I took
the day off and after some pre-game drinks and a march through Wembley we
finally entered the away sector.</p>
<p>After a good performance in the first half Dortmund ended up losing 3:0 (and
three weeks later would be kicked out of the Champions League after also losing
the second leg at home) but it was still a fun night as I can’t see my team live
that much anymore.</p>
<p><img src="https://florianthomas.net/assets/bvb_fans_tottenham.jpg" alt="Away sector" /></p>
<h2 id="whats-next">What’s next</h2>
<p>As of this moment Dortmund is still fighting for the german championship being
second in place with just one point behind league leaders Bayern Munich and
again I couldn’t get tickets for the last two away games of the season so I
reactivated my bot and hope that I’ll be successful this time as well!</p>17.Dec 2018: Champions league draw for the round of 16 and my favourite team, Borussia Dortmund, is in the pot of the group winners. As I’m living in London at the moment I rarely see them live so I’m hoping for an english team to be drawn as our opponent. And it works out: Dortmund will face Tottenham and the first leg will be played in their temporary home, Wembley Stadium.Hacktoberfest 20172017-01-07T00:00:00+00:002017-01-07T00:00:00+00:00https://florianthomas.net/2017/01/07/hacktoberfest-2017<p><a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest</a> is a month long event
(in October as hinted by the name) organised by <a href="https://www.digitalocean.com/">DigitalOcean</a> and <a href="https://github.com/">Github</a> to encourage people to contribute to open source.
Basically they ask to create at least 4 pull requests to public repositories
within this one month to earn yourself a free t-shirt and do good for the community.</p>
<p>The 2017 edition has been the third Hacktoberfest that I participated in. In
addition to the official rules I added some personal goals before the event started:</p>
<ul>
<li>Contribute to projects which I never contributed to previously</li>
<li>Use four different languages (at least two of them should be unfamiliar to me)</li>
<li>Obviously this is not my call but I wanted to try as hard as I can to make sure that the PR will be accepted and merged by the maintainers of each project</li>
</ul>
<h2 id="starting-in-ruby">Starting in Ruby</h2>
<p>The plan for the first PR was to contribute in a language that I’m already very familiar with: Ruby. Via <a href="https://github.com/explore">Github’s explore site</a> I looked for projects which would be interesting to contribute to: They should have at least 100 stars, actively maintained and needed to have a fair amount of open issues.
Because the project should be new to me I couldn’t just add a feature to an unknown project.</p>
<p>After some search I decided to try to contribute to the <a href="https://github.com/skywinder/github-changelog-generator">changelog generator</a>.
It seemed like a neat project and had an open issue which seemed important to a bunch of people to be fixed:
<a href="https://github.com/skywinder/github-changelog-generator/issues/555">skywinder/github-changelog-generator#555</a>.</p>
<p>It took me a while to become familiar enough with the codebase to figure out what
was going on but eventually I was able to come up with a patch which was accepted
shortly after: <a href="https://github.com/skywinder/github-changelog-generator/pull/566">skywinder/github-changelog-generator#566</a>.</p>
<p><img src="https://florianthomas.net/assets/hacktoberfest_2015.jpg" alt="Hacktoberfest T-Shirt 2015" /></p>
<h2 id="off-we-golang">Off we go(lang)!</h2>
<p>Now it was time to create a PR in a language that I haven’t used so far.
I chose Go because I’ve been interested in that for some time.</p>
<p>Given I had very, very limited knowledge of Go I knew that I first had to learn some basic skills before being able to contribute some code to an established project.
Googling for tutorials I ended up using the <a href="https://tour.golang.org/welcome/1">tour of Go</a> on the official go website.
It took me 2 or 3 evenings to work through it but at the end I felt confident enough to look for an issue on a go project that I could tackle.</p>
<p>Taking the same approach of finding a repo to contribute to as in my first PR I ended up with <a href="https://github.com/shopify/toxiproxy">toxiproxy</a>, which is a project from <a href="https://shopify.com">Shopify</a> that helps in testing the reliability of distributed systems by disabling dependencies.</p>
<p>The issue I tried to tackle was about giving the CLI a better error message if an
invalid URL was provided: <a href="https://github.com/Shopify/toxiproxy/issues/174">Shopify/toxiproxy#174</a>.</p>
<p><a href="https://github.com/jpittis">Jake</a>, one of the maintainers of toxiproxy, provided some
really good feedback to a Go noob like me
and at the end also <a href="https://github.com/Shopify/toxiproxy/pull/191">this pull request</a> got merged and I was 2 for 2! 💪</p>
<p><img src="https://florianthomas.net/assets/hacktoberfest_2016.jpg" alt="Hacktoberfest T-Shirt 2016" /></p>
<h2 id="all--in-javascript">All 🆕 in JavaScript</h2>
<p>Having already submitted PRs in Ruby and Go it was time to use a third language.
After my endeavour into Go I wanted to use a language again that I felt somewhat comfortable.</p>
<p>I chose JavaScript, because I knew it already but was nowhere near as confident in it as in Ruby, especially in regards to it’s newish features like ES6 and the entire npm ecosystem.</p>
<p>This time I didn’t look for projects via the Github explore site, instead I tried
to contribute to a project that I saw a couple of times on my twitter feed: <a href="https://github.com/probot">probot</a>.
It’s an automation tool that helps maintainers on Github with certain tasks like
asking issue reporters for more information on their reports.</p>
<p>I looked through the recently update repositories on the probot organisation and
I found a suitable open issue in the <a href="https://github.com/probot/stale">stale</a> repository:
<a href="https://github.com/probot/stale/issues/52">probot/stale#52</a>.</p>
<p>This project didn’t have tests yet but I felt unsure about my JavaScript skills
from the good old jQuery days so I looked for other probot plugins within the
same organisation that included tests and tried to replicate their tests.</p>
<p>This process taught me about modern JavaScript and tools like <a href="https://github.com/facebook/jest">jest</a>
which I never heard of before but now used for testing my JS code. I was able to
create <a href="https://github.com/probot/stale/pull/70">a pull request</a> and the maintainer
was really happy with it so I quickly got my
third contribution for the Hacktoberfest goal merged! 🙌</p>
<p><img src="https://florianthomas.net/assets/probot_pr_feedback.png" alt="Probot pull request feedback" /></p>
<h2 id="compromises">Compromises</h2>
<p>For the fourth pull request I wanted to contribute in an unknown language again. While I was still
debating whether I should go with Rust or Elixir I realised that I had almost
reached the end of Hacktoberfest and because I was also expecting guests for a couple
of days I realised that I wouldn’t be able to learn enough in a new language before the end of the month to be able to contribute. I came to the conclusion that I should focus on a PR in familiar territory (aka “Ruby”) instead.</p>
<p>With that in my mind I saw in one of our <a href="https://deliveroo.engineering">Deliveroo</a> slack channels that someone
had reported an issue on our rails bootstrap project <a href="https://github.com/deliveroo/roo_on_rails">roo_on_rails</a>.</p>
<p>It’s open-source so it was the perfect opportunity to help my coworkers while also achieving my Hacktoberfest goal: win-win! 🎉
It took me only some investigating into the <code class="highlighter-rouge">ActiveSupport::Logger</code> class from Rails
and then I could submit my final pull request:
<a href="https://github.com/deliveroo/roo_on_rails/pull/77">deliveroo/roo_on_rails#77</a>.</p>
<h2 id="summary">Summary</h2>
<p>Overall even though I didn’t quite achieve my initial personal goal I am still really happy about this years Hacktoberfest. I got to know new projects, new languages and each maintainer seemed really
happy about my contribution.</p>
<p>A few weeks ago I received the new shirt for achieving this goal which is always
a really nice reward!</p>
<p><img src="https://florianthomas.net/assets/hacktoberfest_2017.jpg" alt="Hacktoberfest T-Shirt 2017" /></p>
<p>💖 to Github, DigitalOcean and all projects I contributed to this year!
I am already looking forward to the Hacktoberfest 2018 edition!</p>Hacktoberfest is a month long event (in October as hinted by the name) organised by DigitalOcean and Github to encourage people to contribute to open source. Basically they ask to create at least 4 pull requests to public repositories within this one month to earn yourself a free t-shirt and do good for the community.Prevent 404s for missing uploads in your rails app2015-10-01T00:00:00+01:002015-10-01T00:00:00+01:00https://florianthomas.net/2015/10/01/prevent-404s-for-missing-uploads-in-your-rails-app<p>Every software developer has to deal with bugs in their apps and reproducing
those. Some of these cases require a production database clone.
In that case you’d probably just pull the most recent database backup from
production and use it on your local machine.
What is missing in those backups though, are the images uploaded by your users.
This can be very annoying especially if you work on stuff like a media center
which i did recently.
To prevent those 404 errors from popping up in your browser’s developer console, while at
the same time not needing to store gigabytes from user uploaded content
in your locale filesystem, a rack middleware can help you.</p>
<noscript><pre>class UploadsFallback
def initialize(app)
@app = app
end
def call(env)
if is_a_missing_uploaded_image?(env)
[301, {"Location" => '/static/captain_placeholder.jpg',
"Content-Type" => "image/jpg" }, []]
else
@app.call(env)
end
end
private
def is_a_missing_uploaded_image?(env)
is_an_uploaded_image?(env['REQUEST_PATH']) && !uploaded_file_exists?(env['REQUEST_PATH'])
end
def uploaded_file_exists?(file)
File.exist?("#{Rails.root}/public/#{file}")
end
def is_an_uploaded_image?(file)
file =~ /\A\/uploads\/.+[jpg,png,gif,jpeg,JPG,GIF,PNG,JPEG]\z/
end
end</pre></noscript>
<script src="https://gist.github.com/Crunch09/c6dd7928c8ef64a30148.js?file=uploads_fallback.rb"> </script>
<p>As you can see in the gist above, we define a middleware called <em>UploadsFallback</em>
and check in <em>Line 8</em> if the current request is requesting a missing uploaded
image. If that is the case we’ll return a fallback image to the client instead.</p>
<p>Lastly we need to tell Rails to load our <em>UploadsFallback</em> middleware. As we
only want this to happen in <strong>development</strong> we add the following line to the
<em>/config/environments/development.rb</em> file.</p>
<noscript><pre>Rails.application.configure do
# ...
config.middleware.use UploadsFallback
end</pre></noscript>
<script src="https://gist.github.com/Crunch09/c6dd7928c8ef64a30148.js?file=development.rb"> </script>
<p>Restart your app and 404s for missing uploads are gone.</p>Every software developer has to deal with bugs in their apps and reproducing those. Some of these cases require a production database clone. In that case you’d probably just pull the most recent database backup from production and use it on your local machine. What is missing in those backups though, are the images uploaded by your users. This can be very annoying especially if you work on stuff like a media center which i did recently. To prevent those 404 errors from popping up in your browser’s developer console, while at the same time not needing to store gigabytes from user uploaded content in your locale filesystem, a rack middleware can help you.