<h1class="content-subhead">01 Feb 2016, 07:00</h1>
<sectionclass="post">
<headerclass="post-header">
<ahref="https://getgophish.com/blog/post/database-migrations-in-go/"class="post-title">Handling Database Migrations in Go</a>
<pclass="post-meta">
</p>
</header>
<divclass="post-description">
<h3id="why-you-should-version-your-database:664f01dc60472cd080b34187311f6c6f">Why You Should Version Your Database</h3>
<blockquote>
<p>“I got my database schema correct on the first try.”</p>
<p>-No one ever.</p>
</blockquote>
<p>Like most big projects, gophish needed a way to automatically manage changes to our database schema. As new features were being added, we found ourselves in a situation that required us to add or modify columns and tables to store the new data.</p>
<p>In a hosted environment, this is no problem since we control the database and can make schema changes as we see fit. Gophish is different, in that it is software intentionally designed to run on the client’s machine. This means that as we rollout updates to gophish’s backend database, we need a way to easily update (or rollback!) changes to the database structure. A versioning system is a perfect fit, which introduces the idea of migrations.</p>
<h4id="what-is-a-migration:664f01dc60472cd080b34187311f6c6f">What is a <em>Migration</em>?</h4>
<p>A migration is nothing more than a set of SQL commands to make changes to a database. Every migration typically has two parts: how to apply the changes you want, and how to roll them back.</p>
<p>To version control our database, we can create a folder holding multiple migration files. Each file will have an identifier so we know which migration should be applied and in which order. Then, we can store which version our database is currently at in the database itself so if we ever add migrations in the future, we can tell where we left off.</p>
<p>There are tools that can automate this process for us. We settled on a well-known database migration tool called <ahref="https://bitbucket.org/liamstask/goose/"><code>goose</code></a>.</p>
<h3id="introduction-to-goose:664f01dc60472cd080b34187311f6c6f">Introduction to <code>goose</code></h3>
<p>We chose to go with <ahref="https://bitbucket.org/liamstask/goose/"><code>goose</code></a> since it seemed like a mature, fully-featured solution that would be easily integrated into our code. Goose typically works through the use of its command line tool aptly named <code>goose</code>.</p>
<p>To set things up, we first need to create the following folder structure:</p>
<pre><code>| db/
| | migrations/
| `-dbconf.yml
</code></pre>
<p>Our migrations will be stored in the <code>migrations</code> folder as a series of SQL files. Before we can create migrations, we have to specify the configuration for <code>goose</code> to use. This is found in the <code>dbconf.yml</code> file. In our case, we used the following configuration:</p>
<pre><code>production:
driver: sqlite3
open: gophish.db
dialect: sqlite3
import: github.com/mattn/go-sqlite3
</code></pre>
<p>This configuration specifies a single environment, <code>production</code>, that manages a SQLite database.</p>
<p>Now that we have created our configuration file, we can start making our migrations. Unfortunately, this is where the hurdles began.</p>
<h3id="a-little-about-gophish:664f01dc60472cd080b34187311f6c6f">A Little About Gophish</h3>
<p>Normally, migrations are something that is considered early on in the database creation process. Unfortunately, our schema was already defined and we had clients already running gophish. So, we needed to orchestrate <code>goose</code> in such a way that we could create and apply our migrations without messing up any data that was already in the client’s databases.</p>
<p>The first step was creating the migrations. To handle this, we first created an empty migration file using the following:</p>
<pre><code>goose -env production create 0.1.2_browser_post sql
goose: created ~\go\src\github.com\gophish\gophish\db\migrations\20160130184410_0.1.2_browser_post.sql
</code></pre>
<p>This command created a new empty SQL file in our migrations folder that looks like this:</p>
<pre><code>-- +goose Up
-- SQL in section 'Up' is executed when this migration is applied
-- +goose Down
-- SQL section 'Down' is executed when this migration is rolled back
</code></pre>
<p>For our first migration, we decided to baseline our schema to the current version. To do this, we simply exported our existing schema using the sqlite3 tool. That gave us all of our <code>CREATE TABLE</code> statements that setup our tables and default data. We then copy/pasted those statements below the <code>-- +goose Up</code> section of the migrations.</p>
<p>The one change we made was to add <code>IF NOT EXISTS</code> to all of our table creation statements. This meant that if the client already had a database setup, this migration would be applied, but no changes would be made - exactly what we want.</p>
<p>The final step to create this migration was to add the rollback statements. Since this was creating the database, <code>DROP TABLE</code> equivalent statements worked just fine. You can see our final migration file <ahref="https://raw.githubusercontent.com/gophish/gophish/master/db/migrations/20160118194630_init.sql">here</a>.</p>
<p>Now for the next hurdle. Traditionally, migrations work by creating a new migration file and running <code>goose up</code>. Then, <code>goose</code> will compare your database version with the migration files it finds. If there are migrations that need to be applied, it will apply them in order until you are at the current version.</p>
<p>While the <code>goose up</code> command can work if we control the database, there’s simply no way that we can expect our users to install <code>goose</code> and run <code>goose up</code> every time we want to make a database change. Our goal has always been to make the lives of our users easier, so this simply wouldn’t work. This meant that we needed to handle the migrations in our code.</p>
<p>Fortunately for us, the <code>goose</code> CLI wraps a rich library that we can use. We were able to integrate this directly into our <code>Setup()</code> function to apply migrations automatically.</p>
<p>First, we created the <code>gooose.DBConf</code> struct to hold the configuration (a programmatic copy of our <code>dbconf.yml</code> file).</p>
<pre><codeclass="language-golang">// Setup the goose configuration
migrateConf := &goose.DBConf{
MigrationsDir: config.Conf.MigrationsPath,
Env: "production",
Driver: goose.DBDriver{
Name: "sqlite3",
OpenStr: config.Conf.DBPath,
Import: "github.com/mattn/go-sqlite3",
Dialect: &goose.Sqlite3Dialect{},
},
}
</code></pre>
<p>Next, we need to figure out the latest database version supported by our migrations. This gives us the final “goal” migration that we want to upgrade to. We can do this via the function <ahref="https://godoc.org/bitbucket.org/liamstask/goose/lib/goose#GetMostRecentDBVersion"><code>goose.GetMostRecentDBVersion</code></a>.</p>
<pre><codeclass="language-golang">// Get the latest possible migration
<p>And finally, we need to apply our migrations. <code>Goose</code> has a function called <ahref="https://godoc.org/bitbucket.org/liamstask/goose/lib/goose#RunMigrationsOnDb"><code>goose.RunMigrationsOnDb</code></a> which expects an existing <ahref="https://golang.org/pkg/database/sql/#DB"><code>sql.DB</code></a> object. Since gophish uses the ORM <ahref="https://github.com/jinzhu/gorm"><code>gorm</code></a>, we already had a <code>sql.DB</code> object already initialized that we could use to send to <code>goose</code>. This was stored in the <code>db</code> variable.</p>
<pre><codeclass="language-golang">// Migrate up to the latest version
<p>That’s it! You can find our full <code>Setup()</code> function <ahref="https://github.com/gophish/gophish/blob/master/models/models.go#L61">here.</a> To handle any additional migrations, all we need to do is run <code>goose create</code> again, add the SQL that makes up the migration, and push out the new file. The next time clients update gophish and restart the executable, the database migrations will be applied automatically!</p>
<p>If this kind of stuff is interesting to you, and you want to see a full example of a web app written in Go, check out gophish by clicking below.</p>
<p><em>Tl;dr - Download the release <ahref="https://github.com/gophish/gophish/releases">here</a></em></p>
<h3id="the-wait-is-over:1cea0120cd31cba0f7863bc47631176f"><strong>The wait is over!</strong></h3>
<p>The gophish team is excited to announce our first public beta version of gophish - version 0.1.1! This blog post will be a short introduction into what gophish is, as well as some of the insanely awesome features we’ve created.</p>
<h3id="what-is-gophish:1cea0120cd31cba0f7863bc47631176f">What is Gophish?</h3>
<p>Gophish is a phishing framework that makes the simulation of real-world phishing attacks dead-simple. The idea behind gophish is simple – make industry-grade phishing training available to <em>everyone</em>.</p>
<p>“Available” in this case means two things –</p>
<ul>
<li><strong>Affordable</strong>– Gophish is currently open-source software that is completely free for anyone to use.</li>
<li><strong>Accessible</strong>– Gophish is written in the Go programming language. This has the benefit that gophish releases are compiled binaries with no dependencies. In a nutshell, this makes installation as simple as “download and run”!</li>
</ul>
<h3id="time-for-features:1cea0120cd31cba0f7863bc47631176f">Time For Features</h3>
<p>Ok, ok, enough with the intro. The idea of a phishing simulation platform isn’t new. Let’s take a look at some of the features that really set gophish apart and make it awesome.</p>
<p>There are many commercial offerings that provide phishing simulation/training. Unfortunately, these are SaaS solutions that require you to hand over your data to someone else.</p>
<p>Gophish is different in that it is meant to be hosted in-house. This keeps you data where it belongs - with you.</p>
<p>For the few existing in-house solutions that exist, setup can be a <em>huge pain</em> (looking at you, Ruby gems). Your time is too valuable to be spent wrestling with dependencies trying to create the perfect setup that somehow magically allows the program to run.</p>
<p>Gophish was written in the Go programming language for this exact reason. To install gophish, all you have to do is download the zip file, extract the contents, and run the binary.</p>
<p>By doing this, you just started two webservers, populated a database, and setup a background worker to handle sending the mails. Now, your time can be spent making campaigns. Easy peasy.</p>
<h4id="api-s-for-everything:1cea0120cd31cba0f7863bc47631176f">API’s for <em>Everything</em>.</h4>
<p>Gophish was built with automation first. This means that you can create scripts and clients that automate all the hard work for you. In addition to this, we keep up-to-date <ahref="/documentation/api/">API docs</a> that describe each API endpoint in detail.</p>
<p>Speaking of API docs, we take documentation very seriously. We take documentation seriously because we take our user experience seriously. If you can’t find what you need to use and troubleshoot gophish, we’ve failed. Just take a look at our comprehensive <ahref="/documentation/Gophish%20User%20Guide.pdf">user guide</a>, <ahref="/documentation/api/">API documentation</a>, and even <ahref="http://godoc.org/github.com/gophish/gophish">fully documented code</a>.</p>
<p>If you ever find something missing in our documentation, <ahref="/support">we want to know!</a></p>
<p>While the API is the core of gophish’s functionality, we also provide a gorgeous admin UI. This UI is simply a wrapper on top of the underlying API. Nothing says more than screenshots:</p>
<figcap>Viewing the Timeline for a Target</figcap>
</figure>
</p>
<h3id="take-gophish-for-a-spin:1cea0120cd31cba0f7863bc47631176f">Take Gophish for a Spin!</h3>
<p>These features only scratch the surface when it comes to what makes gophish great, and we aren’t anywhere near done yet. To explore these features for yourself, take gophish for a spin!</p>
<p>We hope you enjoy gophish and are excited for all the new features that will be released soon! In the meantime, if you ever have any questions, comments, or issues, <ahref="/support">we want to hear from you</a>!</p>
<p>This is the official blog for <ahref="http://getgophish.com">gophish</a>, a phishing toolkit designed to make rock-solid security awareness training accessible to <em>everyone</em>.</p>
<p>Check back here often to find information on gophish updates, how to leverage gophish in interesting ways to test the security of your organization, as well as general tips and tricks on securing your email infrastructure.</p>
<p>The gophish team is excited to release the alpha version of gophish soon! In the meantime:</p>
