Binary Search

I am rewriting an old app I made where people can record measurements over time (days or weeks). One log has many measurements. The last time I did this, I used PostgreSQL, and I made a one-to-many relationship between the logs table and the measurements table. So there was a foreign key in the measurements table referencing the id column of the logs table.

The way the app worked, it pull 100% of the measurements for a log when presenting the log to the user on the Log "show page". I think this was a slow way to retrieve measurement data. Using an index to lookup individual, small, measurements seems very slow. Doing a scan of the measurements table instead of using the index also seemed slow.

Besides collecting the measurements, I asked the Postgres sort them by date. I needed (wanted) them to be in sorted order to render them in a visual graph.

This time around, I am still using PostgreSQL, but I am taking a no-SQL approach. I am storing all the measurements on the log in a JSONB column in the logs table.

To make this approach work, I have to keep the measurements sorted. There is probably a way to have Postgres sort by a field in the JSON but I don't want to bother with that, neither its programming support nor its computational cost. It just seems way simpler to maintain order among the measurements every time I write a log record to disk.

This means that when I add a measurement, I have to do a search to find the measurement's insertion point. Preferably a binary search.

I didn't have to do this when I was depending on SQL to do all the work for me. But aspiring for what I've stated is better performance and simpler data flow implied figuring this out for myself.

I wrote an implementation of all of this. I found bugs in my program. I initially tested the binary search and it looked good. But my graph was not rendering correctly at all. I investigated things on the front end and discovered that the measurements were not being added in sorted order. This was despite my early database unit tests that suggested that things were working. I invested things on the backend and found that I was miscalculating the insertion index. I investigated the insertion index calculation and found the bug. But how to fix it? This algorithm, which isn't really complicated in theory, was turning into a 4+ hour endeavour across two days, and it reminded me of how stupid I can be. I eventually, after some hacking and paperwork, found a simpler algorithm than the one I originally came up with, and a correct algorithm.

Shortly after the above, I had to implement binary search on the measurements array on the front end. Having the backend approach as relevant experience helped.

Enter Log Levels, Mike

I'm having to figure out how to handle weird situations in my app.

I have a singleton app state holder that doesn't really get its state until DOMContentLoaded. It gets its state from inline Javascript that takes time to execute. I can't just expect it to have taken effect as soon as my code comes alive. The inline Javascript in question attaches some values to the global window object, and the app state holder gets it from there. But it has to wait for DOMContentLoaded to do this.

This leads to small problems because other UI components in the app will register as subscribers to the app state holder before DOMContentLoaded has fired. They will even try to request state information, not knowing whether they have beaten DOMContentLoaded or not.

Thus, despite not being ready, the app state holder may get a request for information. What is this situation? Should I:

• Throw an exception? This isn't really that level of an emergency. It's not a logic error. So no.
• Resort to showing an alert to the user? No. I should handle this. It's not a burden on the user experience.
• Log something at error level? No. This isn't an error.
• Log something at trace level? No. This is more interesting than trace level information.
• Log something at debug level? I chose this. This meant adding log levels to my singleton logger.

This also implied setting the log level to something higher than debug, because I'm normally not interested in hearing about this information. I don't even show the log console normally. Remember I'm on an iPad, so I don't have access to the Chrome console. I've built my own logger that renders to an absolutely-positioned div at the bottom of my web page's viewport.

I've never really had such a need for log levels until now. In the past, I've normally considered throttling logging when I'm investigating a problem. But this is like a small, inconvenient situation that I just want to keep an eye on. I'm not sure what to do with it. I think there is probably an organized way to prevent my UI components from asking for stuff from the app state holder until the holder is ready (DOMContentLoaded has passed). But I don't know what that solution is.

So log levels provide a way for me to monitor the issue for now.

When the app state holder encounters a subscriber request too early, besides logging at the debug level, it either returns null (if a return value is expected) or it returns early if it's a void function. Any interested UI components are assured of hearing about app state updates via their subscription.

This use of an app state singleton, and accompanying subscriptions, is an easy, old-school, lighweight alternative to Redux, and all of the additional programming that comes with actions and reducers.

This woke me up to other areas of improvement in my coding quality of life. I had many places where I was calling logging.info when really I should have been calling logging.error. This makes it easy for me to do debugging with logging.info calls, because when I need to do a search for where I've littered my code with temporary logging for problem-solving, I can search for the string logging.info and not see all of the permanent logging statements that are really for special, error-level information. The list returned in my text search is way shorter, and contains only the info logging I'm trying to clean up.

Coding Lately

There was a time when I thought I didn’t want to let people down. I thought that coding as a hobby and means to get a job was a way to hold my friends on Twitch up.

I met a person on Twitch in March who I was in loose contact with and I didn’t want to let that person down. This pressure was something I felt in the middle of this month. But after going a week or so further, I changed my mind. I decided that the life I’d led in my 20s and 30s was enough to have shown this friend that I wasn’t a tool of a person. At that point, saying that I didn't want to let down my friends on Twitch wasn’t really valid anymore. I didn’t let my Twitch friends down. Why I work now is part of a different phase in my life. I don’t feel like making X dollars a month constitutes holding my friends up.

I had this recent weekend where I was watching a lot of dog and cat videos. I felt not well. When I finally started coding at the end of a day, I felt much better. Coding is what makes me happy.

Per unit time, there is a limited amount of things to do on YouTube, Discord, and Twitch. You can exhaust the content coming out of those machines in about 3 hours, and be starved for the next week. Coding is how I make my life interesting and how I provide conversation fodder.

I had this experience about a year and a half ago where I was working on this rather elite team. I was one of two coders on the team. But there were others attending the daily work meeting who were well-versed in a database system that the company used everywhere. One person in particular was a vice president at the company, which was a few hundred people large. He was both respected and feared, due in no small part to his combination of expertise and diligence.

One day he messaged me and we were talking about a programming req. I told him that I was watching music videos (which I was) and would resume working soon, to attend to the concern we were talking about. He didn't get mad at me. He didn't even balk.

This anecdote illustrates how I see programming. When you're a programmer, you are considered to be benevolent. You are considered to be so elite that (often) no one cares whether or not you wear a suit to work. No one cares if you take a break in the middle of the work day, beyond the lunch break. When the web servers are in panic mode at 11pm, you're expected to be working on them. But if the weather is fine, you can relax and work at your own pace.

At this point in my life, it's not obvious to me whether I will be able to continue my programming career. But I don't feel like I self-destructed. I don't really feel like moving into a different career is high on my list of priorities. Sometimes the world agrees with you, that you are doing good work. But this isn't guaranteed.

One can argue that I should be working so that I can secure financial stability for myself. But if the world got mad at me and no company wanted to hire me, at some point it's not my fault. I believe that I merit a nice paycheck, and that being able to afford my lunch should come easily to me. If the world disagrees, it isn't going to send a team of lawyers and other programmers to prove to me that I'm incompetent. I don't lie to myself about my programming past. I know when I've done good work and when I haven't. That no one sees me as a benevolent leader can be a peaceful disagreement between me and the world.

I program because I like it. Not because I want to be rich.

Am I Falling out of Love with Emacs?

Do I need emacs? Why did I like it so much? Isn't it usually simpler and more portable to just use the command line, instead of an emacs key binding?

Why do I need emacs to do indentation for me when I can use the tab key perfectly fine in Textastic.

Emacs is so complicated that you can't even use the tab key to indent your code. It won't let you. It'll autoformat code according to its internal rules for the current language. Formatting a .tsx file in emacs is impossible (granted, TSX is probably too complicated for us to be using in the first place).

I guess emacs is cool if you're sending lisp snippets into an interpreter. But I don't find that I miss it. It's easy to guess how I feel about tools like Resharper at this point.

Programming on an iPad

I am launching a Java web app. It is 14k lines large. 45% of that I wrote using a 2024 12-inch iPad Pro.

You can't compile Java code on an iPad. You don't even have direct access to a hard drive. One of the first realizations I had when I decided to go down this road was that I would have to rent a server in the cloud just to do development on it. I could install openjdk on it and compile my source code there.

The next big decision you have to make is how to inform your remote cloud machine of the latest keystrokes you've typed to move around characters in a source code file. I read a guide that said that typing in things using an iOS text editor was fundementally too frustrating, and that one should use a web-based text editor that sits in front of your cloud machine. I did not reach this conclusion in my research and approach.

What I did was go ahead and choose Textastic as my text editor. As its accomplice, I chose ShellFish. What ShellFish does is allow you to enter SSH information into it, and it'll sync files via SFTP from your remote cloud filesystem into the Files iPad OS app. Textastic integrates with ShellFish, so you can open files that ShellFish knows about with the Textastic open file dialog. ShellFish was like $30 for a lifetime license. With it and Textastic, you can edit files on your remote cloud machine. Every time you save, your changes get pushed into your cloud machine.

ShellFish also provides support for moving files you download in Chrome directly into your cloud machine. It does this through the "Share" generic function that shows up when you're looking at different things on iOS. This was very handy when it came time to download my SSL certificate and key.

I think Textastic was $10 for a lifetime license.

There is a Git file manager called Working Copy. It's a feature-rich app but I haven't found any reason to use it. ShellFish and Textastic combine to give me all the Git happiness I need (Git is of course installed on my cloud dev machine). But of course to use Git and do a host of other things, you need real ssh access to your cloud machine. This brings us to terminal apps.

Textastic has terminal support but it's not very good. Fortunately there is another (free!) app called Termius. I say free - you can pay a monthly fee for additional features like agent forwarding but I've been living at the free level. Termius is really good. One thing it has to do is use the Location services on iOS to do its work when it's not in the foreground. Which I find fine.

Termius provides an additional need which is port forwarding. When I run my app in the cloud, I need to be able to tunnel in to see it from my iPad. Termius supports such tunneling. So you can open the Chrome browser on your iPad and point it to http://localhost:3000 and as long as you've configured and turned on a Termius mapping from port 3000 on your local machine (127.0.0.1) to whatever port you want to connect to on your cloud machine, you're off to the races. As an aside, I had to use iptables to block public access to the port I listened to on the server. This way no one can see my dev server even though it's on a publicly reachable cloud server.

So with ShellFish, Textastic, and Termius I am up and running. My cloud dev machine I rented from Linode. I use a 4GB dedicated machine. It costs $36/month.

One thing I am pleased to discover that I don't miss is Emacs. People who know me might be surprised that I can do without Emacs. Textastic has a find feature, but it does not have a find-and-replace feature. To alleviate this, I found myself learning to use GNU Sed. You can do a lot with sed at the command line. I had to rename a package name on my app and sed, xargs, and find were able to make that not-so-difficult for me. I had to rename a model and sed came in handy there, too. Just be careful. Sed is powerful and I had to spend 20-30 minutes debugging a problem I introduced with it on one of my renames.

There are problems.

1. Textastic and ShellFish don't work sometimes. Sometimes when opening and editing a file, especially one very recently generated outside of Textastic, one of these two apps starts creating collision-avoidance files. For a file like MyApp.java, the files I'm referring to get names like MyApp-2024-03-26.java, MyApp-2024-03-26-2.java, etc. I wrote a Ruby script that scans my source files and deletes files that look like this. I have to use it about once every 5 hours.
2. Textastic and ShellFish sometimes take a moment to upload a file. This can be really confusing, because you'll see an error that you think you've already fixed. You trust the compiler and feel stupid, when really the problem is that the compiler just has out-of-date source code and your changes haven't propagated up to the cloud yet.
3. The Chrome dev tools are not available on iOS. Neither are the Firefox ones. Goodbye JavaScript console. Goodbye hot-editing of CSS. Goodbye view source (yes, really). I had to write a custom logging feature to give me some feedback on JavaScript print statements I would add to my code. This was somewhat helpful but nowhere near what I would call a solution. Who knows when we'll get the dev tools on iOS.
4. A 4GB RAM $32/month dev machine is not nearly as powerful as a recent MacBook Pro. I feel the pain on every compile and test suite run. Of all of the problems I've mentioned so far, this one is the most onerous. I tested an 8GB Linode, and it went faster, but it didn't feel worth the money to me. The 4GB dedicated CPU machine was an upgrade from where I started - a 2GB shared CPU machine.
5. Sometimes Textastic is so lost that I have to reboot my iPad. I've even had to hard-reboot it once. This hasn't happened to me for about 2 weeks so maybe patches have helped?

I use a fullsize bluetooth keyboard.

The last thing I'll mention is that I am using an iPad with cellular service. I pay for a plan with Verizon. They say they throttle me down after 30GB of network transfer. I regularly exceed that monthly allowance within a week, but the throttled speed is still plenty fast. I've noticed no hardship from throttling.

I love programming on an iPad very much. One of the other things I've been using my iPad for is studying, using Kindle. I like that the act of doing tech stuff has been rendered relatively simple. Do we really need powerful PCs? If and when I work at a normal job, I would be pleased to use a "real" computer on-site. But if I'm doing stuff from a satellite location like my home, I like where development on an iPad is at.

From Ruby on Rails to Relatively Plain Java

The problem with the Ruby on Rails library, compared to other libraries like PostgreSQL, is that Rails is mostly trying to apply the DRY principle across multiple web server projects, whereas the PostgreSQL library is making difficult algorithmic design decisions. That PostgreSQL actually listens to a TCP socket while Rails establishes a shared runtime with your project is incidental to the argument I am making in this blog post. Libraries that are predominantly concerned with DRYing code disenfranchise the developers who make use of them. They add a layer of hardship because you typically don't have easy access to their source code if and when the shit hits the fan somewhere from within their complexity. Beyond the distanced access to the code, they add a challenge because their complexity is daunting even when your app is still relatively simple in scope (again, when problems emerge from within them).

To join my side on this, you have to be able to hold a B-Tree index implementation as distinct from a piece of ORM code that merely generates an SQL Insert statement, based on a struct instance and table schema it found somewhere. Otherwises you're just going to counter that nobody wants to write B-Tree indexing 10 times in their life when they can just count on PostgreSQL to do it one time for them (and everyone else).

If you wrote B-Tree indexing on two projects, and decided to pull that logic out into a library that the two could share, you would not call that "refactoring". "Encapsulation" would be a better term.

I decided about a year ago that I wanted to write a web app using Java. It would be my first time seriously using Java to perform a task. I was coming from a 10+ year background in Ruby on Rails. What were my goals?

I wanted my app to be simple. I started off by investigating Spring, both the framework and the Spring Boot wrapper. I investigated the ORM facilities in it and Hibernate. I decided that I wanted my app to be simpler than a spring project. I didn't even want to use dependency injection. I started out with Spring Initializr and spring-boot-starter-web. Slowly, I pulled out the Spring Boot wrapper, and eventually even spring-beans. I pulled out the spring-webmvc support. I pulled out Hibernate. "Simple" for me meant that I wanted to spend a lot less time reading documentation and more time writing code.

Without Spring, I needed to do some work to get to what I thought was minimal infrastructure on which I could start addressing project-specific requirements. I determined that I wanted infrastructure for at least these features:

1. The concept of a MVC Controller
2. The ability to test a controller action, e.g. a GET request from start to response body.
3. Asset fingerprinting
4. Database migration support
5. Basic SQL statement support
6. SQL transaction support
7. Simple app configuration
8. The ability to launch the app as an executable from the command line
9. Related to 8, a systemd service unit for the app
10. A way to store user sessions within a signed browser cookie

I also made use of some libraries to provide me with some features. These included:

1. Embedded Tomcat and the Jakarta API for the bottommost webserver layer, and basic Request/Response encapsulation, respectively.
2. Hikari for a database connection pool
3. log4j2 for logging to STDOUT with log levels
4. Thymeleaf for templating
5. The Typescript, esbuild, and scss npm projects for building frontend CSS and Javascript
6. JUnit for unit testing, and Mockito for mocking support. I built my own factories.
7. Jackson for JSON help

None of the things on these two lists really had any instrinsic relation to my application's requirements. They were very generic needs. But, I couldn't start to work on my application until these pieces were in place.

I spent what felt like a month migrating from a Spring boot "hello world" project to a place where all of the above was taken care of, without the use of Hibernate, spring-beans, spring-webmvc, or spring-context.

The most obvious consequence of my approach was that writing features took longer. There was one task that I estimated writing in Rails would have taken 10 minutes, but which I spent 3 hours on. It was very easy for me to forecast 5 hours for a feature that I thought was kind of small, but here I was. I didn't have the powerful ORM features at my disposal. I had to write SQL for any database need I had. I had to write unit tests, both at the model level and controller level, for every feature. And of course I had to strongly type my app.

Another concern I have about Rails is that it is dynamically typed. This means that there is no compiler to help you find type errors at a compile phase, whereas with Java and Typescript, you do get support with that. Yes, it takes a little longer to write your code but at least you don't feel hopeless when it comes time to refactor a big chunk of your project. Refactoring in Rails is a nightmare and your unit tests become extremely important in that context. You breathe far easier when you know the Java compiler is going to see what wires you did not reattach correctly.

I do not feel like I escaped writing unit tests. There is the argument that if you're just going to write unit tests anyway, then what is the point of typing? Is it not redunant with the unit testing? My response to this is that as a developer, you want to know in the fastest amount of time that you've made a mistake, and unit tests are not the fastest way to know that. The compiler is faster.

Even though the there was a greater expense of time on this project, the result that I arrived at was more sturdy. If you were developing a video game, wouldn't you want it to never crash? If so, why would you not embrace static typing? Video games are silly wastes of time, as everyone knows. But we still want them to work. Where pride is concerned, I think you want your video game to work just as much as you want code in the NASA Space Shuttle to work. Aren't you willing to wait a longer priod of time for your code to be done, then? This way your app is sturdy. When I say sturdy, I particularly am referring to when refactoring time comes. You can make adjustments and not feel so terrified that you're going to break what was working yesterday.

I wrote above that libraries like Rails disenfranchise the developers who work with them. On this project of mine, I went to use the Devise gem on a toy Rails project while building a tour of some Ruby code. There is a step in the Devise setup where you invoke a Rails generator. All the crypticness of Rails came back to me when I invoked that. I had no idea what it was doing. Why are there ~50 files in a Rails project when I haven't even done anything yet? The Rails library is supposed to support the thousands of Rails projects out there. When you use it, all of the if-statements that have gone into supporting 9,999 other projects besides yours suddenly become your responsibility. It doesn't seem that way, but that's the reality, isn't it? You're responsible for what your app does even if you choose to use Rails. But do you really feel responsible for all of the code inside of Active Record? You're responsible to your customers regardless of what library philosophy you choose. If you choose Rails, you're inheriting responsibility for thousands of lines of code that you know very little about. That places you in an unfair position of powerlessness. And as I mentioned, when it's time to debug that code, it's not in the Zeitwerk watch list, so you have to restart your app every time you edit the outside source. That's after you've navigated outside of your project folder and into the gem sources to find it.

You could wonder how I feel about Sinatra. That's simple, right?

If I were to pursue the above philosophy with Rails, I would write a Rack app without Sinatra. Rack would be where I draw the line. I'd use rspec, still. I'd get controller infrastructure going on my own. I'd use haml or erb for my views. But I wouldn't use Rails, Sinatra, or Active Record.

There is an enormous movement called convention over configuration. I think it would be redundant for me to evaluate it here, given what I've already written. I am much less of a fan of that movement as of today.

Termius Port Forwarding Gotcha

When you setup a tunnel on your iPad Pro to forward port 8080 on a remote host to 3000 on your local host, make sure you give the IP address 127.0.0.1 and not the "localhost" hostname. Or you're gonna have a bad time.

Rails Omniauth OAuth 2 Faraday Getting Unparsable Binary Result from Google's userinfo Endpoint

Problem

My Rails 3 app's OAuth 2 integration was getting binary return values from Google's servers when accessing https://www.googleapis.com/oauth2/v1/userinfo. The below monkey patch, which can be put in a Rails initializer, solved the problem.

Solution

require 'omniauth/strategies/oauth2'

module OmniAuth
  module Strategies
    class GoogleOauth2 < OmniAuth::Strategies::OAuth2
      def raw_info
        return @raw_info if @raw_info
        @raw_info ||= access_token.get('https://www.googleapis.com/oauth2/v1/userinfo').parsed

        if !@raw_info.is_a?(Hash)
          # probably a parse error where we got a binary body...shame on faraday.
          s = RestClient.get("https://www.googleapis.com/oauth2/v1/userinfo?access_token=#{access_token.token}")
          @raw_info = MultiJson.decode(s)
        end
        @raw_info
      end
    end
  end
end

Discussion

Last week near the 26th, my Devise Omniauth Google OAuth2 logins were failing mysteriously.

The way this OAuth2 flow works, you get a 'code' from Google after the user tells it to authenticate with your application, your app then gets an access token with it, and in the middle of the specific Omniauth provider (in this case, the strategy OmniAuth::Strategies::GoogleOauth2), inside of the raw_info method, it makes a call to Google using the Access Token it just got from the code.

This call is performed by Faraday, an HTTP client that can wrap Net::HTTP. Google was responding to it in a binary format. Faraday tries to parse the result with JSON, and if it fails, it fails silently with a rescue and a return of the raw response body that it just failed to parse.

However, I found that if I used the access token in a plain web browser, I could see the expected JSON result just fine.

Why would it work in my web browser but not from Faraday?

I don't know, but I inserted the above monkey patch. If parsing the result from Google fails, it fires off an additional call with RestClient and the access token as a GET parameter - a perfectly valid OAuth2 call. Stuff like this works with Facebook's OAuth (2?) integration, and their Graph API, as well, I think.

The login failures were not always happening. That the above solved the problem in development mode, locally, is evidence that something weird was going on between Devise, Omniauth, Omniauth OAuth2, Faraday, Net::HTTP, or on Google's end.

As an aside, this is one of the most ridiculous bug work arounds I've ever resorted to.

The Importance of Programming Competitions

This is a brief reflection on why I think programming competitions are important for programmers who want to be great.

There should be no argument about this, really, but I try to dig into some detail about it. I believe that programming competitions are immensely more challenging than most work tasks, and they test raw problem solving ability.

The Importance of Programming Competitions from Mike De La Loza on Vimeo.

Deploying a Rails Application to Dreamhost Part 2

I've created an empty Demo app in Dreamhost. I have a working Rails app locally. I need it working at Dreamhost remotely. I need a fast edit-and-deploy process. I need backups.

Assumptions, Versions:

I overload the term "Dreamhost" here to also denote the Dreamhost server that hosts all my domains and servers, including mikedll.com, demo.mikedll.com, and the hypothetical projectname.mikedll.com.

A MySQL deployment database has been created via Dreamhosts' Web Panel. Rails knows its contact info.

My demo app has been launched by Dreamhost using FastCGI, so I'll use FastCGI here (as oppose to Passenger).

Files like config/environment.rb, the logs/ directory, and the vendor/ directory don't need to be synced, but most other files do. A version control system can help with this. Beyond synchronization, it will help me recover when I break a feature. SCP and rsync won't do this. I'll use Git. Hidden Reflex says Git is better than Mercurial and compatible with Trac. Git has Github, which makes it trendy. Git also supports tiered revision, which Subversion doesn't. I've been used to tiered revisions since using AccuRev.

Syncplicity, which I've been using for nine months, will be used for semi-automatic backups.

Local and remote software versions compare as follows:

• Operating System: Local Ubuntu 8.04.3 LTS (Hardy Heron) vs Dreamhost some variant of Linux 2.6.24
• Ruby: Local 1.8.6 vs Dreamhost 1.8.5
• Rails: Local 2.0.2 vs Dreamhost 2.2.2
• RubyGems: Local 0.9 vs Dreamhost 1.3.1

My Ubuntu package manager is behind on versions for both RubyGems and Rails. Besides this problem, most of the community uses RubyGems for package management, not a default package manager.

Given that I'm on Rails 2.0.2 and Dreamhost is on 2.2.2, I anticipate version conflicts. I've encountered a similar situation before.

Develop a Plan

To avoid the version conflicts, I'll sync Rails versions. It'll be an upgrade for me.

I'll manually install RubyGems and use it to install Rails.

I will continue to use Ubuntu's package manager for non-ruby dependencies.

After that, I'll learn Git via the Git tutorial, and then put it in charge of my app.

I can test my edit-and-deploy process by making changes and then expecting to observe them on Dreamhost.

Carrying Out The Plan

I downloaded the following packages and gems into my ~/settings/packages directory.

• RubyGems 1.3.5 (package).
• Rails 2.2.2 (gem)
• sqlite3-ruby 1.2.5 (gem)

I put RubyGems in charge.

# Remove Ubuntu's versions
sudo aptitude remove rails
sudo aptitude remove rubygems

# Install RubyGems.
# The only surprise here is that I have to create a symlink.
tar -xzf ./rubygems-1.3.5.tgz.tar -C ~/
cd ~/rubygems-1.3.5
sudo ruby setup.rb
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

I then installed some dependencies with Ubuntu. The ruby 1.8 development files may be required by the sqlite3-ruby gem. The sqlite3 development headers are definitely required. Without them, the sqlite3-ruby gem gives a "missing sqlite3.h" build error.

# Following above explanations, get sqlite3-ruby dependencies
sudo aptitude install ruby1.8-dev
sudo aptitude install libsqlite3-dev

While installing gems. I explicitly specified the Dreamhost versions. RubyGems recognized and used the downloaded .gem files, which were in the current working directory.

sudo gem install -V sqlite3-ruby --version 1.2.5
sudo gem install -V rails --version 2.2.2
cd ~/myapp
rake rails:update  # update my app's files to 2.2.2

Here, I deleted deprecated calls to cache_template_extensions= and cache_template_loading= in some config/environments/*.rb files.

I then learned and setup Git.

# On Dreamhost, create a remote repository:
mkdir -p ~/git/projectname
cd ~/git/projectname
git init

# On local laptop:
cd projectname # This is folder of my existing app that is not yet versioned
git init
git add .
git commit -m "First revisioned copy of my code."

# Still on local laptop, configure it as a clone the remote repository.
# The remote repository will then be "upstream" of this local one:
git remote add origin mrmike@mikedll.com:git/projectname # Name remote repo 'origin'
git config branch.master.remote origin               # Track remote 'origin'
git config branch.master.merge master                # ... the master branch
git push                                             # Inform 'origin' of commit

# Back to Dreamhost, we create a deployed working copy:
git clone ~/git/projectname ~/projectname.mikedll.com
cd ~/projectname.mikedll.com
git pull

My code was in deployment position. I reviewed my demo app experience to fix file permissions and configure FastCGI. I reproduce a couple steps from that experience here:

# Added this to my .bash_profile, for running rake commands
export RAILS_ENV='production' 

# Initialize the remote mysql database. Here, rails worked flawlessly.
rake db:migrate

My applicaiton should now be deployed and under version control.

Review the solution

I test my solution by editing files locally, commiting them with Git, logging into mikedll.com and pulling the changes, and viewing the website at its public URL. I expect to see my changes.

Then I test my backups by pulling changes into the Windows working copy of the central repository, letting Syncplicity see the changes and back them up to its servers, and viewing my backed up files through the Flash interface at the Syncplicity website. I expect to see the changes reflected in those files.

These tests passed. I had successfully deployed my web app to Dreamhost.

I could also use Git to send changes from Dreamhost to my laptop. This will be used when bugs are fixed on the remote production server.

git commit by definition doesn't require a password like Subversion would, but git push and git pull do. I setup private/public SSH keys to simplify this edit-and-deploy process further.

I used this result to converting my existing static website, mikedll.com, into an identically setup Rails/Git/Dreamhost configuration.

A struggle while carrying out the plan was that, due to extremely poor internet at the time, I had to manually fetch certain gems including actionpack, activesupport, activerecord, actionmailer, and activeresource. RubyGems kept timing out when trying to download them. I omitted these downloads from the above list because if my internet had been better, RubyGems would've installed them for me.

A failure while carrying out the plan was that I tried to do some things with Git that may not have made sense. Originally, I had hoped to avoid a central repository, and to instead push updates from my local laptop repository to remote and backup repositories. I couldn't get this to work, though. I might have been needlessly running from a centralized setup for the sake of being different, instead of trying to get things done.

Future and Related Work

Heroku offers special Rails hosting such that your edit-and-deploy process can consist of a single rake command.

Dealing with conflicts in Rails versions is a known problem and a popular solution is to freeze rails.

The user experiences that alerted me to the deprecations of cache_template_extensions= and cache_template_loading= were Paul Sturgess' snippets collection and the Morph Labs website, respectively.

A Stackoverflow entry showed me how to configure an existing repository as if it were created with the clone command. This avoids the dirty method of cloning a new local repository and deleting the existing repository, which I had done at first and felt embarrassingly weaksauce.

My local laptop environment is actually Ubuntu on VirtualBox, but Syncplicity only works on Windows. So, I created another windows working copy off the central one just for Syncplicity. It will be my (manual) responsibility to pull updates to this copy when I want a backup that is neither on Dreamhost's hard drives nor my own.

My edit-and-deploy process is still missing some vital steps for it to scale, but it will suffice for now. Months from now as bugs begin to pile up, how will I manage them? How will I integrate tickets and milestones? With Trac? With Lighthouse?

Seemingly, Capistrano helps with Rails deployments and many system administration tasks where you have to login to many servers, but I haven't examined it in any depth yet.

Dreamhost strongly recommends launching rails apps with Passenger instead of FastCGI. I will add that as a future todo for my project.

Create a Rails Application on Dreamhost

1. In the Dreamhost Web Panel, panel.dreamhost.com:

   1. Create demo.mikedll.com. Wait until the domain name is resolvable.

   2. In the host configuration, check "Enable FastCGI" and serve documents from /home/myusername/demo.mikedll.com/public.

2. SSH into the web server and run:

   # Create the app
   rails demo.mikedll.com
   
   
   # Go into the public directory
   cd demo.mikedll.com/public
   
   
   # Recurisvely add world-executable to directories
   find . -type d -exec chmod o+x {} \;
   
   
   # Recursively add world-readable to files
   find . ! \( -type l \) ! \( -type d \) -exec chmod o+r {} \;
   

3. In the root of the application directory, run:

   # Make dynamic-content
   ./script/generate controller demo index`.
   
   
   # Initialize the sqlite database
   rake db:migrate
   

4. Create demo.mikedll.com/public/.htaccess with:

   # Launch dispatch.fcgi on requests for which
   # there is no static file
   AddHandler fastcgi-script fcgi
   RewriteEngine On
   RewriteBase /
   RewriteCond %{REQUEST_FILENAME} !-f
   RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]
   

5. Remove demo.mikedll.com/public/index.html.

6. Done

Expect visits to demo.mikedll.com to display static content.

Expect visits to demo.mikedll.com/demo to display dynamic content.

Pitfalls

• This Apache error:

  [...] [error] [client ...] File does not exist: ... demo.mikedll.com/public/missing.html
  

  occurs if the .htaccess file is omitted.

• Recent changes in the application code are not visible if an already-running instance of the application is not killed with killall dispatch.fcgi.

• This Apache error:

  [...] [error] [client ...] FastCGI: comm with (dynamic) server ".../current/public/dispatch.fcgi" aborted: (first read) idle timeout (60 sec)
  [...] [error] [client ...] FastCGI: incomplete headers (0 bytes) received from server ".../current/public/dispatch.fcgi"
  

  occurs when then application crashes while loading. For example, I get this error when my config/environment.rb has a bug in it. The browser will hang for 60 seconds before the server responds with a terse, generic, rails error.

  To debug such issues, I run ./dispatch.fcgi at the command line. No arguments are required. Usually dispatch.fcgi will crashe, give an informative error, and provide faster debugging feedback.

Similar Work

More Trouble-shooting help is available in the Dreamhost "Rails Wiki".

Another guide on a Dreamhost/Rails/Radiant deployment features an application that has many users. It goes into more depth and draws on more experience.

My guide builds on another guide published early this year. This guide had some screenshots and clear directions. However, the .htaccess file in that guide doesn't serve static content, which was a problem for me. Someone else had the same problem. While solving this problem, I used sample .htaccess files at The Rails Playground and this Snipplr entry.

Related Technology

Passenger is a deployment configuration that is different from FastCGI. Dreamhost supports it, but I haven't gotten it to work yet.

Capistrano has something to do with automating common server tasks, like application deployment.

AccuRev on Linux Quickstart

How do I checkout an AccuRev workspace? How do I work with it?

AccuRev installation is easy on Linux. Get their installer, run it, and follow the prompts. Choose a "client" installation instead of a "full" installation. When it asks for the "hostname", put in exactly that - an IP or a hostname. Don't include the port. It asks for the port in the next prompt.

Here are some commands:

# Create the workspace
# I think -b denotes "backing" as in backing stream
accurev show -p PROJECT_Unreleased streams accurev mkws -w new_workspace_name -b Stream\ for\ Cross-Developer\ Testing -l ./local_directory

# Set your AccuRev username in the environment for commands that follow
accurev login mikedll

# Sometimes the client loses time sync with the server, and we
# had to do this before running commands.
sudo accurev synctime

# shows status of modifed, and not in default group
# changes that need to be promoted/committed.
accurev stat -n

# shows overlap only
# changes that may conflict with changes upstream
accurev stat -o

# merges conflicts in overlapp group
# note that sometimes you have to go in and merge them yourself with an editor
accurev merge -o

# Keep modified group into the default group,
# get updates, and promote the default group
# this will be a typical promote/commit process.
accurev keep -m
accurev update
accurev promote -d

If you get the error Element would be stranded, this could be avoided by "purging" your private changes of a file that has been defuncted, or otherwise messed with, in the parent stream. Unfortunately I don't have that command here.

Generating SSH keys for automatic Login

Here is a copy of what I used off an O'Reilly guide - I've ammended it with my changes.

To use public keys with an ssh server, you'll first need to generate a public/private key pair:

$ ssh-keygen -t rsa

After you enter the above command, you should see something like:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/rob/.ssh/id_rsa):

Just hit Enter there. It will then ask you for a pass phrase; just hit enter twice. Here's what the results should look like:

Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/rob/.ssh/id_rsa.
Your public key has been saved in /home/rob/.ssh/id_rsa.pub.
The key fingerprint is:
a6:5c:c3:eb:18:94:0b:06:a1:a6:29:58:fa:80:0a:bc rob@localhost

You can enter a password at the next part if you like, but then you'll have to enter that password every time you try to use the key, which almost defeats the purpose. A compromise is to enter a password, and then use:

$ ssh-agent sh -c 'ssh-add < /dev/null && bash

Back to the original use case - this created two files, ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. To use this keypair on a server, try this:

$ ssh server "mkdir .ssh; chmod 0700 .ssh"
$ scp .ssh/id_rsa.pub server:.ssh/authorized_keys2

.ssh/authorized_keys2 is a file, not a directory. the .ssh directory has permissions 700, as above. The subdirectory has permissions 644, I think.

This task took about 5-10 minutes, and another 5-10 to create this writeup.