Note: hashtags.org has banned me for spam and I understand. Sorry guys! The experiment was stopped as of 11:00 JST

Last night I did a little art experiment: What happens when you use a service like TwitterFeed.com to feed your Twitter status updates back into itself.

• created a user called loopyloop
• used twitterfeed to update both the RSS and Atom feed
• used the great service hashtags.org to keep track of the activity

I was hoping to see some interesting emergent behavior, but the end result was a simple update every half an hour with 5 updates.

I do think it is interesting and important to think about full-artificial organisms living and looping through our cyberspace.

This is a more extensive writeup of a problem I have talked about before. It’s about something many of us web innovators face today. Like many problems, they can be turned into opportunities if implemented well.

The new Web – let’s stop using 2.0 – is much more than social networking sites and round cornered layouts. As Tim O’Reilly said, people who think Web 2.0 is about AJAX are completely missing the boat. This is true, but it is important to keep in mind that the shift from desktop products to web services is a big deal. Online services are already gaining a lot of productivity for people and thanks to the characteristics of the web the companies developing it can distribute and evolve much better. So this means that certain value adders have now migrated to the web.

What is happening now is that many of these online services want to be able to harness the power of the new social characteristics of the web. People have their friends online now and there is a lot of activity between those people and services (UGC). Everybody wants to tap into this attention stream between people. It helps contextualize your offering and helps spread the goodness between like minded.

Unfortunately, getting this social activity going around your service is not easy. And frankly, most of the time it results in creating another lousy SNS with worse social features. Fortunately, most online services open up their API’s allowing you to integrate with their services to create a mashup.

• rate, review and discuss cheeses
• have support groups like: ‘Why is cheese so expensive in Japan?’
• add cheeses they’ve eaten, for example through IM or mail
• upload/import pictures of their cheese related experience
• provide little cheese diary entries for the real fanatics

• using Twitter to add and rate cheeses
• using Flickr to upload/import pictures about cheeses
• using Tumblr to provide journal functionality

We could use one of those create-your-own-community sites like Ning, but than we can’t build our core business around it: selling and recommending cheese based on their attention profiles. Fulfilling all cheese needs to our customers is why we exist.

Mashing up a service like this using Twitter, Flickr and Tumblr is easy. Turning that mashup into a success is nearly impossible. The problem lies in the shallowness of the way we do mashups now. People need a Twitter, Flickr and Tumblr account to use all functionality in our system. Explain that to the 60 year old French cheese farmer who just plugged in his DSL modem!

Now, I don’t want to start debating about web savvyness and targeting certain markets. My main point is that there is a certain flaw in all API’s: You need to register with the API provider in order to make use of the API anywhere.

• our cheese shop is using OpenID
• our cheese shop can create a new OpenID easily
• the services we integrate with are using OpenID
• the services allow first-time use of their service without ANY burden to our customers (truly seamless)

I think OpenID is the path we are moving towards, although I’m sure that at first most services will not be prepared for the seamlessness we want. Also, I have an interesting scenario for the case that OpenID doesn’t take off (will save that one for later).

Services will be able to focus on their core added value by specializing. Building things that have been build before is an extra baggage that seamless integration can fix. Also, when a service continues to specialize it needs to be thinking from the very start about it’s own integration in the collective. In our Cheese example, our service should have an API from day 1.

But right now, most services are not ready for this seamless integration. If FlickR would allow use of their API without showing any FlickR logos or Flickr ads, how would they make money? What if we want to display our own CheesR logo everywhere and completely exclude the user from knowing FlickR is used?

This is where Freemium Integration is born.

Right now, most popular services offer API’s. If you look carefully most of them don’t guarantee any service with the exception of Google’s. API’s are in their very early stage.

Soon, API’s will become the core of any successful online service. They will be the door to most of the revenue. Services will continue to specialize in the things they do well, recommending wine, recommending books or providing entertainment you appreciate. This specialization might go on to a point where there will no longer be an actual site, but just a widget, API, Facebook Application or some other integratable entity.

These integratable entities – that I will just call API’s for now – will have several ways of generating revenue for their creators.

• Free Integration. This could be either seamless or semi-seamless integration together with in-content contextual advertisement. In order to make these advertisements contextual there will need to be some kind of Attention Profile exchange between the two parties.
• Premium Integration. Seamless integration without brands or attention debt. The client-party will have to pay the provider an Amazon AWS like compensation for usage.

I think over the next short-term march towards the first intelligent agents, it is important to keep the API in mind. Make sure that your online service can be used with or without OpenID and that third parties can integrate seamlessly without any hassle. This also means changing your mindset towards branding since this is a concept likely to change very rapidly soon.

The purpose of this article is to spark discussion from the community. Have any of you encountered a similar discussion or desire for seamlessness? Any opinions welcome!

Yesterday I hit the Digg front page with my Building a .com in 24 hours article. Apart from that I was also number one and number nine in the delicious rankings. Very exciting!

What is also interesting is that my rapidly build website – wigitize.com – got a lot of hits. My blog got about 25000 uniques, Wigitize about 10000. All these hits caused an incredible load on my small 256MB memory server so I quickly bought a second one and scaled up. Yesterday, I could finally stabilize the backgrounding on the website and everything is running fine ever since. Someone offered me a free slicehost slice too last night!

It’s also funny that people commented on how I must have too much time on my hands. I even got offered participation in startups and companies haha. In fact, the opposite is true, but I do love hacking away in my free time. Besides, it was xmas, that means coding, drinking and eating right?!

While combating the performance issues I had yesterday, I learned these technical lessons:

Make sure you’re running the right version of tools. Apparently I was running a very old backgroundrb version. So I now upgraded to Ezra’s 0.2.1 version

Debug things in production-mode on your local machine! Some code – especially funky code like backgroundrb – will behave differently on production as opposed to dev. Often people like me are strictly developing in ‘development’ mode and than publishing to staging/production environments. I found a mayor issue that was screwing up everything in backgroundrb by debugging things in production mode on my MacBook.

1

/<link.*href=['"]*([^\s'"]+)['"]*.*application\/rss\+xml.*>/.match(html)

1

/<link.*href=['"]+([^'"]+)['"]+.*application\/rss\+xml.*>/.match(html) 

Update: Last 18 hours I have been combating performance issues because I got digged. Things seem to work more stable now, but still the background processing fails on me sometimes. Good lessons are being learned here so I will wrap that in a nice article soon!

Update: I just added another 1 GB slice, that should get us going for a while. Now just have to wait for the damn DNS entries to update.

I am preparing a new machine as we speak. Wow, did not to run into these luxury problems already! Thanks for the support!

If anyone thinks of buying a slicehost.com slice, please use this referral URL to donate:

https://manage.slicehost.com/customers/new?referrer=129028580

Machine 1 (256mb memory, how I started out with 2 days ago):

Machine 2 (1024mb memory) new

This is about how I spend 24 concentrated hours spread out over 4 days during Holidays to build the online service Wigitize.com. It is part of my ongoing learning process on how to run a successful web startup.

Even though I’m a super pimple-faced code-geek, I strive to be a creative entrepreneur who can utilize modern day tools and navigate the chaos to build cool stuff. What I tried to do for this project is use some new methods/tools out there to solve practical problems in my weakness area: design, frontend coding, system administration and SEO.

Purpose of this article is to show my thought process on the multidisciplinary aspects of this project. Also to invoke the discussion on how these things could be done much better (correct me!) and hopefully also to educate other entrepreneurial minded hackers.

Wigitize.com, A geeky Idea

While working for my current client here in Tokyo, I’m often running into problems where I wish there was a third party web service that could solve them. For example: Uploading/Managing pictures for user generated content. We want to build cool shit, not reinvent the wheel.

Another one of these ‘problems’ – by saying problems I mean ‘important new features’ – was allowing users to import their blog’s RSS feed. When users input the URL of their blog, a listing of their most recent blog entries will be displayed, sounds simple enough right?

Several weeks before, I already wrote a feature that allowed users to display their latest Twitter ‘tweets’. Building this ‘passive twitter integration’ seriously did not take any longer than one hour! This is because Twitter provides a blog badge or widget. More and more sites are starting to provide these widgets making it easier for people to take their data and display it on their blog. And that’s great!

However, the most popular data feed format – RSS – does not have these widget benefits. Widgets rely on a smart technique called JSON which allows your browser to fetch the actual data. With RSS feeds you need relatively complicated server-side processing to display the data.

So wouldn’t it be cool if there was a web service that allowed you to convert any RSS/Atom feed into a embeddable widget? I’m sure some of you will suggest similar services now :]

A service like that should be:

• simple, with one big ass URL input
• smart, it should detect these data feeds
• integratable (another adjective!), by providing an API for other web services

Aahhh a new project has been born. Let’s open up MS Word and fill in our Prince 2 Project template. Herein we can properly plan for all things we need to be doing and all risks and…

Neh, just kidding! Let’s open up that bottle of wine and… just fucking build it!™

Design Mockup, 3 hours

For me, this is one of the more tricky parts. But fortunately, the project is simple and Web 2.0 designs simple too! Since I don’t aspire to be the next whoever cool designer, I will just give you my Idiot’s view of doing design:

• being able to draw is not a necessity
• always start by putting down your content first
• learn a bit about colors: Color in Motion, Vleere and Kuler (not cool but useful)
• don’t force making a design, do it in the evening when you find the creative flow – or in an onsen if you have a waterproof mac

The weapon of choice: Adobe Photoshop (CS3, the latest shizne!). (Any good web alternatives yet? :) They say Photoshop has a steep learning curve and I agree. Still there are some things you can do to speed things up:

Learn the most important shortcut keys, you will use them a hell lot (ordered by usage):

• V, get the arrow key to start dragging shit around
• M, marker to select things
• T, text tool
• I, eyedropper to pick a color
• CTRL+T, transform current selection
• G, paint bucket to fill in stuff
• W, magic wand to select stuff of the same shape/color

There are some things you will be seeing a lot in particular when doing Web20-like designs:

When performing blending options on a layer, you should be playing a lot with the following marked options:

You can learn here and here how to make these babies:

Woops, I shouldn’t get to deep into the details. Anyway, based solely on the content a design like this rolled out:

The colors can be justified as soil and nature, yin yang, peace man!

Storing your Files in an online Project Manager, 30 minutes

After having made those designs, I’d like to throw them online somewhere right away. Preferably in a SCM repository like Subversion. Luckily, I remembered an article on ReadWriteWeb about such Developer Tools

Basically there are two startups that provide freemium SVN and project management hosting. I chose Assembla for disk space, but Unfuddle was a close second.

That said, 10 minutes later I could start throwing things into my very own http://svn2.assembla.com/svn/wigitize/trunk

Hell, it even puts changeset notifications in my Junk box!

Playing with Rails 2.0, 4 hours

Weapons of choice: RubyOnRails 2.0 and Textmate.

Note: If one piece of software is worth paying for, it’s TextMate – am doing so now

RubyOnRails is very suitable for me, but I’m not sure if it is that suitable for you. I myself have a clear CS-engineering background and am very comfortable with digging in deep. On the other hand, depending slightly on luck, you can do a lot with this framework by just modestly hacking away and watching a screencast every now and then.

Rails is quite controversial in terms of scaling and production-ready. However, I think things are changing fast and as I will show you later. Serious innovation is done on how this framework plays with it’s systems layer and technologies like Amazon’s Elastic Computer Cloud

Like making the design, the coding also starts with content. First you jack in the important texts/inputs you have in your design and than you enclose them in the necessary divs so they can be CSS-styled later. In Rails this starts with making a home/welcome controller and writing the default layout: app/views/layouts/application.html.erb. Yes we are using the new Rails 2.0 way of html.erb.

A very important part of Rails I think is the config/routes.rb file. This file holds all your pretty URLs like: http://wigitize.com/json/for/http://dominiek.com/ . In a way routes.rb lays out all the abstract functionality of your web application. Rumors are that the routes part of the framework is internally the most complex one.

Some quick hinters on Rails practices that will speed up things:

• Never do things like: link_to(’’, :controller => ’’), use Named Routes
• If something seems complicated, break it down into simple steps and write a unit test first
• Breathe consistentness, everything you code – even whitespace – has perfect reason. This also means applying Don’t Repeat Yourself all the time as a habit.
• Naming can be a bitch and you can take up a lot of time. Do your best and pick a name consistently, you can always change it later.
• In Rails, take Skinny Controller, Fat Model to extreme heights, it will make your life more easy
• Do it in AJAX right away, it’s often much more simple than supporting plain HTML CRUD in the first place

There is a lot of material out there available on coding Rails in general, so I will let go here.

These first two hours are basically setting up all the routes, filling in the HTML and making things work in a very basic sense. For wigitize that meant:

• hooking up the URL input to a feed detector and aggregator (most of that code is from here)
• making sure that the aggregated data is stored in JSON format (In rails that means calling .to_json on any Object, easy as pie!)
• adding a Widget model that can actually hold the URL, detected feed URL and JSON data:

I spend the next 2 hours seriously code-monkeying on the feed detecting and parsing part of the system. I will soon open source it under the name feedeater.

Style that HTML! 1 hour

Styling, already? Yes, I think it’s good to style quite early in the process. For me, there is one single argument: Flow.

When you are making something work it is nice when things already look quite tangible and usable. When you apply styling in an early stage you can see direct usable results of the things you are building, increasing the psychological state of the Flow.

Ok, so I also kinda suck doing CSS, but I learned enough to turn an image into a web page. These are my bullet point lessons:

• Develop for Firefox first, using the tool Firebug. If you aren’t using Firebug and doing web development you’re either slow or an imbecile ;]
• Always put in some basic CSS, I got some here and made it into my own html.css which I can include as a base all the time.
• Padding and margin: These things are great and you need a lot of them. However, padding often gives you shit so try to choose margin over padding. Also, choose margin-bottom over margin-top since everything tends to float upwards (Thanks Simon)
• Current fashion: Get some nice fonts going on, I’m using Trebuchet MS a lot for Wigitize and I mix it in with plain Arial.
• Current fashion dictates that you use a less fierce black for your text, make it #111 or #222.
• For mozilla/safari I’m using -moz-border-radius/border-radius, will never work in IE6 – fuck them, being a plain user is fine, but you’re not getting any round corners! Besides, isn’t Microsoft planning a forced upgrade soon?
• You only need tables for tables. They will make your life a pain and you will not be cool if you use them for layout. Other than that it’s fine.
• Little side note about AJAX: don’t use AJAX for navigation EVER! There are strong SEO and usability arguments against it (learned that the hard way, like most things in my life).

Spin it Baby! 2 hours

OK, I must admit that I spend way WAY too much time on this! However, when you’re doing things with AJAX, you need to put a spinner somewhere to indicate loading.

This so called spinner is pretty kickass and based on a piece of Javascript I wrote earlier:

Yeah Baby! It’s all hypnotic and stuff.

Note: As someone pointed out in the comments, on ajaxload.info you can get a lot of spinner images, like the one I’m using here.

Designing and Coding the Footer, 2 hours

I really like footers and I think they are becoming more and more important. Nowadays footers are used as sitemaps and often they are contextualized as well. These are nice examples of footers: last.fm and snooth.com

For Wigitize the footer is rather small since it’s a small site:

Making the JSON Embeddable, 3 hours

<div class="feed_widget">
  <ul id="feed_widget_34"></ul>
  <script type="text/javascript" src="http://wigitize.com/javascripts/wigitize.js"></script>
  <script type="text/javascript" src="http://wigitize.com/feeds/34.json"></script>
</div>

This works in three steps:

• define a containing list (ul)
• include a JS library that has a special callback function, in our case wigitize_feed()
• include the JSON file that will call wigitize_feed with the appropriate data

Providing styles obviously complicates things a bit. When choosing a style, it will include a generic wigitize.css and assign a class to the containing div.

Near future improvements:

• Provide a ‘grab the grabber’ so that people can provide widgets of their own feeds (eg ReadWriteWeb providing a last-10 articles widget). This could have a lot of potential if provided in a simple feed-burner like button.
• Put in better default styles than the lame ass ones I have now.
• Option to include data (useful for photo feeds).
• Option to display other kinds of aggregated data, eg microformats.

Making things run in the Background, 3 hours

Running things in the background – dubbed backgrounding – is an important part of production ready web applications. It’s a relatively new concept, since web applications used to be less complex. Now however, we are marching towards the Semantic Web where web apps are expected to become intelligent (the Intelligent Agents are coming, just like in the Langoliers!). I think being able to make your applications autonomous now will already reap you benefits (to be continued as an article).

There are several ways to achieve backgrounding in Rails, but the far away most easy one is using BackgrounDRb. BackgrounDRb is a plugin for Rails that allows you to easily kick off background processes and schedule regular tasks. Good for our purposes: detecting and fetching feed data.

I don’t agree with all of Zed Shaw’s big rant about the Rails community being a ghetto, but I sure do agree that there are a lot of idiots out there that produce things that can screw over your production apps. BackgrounDRb has become one of these projects and I strongly recommend that you do not use the latest code. If you start comparing code and read the mailing list you will see that a new guy has taken over Ezra fine project and I suspect that he has lowered the project’s level to pre-stable. I don’t hear any signals from the community and that worries me. Either I’m seeing ghosts or people are blindly accepting anything that’s marked as stable. In any case, I’m using Ezra’s version which works fine.

Finishing up the API, 4 hours

The Wigitize API for now is quite quick and dirty. There are two simple ways of using the API and examples are provided. In this area, a lot of improvements will be made down the line since it’s a key point in making any future freemium revenue.

Domain and Domain Email, 30 minutes

When I buy a domain I always buy a DNS managing package with it as well. This means that I can login somewhere and setup subdomains and set mail records. So the total price of Wigitize.com was 20$ per year.

Providing info@domain.com email is easy, just get yourself a free Gmail for organizations account. In your Google Apps domain manager you can simply add your domain and in your DNS tool you set the MX record to ASPMX.L.GOOGLE.COM. Now you can use gmail and IMAP to read mails sent to your domain.

Setting up a Production Server, 1 hour

I was really eager to put this project on Amazon’s Elastic Computer Cloud until I calculated my monthly costs. Running costs alone for simple projects like these will cost you 60$ a month. Still I think it’s worthy to look into this once you scale beyond simple project.

After digging around I figured out that slicehost.com would be a good cheap second. For 20$ a month I have a 256 memory slice with 100GB in data transfer – awesome.

Setting up your slice takes 5 minutes with a credit card. This slice is essentially a virtual machine with an IP address, completely yours. And the best thing: you get an awesome web console to control everything! Adding a new machine is a no brainer.

Now, I’ve used linux/unix for a long time as a working station. Eventually I got lazy and switched to my current MacBook. Fortunately, you can be lazy for the systems side of things too. All thanks to a lovely tool called Deprec.

Deprec allows you to install the complete Rails stack with a small set of commands. Shortly thereafter you can deploy your application to your production server by typing cap deploy_with_migrations. Please note that for Deprec you need to install Ubuntu Linux on your machine which you can do as follows:

Deprec installs the Rails stack as: Apache, Mongrel cluster (default of 2 instances) and MySQL. As I’ve written earlier, NginX is a nice nano-alternative for Apache. I would like to see that in my Rails stack someday, but I’m not going to worry about that now. Clock is ticking!

Little Pimps and Tweaks, 3 hours

I think it’s good to prepare a little bit for the storm (and I felt like doing something else for a bit), so I’ve created a nice maintenance message for in case there are system/scaling problems. In here I think it’s important to give people an extra reminder to bookmark and come back.

Which brings us to another great service, addthis.com. AddThis provides you with a button that makes it easy to bookmark on multiple social bookmarking services.

Another one of those little tweaks was proper error checking and displaying it to the user. I am using a pink error message to make it look more friendly (maybe I should go even further and make it yellow or something):

Statistics and Search Engine Optimization, 1 hour

Statistics is another 10 minute no-brainer by using Google Analytics. However I’m on the lookout to find something more real-time like Mint (but than free). Any suggestions?

Note: I just added getclicky.com to get more realtime stats than GA

I found a blog which is solely about Rails and SEO which I thought was very promising but in fact doesn’t have much content. I do found something on how to provide different meta tags in rails which I applied right away.

After looking around a bit I also saw some discussion about whether to use www. or not. The way to roll with this is: you permanently redirect your www. domain to say http://wigitize.com. It makes sense, www is old and architecture centric, http://wigitize.com is less typing and pretty.

On that note, make sure that you always write pretty URL’s by practice. This means thinking them through extra carefully, because changing URL’s is painful after going live.

Wigitize.com was also chosen with SEO in mind. The word is a mis-spelling of the term widgetize and yields 105 Google results (at this moment). Additionally, it is a verb, which lubricates the prettiness of the URLs :]

• Making a list of ‘last wigitized sites’ and ‘most wigitized sites’, those pages can be accessed by the search spiders and thus associating external content with Wigitize.com. I think this might improve search rankings.
• Providing a sitemap.xml, would that help?
• Focus on the viral aspects of these widgets. For example a FeedB urner style button for on popular blogs.

Let’s throw it out there!

While writing this article – which took quite some time. Wigitize.com is already running and doing it’s job. However, I’m sure there are still some kinks to work out which I will do over the past coming days (eg IE6 support, SEO/viral tasks).

Also, I didn’t discuss anything about an important aspect: How to make revenue with Wigitize.com? I’m not sure yet, but since I’m solving a problem for myself, I’m sure others out there had it. Besides, the costs are extremely low at this point so I will worry about monetizing later. Although I would like to hear YOUR thoughts about it!

I realize that this is a geeky project and I must say it’s quite different than the web apps I normally work on. It was fun for me however to write down my thought process, especially on the non-tech parts of building which I find increasingly interesting.

Hey, I think I’ve done this before :] A new blog design for the new season. Still not very colorful, but a lot more focussed on the actual content.

This is all in preparation of a series of articles I will write about:

Wigitize.com – Building a Production Ready .com in 5 days

BackgroundRB was a nice project that allowed you to easily do background jobs in Ruby and Rails.

Now, it is maintained by some PRICK who is rewriting everything making it IMPOSSIBLE to understand. (Maybe Zed Shaw has a Point?)

Today and yesterday I’ve spend another couple of hours coding and designing away on my (autistic) little project Wigitize.com. I’m keeping an exact log of the amount of hours I’m spending and what’s happening so I will publish some full-exposure articles about how to build a project like this.

Currently these are the things that still need to be done:

• enabling people to choose a style for their widget
• write a simple API and a page to show how it works
• optimize backend so that a background worker will be used
• allowing people to ‘grab the grabber’ so that they can offer widgets of their own content to their users
• renting a machine with bandwidth, I’m thinking of slicehost.com (much cheaper than EC2)
• rethinking the name WIGITIZE. It sounds good to my Dutch-English, but it might not for Natives.

Another little sneak preview:

click image to expand