The technology development landscape has changed a lot since we started Flurry over three years ago. At the time the dominant development language was Java, all the smart people were programming in Python, web applications were being developed in PHP and new companies were running their applications on rented Linux servers. Times have changed, and these are some current trends in the valley:
- Virtualization. New services are being built on Amazon EC2 or virtualized services like Slicehost. Even the larger managed hosting providers are running Xen instances insead of actual servers. The old art of renting a rack, buying servers and installing them yourself is going by the wayside unless you are a very large company.
- Ruby on Rails. RoR has emerged as the primary web development framework, mostly because it makes building web applications so easy. Other languages have developed good web frameworks, and some languages are easier to program in than Ruby, but none have been able to duplicate the ease of use of Rails.
- Erlang. The rise of IM and VoIP has forced many companies to deal with handling many simultaneous synchronous connections. The demands for these apps are much different than the demands on asynchronous systems like HTTP for web servers. The current solution is Erlang, which is optimized for handling many different processes.
- Git. Subversion has successfully pushed CVS aside as the primary source code management system, but it is still very hard to use with very large teams. Git, originally created by Linus for the Linux kernel, has arisen as preferred source management platform for top developers. It can be used like SVN, but makes branching and merging much easier. What you gain in functionality, you lose in usability as it’s got a very steep learning curve.
- Hadoop. More and more services are requiring very intense computations. Historically, distributed computing environments were largely proprietary. Today, Hadoop is a great open source solution for distributed computing and file storage. It’s based off of the original Google File System and MapReduce papers from Google. HBase, a BigTable implementation based on Hadoop, is in progress.
- AIR. I was a skeptic at first, but Adobe AIR has changed how desktop applications are built. My favorite desktop applications (Doomi, Snackr) are written in AIR and are must easier to use than native Windows applications.
It’s fun to think about how different life would have been with these tools 4-5 years ago. It’s even more fun to think about what we’ll have in another 5 years. Some clear gaps that I think will be met:
- OLAP. There has yet to be a good open source OLAP system that scales well. Mondrian is the only one I know of and it relies on a single JDBC datasource. As services become more complex, OLAP will be more important.
- Distributed Database. Maybe HBase will solve this problem, but there is a great need for an open source BigTable implementation. Today everyone runs MySQL with replicated slaves but the problems with master failover mean that everyone runs into MySQL scaling problems. We need a more scalable, distributed option for datasets we know will grow to be huge.
I do suspect that in 5 years most of the services we’ll use to build applications will be in the cloud and not open source software. At that point, most technologies companies will shrink even further from 100s in the dot com days to 10s today to 1s in the future.
That will be fun.
I’ve discovered a lot of great stuff through the “Free on iTunes” section of the iTunes store. In addition to promotional music, you can often find television series premieres for free. On such premiere I found was for Startup Junkies, an online show chronicling the adventures of Earth Class Mail as it tries to build a business from scratch.
Most shows about startup companies completely miss the point and over-glamorize the process of building a business (see Bobby G: Adventure Capitalist). Hence I have developed an adversion to such shows and only downloaded SJ because it was free and I was already downloading some other things. I’m glad I did.
If you’re not familiar with the tech startup world, the series premiere of SJ will give you an excellent idea of what it’s like. Watching it was very strange for me as everything that happens to them has happened to me in my own startup journey: technical outages (including DNS problems), funding problems, team building, etc. The most poignant for me is the detail they use in presenting the fundraising process. If you haven’t done it before (and even if you have) the process of raising venture funding is very challenging.
So, if you want to get the feel for a startup but don’t want to take the risk give it a try. I can’t recommend the series, just the first episode.
I came across this article right after that last post about global warming:
The replacement of traditional fuels with biofuels has dragged more than 30 million people worldwide into poverty, an aid agency report says. BBC News
To extend the tragedy of the commons metaphor, creating fuel from food is a lot like feeding your sheep with your neighbor’s kitchen to save the commons. The commons get saved and everyone is happy except for your neighbors who starve.
Twenty years ago James Hansen made a name for himself by declaring to a Senate committee that global warming had already arrived. That was 1988. On Monday he revisited Capitol Hill to tell Congress that if we don’t take drastic measures now the earth is toast.
I agree. Climate change is a lot like having termites in your house. You can ignore the problem for a while to save some money, but the longer you wait the higher the cost to fix the problem. If you wait too long you pass the point of no return and your house turns into saw dust.
Unfortunately, global warming is a tragedy of the commons. The concept is explained best by Wikipedia:
The tragedy of the commons is a type of social trap, often economic, that involves a conflict over finite resources between individual interests and the common good. It states that free access and unrestricted demand for a finite resource ultimately structurally dooms the resource through over-exploitation.
The metaphor often used is that of a town commons, which is available for grazing by any of the towns people’s sheep. If all the towns people worked together to limit their access to the commons, it could easily sustain itself and provide for the sheep. However, without any explicit regulation, the towns people act in their own self interest and over-graze the commons leaving it barren. It’s a simple example of short term self-interest overriding long term community well being.
While everyone wants to help fix global warming, there is no short term incentive to so. Without any kind of oversight, people will continue to use resources at will until the commons is barren. Only then, when the effects impact the average person, will real change happen.
The good news is that the real effects are coming. Artic ice melting, abnormal weather patterns, flooding. People will be driven to action and I trust in our ability to overcome the adversity of global warming when we put our minds to it.
Let’s just hope the house isn’t made of saw dust when we decide to act.
Recently, Barak Obama went back on an early campaign promise to use public campaign funding and will instead opt to raise and spend an unlimited amount of privately raised funds. If it surprises you that a politician would go back on a campaign promise so quickly, look no further than John McCain who has essentially done the same thing.
The U.S. public campaign finance program provides matching of donations with public dollars in exchange for the candidate committing to a maximum amount to raise. It makes perfect sense for underdogs whom receive guaranteed money and might have trouble hitting the maximum amount anyway. Both Barak Obama and John McCain committed to the public campaign finance program when they were large underdogs. Despite the obvious economic benefits to an underdog, committing to public financing is a tool to try and force the front runner to limit their spending. If you have no chance of spending as much as the front runner, try and limit how much they can spend by getting them to commit to public financing. If that doesn’t work, criticize them for bypassing the public system. Typical guerilla tactics.
The surprising part isn’t the back tracking on public campaign finance (although John McCain has been a proponent of campaign finance reform for years), as today’s campaigns require far more money than the public system allows. Barak Obama, for example, would have to give up his record-breaking fund raising advantage in the public system - taking him from first to last in money raised. Additionally, 527 committees are political organizations which are not associated with any particular candidate but can run campaigns based on issues and are not regulated by the FEC. 527 committees were used to great effect in sinking John Kerry’s campaign in 2004 and Barak needs the money to combat them in this election. For Obama, bypassing the public campaign finance system is the best decision. McCain has no choice but to follow or risk being outspent.
The surprising part is that we focus so much on any perceived change in a candidate’s position on any issue. Today’s politicians are trusted so little that we look for any sign that they are lying or misleading and immediately label them as a fraud when we find it. In general, everyone prefers a leader who makes the best decision in a given situation even if they have committed to another path in the past. No one wants to be handcuffed by past decisions if the facts change. Politicians, however, are measured by their policy positions going back decades. Any deviation to the ideas you had starting in college is seen as a sign that you flip-flop on issues.
Candidates should be held accountable for their policies and we should be confident that they won’t change their minds at the drop of a hat. However, they should be allowed to change their minds when the facts change and we should respect that. The paranoia that these new candidates face is a product of the US politic system, so they do bear some of the burden. The best way to fix this problem is transparent government, sound policy decisions and realistic commitments to the American people. Until then, politicians will have to learn to deal with the scrutiny they have brought upon themselves.
An entirely different issue is how much is spent on presidential elections in the US. No optional public campaign financing program will curb the out of control spending. Instead, real regulation is required before we spend more on election campaigning than we spend on helping people in need.
It goes without saying, but if you’re reading this you can see. The eye is one of the most complex products of evolution, and while we take it for granted everyday it is something we should appreciate more often.
Melanoma runs in my family putting me at high risk to develop it myself. Most people know of melonoma as skin cancer but technically it’s cancer of the pigment cells which doctors refer to as melanocytes. Those are the same kinds of cells that make up your retina and allow you to see, so it is possible to get melonoma on your retina. While melanoma of the retina is rare, Melanoma comprises the majority of skin cancer related deaths.
My aunt died of melanoma of her retina. She was a very strong woman and before it killed her it made her blind. Seeing what it did to her was one of the hardest things I’ve ever seen. Seeing how she fought it was the most inspiring.
So, I got to the eye doctors on a regular basis and in addition to examining how bad my eyesight has become (it’s pretty bad) they look extra close for any signs of melanoma. It’s a sobering reality check on a regular basis.
A great way to appreciate how lucky you are to see is to try living with a blindfold on for any length of time. I strongly suggest doing it with someone else around and only in your home lest you hurt yourself. Doing it for even five minutes can make you very thankful for the little things.
For me, I consider sight a blessing that could be taken away at any time with no warning - just like my aunt. Any given trip to the doctor’s could come with bad news.
So, I appreciate it while I can. So should you.
If you haven’t noticed at the gas station (or been reading my blog), the price of gas has skyrocketed in the past year. Most of that has to do with the price of oil skyrocketing during the same time and, well, gas is made from oil. Seeing the price double over the course of two years is a good sign that there is speculation going on in the market, even if some of the growth is due to the falling value of the US Dollar.
To prove this is really a bubble, you just need to compare it to the stock market and housing bubbles we know so well. See the following graph courtesy of Paul Kedrosky’s great blog Infectious Greed:
My main observation is not that there is a bubble or speculation, or even that this bubble is similar to the last two. I’d like to point out that in many ways this bubble is the opposite of the stock and housing bubbles - what I’ll call an anti-bubble.
Why is that? Here’s a comparison:
|
Oil Bubble |
Stock/Housing bubble |
| Who benefitted during the bubble? |
Speculators and oil companies |
Retail investors and homeowners |
| Who suffered the most during the bubble? |
Retail consumers |
|
| Who gains the most from the bubble bursting? |
Retail consumers |
Speculators |
|
Unlike previous bubbles, the oil bubble hurts retail consumers instead of generating wealth. And, unlike previous bubbles, this bubble bursting will actually help wealth creation instead of destroying it. Hence, an anti-bubble.
As with any bubble, the rationalization of the bubble tells the most about whom benefits and who suffers. With the stock market and housing bubbles, there was wide-spread belief that the amazingly high prices were driven by a fundamental shift in the economic landscape. In both cases, amazing innovations in technology and financing had opened new doors to continued growth that justified clearly ridiculous prices.
Now, with the oil bubble, there seems to be a belief that there is no fundamental shift and we can in fact get things back to where they were. The government will allow drilling for more oil off the coast, OPEC will produce more oil and eventually we’ll get prices back to where they should be. Some attention is given to the rising consumption of China and India and even less to the weakening dollar - both of which are actually fundamental economic shifts.
Speaking of which, if you are hoping that when the bubble bursts the price of gas will go back to low prices of yesteryear you’ll be disappointed. Even if you take speculation out of the equation, the price of gas would likely still land around $3.50.
Whew. We’ll need another bubble soon if we’re going to afford to buy gas.
And now for something completely different. Here’s a poem I wrote a few weeks ago.
Broken Heart
Quietly the fizzures show
Broken like a jagged stone
Coldness creeping in below
Broken, barren, cold, alone
Quickly, now, before sadness creeps
Forget it all but don’t let go
Push it down to endless deeps
As time itself begins to slow
Catch a glimmer, bursting bright
Dare not hope, dare not smile
For fake is hope behind the light
And awaiting is the lonely mile
Be still my heart as love is gone
The day is breaking, but with no dawn
Despite numerous reports of me going into hiding I was in fact back east for the past week spending some time with my family. Have no fear, I have returned!
Unfortunately, in the meantime I wasn’t able to provide my unique perspective on the exciting events of the past week. To make up for it, here they are in rapid fire:
- Tim Russert died on the set of Meet the Press. Tim was one of the few journalists who was willing to really challenge politicians on issues. Unlike almost every other journalist on TV he was able to make politicians visibly uncomfortable by asking questions that regular people would ask. He brought some honesty to politics and will be missed.
- Hillary Clinton acknowledged Barak Obama’s victory in the Democratic nomination. While the news quickly shifted to McCain vs. Obama there is a secondary storyline about Clinton, her supporters and what really happened. There are claims of sexism, campaigns to have Clinton as the VP candidate and promises by former Clinton supporters that they won’t vote for Obama. PUMA (Party Unity My A$$) is a great name for an organization, but all of these people are so focused on the fight that they have lost perspective. As for sexism, this is a comment I left on Jeff Jarvis’s BuzzMachine blog where he highlighted the sexism issue:
The irony here is that if Hillary won the nomination, this discussion would decry the racism prevalent in the US that prevented Obama from being treated fairly. I don’t think blaming sexism for a poorly run campaign is what we need right now.
If there really was such blatant sexism, Hillary never would have been considered the front runner for the nomination.
- The price of gas keeps going up. It’s a commodity bubble, no asset appreciates in value that quickly. It will come down, but not quickly and definitely not to the level it was at before. The real story that comes out of this is how hedge funds and speculators are beginning to affect markets to make profits instead of just making smart investments.
- The financial services industry in the US is valued at roughly the same level it was back in 2000. The cliff it fell off last year is really apparent in the value of the iShares financial services ETF IYG. Recent rumors of Lehman Bros going out of business and continued problems in the credit markets have combined with climbing gas prices and inflation to keep them low. I suspect they bounce back in 2009 so if you’re an investor take a look. (Disclosure: I’ve been investing in the US financial services sector this year, with a focus on LEH and WFC).
Consider yourself informed.
Powerset is a much-hyped search engine that has been on the verge of launching for what seems like a decade. Just like every other search startup they have been labelled a Google-killer for their natural language processing capability, which more accurately extracts your intent from your search query. The service right now only allows you to search Wikipedia (and a few other sites) but works quite well.
It’s very easy to dismiss Powerset as a gimmick as it lacks the internet-wide coverage of Google’s search index. Intuitively, it seems like searching only Wikipedia would produce worse results than the entire internet. Luckily, intuition lets us down again. Wikipedia is not only one of the top 10 trafficed websites but it also one of the top results for many searches on Google. It is at the top of so many searches, in fact, that Google launched a project called Knol to try and retain some of that traffic. The reality is that Wikipedia is one of the scariest things to a company like Google.
How can something possibly scare Google? A dominant player like Google (and Yahoo, and Microsoft before them) does not really fear competition as their advantage is far too great to compete head on. What they fear is a change in the rules of the game. For Microsoft it was a shift from the OS to the web as the power platform for software. For Yahoo it was a shift from web portals to search/aggregation. For Google, it is a change in the way information is organized on the Internet. They fear a shift that makes their competitive advantage meaningless.
Google’s core search advantage was originally formulated in the late 90s. It was built on the assumption of a web of links between websites that helped find the answers that were hidden throughout the landscape. The best answer to this question was over here, another question over there. The data was poorly organized at best, but with complex math you could look at the big picture and find interesting things. In this world, Google became the king.
Google, however, was not designed for a world where a single website had all of the answers. Google was fixing a simple problem: the internet’s information was spread out and poorly organized. If there was a single website will all the internet’s information well organized, all of a sudden Google is not as useful. The rules would have changed. Wikipedia is an example of what that world would be like.
Luckily for Google, Wikipedia is a non-profit company and cannot undermine its business in the way Google undermined Yahoo or Yahoo undermined Microsoft. In fact, Google can use Wikipedia to make its search results more accurate and improve it’s value.
Unfortunately, so can new startups like Powerset. If the world becomes more organized then the advantage goes to whomever provides the nicest card catalog. It’s a lot easier to build a card catalog than a library, and the barrier to competition is also lower. Friction points disappear, and with them entire business models.
Powerset won’t topple Google because Wikipedia won’t help. Google’s business will be strong for a long while, but the rules will change eventually.
So, don’t count out players like Powerset just yet. They are just waiting for the whistle to blow.
