SOLR, Lucene, Java, .Net and Microsoft SQL Server

@ Hurriyet, we finally launched our new search http://arama.hurriyet.com.tr/
It runs perfect.

I am going to tell you about the technology behind it.
After long time of research and evaluation, we finally decided to go with Lucene, which is a Java Library. But the problem was, we needed a solution that supports any technology. Then we come up with a sub-project of Lucene called SOLR. Which is a HTTP gateway for Lucene. This way we could support any kind of technology and use search as a service (SaaS :P).

SOLR is an highly extendable application that runs behind Tomcat. The output results are XML (but can produce JSON, PHP as well). Each search query is a HTTP GET request and each update (including delete) is HTTP POST request. That's it, this was what we needed.

So basically we had two Linux Servers, each running almost same configuartion with Tomcat and SOLR installed on a port other than 80. One is named "Master" the other named "Slave". Master is responsible for collecting data from SQL Server and update the Search Schema. This is a cron-job that runs in every 15 minutes. Only modified records are updated on Master. Master also has the ability to fully import data from scratch (this can be done in 40 minutes). Master creates a snapshot of the archive in every 15 minutes. Slave is the actual box that is queried. All select/search requests go to Slave. It updates its own schema from Master. This way we can increase the number of Slaves and load balance them.

The Search web server asks search requests to the Slave instead of a regular SQL Server. Web server is an ASP.Net application, that's running on Windows IIS and makes HTTP requests Slave, to get XML results. We also created an Application Block with Enterprise Library, so that Search concept is abstracted from the Web Application.

With this design we get the following benefits:
1. Amazingly fast web requests
2. Clever search engine
3. Ability to scale (both on backend and frontend)
4. Failover clustering
5. Highly customizable design

Google App Engine to support Java and Amazon announces Elastic Map Reduce

On April 7th, 2009 Google App Engine announced to support Java on this blog
http://googleappengine.blogspot.com/2009/04/seriously-this-time-new-language-on-app.html

Although I was happy with Python, now they are also adding Fortran 77 support.

Also they have Eclipse support as well
http://code.google.com/eclipse/

Second thing, Amazon announced Elastic Map Reduce which is based on Hadoop.
http://aws.amazon.com/elasticmapreduce/
Note that Amazon have amazing services for web as well as Google.

I believe, soon the way we program and host our applications is going to change again.

My Linux Adventure

It's been couple of months since I switched my desktop to Linux. On recent days I was working on many open source projects, then I decided why not switch the desktop as well. I think I made a nice choice. One can live with Linux desktop within a Microsoft environment.

I first tried Fedora which was 9 then upgraded to 10, but I've had problems on my laptop. Recently I am using Ubuntu without any problems. It is much easier to install Linux when compared to previous years. There are many Live CD distributions. Just pick one and go with installation.
A Vista installation still stays on my laptop, but I am not using it anymore, I keep it in case of Microsoft emergencies.

I'll blog more on this later.

Mert 3.0

I have upgraded myself. I am jumping to the open-source bandwagon, "finally".

In couple of months (Since last May), I have learned great deal of information about open-source world. Every year, couple of times, I was checking what was going on. I found out that, now it was mature enough and evolving much faster than any other trend.

It all started when I bought my iPhone. I should thank Apple for this product. I found out that it was running on Unix and I could install any program on it (by hacking the phone, but it works, I don't mean any harm). So I first met SSH, for uploading files. Then I've seen Lighttpd and Apache. After that I needed some interpreted language, which I've found Python (not PHP). With these technologies, I needed some server platform, Microsoft was doing fine, but it wasn't meant for these applications, so I've installed Fedora 9 on my laptop.

After a while I didn't need any Microsoft application. Everything was running fine on Linux. So I am running Fedora 9 as primary hardware now.

Mert 1.0 -> Commodore 64
Mert 1.5 -> PC + dBase
Mert 1.6 -> Microsoft + Pascal, C etc
Mert 2.0 -> Microsoft Windows
Mert 2.4 -> Windows + Web
Mert 2.6 -> .Net
Mert 2.7 -> .Net + Agile
Mert 3.0 -> Open Source (*-nix)
Mert 3.x -> Not decided ("Web" for sure, but Java or Python, may be PHP, who knows)

Enterprise Library 4 finally released

The long waited "Enterprise Library 4" is finally released, you can download it from http://www.codeplex.com/entlib .
The new library introduces "Unity" as a new application block. The library itself also uses "Unity" for dependency injection.

Other than that, there might be minor changes. The library is now ported to Visual Studio 2008. It also now can run in Partial trust mode. I noticed that it is a 30MB download with More than 20MB documentation. The signed binaries are like ~7MBs.

I am using this library for my "MS.Serices" application. Now I am going to port it to the new library and do a lot of testing...

Adobe Flex 3 and AIR - Silverlight???

Finally I have a chance to write about my dream product.
"Adobe Flex Builder 3"
Recently it has been released. With AIR (Adobe Integrated Runtime) support, now Flash is not dependent on browsers and is supported on Apple and Linus as well as Windows.

Flash 9 is compiled, now with Flex 3, Flex Runtime can also be cached on Flash Cache, which means smaller and faster loading SWF files.

Now, I can bind my server-side data on the client in an Object Oriented environment.
I can also have source and version control for my Flex Applications.

In my opinion comparing Silverlight against Flex 3 is not yet possible. While I am a Microsoft oriented developer, I believe Silverlight need more time to catch up with Flex, but I dont think Adobe will wait for Microsoft ;)

I want to use Microsoft on the server and Adobe Flex in the client.
I wonder if there are any people like me?

MS.Services 1.1.0 published

I have published the latest release of MS.Services. you can find it here (http://www.codeplex.com/msservices).

The latest release includes more database support (actually it is inherited from the stucture of Enterprise Library) MySql, PostgreSQL and infamous SQLite.

A new executor is introduced; the "SQL Executor" which executes SQL statements, also handles injection attacks. "SQL Validator" and "SQL Shield" are introduced respectively. This support is for directly executing SQL Statements, from a datasource.

Flex and AIR support (ActionScript) is available with MS.zip and .asdl file.

Also "MapFieldsToRecords" property is added to JSON generator. If this value is true, the "RecordSet" Property of JSON output is array of hashtable(Javascript Object) instead of two dimensional array.

I am planning to add generic generator, executor, validator and shield structure which can be added without modification of thoe source, in the next release.

"Think"

The most precious, beautiful and powerful ability in the universe is "the ability to THINK".

Because I can think whatever I want and they only belong to me. I can share any of my thoughts the way I want to share. In my thoughts, I can be in heaven or in hell. Some people think the way I do, the others don't.

Someone invented internet with his/her thoughts.

In conclusion, I am what I think.

Turkish toUpperCase function for Javascript

Javascript toUpperCase function for Turkish;

String.prototype.toTrkUppercase = function() {
    var str = [];
    for(var i = 0; i < this.length; i++) {
        var ch = this.charCodeAt(i);
        var c = this.charAt(i);
        if(ch == 105) str.push('İ');
        else if(ch == 305) str.push('I');
        else if(ch == 287) str.push('Ğ');
        else if(ch == 252) str.push('Ü');
        else if(ch == 351) str.push('Ş');
        else if(ch == 246) str.push('Ö');
        else if(ch == 231) str.push('Ç');
        else if(ch >= 97 && ch <= 122)
            str.push(c.toUpperCase());
        else
            str.push(c);
    }
    return str.join('');
}

////Sample;
//alert("ğüşiöçıĞÜŞİÖÇI".toTrkUpperCase())
////Output: ĞÜŞİÖÇIĞÜŞİÖÇI

String.format function in Javascript

Javascript String Formatting function (just like in C#);

String.format = function() {
    if(arguments.length == 0)
        return null;
        var str = arguments[0];
        for(var i=1;i<arguments.length;i++) {
            var re = new RegExp('\\{' + (i-1) + '\\}','gm');
            str = str.replace(re, arguments[i]);
        }
        return str;
}
////Sample;
//var a = String.format("{0} {1}", "Hello", "World");
////Output: Hello World

SQLite ve Tarayıcılar

PHP ile birlilte gelen SQLite diye bir ürünü keşfettim. Kendisi, 3. versiyonda olup, 200KB'lık C ile yazılmış bir Open-Source projedir. İşin güzel tarafı, pek çok platform için destek vermekte ve her türlü projeye kolaylıkla gömülebilmektedir.

In-Memory veritabanı arıyorsanız kullanabilirsiniz. Dosya tabanlı olduğu için çok rahat yönetilebiliyor.

Esas ilginç tarafı ise, kullanıldığı yerler giderek artıyor ve ilerde yeni nesil web uygulamalarında kullanımına alışmamız gerekecekmiş gibi geliyor. "Vay be, adamlar nasıl bu kadar hızlı yapmış, nasıl oluyor da oluyor?" gibi soruların cevapları, bu uygulamanın gömülü olduğu ortamların kullanılmasında yatıyor olacak.

En son Firefox 3'ün SQLite gömülü olarak yayınlanacağını duydum. IE'ye acaip fark atar. Ayrıca Google'ın hala beta'da olan Gears ürünü de çok enteresan. İlginç problemlerin çözümü için ideal. İndirip incelemenizi hararetle salık veririm.

Adobe'nin AIR (eski adı apollo) adlı ürünü de, SQLite desteği verecek. Bir de Flex 3'e de koyarlarsa mükemmel olur.

Bence Web, enteresan gelişmelere gebe. AJAX dönemi bitti! RIA (Rich Internet Application) dönemi başlıyor.

Web 4.0

Arkadaşlar,

Artık Web 2.0 ve 3.0 kapıldığına göre, Web 4.0'ı da burada tartışmaya açmak istiyorum.

Mesela Web 4.0, evimde buzdolabımda ne olduğuna bakıp, benzin istasyonundan arabama benzin koyması için, http://www.benzin.com.tr adresine istekte bulunup, aracıma benzin koymaya gelen benzinciye de, yorulduğunu düşünerek bir bardak soğuk su ikram edebilmeli diye düşünüyorum.
Bir de Web 4.0'da mutlaka Türkçe karakter sorunu çözülmeli.

Hmmm Web 5.0 da, tatil yapmak istediğimi sezip, bana Bahamalar'da yer ayırtabilir sonra paramın yetişmemesi durumunda (Bunu banka transactionlarını hissederek algılıyor), beni otobüsle Kapadokya'ya gönderebilir. Kızgınlığım üzerine de bedava bir balon turu ikram eder.

Web 6.0'da da bütün gün evde yatar uyurum artık!

Herkese mutlu Webler.

(Not: bu yazıyı aynı zamanda cember.net'te de yazdım)

Openlaszlo

Geçen haftalarda "OpenLaszlo" diye bir ürün buldum ve yepyeni bir dünya ile karşılaştım.
Bir Microsoftçu olarak, Java ile çok enteresan şeyler yapılabildiğini keşfettim. Eğer internet üzerinde uygulama yazmayla ilgileniyorsanız, mutlaka bu ürünü incelemenizi tavsiye ederim.
Ürünü incelerken "RIA" gibi bir kısaltmayla karşılaştım. Rich Internet Application anlamına geliyor ve anlayınca pek çok soru işaretinin ortadan kalkmasına, internetin geleceği hakkında farklı düşüncelere kapılmanıza sebep oluyor.

OpenLaszlo'yu mutlaka inceleyin!!!

Elveda GotDotNet

Bir şekilde bir zamanlar, Microsoft'un etkili bir şekilde tanıtımını yaptığı GotDotNet sitesi kapanıyor. Sanırım aynı işlevi daha etkili bir şekilde gören pek çok site var ve bu siteler yeni teknolojileri kullanıyorlar. (Wiki, Blog, BugTracking, Source Hosting, RSS desteği vs.)

Benim MS.Services projesi de buradaydı. Mart sonunda yayından kalkacağı için CodePlex'e taşıdım. Enterprise Library nereye giderse ben de oraya gidiyorum.

Bu arada Nisan ayında Enterprise Library 3.0 da yayına girmek üzere.