TTIWBot is lightning fast now! I knew it was not the code, and it couldn't be the server.
I'll do some little tinkering with it, but I'm taking the weekend off.

It's the dns resolutions. It's taking an average of 5 seconds for the server to solve Amazon when doing an HTTP request (even via Lynx). That's what is making the bot so slow. We are working on it. But, we'll have to wait for the response of the people at our server provider tomorrow.
But, yep, doing a request directly from the ip takes an average of 80% less than when going with the domain name. So that's it. :-)

Emiliano

I added a small new feature today, you can now order your wishes by Priority, Date Added or Site you add it from (Appart from alphabetically and price). It was a small thing but I had promissed users this and somebody asked for this today and well, it was a half an hour thing so there it is. It's only for managing. I'll add sorting (not ordering) for gifting during the next days, which I won´t be working that much at this, there is some real world stuff I need to do and I want to do some stuff with hCard. But I will be adding little small thingys like this one, I'll keep you informed through the blog.

I did a full benchmarking of the different processes of the bot and I have finally found what is slowing it down. First, I thought the problem was with the new contextual stuff, since I added this at last minute. I had some dead time while I waited for Ernesto to finnish something and I realized that I could do this from something that came up on the discussion at the microformats list. I figured that with the data the bot was already picking I could do some of the parsing based on contextual variables, and I had some dead time so I added that to the bot. This was kind of rushed to be honest. But I benchmarked this and it slows the bot down for about 10% to 30%. So I started digging further and I realized that querying amazon from my machine generally took 1 second while It took 6 from the site, this means that the getting of the page or XML document from the site is trully slow, I'm guessing that we are probably behind a proxy or some cache mechanism ( This would also explain why if you add an item then add it again, the second time takes 1 tenth of the time of the first one. The parsing is exactly the same and it's not cached, so the problem is when picking the page in itself.). So I will be contacting the people at our server provider regarding this. Hopefully they will give us some sort of solution since we are consuming data from all sort of sources and we plan of doing web services in the future.

Well that's pretty much it!

I added Travis amazing plug-in to the add to blog page.
Well we have finally fixed emails. The MX is still updating though. I'm hoping it gets updated quickly. I'm generally reachable through emilanomartinezluque-NoSpam-yahoo.com, which is the email I used when travelling. I really need a holliday. Well anyhow if you want to reach me, for some reason or the other, you can always use that email.

That's pretty much all for today.

There are still some validating quircks but the application in itself is all working. I've fixed all the bugs I was missing tonight. I'm off to sleep now.

Emiliano
Travis: I'll add your Plug in tomorrow I promise!

Fixed the problem with deleting tags, the problem with & and < when adding items and changed some stuff on the Bot to make it faster. I had to change the function for decoding html entities this fixed the extrange characters Issue without making the bot slower. I also reindexed some Amazon Items yesterday night. And fixed some visual issues.
But from this changes I've moved this to Beta 2.
I will keep working through the night, hopefully it will be done tomorrow.

I'm really hoping Ernesto gets emails done by tomorrow.

I have added the deffiles to the bot, fixed what broke with catalog and Suggested Sites and the amazon thingy has been fixed. I will clean up some visual quircks tomorrow, add Travis awesome plugin and move the thing to final version.
Hope you enjoy the new version!! We worked a lot on this one.

You should add the following link to your item's pages:

javascript: location.href='http://www.thethingsiwant.com/wishing/addtolist_s.php4?type=fromsite
&amp;URL='+encodeURIComponent(location.href)+'
&amp; Desc='+encodeURIComponent(this.document.title);

Then you send us an email, we customize TTIWBot, and when somebody clicks on the link, TTIWBot does all the job of finding the item info.

I added an example here to show how it works (please, pay no mind to the page design, it's just an example.): http://www.metonymie.com/testeo.html
(It should work now.)

The bot should be fast now. I also added a bunch of new sites to it.

When I finnish with the sites (in a couple of hours), I'll fix the suggested sites section and then I think we are ready to move to beta 2.
(I have to fix the Amazon thingy to move to Final yet).

If you visit http://www.thethingsiwant.com/tag/babys%20room/ it provides the same results as if you vist http://www.thethingsiwant.com/tag/babys_room/ or http://www.thethingsiwant.com/tag/babysroom/ or http://www.thethingsiwant.com/tag/Babys Room/.

If you tag something as "baby room" or as "babyroom" it shows what you typed but it gets normalized in the database. It also stores "baby" and "room" with a different level (I'm not showing that yet) in the db.

Test the following sites:

http://www.ginza.se/?artnr=8170&referer=kelkoo
(Finds out the site has Frames and provides a warning)

http://www.megastore.se/template/next%2CProduct.vm?itemid=768498
(It contextually finds Swedish kroner)

http://www.isavezone.com/goods/product/shopping_productinfo.jsp?event_no=4529&TG_GOOD_NO=90186449
(It contextually finds Korean Wons)

http://www.p1000.co.il/p1000/hot_sale_product.asp?g_id=5877011&came=3820
(Is contextually finds Izraeli sheckels)

http://www.bedbathandbeyond.com/Product.asp?order_num=-1&SKU=13546827
(For sites with definition files, it takes the name from inside the page)

There's something on the Apache and Php conf that I have to fix, cause though it's still faster than before, It's still not as fast as it should be.

I fixed the bot now.. it should be faster. We had a problem while migrating the versions. We had 2 simultaneous versions cause we had 2 simultaneous dev environments.
I'll add the rest of the deffiles later.
Ernesto (the sysadmin) had trouble setting up emails, so we couldn't move the MX yet.

I still have to upload the site definitions for the bot.. will do that in some time but I'll get something to eat first. I'm really tired today. This last weeks have been the worst ever (everything that could go wrong with the new server just went and did!). I literally broke appart a couple of times. Hope you like the new version.. I may write a post in about our server troubles on couple of days.

Well, This is the new URL syntax:

User home: http://www.thethingsiwant.com/emiliano/
User List: http://www.thethingsiwant.com/emiliano/list/my wishlist/
User Tag: http://www.thethingsiwant.com/emiliano/tag/guitar/
User Profile: http://www.thethingsiwant.com/emiliano/profile/
User Address: http://www.thethingsiwant.com/emiliano/address/

Everybody's Tag: http://www.thethingsiwant.com/tag/led zeppelin/

I think that's it. I'll add this info to the help pages latter on.

Go to http://www.thethingsiwant.com/commercial/addservice.php4to learn how to add the service to your site.

And I've also noticed that the catalog and Suggested sites session broke down. I'll fix that today (actually I guess it's gonna be tomorow, cause I need to go to sleep!).

Emiliano

Well, we´ll be migrating in an hour or so.
I have not decided yet on a couple of issues on the Url syntax. All the deambiguation logic between managing and gifting is allready done, but there are some issues regarding the URLs per se that I have to test on the new server, because there are some tests that I couldn´t do on our current development environment. I will post what we decide tomorrow, when I take the site up.
I'm expecting the new version to have some bugs, like always, and since I need to take a break, we will only work on fixing the bugs that may come up during the next 3 or 4 days. I also want to do some stuff with hCard so the next week will be a little quiet here, I really need a break and also to clear my head a little and think of some other stuff for a couple of days. Hope you like the new version, we worked a lot on this one, and we went through some outside (real world) stuff while doing it. We will be working on the GUI during the next months, we´ll try to go for Ajax (just where I think it will be usefull), Semantic markup (the w3c validation broke on v3.2 , that was pretty bad), unified searches, more Social Features and clearer texts. We will probably go one step at a time and with more periodic releases. I will also try to start releasing some helper classes or functions and writing about how we solve some issues. I didn't did that before because the last couple of years have been quite tough for me (except 2003 wich was awesome!), and I just didn't felt confident enough to do stuff like that.
I wanted to get this done now, cause it really was the foundations of the APP, so that we can get the GUI ready for shopping season. Next updates will be quicker, since they are more specific and less application wide. Well, hope you like the new version!

The Service will be unavailable bettween Tuesday the 20th 11 PM (EST) and Wednesday the 21st 11PM (EST) while we migrate to a new server, the DNS gets updated and we upgrade to version 3.2.8 of the Service. We are sorry for any inconvenience this may cause.

We will be sending a service Email and we will add a Notice to the Homepage in the next few hours.

What´s new on the next version.

• The whole Application has been moved to UTF-8.
• TTIWBot Version 0.3 which:
  • Is Faster, Leaner, Full Object Oriented Code (Version 0.2 was just a proof of concept version).
  • Has all of it´s old Bugs fixed including the Base problem and the strange characters one (now we´ll have to fix the new ones!).
  • Has become Context aware for certain things (I won´t explain this cause it´s quite complex, just test it on some international sites and you´ll see).
  • Has better handling of Redirects and Error pages (The HTTP handling has been highly improved).
  • Translates all data to UTF-8 (The data that that we use. Some charsets are not supported though, but all the common ones for all languages are (except ISCII which I couldn´t find a translation table and it´s on our todo list.)).
  • Find if a Site uses Frames on it´s own (Old versions found this from the Bookmarlet, which was not reliable at all).
  • Is more customizable for Picking up info from sites (You´ll see), and it´s easier to write the deffiles for new sites.
  • Accesses Amazon through Rest instead of SOAP which is way faster.
• A better Tag sintax. You´ll have more freedom when writing Tags (you´ll be able to use "'" and "-", and UTF-8 characters for example) and yet it will get normalized when it´s stored in the database (We went for a Signifiant, Signifié approach (with my appologies to Saussure)). And also different levels of tags for tags with multiple names (the phoneme approach.. hmmm, actually morpheme approach (more, and more appologies to Saussure)) though this last bit will not be noticeable on this version, which is about normalizing the DB. (We continue with the Official Motto of versions 3.2: "Things will become clear as we move along".)
• A simple yet easy way to deactivate from the service (I will add this tonight to current version, so that if you receive the email and you haven´t used the service in some time and want to deactivate, you´ll be able to do so after you recieve the email (Which will only be sent to Users that have choosed to accept Service Emails, of course.)).
• A better URL Scheme. I´ll write a full explanation of this tomorrow. ( We have made sure that your old links will still work, even though we´ll prefer if you use the new ones from now on.)
• A simple way for Small Sites to Add the Service to their Sites.
• Some Bug fixes through data normalization (The Order by Name issue). And a lot more Data Normalization that I won´t explain here because it´s trully geeky.

Well that´s pretty much it, hope you like the new version!!! And now we will concentrate on the GUI as promised!!! Sorry for all the delays!!

Well, most users don´t look at this, so it really doesn´t matter. But anyhow we were delayed again.. right on! :-).

We basically had a bunch of problems while testing the migration path and I also found a little quirck on the new bot, and being as obsesive as I am, I had to go there and rewrite the full component.

Man, this whole thing was wierd.. I mean, I was really used to the way we were working before, but getting a new server and adding Ernesto to the team (we are actually a dev team now!!!.. not just me and my imaginary friends..:-).. ) was more complex than I thought. We had to setup a new development environment and get the new server ready ( We tried installing Debian on it, but decided that it was too complex without physical access to it, so we are keeping it´s preinstalled Fedora ). And coordinating work was also complex, I was just too used to doing the whole thing myself (yep, believe it or not, we managed to get to version 3.2 with a devteam of just one guy (TTIW Core is me on Code + Guy on Financials, Management, etc.), though version 1 had a designer, Sebastian who did the logo and the original design.. but version 2 to 3.2 was just me!! I even did the stylesheets and the little icons on the menu!! and replied to your mails!!! Though Ernesto helped from time to time, ( and Guy sometimes answered emails too ), but now Ernesto is on a more regular basis.. we´ll probably get some designer help for the next steps when we will focus heavily on the GUI and on more social features (you´ll like it, it´s a really nice concept). Anyhow, I do believe that by today´s standards we are sort of a beta actually. I just started versioning the thing cause I wanted to send the message that we were doing Software here. ( That´s also the reason we have "the mythical man month" on the left. If you haven´t, go read it it´s a classic. ))

Anyhow, I´m trying to get all ready to migrate tomorrow, but it seems more realistic to settle for saturday or sunday (as always the final date will be the one on the service email).

I got this question about wether we were gonna go for the space separated tags like del.icio.us .. and no, we are keeping the commas. I mean our audience is less techy than the people that use delicious (which I love dearly BTW, I think their whole concept is Brilliant) and commas are more intuitive... and you know, there are also certain ideas which go better with 2 words. It´s just that "art nouveau" c´est pas la meme chose qui "nouveau art", n'est pas? . But we are doing something that will make it so that if somebody tags something as "nouveau art" you can still get it in the results if you search for "art". And you know that is one of the "3 innovations of TTIW", which I will proudfully enumerate (just because I feel like it :-).. ):

1. First site to send a bot on demand to fetch info from a page. (We did that when we added the amazon thingy back in version 2 (And I coded that from an internet cafe in beautifull Prague!! In case you´ve been to that amazing city it was on Easy internet cafe, not the one in Vinohradska but the one in the basement near Winceslas square. "TTIW, the webap coded from five mayor cities in the world".), funny thing is we were working on a better bot when other sites came up with the same concept. I haven´t seen that done before on any site but I think it will become commonplace now with the emergence of microformats. Though I made a complete mess of myself in their mailing list (I have the social skills of Attila the hun).. I´ll try to compensate it by releasing some code I´ve been thinking of latelly (I actually didn´t have the time to work on anything else lately, I haven´t shaved in 3 weeks!!!))
2. Separating access by Private, Guest and Public. And having the possibility of doing a link that gives special access. Again I haven´t seen that before, and now Yahoo´s My web has something similar, that´s pretty wierd.
3. And last but not least, using commas to separate Tags.

And that´s the end of the "3 innovations of TTIW".
The rest is all copied.. :-)

Well that´s it for today, I have to go test some stuff now.

But before I leave, I have this really great news: Travis of gunnerweb.net did an awesome wordpress plug-in for displaying your list as a full page on your blog. It´s really, really nice and it´s way more elegant than our current version. And though it´s still an alpha, we will be adding it to the next version, cause it´s really nice. And I just found out while I was looking at his Blog, that he and Rachel did this amazing service called Swap-Bot where you sign up to swap stuff with people (like postcards or homemade tapes of your music or your art.). It´s a new, new service that has just started, but it´s a great, great idea, so go check it out!!!. I´m working on some experimental electronic music, when It´s ready I´ll start a swap there.

Emiliano

We have finnished with the development side of the new version. We will be testing the new version and a server migration path during the next week. We'll probably move to the new server next friday or saturday. Service will be unavailable during that day while DNS servers are updated to our new address. The new version is really solid and addresses all of the core issues. A better Tag sintax will be deployed too (It won´t affect your current tags), a simple (yet secure) way to deactivate from the service will be available and a simple way for small ecommerce sites to add the service to their sites will be (finally) available. We will send a service email next week to alert of the exact date the service will be unavailable.