Čtvrtek srpen 07, 2008

Battle with 64bit beast

As a part of my work on in-place upgrade project I need cleanup code and prepare unified API for page content access. The work went smoothly and I thought that I'm ready with first prototype. Everything worked fine and regression test passed without any problems. However, I has expected that new interface should have a performance penalty and I wanted to know how big the impact is. I asked my colleagues from performance team if they can help me. Paul van den Bogaard offered me that he helps me but he needs 64bit SPARC binaries. I answered "It's no problem". I used same flags for compilation as we use for Solaris PostgreSQL official packages. And of course as a true hacker a tried gmake check at the end. Upsssssss...

...
     bit                  ... ok
     numeric              ... ok
     txid                 ... ok
     uuid                 ... FAILED (test process exited with exit code 2)
     enum                 ... FAILED (test process exited with exit code 2)
     money                ... ok
...

======================================================
 27 of 115 tests failed, 1 of these failures ignored. 
======================================================

Hmmm. It does not look really good. Quick look into regression.diffs confirm my doubt that it is in hash index.

  -- btree and hash index creation test
  CREATE INDEX guid1_btree ON guid1 USING BTREE (guid_field);
  CREATE INDEX guid1_hash  ON guid1 USING HASH  (guid_field);
! server closed the connection unexpectedly
!       This probably means the server terminated abnormally
!       before or while processing the request.
! connection to server was lost

Ok. Core dump shows more. Fortunately I set system to store core into special directory and with "core.%f.%p" pattern (see coreadm(1M)). It helps to keep all generated core files separately. And now what pstack(1) shows:

 00000001000858f8 _hash_metapinit (1006c2818, ff00, fc00, 1000038c9bc, 13d, 2) + 260
 000000010008238c hashbuild (ffffffff7fffa1c8, 1006c2708, 1006ba670, 100645840, 1004e2000, 1006c2818) + 84
 00000001002c66d8 OidFunctionCall3 (100082308, 1006c2708, 1006c2818, 100645840, 3, ffffffff7fffa570) + 48
 00000001000d1eb8 index_build (1006c2708, 1006c2818, 100645840, 1006c24c8, 1006c47f0, 0) + 3c
 00000001000d16c0 index_create (0, 1, 4085, 100645840, ffffffff7fffa7b8, 1006c2818) + 78c

Good, I adjusted HashMetaPageData structure and added new HashPageGetMeta macro for get pointer to the correct place and of course the change must be backward compatible.

--- a/src/include/access/hash.h Fri Aug 01 13:16:08 2008 +0000
+++ b/src/include/access/hash.h Sat Aug 02 18:04:16 2008 +0200
@@ -138,7 +138,6 @@
 
 typedef struct HashMetaPageData
 {
-       PageHeaderData hashm_phdr;      /* pad for page header (do not use) */
        uint32          hashm_magic;    /* magic no. for hash tables */
        uint32          hashm_version;  /* version ID */
        double          hashm_ntuples;  /* number of tuples stored in the table */

...

+#define HashPageGetMeta(page) \
+ ((HashMetaPage) (((char *)(page)) + SizeOfPageHeaderData + sizeof(ItemIdData)))

Because I got SIGBUS (Bus Error) there is problem with memory alignment but start address of hashm_magic is on correct place. Strange, time to take debugger:

_hash_metapinit+0x260:          clrx      [%l4 + 0x24]

After investigation (thanks to Vita Batrla for help with SPARC assembler). Register l4 contains pointer to the page. It means that it points to hashm_ntuples which is double and double has to be located on 8 bytes aligned address. But how can it happen when structure starts on same address? The problem is size of PageHeaderData structure. It is 28bytes long and of course it is not divided by 8. It means compiler has to put 4 padding bytes between hashm_version and hashm_ntuples in original structure. But padding is not necessary in the new one and unfortunately new structure is different. I compare it with code generated from head and difference is clear:

_hash_metapinit+0x248:          clrx      [%i3 + 0x28]

During investigation I found another interesting thing. When I compiled PostgreSQL without optimization problem disappeared. Close look into code shows what happened:

_hash_metapinit+0x32c:          std       %f36, [%l0 + 0x8]

The STD instruction works fine with nonalignment floating point.

And what is conclusion? I can break backward compatibility and cleanup hash code or create several version of HashMetaPage for different page layout. Everything depends how new hash index implementation ends up. When it will be integrated into PostgreSQL 8.4 compatibility will be broken anyway and clean up can be done.

Čtvrtek květen 29, 2008

PGCon 2008 - Lets go Elephant

PGCon 2008 is over. It held in Ottawa, Canada last week and it was really busy week. Conference began on Tuesday with two days of tutorials. And two days of talks followed afterwards in three separate tracks. Quality of tutorials and talks were very high. It was very nice to be there again and met with people. Especially most of Sun's Elephants were there and we could meet personally.

The conference is not only about talks. Social events are big advantage of the conference. Meet and discuss with other developers or PostgreSQL users is very useful. You can see photos for example on Magnus Hanager's photo album. It is nice to connect face with email address.

My main task there was present about in-place upgrade project. I discuss this topic with many people during the conference and it is really big issue. One TB database is usual size today and migrate data from old version to new one is DBA's nightmare. Everybody asked me if I have some code for testing. It has been big motivation for me. I'm now focusing on storage upgrade and unfortunately, code which I have now is only experiment. The changes between 8.2 and 8.3 version of postgreSQL are big and I thought that online conversion will not be possible without major rework. Fortunately, I got idea how to process old data structure online during my long flight to Ottawa. I discussed it with Tom Lane during a lunch and on Friday morning I had to adjust my presentation. It seems that my presentation has been accepted. It is mentioned in Bruce's and Josh'es blogs. I think my mission was accomplished successful. I have green light and now, I can start coding.

Thanks all who gave me a feedback and who offered me a help.

Úterý duben 08, 2008

Sun Techdays St. Petersburg

Event Last week me and my colleague Julo attended Sun Techdays in St. Petersburg. We spent three days in a booth to present PostgreSQL and JavaDB. MySQL guys had a booth next to us and we created opposite to big Oracle booth. It was big action and only acceptable place was Ice Hockey Hall. I spent over 10 hours each day to answer questions about PostgreSQL. Some people returned again next day with follow-up questions. And at the end of day I was really looking forward to fall in a bed. Small complication was a language barrier because not all attendees spoke English or their English was not so good, but together with my poor Russian knowledge we were able to talk. City I have never been in St. Petersburg. It was my first visit. I heart lot of recommendation about heritages and city downtown at all. However begin of April is not good time for visiting. There was rest of snow/ice in some places and dust from sand and salt were everywhere (rest of road protection during of winter). What I really didn't expect it was heavy traffic. Six lanes roads are common in US but they are not common in European city center. It was difficult to crossover. It seemed like new form of Russian roulette. We have only one afternoon to walk through the center and it is really not enough. I personally plan to return back in the future and spend at least one week to explore a city and museums. Sun office St. Petersburg trip was also good occasion to visit Sun office and meet with our colleagues from Sun Studio development team. I specially interested in compiler optimization and how to compile PostgreSQL for best performance. C compiler has a lot of optimization switches but not all are safe to use them in all situation. I got a lot of tips and I plan to test them soon. Thanks to Boris and Michael.

Posted by hlipa ( IV 08 2008, 05:31:11 odp. CEST ) Permalink

Pátek únor 15, 2008

Prague PostgreSQL Developers' Day 2008

Prague PostgreSQL Developers' Day 2008 conference is gone. Big thanks to all who helps me and Marek Kocan to prepare and organize it. Big thanks to all speakers and all attendees for friendly atmosphere. And big thanks to Sun microsystems, CZ Nic and Czech Technical University in Prague for their support. Conference was recorded by AVC center (big thanks) and video should be available soon.

About 76 people visited the conference and three of them arrived from Košice, Slovakia. We can say that it was international conference :-).

You can find more infromation on Julo's [1] and Peter's [2] [3] blogs.

Pátek leden 18, 2008

Arise a Elephine

Do you know it? Did you hear or read about it? Probably yes. I had severals calls, I got many emails and uncountable instant messaging messages. Do you still not know what I talk about? Yes, I talk about MySQL acquisition. I had to read Jonathan emails three time to really catch a point. It was something unbelievable and it happened in January not on first April. It had to be true. OK. It was shock for me, because I'm member of PostgreSQL team and I spent last two years to conviction people inside and outside Sun that PostgreSQL is better database then MySQL. What will I do now? What will happen with PostgreSQL support they were my first questions in my head.

I have been reading a lot of articles, blogs and comments about this deal for two last days. And some comments were really funny. I read that we are planing probably to put all three databases together (PostgreSQL, JavaDB and MySQL). Whauu ... It is fantastic idea. How will we call it? Elephine seems to be good name. And how this animal looks like? You can see it on the following picture (thank to Petr Zahradnik).

OK. Does anybody still think that it is good idea? It is nonsense. All these three databases have different architectures and cover different markets with some overlaps. Support all of them makes more sense then putting them together or stop support one of them.

Neděle srpen 05, 2007

Czech Republic Youth Programmers Championship 2007

Czech youth programmers met in Janske Lazne in Krkonose Mountains during the last weekend of June, and they competed who is better programmer. It was 21st year of this championship and second when Sun was one of sponsors (thanks to Pavel Suk). The task was to create easy music score viewer/editor which shows music score loaded from XML file. The task was relatively easy, but four hours limit for solving the problem is not many. The competition has been divided into two categories based on the competitors age. First category is up to fifteen and second category is up to nineteen.
	Sun also prepared some presentation about its own technologies. Martin Ryzl presented Mobile Java solution. He described what ME Java is what is different from standard Java and how to write application for cell phones. He used Netbeans and its mobility pack to demonstrate examples.
Jakub Jermar presented basic concepts of operating systems and these concepts described on Solaris implementation. I was surprised how many people were interested in this theme. Jakub had to answer a lot of questions and some youth were interested in short mdb presentation.
	Each year is difficult to determine first three/four positions in each category. Jury must examine each detail and they must make a decision which solution is better. It is not an easy and often causes hot discussion. Fortunately, jury always makes a decision and the championship is successfully finished. I'm looking forward to the next year and I hope Sun will be there again.

Posted by hlipa ( VIII 05 2007, 08:40:24 odp. CEST ) Permalink

Pondělí květen 28, 2007

PGCon 2007

PGCon 2007 went ahead last week in Ottawa. It started on Monday May 21st with two days of tutorials. I attended only Niel Conway's tutorial Introduction to Hacking PostgreSQL. Niel presented how to write PostgreSQL code, what kind of tools is useful for debugging and he also described how to use memory management and handle errors. He mentioned DTrace as a good tool for debugging, but he does not have experience with it. I had a few OpenSolaris Developer Express Starter kits and I gave him one copy.

Official registration started on Tuesday evening in Royal Oak pub. A lot of hackers and users were there when I came and they discussed a lot of topics.

My and Jim's mission there was to discuss PostgreSQL upgrade project. Nobody want to have eight versions of PostgreSQL in Solaris and there is not easy way how to upgrade from one version to another, especially when database is very big. We made decision that only on-fly upgrade, when new PostgreSQL version will understand old database structures, is correct. However, this solution requires a lot of changes in the current code and it is not possible start without wide acceptance. We started discussion with EnterpriseDB folks, because they work on PG Migrator tool. PG Migrator tool is useful for migration between 8.1 to 8.2, because there are mostly only catalog changes (disk layout is not touched exclude two datatypes). However for 8.2 to 8.3 migration PG Migrator is not suitable. It is designed as offline utility and conversion of Tera byte database disk structures take a long time. After hot discussion about technical details the fully integrated on-fly upgrade has been accepted as possible and correct solution.

I also had 5 minutes long lighting talk about it and a lot of people interested in this solution. Only two things are now necessary to do ... design and implementation.

We (Sun's folks) had also two other presentations. Jim Gates, Robert Lor, Josh Berkus and me presented Useful Solaris tools for PostgreSQL DBAs. I think there were over fifty attendants. Unfortunately, our presentation did not fit in one hour and we skipped SMF. People mostly interested in DTrace and ZFS features. Robert spent a lot of time after presentation with people who interested in DTrace and they wanted to learn more.

Julo Stroffek and Tomas Kovarik presented Execution plan optimization techniques. PostgreSQL uses genetic algorithm to determine execution plan for twelve and more tables join optimization. This algorithm is not optimal and Julo and Tomas presented summary of suitable optimization methods. Their presentation was accepted and PostgreSQL hackers now interest in real result. However, some pluggable interface for optimizer is first step in their effort.

There were a lot of excellent presentations and most of them are available on conference web site. If I could recommend one presentation Scaling PostgreSQL on SMP Architectures -- An Update is very interesting.

I think it was excellent conference and I'm looking forward to next year.

Sobota květen 26, 2007

Ottawa Senators ... Lets go

This week I was on PGCon 2007. The conference was located in Ottawa, capital city of Canada. I arrived on Saturday in time when Senators wons its NHL conference. A lot of people celebrated this occasions and lot of cars with the Senators flag run around. During my trip from the airport to University of Ottawa campus I saw of a lot of trees and green areas in the city. The conference started on Monday and I made a decision to visit city down town. University campus is located very close and it takes fifteen, twenty minutes by walk.

Whau... I crossed a bridge over Rideau Canal and first Gothic building appeared. What Gothic building does in Canada? How I read a later it is Victorian Gothic style and it was very popular during 19th century when also Canadian cities growing up and these buildings still stays there ... wonderful, because in many cities in US and Canada old downtown was destroyed and ugly skyscrapers replaced them.

I passed City Hall (It is a modern nice building) and after ten minutes I reached Parliament Hill. It is really beautiful place - Parliament, Peace Tower, Library ... There is also nice view on Ottawa river and Civilization museum building on other side. I strongly recommended to visit this place. Ottawa is nice, quite and clean city. And I hope Ottawa Senators wins Stanley Cup this year.

Středa březen 07, 2007

Elephant in Berlin

The last week (Feb 27 - Mar 2 2007) German Unix User Group organized the first OpenSolaris Developer Conference. I had a pleasure to attend this conference and present there about PostgreSQL and its integration into Solaris. I was also surprised how strong Unix community is in Germany. The first day focused on technology inside SMF, RBAC, DTrace and also on OpenSolaris project itself. The second day was about alternative Solaris distributions, like BeleniX, Nexenta and others.

All presentations were very good and I learned new and interesting information. If I should choose only one presentation, Looking into the black-box - how the kernel may impact your application presented by Thomas Nau would be the winner. Thomas Nau described how to use a lot of system tools include D-Trace for analysis application bottlenecks and he demonstrated these tools on a real problem.

My presentation started at 4:30pm when many people already left the conference, however, ca thirty five folks still stayed to listen to my talk. The talk presented how SMF and RBAC is implemented for PostgreSQL and also showed how DTrace user probes are integrated into PostgreSQL 8.2. Presentation generated a few questions and most of them focused on high availability and load balancing solution for PostgeSQL. Unfortunately, PostgreSQL does not have functionality similar to Oracle cluster. However, SunCluster version 3.2 provides failover data service for PostgreSQL.

Me in speaker role.

Conferences are also good place to meet people and discuss various things. I talked with Sasa Gorshenev about Sun Studio compiler and its optimization. He gave me some ideas how to tune PostgreSQL. Frank Batschulat also joined the conference and we discussed how to setup disk access to get best performance for databases.

And what to write at the end? It was amazing conference and I hope I will attend it the next year again.

Sobota srpen 12, 2006

The Freeze Elephant

The Freeze Elephant is not next volume of the Disk world book series from Terry Pretchet. It is not historical discovery as well. How many of you know Elephant is Postgres logo and August 1st was date for new release feature freeze. All patches received before this date will be processed and integrated after their review to the release 8.2. And what will be new in the 8.2? This question is heavy discussed in the hacker mailing list, because there are many improvements and small enhancements, but there are only few new major features. However, some hackers and users expect more. It triggered discussion about release cycle, because Postgres has the fastest release cycle in the database market and major features request more time for implementation.

I think that sometime is good to take time for improvements and cleanup in the old functionality than create new and new features. Some new features from 8.0 will be not until fully functional in the incoming version. It is probably necessary, because TODO list contains a lot of new major features, but many of them request a lot of time and high experience. The time for full implementation is in many cases longer than release period. Partial implementation makes possible fast integration into main branch. Review of this code is much easier and developer has fast feedback. It avoids a lot of problem with the source code merge as well. I expect a lot of major features in the 9.0. And when we will expect it? Nobody knows it.

And when and what will be in 8.2? Bruce expects that Postgres 8.2 will be released in October or November, but it depends on speed of stabilization process. We can expect following major features:

• GIN (General INverse index)
• bitmap index
• udatable view
• PL/SQL debugger infrastructure

Most of them are not committed yet and some of them should be still rejected.

Somebody may ask what Sun brings to the new release? The Sun announced support of Postgres in the December 2005 and the new support team has established. This team (yes I member of this team) started integration Postgres with solaris and has helped community implement some items from TODO list. The short list of our work you can see bellow:

• D-Trace
• Kerberos support on Nevada
• Improvement in GUC (Genereal User Configuration) module
• The 64 bit integer support in some SQL commands

Someone can say that it is not too much, but I think the D-Trace is Sun's big contribution for Postgres. Community started to use it immediately for searching concurrence locks problem on multiprocessor machine with more than four CPUs and it will be useful in many other cases as well.

I think it is good start and Sun has big power to bring more in the future. Now we do not have to wait until elephant will defrost and we can continue in participation on the Postgres development.

Pátek červenec 21, 2006

PostgreSQL code sprint

The Postgres conference finished on Sunday, but postgres hackers stayed in Toronto until Tuesday at the Code Sprint sponsored by SUN. Over 30 hacker were in one room and spent these extra days on coding the last features to postgres 8.2. Flipcharts were written up very quickly during brainstorming sessions. Tom Lane and Bruce Momjiam did not have time to have a rest. Many problems were solved (at least on the flipchart) and I hope that postgrest 8.2 will be released soon. Feature freeze has been announced to July, 31st.

Code Sprint

Bill Franklin attended this code sprint as well. He spoke with many hackers and was interested in their work. I think that the conference was a good experience for me, my colleagues and for Sun too. See you on the next code sprint ...

Sun postgres hackers

Josh, Robert, Jim, Zdenek, Dhanaraj

Čtvrtek červenec 20, 2006

PostgreSQL Anniversary Summit

PostgreSQL Anniversary Summit performed in Toronto from 8th and 9th July 2006. This summit was the first occasion to meeting postgres hacker face to face in its ten years history like opensource and free project. The conference unofficially started day before on 7th July evening with small refreshment in the hotel lobby. Many people met first time and everybody tried to assigned mailing address to real person. It was a little bit funny. I met with my Indian colleague Dhanaraj in hotel day before, but I have never seen my US colleagues. For example Jim's photo is little bit out of date. Fortunately, Jim identified us. The all group of hacker moved to boat for dinner and for ongoing person identification and unlimited discussion about postgres.

Toronto evening

Main conference started on Saturday morning by Josh Berkus. He invited all conference participant and said thanks to conference sponsors. Sun was one of them. After this he gave floor to Bruce Momjian. Bruce is one of people who ware presented releasing of postgres source code ten years ago. He talked what happened during this time.

Next presentations had been divided to two separate tracks. Many of presentation ware related to present new projects. Korry Douglas showed PL/SQL debugger based on postgres proxy server which is stored close to main server. Communication is based on standard plsql protocol and debugger features are implemented as standard stored functions. Simon Riggs continued with presentation about performance problems. However, some problem generated big discussion and unfortunately, Simon did not have time to finished his all presentation.

Saturday afternoon was dedicated to Neil Conway's and Gavin Sherri's introduction to postgres hacking. It was very useful practical presentation based on real feature implementation.

Contributors dinner was closing Saturday. I met and talked with many people. I spoke with Luke Lonergan, CTO of GreenPlum company, about ZFS and its scalability. He connected from his PDA to his home server and shown me version of running opensolaris. Yes, My OpenSolaris T-Shirt identified me and many people ask me about our postgres support and future strategy. Many people trust us but not all of them. I spoke with Skype developers from Estonia. I asked them if they plan skype version for solaris. Unfortunately, no. They have version for linux, but they afraid that sound subsystem is different on solaris and is not easy to port skype on solaris. I'm not solaris sound guru, but if anybody interesting in it, I have a their email addresses. The local band interrupted all discussions and I went to the hotel.

Second day was opened by Robert Lor and his famous D-Trace presentation. Robert shown only part of D-Trace capabilities, but it was enough to persuade community. Community have accepted D-Trace and it will be implemented in the postgres 8.2. Andrew Dunstan presented buildfarm framework status and its problems and their solution. However, some buildfarm problems are still opened and is need of more investigation. Oleg Bartunov and Teodor Sigaev spoke about new features and improvements in the T-Search 2 project (full text search) and shown a big TODO list as well.

Robert Lor

Sunday lunch I spent in Elephant pub on Yonge Street (longest street in the World). Bruce with his colleague came little bit later into this pub and Bruce invited me to join the lunch with them. Yes, really I had a lunch with Bruce Momjiam. You can see it on picture below. I asked him about future of his TODO list, if there are any plans to replace this TODO list by some bug tracking system. He answered that there were many discussion about it but all bug tracking system is too complicated for postgres purpose and hackers using primary mails for solving problem and there no exist any relevant system integrated it.

Me and Bruce

Peter Eisentraut and Josh Berkus closed the conference and all attendants went to take group photo.

And what is conclusion? Italy won World Championship in Football (for US citizens soccer :-). And conference had big success and big benefit for all attendants. Interest of this conference was three time bigger than available capacity. We can expect that next conference will be early maybe next year. I'm looking forward to it.

Neděle červenec 16, 2006

Czech Republic championship in Programming for Young People

How you can read on Tomas Heran or Roman Strobl blogs, the Czech Republic championship in Programming for Young People occurred on last June weekend (23rd - 25th June) in the Janske Lazne. Forty youth have been divided by age to the two categories and they spent four hours to solve their task. It was 20th volume and first when Sun was co-sponsor of this competition. There is a place to say thanks to Pavel Suk and Lukas Rovensky for their interest in it. The Sun presentation described Tomas and Roman on their blog and I will not repeat it. I want focus on another think. I want focus on the history, present and future of this competition.

How I wrote it was 20th volume. I started in this competition at 1988 like competitors. It was age of 8-bits computers like ZX Spectrum, Atari, Commodore ... All competitors wrote their program in the basic. Final version of program must be rewritten to the paper, because hardware and storage media were unreliable and jury spent lot of time to check all solutions. However, There were many youth and adults which interest in. I remember that one competitor toke iron and started to solder his modified Spectrum during competition.

The turbo pascal and quick basic on MS-DOS replaced basic on 8-bits computer since 1991/92. Only two of competitors used C. However, C is good language, but not for speed programing.

How time went I was too old to stay like competitor and I have started like member of Jury (since 1995). We have started to submit one complex task instead many small tasks. I remember one task was text editor with blocks feature. Four hours were enough for many competitors to solve this task and it was very difficult to choice winner. Pacman was task next year and it was difficult to avoid to play pacman and finish evaluation.

Next change came about 1998 when MS Win push out MS-DOS. RAD tools (Delphi/Visual Basic) started to play major role in programming and task was adjusted for new programming environment. But what happened. Number of competitors in all Czechia go down and their quality too. We started to search where is problem. What is root cause?

I think there are many causes. First is that RAD tools are good for quick user interface design, however, implement algorithm is difficult and people use only prepared component without knowledge of their internals. Programing requests bigger and bigger knowledge and is difficult to start. There is no easy way how to write program "Hello world!". The barrier is multiply by the Windows which covers everything under graphics interface.

The next cause is quality of programming lessons. Many good programmers taught children at the beginning of nineties. They taught very often in the centers of children free time. However, they have lost time for teaching and new teachers don't have experience. Pavel asked me, if I know about some place in Beroun where his child could start to learn programming. I said him "Beroun had good center for children with very good teachers." I tried to find some contact person, but I must say again "Beroun had ...". Nobody teach children to programming in Beroun now. It is common scenario in all Czechia. Only few centers teach children to programming and many children prefer game playing.

The Sun cooperate with many Universities and it shares our technology knowledge with students. It is very good. But Universities expect basic programming knowledge from their new students. It should be problem in the future. Many excellent musician start play when their was young children. It is similarly with programming skill. The question is where young children can start learn programming and who will be their teachers?

I afraid that if we can solve it quickly we will have problem find excellent programmers in the near future.