GUADEC dtrace, normalized time spent in Cairo

Friday Jun 30, 2006

Those attending the GUADEC performance bof may be interested in this little dtrace script based on normtime.d:

#!/usr/sbin/dtrace -s
#pragma D option quiet

BEGIN
{
        /*
         * Get the start time, in nanoseconds.
         */
        start = timestamp;
}

pid$target::*cairo*:entry
{
        @func[probefunc] = count();
}

END
{
        /*
         * Normalize the aggregation based on the number of seconds we have
         * been running.  (There are 1,000,000,000 nanoseconds in one second.)
         */
        normalize(@func, (timestamp - start) / 1000000000);
}


run it with cairo.d -c gedit and play with gedit:
  _cairo_ft_unscaled_font_keys_equal                                0
  cairo_font_iface_init                                             0
  cairo_font_map_iface_init                                         0
  cairo_font_options_set_hint_metrics                               0
  gdk_cairo_rectangle                                               0
  pango_cairo_fc_font_class_intern_init                             0
  pango_cairo_fc_font_map_class_intern_init                         0
  pango_cairo_fc_font_map_init                                      0
  pango_cairo_font_map_new                                          0
  pango_cairo_renderer_class_intern_init                            0
  pango_cairo_renderer_get_type                                     0
  pango_cairo_renderer_init                                         0
  _cairo_cache_init                                                 0
  _cairo_font_face_init                                             0
  _cairo_ft_unscaled_font_init                                      0
  _cairo_hash_table_create                                          0
  _cairo_unscaled_font_init                                         0
  cairo_font_options_set_antialias                                  0
  cairo_font_options_set_hint_style                                 0
  cairo_font_options_set_subpixel_order                             0
  cairo_line_to                                                     0
  _cairo_ft_font_face_create                                        0
  _cairo_ft_font_face_scaled_font_create                            0
  _cairo_ft_scaled_font_create                                      0
  _cairo_ft_scaled_font_font_extents                                0
  _cairo_ft_unscaled_font_create_for_pattern                        0
  _cairo_scaled_font_font_extents                                   0
  _cairo_scaled_font_init                                           0
  _cairo_unscaled_font_destroy                                      0
  cairo_ft_font_face_create_for_pattern                             0
  cairo_scaled_font_extents                                         0
  _cairo_matrix_get_affine                                          0
  cairo_ft_font_options_substitute                                  0
  pango_cairo_fc_font_map_context_substitute                        0
  _cairo_ft_unscaled_font_init_key                                  0
  _cairo_ft_unscaled_font_map_lock                                  0
  _cairo_ft_unscaled_font_map_unlock                                0
  _cairo_hash_string                                                0
  _cairo_hash_table_insert                                          0
  _cairo_hash_table_resize                                          0
  _cairo_matrix_compute_determinant                                 0
  _cairo_matrix_compute_scale_factors                               0
  _cairo_surface_composite_trapezoids                               1
  _cairo_traps_extents                                              1
  _cairo_xlib_surface_composite_trapezoids                          1
  _cairo_array_append                                               2
  _cairo_array_grow_by                                              2
  _cairo_clip_intersect_to_region                                   2
  _cairo_surface_fill_region                                        2
  _cairo_user_data_array_set_data                                   2
  cairo_surface_set_device_offset                                   2
  cairo_surface_set_user_data                                       2
  cairo_xlib_surface_create                                         2
  gdk_pixmap_ref_cairo_surface                                      2
  gdk_x11_cairo_surface_destroy                                     2
  gdk_x11_ref_cairo_surface                                         2
  pango_cairo_font_map_create_context                               2
  pango_cairo_font_map_get_default                                  2
  pango_cairo_context_set_resolution                                2
  cairo_font_options_create                                         2
  pango_cairo_context_set_font_options                              2
  _pango_cairo_fc_font_new                                          3
  pango_cairo_fc_font_get_type                                      3
  pango_cairo_fc_font_init                                          3
  pango_cairo_fc_font_map_create_font                               3
  pango_cairo_fc_font_map_get_type                                  3
  cairo_matrix_scale                                                3
  pango_cairo_fc_font_map_context_key_copy                          3
  _cairo_gstate_clip_and_composite_trapezoids                       3
  _cairo_gstate_fill                                                3
  _cairo_surface_clip_and_composite_trapezoids                      3
  _cairo_surface_fill_path                                          3
  cairo_fill                                                        3
  cairo_fill_preserve                                               3
  _cairo_image_surface_assume_ownership_of_data                     3
  _cairo_ft_scaled_font_glyph_extents                               3
  _cairo_ft_unscaled_font_create_glyph                              3
  _cairo_image_surface_create_for_pixman_image                      3
  _cairo_pixman_format_create                                       3
  _cairo_pixman_format_create_masks                                 3
  _cairo_pixman_format_destroy                                      3
  _cairo_pixman_format_init                                         3
  _cairo_pixman_image_create_for_data                               3
  _cairo_pixman_image_get_data                                      3
  _cairo_pixman_image_get_depth                                     3
  _cairo_pixman_image_get_height                                    3
  _cairo_pixman_image_get_stride                                    3
  _cairo_pixman_image_get_width                                     3
  _cairo_scaled_font_glyph_extents                                  3
  cairo_image_surface_create_for_data                               3
  cairo_scaled_font_glyph_extents                                   3
  _cairo_clip_reset                                                 4
  _cairo_gstate_reset_clip                                          4
  _cairo_gstate_translate                                           4
  cairo_identity_matrix                                             4
  cairo_reset_clip                                                  4
  cairo_translate                                                   4
  gdk_cairo_create                                                  4
  gdk_window_ref_cairo_surface                                      4
  cairo_font_options_destroy                                        5
  _cairo_unscaled_font_reference                                    6
  _cairo_gstate_show_glyphs_draw_func                               6
  _cairo_scaled_font_get_glyph_cache_key                            6
  _cairo_scaled_font_show_glyphs                                    6
  _cairo_surface_show_glyphs                                        6
  _cairo_xlib_surface_show_glyphs                                   6
  _cairo_xlib_surface_show_glyphs8                                  6
  _cairo_ft_scaled_font_glyph_bbox                                  6
  _cairo_gstate_backend_to_user                                     6
  _cairo_gstate_ensure_font_face                                    6
  _cairo_gstate_ensure_scaled_font                                  6
  _cairo_gstate_set_font_face                                       6
  _cairo_gstate_set_font_matrix                                     6
  _cairo_gstate_set_font_options                                    6
  _cairo_gstate_show_glyphs                                         6
  _cairo_path_fixed_get_current_point                               6
  _cairo_scaled_font_glyph_bbox                                     6
  _cairo_xlib_surface_get_font_options                              6
  _pango_cairo_font_install                                         6
  _pango_cairo_font_map_get_renderer                                6
  cairo_get_current_point                                           6
  cairo_scaled_font_destroy                                         6
  cairo_scaled_font_reference                                       6
  cairo_set_font_face                                               6
  cairo_set_font_matrix                                             6
  cairo_set_font_options                                            6
  cairo_show_glyphs                                                 6
  cairo_surface_get_font_options                                    6
  pango_cairo_fc_font_install                                       6
  pango_cairo_fc_font_map_get_renderer                              6
  pango_cairo_renderer_draw_glyphs                                  6
  pango_cairo_show_glyph_string                                     6
  _cairo_scaled_font_keys_equal                                     6
  cairo_scaled_font_create                                          6
  pango_cairo_fc_font_get_font_face                                 6
  cairo_font_face_reference                                         6
  _cairo_hash_table_lookup                                          6
  _cairo_scaled_font_init_key                                       6
  _cairo_hash_table_lookup_internal                                 6
  gdk_cairo_region                                                  6
  _cairo_clip_clip                                                  6
  _cairo_clip_init                                                  6
  _cairo_color_compute_shorts                                       6
  _cairo_color_init_rgb                                             6
  _cairo_color_init_rgba                                            6
  _cairo_gstate_clip                                                6
  _cairo_gstate_create                                              6
  _cairo_gstate_set_source                                          6
  _cairo_path_fixed_init                                            6
  _cairo_surface_allocate_clip_serial                               6
  _cairo_surface_get_clip_mode                                      6
  _cairo_surface_set_clip_region                                    6
  _cairo_xlib_surface_set_clip_region                               6
  cairo_clip                                                        6
  cairo_clip_preserve                                               6
  cairo_create                                                      6
  cairo_destroy                                                     6
  cairo_pattern_create_rgb                                          6
  cairo_set_source                                                  6
  cairo_set_source_rgb                                              6
  gdk_cairo_set_source_color                                        6
  pango_cairo_fc_font_get_metrics                                   7
  _cairo_matrix_is_integer_translation                              7
  _cairo_pattern_acquire_surface                                    7
  _cairo_pattern_acquire_surface_for_solid                          7
  _cairo_pattern_release_surface                                    7
  _cairo_surface_create_similar_scratch                             7
  _cairo_surface_create_similar_solid                               7
  _cairo_surface_fill_rectangle                                     7
  _cairo_xlib_surface_create_similar                                7
  _cairo_xlib_surface_set_attributes                                7
  _cairo_xlib_surface_set_repeat                                    7
  cairo_xlib_surface_create_with_xrender_format                     7
  get_cairo_context                                                 7
  _cairo_box_round_to_rectangle                                     8
  _cairo_clip_intersect_to_rectangle                                8
  _cairo_gstate_clip_and_composite                                  8
  _cairo_region_create_from_rectangle                               8
  _cairo_pixman_region_num_rects                                    9
  _cairo_pixman_region_rects                                        9
  _gdk_drawable_ref_cairo_surface                                   9
  cairo_font_options_merge                                          9
  cairo_font_options_copy                                           9
  _cairo_font_options_init_default                                  9
  _cairo_array_fini                                                 9
  _cairo_surface_fill_rectangles                                    9
  _cairo_user_data_array_fini                                       9
  _cairo_xlib_screen_info_get                                       9
  _cairo_xlib_surface_create_internal                               9
  _cairo_xlib_surface_fill_rectangles                               9
  _cairo_xlib_surface_finish                                        9
  cairo_surface_finish                                              9
  _cairo_gstate_copy_transformed_pattern                           10
  _cairo_gstate_copy_transformed_source                            10
  _cairo_pattern_init_copy                                         10
  _cairo_pattern_transform                                         10
  _cairo_pixman_region_extents                                     10
  _cairo_pixman_region_intersect                                   10
  _cairo_region_extents_rectangle                                  10
  _cairo_surface_get_current_clip_serial                           10
  _cairo_surface_set_clip                                          10
  cairo_matrix_translate                                           10
  cairo_matrix_init_scale                                          10
  _cairo_path_fixed_fill_to_traps                                  10
  _cairo_path_fixed_interpret                                      10
  _cairo_polygon_fini                                              10
  _cairo_polygon_init                                              10
  _cairo_traps_extract_region                                      10
  _cairo_traps_fini                                                10
  _cairo_traps_grow_by                                             10
  _cairo_traps_init                                                10
  _cairo_traps_tessellate_polygon                                  10
  _cairo_gstate_apply_device_inverse_transform                     11
  _cairo_gstate_apply_device_transform                             11
  _cairo_gstate_identity_matrix                                    11
  cairo_rectangle                                                  11
  pango_cairo_context_get_resolution                               11
  pango_cairo_fc_font_map_get_resolution_fc                        11
  _cairo_filler_close_path                                         11
  _cairo_path_fixed_close_path                                     11
  cairo_close_path                                                 11
  _cairo_traps_add_trap                                            11
  _cairo_get_global_image_glyph_cache                              12
  _cairo_lock_global_image_glyph_cache                             12
  _cairo_unlock_global_image_glyph_cache                           12
  _compare_cairo_edge_by_current_x_slope                           12
  pango_cairo_fc_font_map_context_key_equal                        12
  cairo_font_face_destroy                                          12
  _compare_cairo_edge_by_top                                       13
  _cairo_surface_init                                              13
  _cairo_array_init                                                13
  _cairo_path_arg_buf_create                                       13
  _cairo_path_fixed_add_arg_buf                                    13
  _cairo_path_fixed_add_op_buf                                     13
  _cairo_path_op_buf_create                                        13
  _cairo_pattern_create_solid                                      13
  _cairo_user_data_array_init                                      13
  _cairo_stock_color                                               14
  _cairo_xlib_surface_set_picture_clip_rects                       14
  _cairo_filler_move_to                                            14
  pango_cairo_font_map_get_type                                    15
  cairo_new_path                                                   15
  _cairo_format_from_content                                       15
  _cairo_fixed_integer_ceil                                        16
  _cairo_ft_scaled_font_get_glyph_cache_key                        16
  _cairo_clip_init_copy                                            17
  _cairo_gstate_clone                                              17
  _cairo_pen_init_copy                                             17
  cairo_restore                                                    17
  cairo_save                                                       17
  _cairo_xlib_surface_ensure_dst_picture                           17
  _cairo_operator_bounded                                          17
  _cairo_path_fixed_move_to                                        18
  cairo_move_to                                                    18
  _cairo_cache_shrink_to                                           18
  cairo_font_options_equal                                         19
  _cairo_fixed_integer_floor                                       19
  pango_cairo_fc_font_map_context_key_get                          19
  pango_cairo_font_get_type                                        19
  _cairo_scaled_font_map_lock                                      19
  _cairo_scaled_font_map_unlock                                    19
  cairo_matrix_init_translate                                      19
  pango_cairo_fc_font_map_context_key_hash                         20
  _cairo_restrict_value                                            20
  _cairo_pattern_init                                              21
  _cairo_pattern_init_solid                                        21
  _cairo_path_fixed_fini                                           22
  _pango_cairo_context_get_merged_font_options                     22
  _cairo_clip_fini                                                 24
  _cairo_gstate_destroy                                            24
  _cairo_pen_fini                                                  24
  cairo_pattern_reference                                          24
  _cairo_pixman_region_union                                       25
  _cairo_pattern_fini                                              25
  _cairo_pixman_region_union_rect                                  25
  cairo_ft_scaled_font_lock_face                                   26
  cairo_ft_scaled_font_unlock_face                                 26
  pango_cairo_fc_font_lock_face                                    26
  pango_cairo_fc_font_unlock_face                                  26
  cairo_font_options_hash                                          27
  _cairo_pixman_region_destroy                                     29
  _cairo_ft_unscaled_font_lock_face                                30
  _cairo_ft_unscaled_font_set_scale                                30
  _cairo_ft_unscaled_font_unlock_face                              30
  _cairo_pen_init_empty                                            31
  cairo_matrix_multiply                                            33
  _cairo_gstate_user_to_device_distance                            35
  _cairo_path_fixed_rel_line_to                                    35
  cairo_rel_line_to                                                35
  _cairo_filler_line_to                                            35
  _cairo_path_fixed_line_to                                        35
  _cairo_polygon_line_to                                           35
  _cairo_gstate_unset_scaled_font                                  35
  _cairo_polygon_close                                             37
  _cairo_pixman_region_create                                      37
  _cairo_pixman_region_create_simple                               37
  _cairo_pixman_region_copy                                        38
  cairo_pattern_destroy                                            38
  cairo_surface_destroy                                            39
  _cairo_fixed_integer_part                                        40
  pango_cairo_fc_font_get_glyph                                    41
  cairo_surface_reference                                          47
  _cairo_polygon_add_edge                                          50
  _cairo_gstate_user_to_backend                                    52
  _cairo_fixed_is_integer                                          57
  cairo_matrix_init_identity                                       59
  _cairo_polygon_move_to                                           64
  _cairo_path_fixed_add                                            65
  _cairo_cache_lookup                                              74
  _cairo_glyph_cache_keys_equal                                    75
  cairo_matrix_transform_point                                     82
  cairo_matrix_init                                                89
  _cairo_glyph_cache_hash                                          92
  _cairo_fixed_to_double                                          108
  cairo_matrix_transform_distance                                 117
  pango_cairo_fc_font_get_glyph_extents                           201
  pango_cairo_fc_font_get_scaled_font                             254
  _cairo_fixed_from_double                                        315

When I have more time I'll try this on the benchmark and also try it in a thin client environment where I expect things will be different. But now it is time for lunch!

Slow food - Slow boot at GUADEC

Wednesday Jun 28, 2006

GUADEC was my first visit to Vilanova i la Geltrú, Catalonia, Spain. I've enjoyed the weather, the people, the prices and the quality of food. (Black Rice Paella :-) The relaxed style of eating is polar opposite of the U.S. fast food that I grew up with.

Yesterday afternoon a few of us sat down at a cafe/tapas place on Las Ramblas and ordered salads and sandwiches. I was preparing for my dtrace talk and I'd hoped to give out live OpenSolaris CDs to the audience so that they could play with dtrace during the talk. Dtrace really is one of those tools you have to play with to appreciate.

Sun's Solaris Express opensolaris distribution would have been the obvious choice. It was put together by Sun, with good i18n, documents and accessibility support. GNOME 2.14 was integrated only a few builds ago. Solaris express contains zfs, dtrace and other cool solaris tools. It has passed some Sun/QA and review for architecture stability. Unfortunately it also contains a whole bunch of stuff which has nothing to do with my desktop and won't fit on a CD. Even some of the minimal package clusters contain weird things that are hardly appropriate for a laptop. Do I really need this fiber-channel stuff on a wifi laptop?

Belinix might have been another good opensolaris choice. It fits easily on a CD, but it contains an XFCE desktop instead of GNOME. Schillix also seems to lack a recent GNOME and the last time I used it, it didn't have dtrace.

Nexenta has an interesting opensolaris demo distribution which runs from a live CD. This distribution doesn't have documentation (no man pages?) or some of the Solaris tools. It packages som GNU utilities and user space apps with a solaris kernel and it contains both dtrace and GNOME 2.14. So I burned this distribution onto a few CDs. I intended to pass the CDs out before the dtrace talk so I booted it while eating a tomato and cheese salad at the Vilanova cafe. It was still looking for a dhcp server on my primary network interface when I finished my salad. It searched for a dhcp server on my wifi interface while I finished my sandwich. When I returned from ordering my coffee, it was at a login prompt. I logged in and sipped my coffee while we discussed why Nexena didn't change the default dhcp timeout to something more reasonable for a wifi connected laptop. About 10 minutes later (total time at least 30 minutes) I still wasn't at the gnome desktop. It seems that nexenta didn't take advantage of an alternate filesystem to buffer reads from the CD. (How about something like ZFS compressed filesystem living in RAM?) My battery was quickly running down while the CD was furiously trying to supply the livecd content. I gave up.

So that's why I didn't have a good distribution to share for the dtrace demo. With Sun's new boot architecture there is no reason why boot should take so long and there is no reason why a desktop user needs so much server related cruft packaged with their desktop. Someone told me about a keychain opensolaris distribution but I haven't had time to look at it. Sun has a great kernel, some awesome tools and a 21st century boot architecture. I know it's possible to create a distribution which is neither too big nor too small. Up until a year ago when Solaris was opensourced, it wasn't possible to make a task-custimized Solaris distribution. Now it is possible. I think it's only a matter of time before someone puts together a keychain opensolaris, DVD Opensolaris, CD demo opensolaris and (maybe I'll do this one) an RSYNC/ZFS opensolaris backup server distribution.

GUADEC dtrace

Wednesday Jun 28, 2006

Shortly before Glynn and I gave our GUADEC talk on dtrace, we sat in on Federico's "How fast" talk where he raised a question about profiling evolution memory allocation. I put together a quick dtrace oneliner which gives a distribution for evolution startup:

bash-3.00$ dtrace -c /usr/lib/evolution-2.6 -n 'pid$target::malloc:entry {@howmuch=quantize(arg0);}'
dtrace: description 'pid$target::malloc:entry ' matched 2 probes
CalDAV Eplugin starting up ...

(evolution-2.6:104389): camel-WARNING **: camel_exception_get_id called with NULL parameter.
^Cdtrace: pid 104389 terminated by SIGINT



           value  ------------- Distribution ------------- count
               0 |                                         0
               1 |                                         3414
               2 |@                                        12502
               4 |@@@@@                                    45218
               8 |@@@@@@@                                  71301
              16 |@@@@@@@@@@@@                             117177
              32 |@@@@@@@@@                                84371
              64 |@                                        14154
             128 |@@                                       15120
             256 |@                                        8680
             512 |                                         4517
            1024 |                                         4119
            2048 |                                         1193
            4096 |                                         784
            8192 |                                         1263
           16384 |                                         25
           32768 |                                         112
           65536 |                                         10
          131072 |                                         4
          262144 |                                         1
          524288 |                                         0

Joerg also pointed out that my "where evolution mallocs" oneliner was printing (N X M) stack traces and that one of the pair of output numbers was meaningless because it represented the count of that unique combination:

dtrace -c /usr/lib/evolution-2.6 -n 'pid$target::malloc:entry {@howmuch=quantize(arg0); @where[arg0,ustack()]=quantize(arg0)}

Where N was the number unique allocations and M was the number of unique stack traces. Its not as easy to do this correctly in a oneliner, so as soon as possible I'll put together a little script for profiling where the allocations are taking place.

One thing that is easily obvious from the above histogram is that evolution allocations are weighted towards very small amounts (2 bytes?!!!) Solaris's slab allocator should help in these circumstances. The gtk 2.10 talk was interesting, apparently GSlice uses an allocation scheme similar to that used by Solaris's slab allocator. You can't keep a good idea secret forever, can you?

I'll try to blog more about this if I have time and reliable connectivity.

BTW, why are the GUADEC wifi dhcp-servers handing out the same IP address to several MAC addresses?

Jun 28 15:24:46 sligo ip: [ID 903730 kern.warning] WARNING: IP: Hardware address '00:0e:35:07:69:1b' trying to be our address 010.000.007.027!

I guess the opportunity for IP address collision at an event like guadec is the reason why dhclient uses a very time consuming and paranoid arp broadcasting scheme to make sure that I'm not using someone's IP.

Thanks to whoever attended our Dtrace talk, I wish we had more time so I didn't have to fly through the demos, but it really is a technology you have to play with to fully appreciate.

grepping /etc, who put these pipes here?

Friday Jun 09, 2006

One minor annoyance I've encountered when switching between GNU/Linux and Solaris is that Solaris lays some pipes in the /etc directory. So if you happen to be logged in as root and you do something like this: grep foobar /etc/* Then your grep may hang for a very long time, seemingly trying to read from the /etc/initpipe pipe. The solution is to only grep files (not pipes) with something like this: grep foobar `find /etc/* -type f -print` This searches all of the subdirectories under /etc so it still isn't as quick as I'd like. I'm sure someone out there has a more elegant way of doing this. And I suspect there is a very good reason why rusty old pipes are laying around where folks can trip over them.

Yerkes observatory site to become spa resort, a victim of light pollution?

Friday Jun 09, 2006

The photo is of my nephew in front of Yerkes, an observatory which houses the world's largest refracting telescope. This pillar appears to be decorated with creatures of the zodiac. While on our eclipse trip, the Astronomer Royal for Scotland told us that this beautiful observatory near my family's Wisconsin home was up for sale. Today I learned that the University of Chicago has decided to sell Yerkes observatory and the surrounding land to a developer. The developer plans to build resort and spa on part of the land.

I don't know what will become of the telescope, but if you find yourself in southeastern Wisconsin and Yerkes is still open to the public, it is worth a visit. Our Scottish astronomer friend called it the Taj Mahal of astronomy. Albert Einstein once said that he would rather visit Yerkes than Niagara falls. The two main dome buildings sit on a grassy hill which is surrounded by woods and overlooks Lake Geneva, a small deep blue glacial lake. During one cloudy November leonid meteor storm watch, we were taken inside for a semi-private tour. The enormous refractor fits the stereotype of what a telescope should look like. Yerkes observatory is approximately the same age as Lick observatory, but Yerkes seems to be much more ornate. I'm reminded of Jules Vernes or H.G. Wells' vision of Victorian technological art. The enormous wooden floor under the dome of the 40 inch refractor rises so smoothly that it feels as though you are standing still and the building is shrinking. But when the dome rotates, it feels like you and the telescope are rotating.

I certainly hope the developers are able to maintain the buildings as a public science education resource. Unfortunately, the location for this observatory is now severly affected by light pollution and air pollution from Chicago and other large nearby cities. If it's true that exburbs, including those adjacent to Yerkes, are approaching the peak of a real-estate bubble, perhaps in 10-15 years a science organization can buy the observatory buildings back for an inflation-adjusted million or less. In the meantime, I hope the University of Chicago wisely uses the $8 million generated from the sale.

Toddler photo and kayaking

Sunday Jun 04, 2006

My three year old daughter took this photo. She took another one aiming slightly more to the left so you can see me tieing a string to the kite.

You say to-may-to, I say to-mah-to, you say po-tay-to, I say po-tah-to... One of the odd differences between U.S. english and Irish english is that what I call canoeing, they Irish call kayaking and vice-versa. I've never seen this kind of "roller derby" style kayak racing before.

This rubber inflatable Sevylor thing doesn't fit my definition of a canoe or kayak. It uses double-ended kayak paddles, but it's totally open and nearly impossible to capsize unless you puncture a pontoon. The upper Broadmeadows estuary is so shallow that if we were to sink it, we'd just walk home. After 12 years of Florida sunshine and Wisconsin winters, one of the seat pillows developed its first air leak. I read the patch kit instructions and warnings about the kayak. You're supposed to wear a PFD (we always do), and you're not to burn or eat the kayak. Eat it?