Skip to content

SyncEvolution Status Update: On the Road to 0.8

I have been rather quite for a while – not because I was lazy, but rather because I was actively working on the next major update of SyncEvolution… That work is not yet publicly visible, so let me summarize the status.

Nightly Testing

I have extended the regular nightly (okay, only in some timezones) testing. It now includes a build of the latest experimental Evolution sources and running the SyncEvolution tests against that. I now run both the Evolution Data Server (the background process which stores the data) and the SyncEvolution client under valgrind to find memory handling errors and leaks. I put off that work for over two years because of the large amount of reports generated for Evolution libraries and the Funambol client library used by SyncEvolution.

Due to the design of the Evolution libraries (one thread allocating memory which is handed over to the main thread) it is very hard to tell which call in SyncEvolution really caused the memory to be allocated, but now I finally sat down and went through all reports. I intentionally made no attempt earlier to free memory allocated by calls like e_book_get_changes() because the right way to free the resources is undocumented, and I didn’t want to get it wrong: a leak is better than a crash. Now I got rid of many leaks and the nightly testing against Evolution 2.6, 2.8, 2.12, and trunk has not revealed problems yet – knock on wood.

I’m still not checking for any leaks of memory allocated by the C++ “new” operator because that would include leaks caused by the client library. SyncEvolution uses the RAII principle to track resources (not just of C++ objects, but also of the underlying C libraries!). Therefore I am rather confident that I don’t leak C++ objects, at least in those cases where memory ownership is obviously transferred to and then tracked by SyncEvolution. There were a few calls in the client library where that wasn’t clear, leading to leaks in the past. It should be better now, though, and I look forward to the next release of the client library: Funambol is working on fixing the leaks inside the library.

Configuration Handling

Earlier this year I got the required changes into the Funambol client library on which I could build the new SyncEvolution configuration handling. It is now in a working state, but because I don’t have unit testing for it I am not yet ready to publish a preview. With the new configuration handling come new command line options which completely remove the need to ever edit the configuration in a text editor – although that of course is still possible, and even easier than before. Thanks to storing and/or encoding a lot more information about configuration options, the new command line interface can print lists of properties and help texts for every property, check the validity of a property value, etc.

Migrating to 0.8

The rewritten configuration handling is backwards compatible with SyncEvolution <= 0.7, so in general it is possible to use 0.8 in one sync to try it out and keep 0.7 as a fallback. Only after the configuration is actively migrated to the new config file layout, then falling back to 0.7 becomes harder: restore the old configuration (which is intentionally only renamed, not deleted, for this situation), then do a refresh from client or server (depending on who has the complete data). Without the explicit refresh, a slow sync would be done, which has a higher risk of leading to duplicates.

The exception to this backwards compatibility rule might become calendar synchronization. In order to implement support for detached recurrences, I will have to replace Evolution’s change tracking (which is used in SyncEvolution up to 0.7) with a more fine-grained change tracking. When moving between 0.7 and 0.8, bad things(TM) like asking the server to delete/update/add an already deleted/updated/added event can happen. Redundant delete/update shouldn’t be such a big problem, but adding the same event multiple times could lead to duplicates on the server.

Would it be acceptable to go ahead with the implementation assuming that a big, fat, blinking warning in the release notes will be noticed by users? Please ease my troubled mind and say yes, because I don’t really have a plan B…

Non-Evolution Sync Sources

It has been part of the “master plan” for SyncEvolution to make it extensible so that other data sources also outside of Evolution can be synchronized. As part of the configuration rewrite I also made that a lot easier: I have gradually reduced the number of places that one has to touch to add a new data source. This work is almost done, with some work left in the automated testing program.

As a proof-of-concept I intend to add a sync source which stores items verbatim in local files. Think of this as a backup of the data stored on your SyncML server, or as a convenient way to download all data, modify it via a script and send back the modified items. I actually need this because I want to sanitize the various phone numbers in my contacts, and doing that in Evolution is not scriptable. This new file-based data source is not meant for synchronization of normal files or directory hierarchies, which also would have to deal with synchronizing file names and permissions. This could be added as there is a standard format for file content plus meta data, but typically the free SyncML servers on the net do not support that due to the potentially huge amount of resources that file synchronization can take.

{ 8 } Comments

  1. Peter Palensky | March 30, 2008 at 3:45 am | Permalink

    Hi.

    Great work!! Keep on going!!

    I am syncing evolution with a local funambol 6.5.14 server via syncevolution 0.7. Addressbook works fine.

    One question to the calendar sync: my fist test with a 300k *.ics file worked well. However, my real calendar (1.6MB, 5000 entries since 1997) takes 1.5h to sync.

    Who slows it down? funambol or syncevolution? Any idea or solution?

    Thanx,
    Peter.

  2. Patrick Ohly | March 30, 2008 at 9:37 am | Permalink

    Is this slow during the initial transmission or during each sync? A normal, two-way sync shouldn’t be that slow. If it is slow, please run “top” in a shell window and check whether any process uses unusual amounts of CPU time. Suspects that I think of are evolution-data-server, perl (running the synccompare script) and of course, syncevolution itself.

    You can also look at the client.log (usually stored under /tmp): it records time stamps for each message sent to the server.

    There is one setting which may have an effect here: syncml/config.txt maxMsgSize. The suggested default value is 8192, but this is not based on any kind of “scientific” analysis. If the actual transmission is slow, then you might experiment with larger values.

  3. Peter Palensky | March 30, 2008 at 11:14 pm | Permalink

    I changed the maxMsgSize to 32768 and now it takes one minute only. But maybe the previous tries were slow syncs for whatever reason.
    Thank you anyway! It works great now!

    Peter.

  4. Patrick Ohly | March 30, 2008 at 11:33 pm | Permalink

    Peter, can you test whether the maxMsgSize really makes a difference? Simply try it once more with a smaller value.

    If it works equally fast, then it is mostly a difference between slow sync and two-way sync. It would be worthwhile to try “–sync refresh-from-client”, once with small message size and once with large one.

  5. Peter Palensky | March 31, 2008 at 5:51 am | Permalink

    Patrick,

    I tested several times with maxMsgSize = 8192 and maxMsgSize = 32768 with my large calendar and –sync refresh-from-client . The larger value actually results in slightly longer sync times (6-7 minutes instead of 5). The 1.5h did not happen again…
    Sorry for not being able to reproduce that. If it happens again, I’ll try to track it down and post here.

    The two-way sync is now fast enough (1 minute), I run it in background via cron anyway.

    Peter

  6. Patrick Ohly | March 31, 2008 at 9:36 pm | Permalink

    I bet it was slow because of a slow sync: during a slow sync the same amount of data is sent as with a “refresh-from-client”, but the server has to do quite a bit more calculation on the data.

    Thanks for testing. I’ll leave the default at 8192, as this seems to work well for streaming the data to the server.

  7. Peter Palensky | April 14, 2008 at 5:35 am | Permalink

    Hi.

    Maybe you have some advise…
    I am in the unfortunate position to work on 3 different continents and i have not found any PIM that would support that right (maybe only 4-dimensional desktop environments can view that properly… ;-) .
    I am using evolution, which does accept entries for Johannesburg (SAST) and Los Angeles (PDT) without problems (except that when i move a non-current-time zone event [which are apparently shown with a "pin" symbol in the calendar], the timezone is set to the current one and the “pin” symbol disappears).

    This happens when when i add two 30 min appointments at 14:00 and 14:30 for Johannesburg (SAST). One entry is done as evolution probably wants it (14:30 and timezone SAST), the other with my dirty workaround (chose timezone PDT and time 05:00, which results to 14:00 for SAST). Evolution itself has PDT set as timezone (as i am currently in California). When setting the TZ to Johannesburg, both entries show up at their correct time. The evolution *.ics file contains:

    – cut –
    BEGIN:VEVENT
    UID:20080414T015530Z-26959-1000-1-14@q40a
    DTSTAMP:20080414T015530Z
    DTSTART;TZID=/softwarestudio.org/Tzfile/America/Los_Angeles:
    20080421T050000
    DTEND;TZID=/softwarestudio.org/Tzfile/America/Los_Angeles:20080421T053000
    TRANSP:OPAQUE
    SEQUENCE:3
    SUMMARY:Test LAX
    CLASS:PUBLIC
    CREATED:20080414T015538
    LAST-MODIFIED:20080414T015552
    END:VEVENT
    BEGIN:VEVENT
    UID:20080414T015555Z-26959-1000-1-15@q40a
    DTSTAMP:20080414T015555Z
    DTSTART;TZID=/softwarestudio.org/Tzfile/Africa/Johannesburg:
    20080421T143000
    DTEND;TZID=/softwarestudio.org/Tzfile/Africa/Johannesburg:20080421T150000
    TRANSP:OPAQUE
    SEQUENCE:2
    SUMMARY:Test JNB
    CLASS:PUBLIC
    CREATED:20080414T015621
    LAST-MODIFIED:20080414T015621
    END:VEVENT
    END:VCALENDAR

    – end cut –

    One entry is added as Los Angeles appointment, the other one as Johannesburg appointment (This is ok so far, although i hoped that evolution stores entries in UTC only…). Note that the appointments are at UTC 12:00 / PDT 05:00 and UTC 12:30 / PDT 05:30, since SAST is 2h ahead UTC and PDT is 7h after UTC.

    Now comes the sync magic. In Funambol settings, the TZ of the device evolution is set to PDT, conversion to current TZ is disabled, the TZ of my computer is also PDT.
    After syncing, the Funambol web demo client tells me:

    Test JNB 07:30 21.04.2008
    Test LAX 22:00 20.04.2008

    obviously
    05:00 became 22:00 previous day -> -7h
    14:30 became 07:30 same day -> -7h

    Taking into account that TZ info is lost, the -7h is not the same but even more confusing, since my computer speaks PDT:
    PDT 05:00 became PDT 22:00 previous day -> -7h
    SAST 14:30 became PDT 07:30 same day (= UTC 12:30 became UTC 14:30) -> +2h

    The evolution *.ics file stayed untouched:

    Changes applied during synchronization:
    +——————-|——-ON CLIENT——-|——-ON SERVER——-|
    | | successful / total | successful / total |
    | Source | NEW | MOD | DEL | NEW | MOD | DEL |
    +——————-+——-+——-+——-+——-+——-+——-+
    | memos | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
    | addressbook | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
    | calendar | 0/0 | 0/0 | 0/0 | 2/2 | 0/0 | 0/0 |
    +——————-+——-+——-+——-+——-+——-+——-+

    Further syncs leave everything untouched. Updating the two entries in the Funambol web demo client (typing “touch” in the description) leads to updates on the client side during next sync:

    – cut –
    before sync | after sync
    removed during sync added during sync
    ——————————————————————————-
    BEGIN:VCALENDAR BEGIN:VCALENDAR
    VERSION:2.0 VERSION:2.0
    BEGIN:VEVENT BEGIN:VEVENT
    SUMMARY:Test JNB SUMMARY:Test JNB
    CLASS:PUBLIC CLASS:PUBLIC
    DTEND;TZID=/softwarestudio.org/Tzfi | DESCRIPTION:touch
    le/Africa/Johannesburg:20080421T150 | DTEND:20080421T150000Z
    000 | DTSTART:20080421T143000Z
    DTSTART;TZID=/softwarestudio.org/Tz <
    file/Africa/Johannesburg:20080421T1 <
    43000 <
    SEQUENCE:2 <
    TRANSP:OPAQUE X-FUNAMBOL-ALLDAY:0
    > X-LIC-ERROR;X-LIC-ERRORTYPE=VALUE-P
    > ARSE-ERROR:No value for CATEGORIES
    > property. Removing entire property:
    > X-LIC-ERROR;X-LIC-ERRORTYPE=VALUE-P
    > ARSE-ERROR:No value for LOCATION pr
    > operty. Removing entire property:
    END:VEVENT END:VEVENT
    BEGIN:VTIMEZONE <
    TZID:/softwarestudio.org/Tzfile/Afr <
    ica/Johannesburg <
    X-LIC-LOCATION:Africa/Johannesburg <
    BEGIN:STANDARD <
    DTSTART:19700319T010000 <
    TZNAME:SAST <
    TZOFFSETFROM:+0200 <
    TZOFFSETTO:+0200 <
    END:STANDARD <
    END:VTIMEZONE <
    END:VCALENDAR END:VCALENDAR
    ——————————————————————————-
    BEGIN:VCALENDAR BEGIN:VCALENDAR
    VERSION:2.0 VERSION:2.0
    BEGIN:VEVENT BEGIN:VEVENT
    SUMMARY:Test LAX SUMMARY:Test LAX
    CLASS:PUBLIC CLASS:PUBLIC
    DTEND;TZID=/softwarestudio.org/Tzfi | DESCRIPTION:touch
    le/America/Los_Angeles:20080421T053 | DTEND:20080421T053000Z
    000 | DTSTART:20080421T050000Z
    DTSTART;TZID=/softwarestudio.org/Tz <
    file/America/Los_Angeles:20080421T0 <
    50000 <
    SEQUENCE:3 <
    TRANSP:OPAQUE X-FUNAMBOL-ALLDAY:0
    > X-LIC-ERROR;X-LIC-ERRORTYPE=VALUE-P
    > ARSE-ERROR:No value for CATEGORIES
    > property. Removing entire property:
    > X-LIC-ERROR;X-LIC-ERRORTYPE=VALUE-P
    > ARSE-ERROR:No value for LOCATION pr
    > operty. Removing entire property:
    END:VEVENT END:VEVENT
    BEGIN:VTIMEZONE <
    TZID:/softwarestudio.org/Tzfile/Ame <
    rica/Los_Angeles <
    X-LIC-LOCATION:America/Los_Angeles <
    BEGIN:DAYLIGHT <
    DTSTART:19700308T030000 <
    RRULE:FREQ=YEARLY;INTERVAL=1;BYDA <
    Y=2SU;BYMONTH=3 <
    TZNAME:PDT <
    TZOFFSETFROM:-0800 <
    TZOFFSETTO:-0700 <
    END:DAYLIGHT <
    BEGIN:STANDARD <
    DTSTART:19701102T010000 <
    RRULE:FREQ=YEARLY;INTERVAL=1;BYDA <
    Y=SU;BYMONTH=11 <
    TZNAME:PST <
    TZOFFSETFROM:-0700 <
    TZOFFSETTO:-0800 <
    END:STANDARD <
    END:VTIMEZONE UTC 05:00 = PDT 22:00 previous day
    SAST 14:30 -> UTC 14:30 = PDT 07:30 same day

    I think this is also evolution’s fault (i did the usual Gentoo weekly upgrade without thinking…). Until recently i believe the entries were stored like

    DTSTART:20020313T130000Z

    at least that’s how most of my entries look like (but they are originally from Kalendar, Palm, etc., so i might be mistaken) Now the new entries look like

    DTSTART;TZID=/softwarestudio.org/Tzfile/Africa/Johannesburg:
    20080421T110000

    or

    DTSTART;TZID=/softwarestudio.org/Tzfile/America/Los_Angeles:
    20080425T040000

    and the TZ info is stripped during sync…

    If i – in evolution – edit the “Test JNB” again (putting “touch again” into the description) and sync, the 07:30 entry becomes a 0:30 again…

    So even a potential, ugly workaround – to add all entries as PDT (i.e. local) entries, and setting the time according to the time shift (e.g. an appointment for 10:00 in SAST would result in a 01:00 entry in PDT) – would not help much, since the sync shifts the new or changed evolution entries for 7h. And changing the TZ of your computer would lead directly into sync hell…

    I’ll investigate the Funambol settings once again, maybe the “convert to current TZ” thing helps. I had the impression i need that setting on the cell phone side.

    Thanx,
    Peter

    p.s.: hm… quite a long posting… sorry.

  8. Patrick Ohly | April 15, 2008 at 4:44 pm | Permalink

    Peter, the Funambol server strips timezone information. I think this loss of information is the reason for many real-world problems and therefore would recommend not to use the Funambol server for calendar syncs until timezone support gets improved. If you want to help with that, contact the Funambol developers on their mailing list. Your comment above would be a good starting point.

    If you want a solution that already works, then you might want to consider syncing via http://www.scheduleworld.com. It fully supports iCalendar 2.0 and preserves the timezone. There are some issues related to timezone support in Evolution and I am working on a patch for Evolution, but they mostly occur when importing events with non-Evolution timezone IDs and shouldn’t affect you.

Post a Comment

Your email is never published nor shared. Required fields are marked *