Anatomy of a Software Update + 700p Maintenance Release Info

Product managers like me have needed a place to communicate with Palm customers and fans about what's happening at Palm, and now we have it with the new Palm blog! We're looking forward to future blogging and to reading your feedback and comments.

One of the hot topics on the Palm/Treo community forums has been about software updates and I noticed that folks often wonder why we can quickly post a patch for some issues, but others sometimes take a full ROM/firmware update and take more time to deliver. Let's look at a couple examples...

Shortly after the release of the Treo 680 smartphone, it was discovered that using the camera could cause an excessive battery drain on the device. We quickly diagnosed the problem and figured out that a small change to the camera application could solve the problem. We cranked out the updated app, wrapped it up in an installer and got quick approval from the carrier to post it. Case closed, but how could it happen so fast? Two reasons - it was a small change limited to one app, and that one app happened to be a component that we could replace with a new version in user storage. Since the change was localized and didn't touch any of the radio/telephony software, we could push out a patch quickly.

Now maintenance releases (MRs) are a different story. When we release a MR that replaces the ROM image and radio firmware on a Treo, the process becomes more rigorous and requires more steps than a patch. Once you start mucking with complex components like the radio, power management or the file system, then you are way beyond a patch. Case in point: we started working on Bluetooth improvements for the Treo 700p as a patch so we could get it out quickly. More than a few months into the effort we decided that a patch wasn't going to work because the changes touched too many other components - we'd need to build it into a full MR.

MRs are usually big efforts - the deeper into the wireless functions we dig, the more careful we need to be to avoid breaking something else and the more testing Palm and the carriers have to do. So when we do an MR, we try to make it count. That means we batch together a number of changes that come from a variety of sources - call centers, online forums, carriers, retail stores, etc. Once Palm is done with the MR development work, we beta test the software internally and externally, run through pre-certification checklists, and hand the MR to the carrier for final certification and potentially a final round of changes.

We're now in the carrier certification phase with the Treo 700p MR and as I type this, our best estimate of customer release will be the week of May 28th. In addition to the Bluetooth improvements I mentioned above, the 700p MR includes updates to SMS send/receive performance, MP3 playback, phone app launch speed and other areas. It's admittedly been a much longer project than expected, but we're almost there...

Thanks for reading,

Steve Sinclair

Treo Product Manager

Update:

We have been reading through everyone's comments on the 700p maintenance release update and I wanted to answer some of the recurring questions that have been brought up:

• Why didn't you mention the lag in your blog? "The lag" is a broad term that could mean any of number of symptoms customers have reported, so it would have been a mistake for me to make a blanket statement like "the lag is fixed" or "the lag is not fixed." We've implemented specific mitigations that will benefit some of you: launching the phone app should be faster, we've lessened the occurrence of SMS sending delays and we addressed a problem that was causing the multi-second delays when launching/exiting the browser.
• Will this MR make the device more stable? The short answer is "yes" in most cases, but I can't list every situation where we've made changes or quantify the improvement since each person's device setup is different.
• What are the Bluetooth improvements? We've improved overall stability to Bluetooth pairing and connections. When the MR goes live, we're going to publish an updated list of Bluetooth accessories that are compatible with the Treo 700p + MR software.
• Why don't you share the MR release notes? In this blog and some of my previous correspondence to the Palm community, I've referred to the most discussed issues. Our standard policy has been to not publish an itemized list, but to disclose the top issues that are addressed by the MR when the MR goes live on our web site.

We know that a number of you are frustrated and appreciate that you've all taken the time to share your feedback, and apologies for taking a bit to respond to that feedback.

Comments

[Stephen Mortensen]

I dunno why it disappeared for a bit, but now that it's back, thanks for posting this update on the situation. any chance we could get you to expand on the "other areas" a bit?

Thanks.

[Richard]

And the reason you haven't been working on it for a year is that you've been churning out new phones? So the phone will maybe work after you've determined the end of life for the phone. What kind of company does that? Why was this not a higher priority? It appears, again, that you have deserted the 700P, and nothing you've written excuses that at all. It's been a YEAR.

[Jim Adams]

Thanks for the update.

[BusyME]

Well, I always say, better late than never.

I love my 700p, so wait I will.

=)

[Tim Lopez]

Thanks for letting us know about the progress on the 700p!

[kevin]

Awesome, so a year after most of us bought these phones, we maybe sorta might have an update to fix serious bugs that have plagued most of us since day 1. Thanks for the explanation that the treo is a complex piece of hardware - I guess the $600 pricetag didn't illuminate that for me...

[Nathan]

Thanks for the "update" on the update. It sounds like you guys are going after most (if not all) of the gripes that I've seen. Should this MR also help with overall device stability?

[David W. Dalrymple]

Steve,

The lag has not specifically been addressed in your post. Will it be corrected in this "MR" ?

[Rob]

Wow.

Isn't that exactly a year to the week from when the 700p was released???

Steve, we understand that the process for writing, testing, writing, testing, and then certifying through the carriers is time consuming and challenging. That is really not the issue. Even if it is a 4-5 month process (which seems reasonable based on your previous MR schedules for the 650 and other devices) then you began working on this MR 6 months later or more than you should have. The bluetooth issues alone were obvious to us in June of last year. Were they not obvious to your team???

By July the lag and MP3 performance was written up all over the internet and many had narrowed down the problems. Yet the Palm mgt team still did not begin a cycle for an MR?

I'm sorry but the support of the 700p and it's users has been absolutely horrible. You have lost a number of customers and it will be tough to win them back.

[GB]

This is welcome news!

Thanks for the Blog, and for giving your customers the facts about the Treo 700p situation.

It's a great product, notwithstanding the unfortunate series of glitches in this model.

Can't wait for the Linux version.

[Josh]

I just want to say that this a welcome change in Palm's philosophy! Letting the customers know what is going on will not only help improve their patience but will also give them a sense of quality support and let them know you guys are actually working over there. Thanks for the update on the MR and I hope to hear something about the Treo 750 WM6/HSDPA update soon.

[Bob Clancy]

Thanks for keeping us in the loop Steve. I'm glad to hear of your decision to make this an MR and to give us a "quality product" rather than a patch. This effort restores a lot of hope for me that Palm will support existing products in addition to pumping out new devices. (Without support for existing products, you would have lost me as a customer. You've made the first steps in winning me back!)

Bob Clancy
9 Lives Software Engineering

[Chavous Camp]

THE WEEK OF MAY THE 28TH? ONE FULL YEAR after the device was released? Geez palm - it didn't take you that long to get the 650 update released - and somehow Moto manages to release maintenance releases for the Q without this amazing time delay. Seriously - y'all OWE 700p users A WHOLE LOT, as we've been AMAZINGLY patient and haven't showed up in large numbers at your corp office with the traditional choices of torches and pitchforks! I'm thinking a SERIOUS rebate (and/or REASONABLE device trade-in program - not the restricted "you must buy the new device solely from us and maybe you'll get a rebate from a sketchy third-party company" trade-in game played with the 650 -> 700p). Seriously Palm - you owe us BIG TIME for this. We spent a HECK of a lot of money on these phones - and for MANY MANY MANY of us - THEY JUST DO NOT WORK AS PROMISED.

[Dave]

ARE YOU KIDDING? A FULL YEAR LATER?

You know what, that's fine by me. If the patch fixes everything and makes the phone work, that's great. You know why? Because there's no chance in hell that I'll be going out and buying a 755 or whatever when it first comes out to deal with this again. This is absurd. Once they release the fix to make the 700p work, there will be little to no reason to justify upgrading and giving them more of my money. That's it and I'm done with supporting palm.

[Jim Conforti]

Steve,

I think you all at Palm would have greater credibility if you actually told us IN WRITING *exactly* what you are trying to include in the MR.

Think "Release Notes".

I for one have lost business because you released this phone with a faulty bluetooth stack. Lost calls that I have no record of as the 700p resets before ring even when used with a "Palm Approved" BT headset (JX-10).

If the client doesn't leave a message, I have no call log and POOF likely no more client.

[Scott Kennedy]

I hope that the testing goes well. I've been suffering from the Verizon SMS Lockup bug for months.

But, I do appreciate the update, let's hope that no showstoppers crop up...

Scott
PS> Can I volunteer to be a Beta tester?

[Dave]

DO NOT BUY A 755 WHEN IT COMES OUT - WAIT FOR THE MR FOR YOUR 700P WHICH WILL MAKE IT WORK LIKE A NEW PHONE. DON'T GIVE THEM YOUR MONEY.

[bob]

Paul,

Thank you for your explanation and estimate of the MR release date.
The Treo 700p has been in the consumer's hands for about a year. The issues that you have noted in your blog entry have also been existing and have been noted in multiple on-line discussion groups for the same amount of time.
Why has Palm taken almost a year for these issues to be noted & hopefully fixed?
I have been a long time Palm advocate and was an early adopter of the Treo 700p platform. All I want is a device that works correctly as promised.

I'm still staying tuned.

bob

[Andres Jabois]

Oh, this is great, you have become (now, after a long time) in a company who cares for customers, don't let it drown.

[Mike]

Maybe you guys at palm should setup the firmware as modules so you can rewrite or update problem areas and then piece the firmware together and release the firmware (ie. phone, radio, blue tooth, etc) then if you change the blue tooth module you may only need to update the sound module and piece the firmware together and test it and maybe it would be quicker because if you don't change how the radio works then you wouldn't have to go through a long cert process.

[Alan Grassia]

Hello Steve.

Thank you for providing us with an update on the status of the Treo 700p maintenance release. I do appreciate the new open lines of communication with Palm's customers.

For the most part, owning and using the Treo 700p has been lots of fun. I occasionally see the lag issue when launching Blazer or Documents To Go, however, I still think that the Palm solution is still the best all around smartphone solution. The Bluetooth stack has recently started giving me trouble now that I have a Bluetooth keyboard. I am eagerly awaiting the MR so I can start using that keyboard.

I'll be sure to cover this news in this week's 1SRC podcast.

Cheers!
Alan G

[waldo15]

Steve,

I have mixed feelings.

First of all, recognition. Palm finally has a public storefront to address the community in a somewhat direct fashion. This blog came out perhaps a tad too late, but nevertheless becomes a very valuable tool for directly addressing a very active, lively and vocal community.

Having said that I applaud you giving us a refresher on when the ROM update (or MR as you call it) could be out in the wild. The date has been and still is a moving target nonetheless, let us all hope the last week of May holds true.

But on the other hand, several issues arise. No mention of lag and/or stability enhancements in the MR make it look like the 700p will still be crippled in that sense. Also, the date as temptative as it may be is a year after the release of the device. This sets a not so nice precedent in what it was a good reputation on Palm support for its devices. The 650 maintenance came out no more than 6 months after launch, and let's face it it's not like the 700p problems were unknown until recently. Anyone who owns the device experiences the issues you described almost on a daily basis.

We can only theorize on why it is taking so long, could it be because of the new products you guys released in the past few months? Could it be because you re-directed resources to Linux development? Why the MR took so long only you guys know, but the year long delay leaves a bad taste in all of us power users as well as prosumers that got their hands on what it was a very sought-after device. Comparable devices in the industry do not go that long without some kind of patch release and/or firmware update.

In any case, we will see come last week in May if the 700p can become the device it always has been promised to be... one of the best (if not THE best) Palm OS devices you guys have ever released

[Will]

You should be absolutely ashamed of yourself. Do you really think we're buying all this "oh it is serious, complicated business" noise that you're giving us? Of course it is complicated, that is why we paid so much for the phone. You know--to actually perform the features listed on the box. If the phone says it supports Bluetooth it should work correctly out of the box. Not half the time, not randomly un-pair, not with a specific set of headsets. That means you should have adequately tested the phone before bringing it to market. Not fix it after with an update and certainly not after making your customers wait a year for the bug-ridden phone you foisted upon them.

Do you have any idea just how much you've ruined your customer base with these antics? Are you aware that you have competition in this market? Do you really think people are going to come back to you for another phone? Do you think people aren't telling their friends and family about the lousy support from Palm, with all their delays on promised updates?

[twitch3]

Palm should really try to get US Cellular to start carrying POS products. They are the only other company that I know of that take a year to release anything which is why I dropped them in the first place. I hate to be so negative but I've been a Palm customer for seven years, and while I have defended most of the Palm products, software, and decisions; I can't help but to start feeling like 700P users have been abandoned. There are other products out there and they have to deal with the same carrier testing but their MR's have been released. Honestly I haven't had any major issues with my 700P but MANY people have so a release of the MR a year later is truly inexcusable.

[WI]

Thanks for the update.

Release notes would be very helpful.

The lag in my device is getting worse and is so frustrating! Will it be included in this MR?