20 Feb How I killed the new Mac Pro
Us tech folk over at Key Code Media got a shiny new Mac Pro. 12 core, 32GB RAM, dual D700s with a a 512 stick of flash memory. My first 2 cars combined cost less than this beast.
We got the system current via Software Update, and installed all the apps we needed, and began benchmarking and demoing the system. It was smooth sailing for the next week and a half. We then noticed that the unit began locking up. The cursor would become a beachball, and all functionality (short of mouse movements) was lost. No keyboard commands worked, and even the menu bar, which showed realtime statistics on a number of processes, froze. And thus, I began basic troubleshooting:
- Check and/or replace cables. Try different Thunderbolt and USB ports.
- Remove all peripherals, except keyboard and mouse.
- Swap out monitors and Thunderbolt to DVI adapters.
- Verify all apps installed are Mavericks compliant.
- Remove any apps which may be borderline in terms of Mavericks compatibility or basic stability (i.e shareware and freeware)
- Repair permissions and examine hardware via Disk Utility.
- Use Onyx.
No dice. Mac Pro still locked up within 10 seconds of boot.
Many times, I had to do a forceful cold boot: Power down while the CPU is running (but frozen), via the physical power button. Unplug the CPU, and again press the power button to drain the trickle charge of the unit. Why? Often, a force shutdown via the power button would bring up an error on reboot telling me that the machine had been shutdown incorrectly. I would then get a beachball, and I’d be unable to continue. Draining the trickle power seems to remedy the error message on boot.
Upon viewing the system.log files, the logs revealed a few things that showed that the indexer for Spotlight may be causing issues. The log was littered with entries showing that the sandbox – mdworker was having issues accessing files.
Here is a sample line from one of the entries:
Feb 13 11:58:36 CPU5-Trash-Can-Mac-Pro.local sandboxd (): mdworker(392) deny file-read-data /Library/Application Support/Final Cut Pro/Library/Content.localized/Images.localized/.localized (pre-plugin fst
ype:hfs fsflag:480D000 flags:50000005E diag:0 isXCode:0 uti:public.folder plugin:internalPlainTextImporter – find suspect file using: sudo mdutil -t 1090659)
As you may know, mdworker is tied to Spotlight. Why does Spotlight seem like a possible candidate?
- The logs are littered with these entries and this was the last entry before some garbage at the end of the system.log. There were over 1350 entries in the last few minutes.
- Google shows there was at least one mention of these seemingly hanging a Mac system.
- There are additional entries in the logs showing other issues with the mdworker:
local mdworker: (Normal) Import: Using too many resources after 1408 files (wired: 0 resident: 142550 swapped: 0 regions: 2637), hit usage threshold importing /Library/Ap
plication Support/Adobe/CEPServiceManager4/extensions/Adobe Mini Bridge 3/CSXS, exiting to clean up now.
I was able to postpone the inevitable lockup by immediately putting the Mac OS drive into the Privacy section of Spotlight. The lockups still happened, but it would take a few minutes of use before the freeze.
I also attempted “sudo Mdutil -E” to stop Spotlight and re-index – again, this just slightly delayed the inevitable.
When the lockups and unpredictability became too much for reliable work, we decided to do a clean install. This is where it got really fun.
Booting from the recovery partition: Disk Utility did not see the Mac OS partition. When it occasionally did see the partition, the OS volume would appear, but not mount. When the volume did occasionally mount, and the Mavericks install process began, the Installer did not see the OS partition as being able to handle Mavericks (i.e. no option to choose the OS partition). See picture.
Internet Recovery : Similar issue. Mavericks could not be installed on the partition – the OS partition was not seen.
Apple Diagnostics (formerly AHT): no problems found.
AST: As we are an Apple Certified Repair Center, we have a network AST server. This found no errors.
Download Mavericks on MacBook Pro, create bootable Mavericks installer USB stick: creation of USB stick is successful, but the new Mac Pro is unable to boot from the stick. Tried 2x with different stick creation methods.
Target mode from a 10.8.5 MacBook Pro: Ran Diskwarrior. No “disk” problems, and only a few permissions issues on 3 files. As a safety, I quarantined those files. Lock ups still occurred.
Target mode from a 10.8.5 MacBook Pro: Ran Mavericks installer from the App store, in order to target the Mac Pro. Surprise! The version of Mavericks on the App store is older than the version on the MacPro. FWIW, the Mac Pro Mavericks version was 10.9.1 (13B4116)
Downloaded Mavericks from the App store on the flakey MacPro. I began the installer. Error Message: Mavericks cannot be installed on this computer. See picture.
Second new Mac Pro to Target from, do diagnostics from, etc.: The additional Mac Pro cupboard was bare.
I should add that at any time, I could *still* boot up normally, and use the computer for a few minutes.
Not wanting to delve more into additional Terminal work (plus, I really shouldn’t *have* to on a brand new machine), I gave in and called Apple support, and got through within 2 minutes. Extremely courteous technician. Sadly, I was told “You’re the first [new] MacPro call I’ve received!” He went on to say the machines were very stable, hence no calls, but unfortunately, I took it the other way: “…we don’t have a ton of experience with the machine.” His courteousness and willingness to help overshadowed this concern.
After an hour or so of discussing, troubleshooting, and being placed on hold, I was transferred to a Senior Technician. Also, completely professional and polite. They verified that if I opened up the hood and did some minor work on the engine, I would not be breaching warranty. They had me reseat the flash memory (you need a T8 torque screwdriver, BTW). Once reseated and closed up, the machine immediately exhibited the same issues.
Apple then decided they would swap the machine for a brand new one, matching the config of the initial unit. 3 days later (Sunday) I received a shipping label from Apple. I received a followup call on Tuesday (Monday was a holiday) asking if I received the label, and that as soon as the unit was scanned, they would ship out the replacement unit.
I, of course, have many questions as to why none of these techniques worked; and I’m not so sure it has to do with bad flash memory. It seems, to me, that perhaps there was an oversight by Apple. Why wouldn’t the Internet Recovery recognize the new machine’s OS? Why wouldn’t the LOCAL recovery recognize the OS? Why would the version of Mavericks on the App store not be current or applicable for a new Mac Pro?
Regardless, we await our new (x2) Mac Pro for more testing.
*Many thanks to those thoughtful tech folk who made sure I had done everything humanly possible: @octothorpe @natums @NEO_AMiGA @DCREELS @comebackshane @dwolfmeyer @philiphodgetts and Dan Bright of StorageDNA.
Edit 2/25/14 : Apple called today, 7 days after we shipped out the unit. They apologized for the delay, and told us their initial ETA for a replacement was incorrect. Since the Mac Pros are backordered until April, we are put on high priority, but this still means 2-3 weeks wait time. Oh, and it was also mentioned that there have been ~70 cases similar to mine.
Edit 2/28/14 : New Machine has arrived. But…with wrong GFX cards. Contacted the tech individual who has handled this case from the beginning. Our notes were in sync, he’s not sure where the disconnect happened. I was told I’d get a call tomorrow (Saturday) with the plans for the next step.
Edit 3/19/14: We shipped the MacPro back, and had arrangements made for the correct one to be shipped. We, again, were put in a queue. In the 3 weeks since the last update, we received the new unit. All appears to be working well, and Apple, as a show of good faith, offered up our choice of technology from their store.