Spotlight failing on 10.8.2 (and how to repair that)

After updating my MBP15 to 10.8.2 it started constantly running on 100% CPU. Who needs QA anyway, right Apple?

A quick look into top revealed constantly respawing 'mdworker' threads, process number where already close to 100k.
Console revealed that fact even more, with 50+ messages streaming by every second. 'mdworker' was crashing and immediately respawned by launchd. The crash reports cycled so fast, I needed to do a screen shot to read it:

Ok, the first thing I did is stop spotlight from indexing, and resetting the index:

In the terminal:
sudo mdutil -a -i off
sudo mdutil -E

I put the whole HDD in the privacy tab in the spotlight prefs for good measure. After rebooting the log storm was gone, but Spotlight obviously as well.
I didn't really know I had so much stuff in my application folder, being used to Spotlight. I found myself screaming in frustration, navigating  hierarchically (wow, google spell checking rocks, I entered 'hirachicly' ^^;) though all this crap.

Anyway, I googled around, tried a few fixes and finally found a solution here. In post #60 phobox explains that the errors stemms from the ML sandbox (or failbox as it will come to be known) blocking mdworker from indexing.

The solution:

Append the following to /System/Library/Sandbox/Profiles/system.sb

;;; Spotlight fixes  (allow mach-lookup (global-name "com.apple.ls.boxd"))  (allow mach-lookup (local-name "com.apple.ls.boxd"))

Watch what your doing, by deleting, overwriting or generally messing in this file you can lock yourself out of the OS. 

In true try-all-fixes-at-once-so-that-you-do-not-know-what-helped thoughtlessness, I cleaned my caching folder as well 

sudo rm -r /System/Library/Caches/*

And then did a safeboot to clean up permissions (hold shift key at startup, you will see a progress bar).

Huzza, my spotlight works again! I can again close the lid at the mess in the folders, and just start the app I want to start. ^___^

Numlock in Mac:Office 2011

Just a small hint for Excel 2011:

If your keypad does not work as number pad anymore (like, printing numbers when pressing keys), but moves your sheet around, try pressing Numlock.

There is no numlock in MacOS you say?

Right, but there is one in Mac Office! Press Shift + Clear (Thats the key between 7 and F16 on your keypad). You will get no reaction whatsoever, but you should now be able to enter numbers again. Thanks for consistency Microsoft.

A small truth about Open Directory passwords...

We had a problem today, which gave me some insight in the way OD handles its passwords.

I created a new user in OD and, being based in Munich, made an Oktoberfest induced error.
(At least that's my excuse...)

I deleted the value of "Authenticaton Authority" in the Inspector window in the user account. I tried to fix that by copying over that attribute from another user. Seems to work, everything fine.

After a while, the user from which I copied the attribute showed up and complained that his password suddenly seemed to be invalid.  Hmm.
Maybe just a coincidence? I changed his password, seems to work, everything fine. Back the the headache.

Shortly after, the new user showed up, her password was broken now!
Ok, time for some research...

The open directory does not save the passwords in the ldap tree (as hash, hidden field, etc...) . Instead is utilises a secondary password service (kerberos) which holds and manages the password. The link to this password is established by the "Authentification Authority" field in the OD. It contains an ID that references the password in the service.

Copying the hash from one user to the next gave both users the same entry, and thus the same, linked password. The problem was solved as I deleted the complete entry "Authentification Authority" of the new user, and changed the password. The OD created a new entry in the ldap and the password service, and it really worked, finally everything fine.

...don't drink and root. ^_^


<edit>
A college found this on the web:
http://flylib.com/books/en/4.395.1.68/1/

It explains the relation beween LDAP, Kerberos and the Apple password service in detail. Interesting stuff.

It's from 2005 but seems to be correct in most regards. One thing that seems to be deprecated is the note that the change of a users password with kpasswd does not change the password in the Apple password service. Tried it, it does. (SL Server, ML Client)

________
Appendix:

Get the Last modification stamp of the users password (on the OD host):
sudo kadmin.local -q 'getprinc user@DOMAIN.COM'

New shell commands in Mountain Lion

Just a quick link to a useful post from patrix:

http://apple.blogoverflow.com/2012/07/interesting-new-unix-commandsbinaries-in-os-x-mountain-lion/

There is some cool new stuff under the hood. =)

Recovering a locked down system with broken mobile user

Yesterday I set up a brand new MBA13''. Nice, but I prefer the MBPwRD for the same money. 
As usual I created an local admin account to access the system if anything goes wrong. The user account was an mobile user for obvious reasons. Both users had access to the filevault2. 

I gave the system to the user for data migration, task done.

Today I got it back with the trademark words "It Just Broke, I Did Nothing". First thing I noticed as I switched the thing on, no local admin account. And guest access was activated. Huh?

Logging in with the users password started the system, but dumped me to the normal login screen where only the guest user was visible and the "more..." button for other (normally network) accounts.

Trying to sign in with the "more..." button yielded nothing, so off to start from the recovery partition.

First bump: filevault2 encrypts the whole disk, how do I start the Recovery?
Quick Google: holding cmd-R works, even if the Recovery partition is not listed on the alt boot menue.
I still could enable the filevault2 partition with the old user password.

I tried to reset the password of the user with resetpassword (See blog post in Febuary) but it did not work. Well duh, it's a mobile user, of cause it does not work. The local admin stayed gone.

Well then, of to single user mode! (cmd-s at startup)
First, I fixed the file permissions like this:

> /sbin/fsck/ -fy

A look into the /Users directory showed the local admin account as deleted. Well, someone did something...

Ok, the user account was somehow borked, the local user was gone... what to do?

Easy:

(in single user mode) 

> mount -uw /
> rm /var/db/.AppleSetupDone
> reboot

After the reboot, the fresh installation assistent greets you, wants to register your device, show you how to natural scroll and creates a user with admin rights!

I took a gamble and gave it the same name and credentials als the deleted local admin account, and it worked like a charm.

Ok, after the system was accessible, the thing that caused all this was obvious:

The "Allow network users to log in at login window" login option was deactivated. Well durr, good lock signing in Mr Mobile User. Together with the deleted local admin account this was, well, stupid. 


At least I can now be sure that the local admin stayes on the system, that my colleges got a good laugh and I a blog post out of it. =) 

Kerberos Auth works again!

After trying again to analyse the problem, I've found that only the MCX prefs I've applied to the main group of the users would not get pulled down to the clients. 


Looking for this specific problem, I've found this:
http://lists.apple.com/archives/client-management/2012/Jun/msg00005.html

Durr, now everything is clear.
Just as the post stated, I've created a new group with the AuthWhiteList preference set and ... voila!

Chrome Kerberos auth breaks with MacOS 10.7.4 patch?

I have installed the new 10.7.4 update on my MacBook and promptly the Kerberos Auth of our Trac and FreePBX broke down in Chrome. I have not a clue why this is so, but it seems to affect only Mobile (account) users...

Puh... work, work.

- update -

It's still possible to give Chrome the Whitelist parameter manually:
open /Applications/Google\ Chrome.app --args --auth-server-whitelist="*.example.com"


- update 2 -

I've found the source file with all cmdline switches for chrome/chromium. Took me long enough to find, so I'm documenting it here:

http://src.chromium.org/svn/trunk/src/chrome/common/chrome_switches.cc

Get Chrome to work with Kerberos

A lot of stuff regarding Kerberos is broken on Lion. Some say everything. 

One thing that hit us soon after trying out Lion was that Chrome was not working with Kerberos anymore. We found the problem was with the "AuthServerWhitelist" parameter. 

We could fix it for network users with a change in the OD: 

• Access the opendirectory in the workgroup manager
• choose your main user group
• switching to "manage preferences" in the button bar 
• going to the "Details" tab
• Press +
• choose Google Chrome
• Select "Always"
• click on "New Key"
• Enter "AuthServerWhilelist" and give it the string value "*.<yourdomain.com>".
• Apply, and your done. 

It's more difficult for local users. There you need to access the local directory on the client. 

You can do this in Lion with the building "Open Directory Utility" but it cumbersome. 

I streamlined this by learning a bit of dscl: 

sudo dscl . mcxset /Users/<USER> com.google.Chrome AuthServerWhitelist always '*.<yourdomain.com>'

Replace USER and YOURDOMAIN, enter into terminal. Done.  

Kerberos client config with OD on SL and Lion

I've dived into the murky, two-colored waters of Kerberos on MacOS SL and Lion. Our installation works, but the tickets are issued without the Renewable and Forwardable flags, which complicate usage with Linux. I needed to find a way to config the Kerberos Client on the Macs….

After a lot of searching, I've found that Snow Leopard clients with network users read an Open Directory config entry and uses it as its krb5.conf

This entry is (seen from the OD host (SL Server)): 

/LDAPv3/127.0.0.1/Config/KerberosClient/XMLPlist

You can edit this list with the Inspector tab in your Workgroup Manager (again SL). The entry is formatted as an XML file, but you'll see the similarities to a normal krb5.conf right away. 

Don't forget to increment the GenerationID at the end of the XML to apply changes on your clients. The entries should be applied on the next login but you can enforce the sync by using the tool "kerberosautoconfig".  Add the parameter "-v 4" to it to get some feedback: 

$> sudo kerberosautoconfig -v 4 

Alternatively, you can copy a regular krb5.conf file to the file /Library/Preferences/edu.mit.Kerberos.

Lion Clients ignore the entry in the OD. They hilariously still accept the edu.mit.Kerberos file, even though Lion uses Heimdal Kerberos instead of the MIT flavor. Lion is missing the "kerberosautoconfig" tool as well, but the SL version still runs on Lion. Get the file from somewhere and copy it to "/sbin/".

You can generate a local "/Library/Preferences/edu.mit.Kerberos" file by running: 

$> sudo kerberosautoconfig -f /LDAPv3/<yourODhost>  -v 4

The configuration does not refresh by itself, so you need to run this again if something changed. 

I still have a lot to learn about Kerberos on Mac, I'll keep you posted. 

Referencing files via VolumeID and InodeID

I found a strange way to adress files in MacOS: Direct referencing via VolumeID and InodeID.

MacOS has a special folder for that: "/.vol"

If you list all files in there it seems empty...
But if you try this for example:

cat /.vol/234881026/4321840

you will get the file referenced by the volume (first part) and the inode (second part).

An example:

>:/$ stat /etc/hosts
234881026 4321840 -rw-r--r-- 1 root wheel 0 236 "Feb 22 12:04:33 2012" "Feb  2 21:25:28 2012" "Feb  2 21:25:28 2012" "Feb  2 21:25:28 2012" 4096 8 0 /etc/hosts


>:/$ cat /.vol/234881026/4321840
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1 localhost 



Groovy! =)

Create a bootable Lion USB Stick

To install Lion on a new HDD or SSD you need some kind of installation medium (at least on older Macs). Here is an easy way to create a bootable Lion USB Stick from your "Install Mac OS X Lion" App. You'll need an empty 8 GB USB Stick. 

0. Redownload the "Install Mac OS X Lion" Program from the App Store: 

"(While holding down the OPTION key, click on the “Purchases” section

You should see “OS X Lion” and “Install” should now be gray and you can click that to re-download Lion (you may have to re-authenticate within the App Store with your Apple ID)."

(Thanks asorta from MacRumors)

1. Locate the "Install Mac OS X Lion" program in your application folder with the Finder. 

2. Right click on it, and choose "Show Package Contents".

3. Navigate to "Contents/SharedSupport/InstallESD.dmg". This is the install disk.

4. Start "Disk Utility" 

5. Click on your USB Stick (the device not the partition) and click on the "Partition" tab 

6. Click on "Current" and choose "1 Partition". Choose "Mac OS Extended (Journaled)" as Format.

7. Press Apply. Now, choose the "Restore" Tab. 

8. Drag your "InstallESD.dmg" disk image from the Finder into the left side of the Disk Utility. 

9. In the Restore Tab, choose the InstallESD.dmg Image as Source, and your USB Stick as Destination. 

10. After 30 Minutes (depends on the Stick), you are done. You can test it now by booting from your stick. Hold "alt" at startup to go into the boot device selection...

This stick is good fixing a broken installation, for installing Lion from it, reseting passwords or change the partition layout of your disks. Have fun! 

Reset local user passwords in Lion and SL

Who doesn't know the pain of user leaving the company in anger, just throwing his equipment in the corner? Or someone who comes back from 6 weeks rafting in Canada, barely remembering where he works? 

With a fully directory enabled network this poses no problem, you just reset the password as admin. But with local users (and no local company admin account) this means some work.

Lion (10.7)

In Lion, you can do this via the recovery partition. There is a tool called "resetpassword" that you can start in the terminal. Here is a screenshot:

That's quite comfortable. Just enter the recoverypartition by pressing "alt" while turning your mac on, and choose "Recovery HD".

Alternatively, you can do it in single user mode:

• Hold "cmd-s" for Single User mode at start up
• Use the following commands (without $>):

$> /sbin/fsck -fy  # Check Filesystem
$> /sbin/mount -uw /  # Remount / as rewriteable
$> launchctl load /System/Library/LaunchDaemons/com.apple.opendirectoryd.plist  # Load Directory Service
$> ls /Users  # Show Users in System
$> passwd <user>  # Change PW for User
$> reboot

  
Snow Leopard (10.6)


In SL you can use the "resetpassword" tool as well, but you have to boot from your install disk. 
Reseting your password in Single User mode is a lot faster, and does not need a cd.  

• Hold "cmd-s" for Single User mode at start up
• Use the following commands (without $>):

$> /sbin/fsck -fy  # Check Filesystem
$> /sbin/mount -uw /  # Remount / as rewriteable
$> launchctl load /System/Library/LaunchDaemons/com.apple.DirectoryServices.plist  # Load Directory Service
$> ls /Users  # Show Users in System
$> dscl . -passwd /Users/<user> <password>  # Change PW for User
$> reboot

I've tried both, works for me. 

Software Update, the client side of things.

Ok, now that the Server is running, we can turn to the clients.

We have a pure Open Directory without the smallest stain of Active Directory in our network.

Which is not to say that I dislike the AD. I works great, and it would have been a lot harder to f**k it up like the OD we have here. But I'm embracing Apple fan-boyishness to fulfill my role as Mac admin. I even started to cut out the little Apple logos on the accessory packaging and put them into my sideboard to treasure them...

Uhm, yeah, clients.
There are three four ways to get your clients to access the local Software Update repository:


1. Use the OpenDirectory to push out the network path to the server.


This is accomplished in the Workgroup Manager in Preferences. It seems to work out of the box, but all of your users must be in OD, and login via OD as well.

Sadly, that's not the case with our users, which are all local users that only use Kerberos tickets to access the fileshares and linux.
I am going to change that, but did not find a good way to migrate all these local users to mobile users jet.


2. Update your local SoftwareUpdate.plist to use the new path


Run this in the terminal (replace the <...> with your update server):

sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL '<COMPLETE_URL_TO_SERVER>'

Some other how-tos state that you can enter this line without the complete path to "com.apple.SoftwareUpdate". At least in my setup, this is wrong.
I've tested it with Lion and SL clients, works for me.

Remember, this is an SnowLeopard Server. I think this will not work anymore with a Lion Server as something changed.

You can go back to the original Apple servers if you delete the entry again:

sudo defaults delete /Library/Preferences/com.apple.SoftwareUpdate CatalogURL

3. Change you local DNS Server to redirect the clients to your server


This could be seen as a horrible hack, but has some nice effects, not possible with other solutions:

• unmanaged clients are caught as well, you don't have to touch your clients
• MacBooks can still access Apples servers if outside the LAN

To integrate this into you network you need to create a new DNS zone, modify the "hosts" file of your update server and use a webserver to redirect some path. This is necessary because the original Apple update server uses different paths then the Software Update service. If you are already using a webserver on port 80 on the Software Update server, you can use another one (but you'll have to change the redirect paths).

This will need some maintenance in the future!

If the Apples swscan.apple.com changes its IP...
If Apple introduces a new DNS name for their update server...
If a new version of MacOS X comes out...
If some automatism in a update changes some part of this...

But it works for now. We use this for Lion and SL client.

First the DNS Zone:

• Go into the Server Admin tool and open the DNS Service. 
• Add a zone "swscan.apple.com."(mind the dot at the end!)
• Enter you local DNS server as "Nameserver Hostname"
• Add a A record in this zone: "swscan.apple.com." (mind the dot again!!)
• Don't forget to save. 

That's it. You can try it by using the "host" command in the terminal:

original:


         :~$ host swscan.apple.com
    swscan.apple.com has address 17.250.248.95


modified:


    :$ host swscan.apple.com
    swscan.apple.com has address 10.0.109.204


While you are on the commandline, you need to add the original IP of the Apple server to the /etc/hosts of your update server. If you fail to do that, your server will not be able to get new updates.

As root:
    echo "17.250.248.95   swscan.apple.com" >> /etc/hosts

Second, the web server:

• Enable the web server in the Server Admin tool (Settings/Services).
• In the Web service page, edit the default root site (the one with the *) 
• Give it the Host Name "swscan.apple.com
• Under Aliases, add the following entries as Redirects in the bottom box: 

        /content/catalogs/index-1.sucatalog 

        http://swscan.apple.com:8088/index.sucatalog

       /content/catalogs/others/index-leopard.merged-1.sucatalog

       http://swscan.apple.com:8088/index-leopard.merged-1.sucatalog

       /content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog

       http://swscan.apple.com:8088/index-leopard-snowleopard.merged-1.sucatalog

       /content/catalogs/others/index-lion-snowleopard-leopard.merged-1.sucatalog 

       http://swscan.apple.com:8088/index-lion-snowleopard-leopard.merged-1.sucatalog 

• You can disable the default services (wiki, blog, calendar) in the tab "Web Services"
• Don't forget to save

You should now be able to download software updates from your server, without any editing on you system. Try it by entering the URL in you browser:

http://swscan.apple.com/content/catalogs/index-1.sucatalog

The Software Update access log (in Server Admin) should show you your access.


4. Use your HTTP proxy server to accomplish the above 


A college showed me this, I didn't try it out but this is an real alternative:

You are using your squid installation to redirect http traffic to you local update server. This gets you the same benefits as the DNS method, without messing in you DNS config.

Here is the source link (german!):
http://www.heise.de/mac-and-i/artikel/Update-Zweigstelle-1424907.html

Software Update, go!

I spend the day installing a centralized update server on on of our Mac Mini Servers (on SL).

Apple provides a service called "Software Update" to accomplish this. 

Configuration is strait forward, just enable it in the Server Admin, configure the destination directory for the update cache and wait for it to finish downloading. Easy as it should be. 

As we have a mixed environment of Snow Leopard and Lion, the update server needs to host both types. 

This needs a few modifications in the config files. And this is where it gets ugly. 

Apple has documented the configuration here:

http://support.apple.com/kb/HT4771

Sadly, it does not work like that. Maybe it did one time, it does not anymore.
In short, two files need to be edited but these get changed back every time the service starts again. Apple fail.

Thx to leifsehn, I've found the correct way to do this: 

Hello All...I originally was having the same issue with the files reverting back to the original but I was able to get the lion updates working by doing the following: 

1. Login to your server as root, turn off SUS and Quit Server Admin.
2. Goto /System/Library/PrivateFrameworks/SUServer.framework/Versions/A/Resources/
3. Edit the swupd.conf to contain the new lion entry as per apple.
4. Delete the swupd.conf that is located in /etc/swupd
5. Run the command "sudo /usr/libexec/PlistBuddy -c 'add :otherCatalogs:2 string index-lion-snowleopard-leopard.merged-1.sucatalog' /etc/swupd/swupd.plist"
6. Resart the server
7. Start the Server Admin and Start the SUS.
8. Now the server should be reflecting the new Lion Updates.

Tested this with a managed computer and after I enabled the Lion updates I was able to instalt the 10.7.1 update.

Hope this helps. 

Source: https://discussions.apple.com/message/15962462#15962462

You can check if it's working correctly by accessing the Software Update repository in a browser:

http://softwareupdate.pretentco.com:8088/index.sucatalog

You should see an XML starting with "<?xml version="1.0" encoding="UTF-8"?>"

Great, so now SL and Lion updates are on the system, nothing left to stop us, right?
Well, the clients need to be configured. But I'll do that tomorrow...

And here I am.

I am a newly made MacOS administrator!

I've got a lot of experience with linux and unix systems but used MacOS only as a "couch device", so my experience was limited. I'm now 4 weeks into the new job, and am torn about the system:

- Yay, MacOS combines the wonderful core and shell of unix/bsd, some cool new concepts and a beautiful, usable, stable gui into a great package, fit for a lot of users from mums to developers.
- Boh, Apple obviously does not care for the enterprise. Bad documentation, delayed security fixes, incomplete or broken features, stupid licensing... it goes on like that.

Well, the mission of this blog is to document my work and experience with MacOS X in an enterprise environment.

I hope I will keep up writing, and that some of the posts will help others. =)

-Tarwin