BP 401 - Admin Zero to Hero in 60 Minutes

The question is no longer, "How can we?" The question now is, "How should we?" Andrew Pollack, President Northern Collaborative Technologies

Language Note
 I realize that for some of you, English is not your primary language, and for others, my accent is not the same as yours.  If you are having trouble understanding me during this talk, please raise your hand and I will try to slow down and speak more clearly.  Thank you.

Wireless Devices

 Wireless device noises are rude in any language. Please take a moment to turn off any of the following:
 Cell Phones  Scheduler Devices  Pagers  Alarm Clocks  Pacemaker low-battery warning alarms  Anything else you are carrying on or about your

person

which may make noise during this presentation.

About this Presentation

 A "best practices" session is different
 This is not a list of product features.  This is a practical 'field guide' of which ones to use, and why.  Focused on What and Why, pointers to resources for how.

 Designed for re-use

 These are not empty bullet points.  The details you need are in this text.

 The Goal of this Presentation

 Provide an overview of what you should be thinking about as an

administrator
 Provide a trail map for finding out more, and implementing the ones you

find of value
 Help you start thinking in terms of the big picture rather than being

constantly swamped by the details

Agenda
 Who am I to be telling you anything?  The Scenario Setup  Server Stability Management  Security Management  Mail Management  Database Management  Client Software Management  End User Support

Who am I To Tell You Anything?

 Andrew Pollack
 President, Northern Collaborative Technologies  2003 IBM Lotus Beacon Award Winner  1999 Lotus Beacon Award Finalist  Administrator & Developer since version 2.0  Member of the Penumbra Group  Firefighter Cumberland, Maine!
 Lieutenant of Engine 1, Ladder 7, Heavy Rescue, RIT, Special Operations  In firefighting, just like Server Administration it's all in the planning

 Why We're Here

 To learn and grow as human beings  The question has changed, now it isn't "How Can we," it's "How Should We"

 Also, I'm here because it makes the phone ring more

A Typical Environment
 Three Offices
Southeast The Home Office
Mid Sized, easy to get to, excellent

net connection
Southwest A Production Facility
Mid Sized, easy to get to

Northeast R&D
Small Office Terrible Airport Access Heavy Ground Traffic Weather & Power Issues Expensive Travel Costs

 Then theres you

The new Domino Administrator

Server Load & Hardware Choices

Clustering vs. Giant Boxes

 Benchmarks are just statistics, and we know how much we should trust those.  Would you really put 12,000 users on one server? 20,000? More?  Domino clusters do not shared any hardware or part of the same operating system. They are fully redundant.  Balance the load across all the servers in the cluster, but make sure that if one goes down, the others can handle the load without crashing.
 A performance drop is acceptable for a brief period in most shops.

Clusters Provide High Availability, Low Cost

Domino Clustering is REALLY Easy

 Put databases on both servers
 Make sure they replicate, and have proper access

 Select the servers in the directory  Click "Add to Cluster"

Considering Peak Loads

 We think of number of users dont do that.  Think number of concurrent users.
 If you run three shifts, and only one shift is active at a time, you may be

able to use smaller hardware.

 Think total disk space.

 Disk usage is critical on the server, even if it isn't in use it costs the server

resources to keep indexes and run checks.

 In some customer sites, mailbox size dictates server count because of

drive space limitations and the cost of massive storage networks.

 For more information about clustering

 JMP102 An Introduction To All Things IBM Lotus Domino Clustering --

Gabriella Davis

Software Version Management

Operating System Choice

 Which operating system is the best?
 Avoid politics, religion, and operating system preference discussions at the

dinner table
 Either choose an OS that your staff knows well, or send them to school  All operating systems need to be patched and updated. Keeping up with

these is required for stability

 Make a choice that is not unique in your company  Test, Test, Test

 Watch out for case sensitivity when moving off Win32

 Debugging can be very difficult because the initial hit to a resource is case

sensitive, but once the object is in the cache, it may not be.
 BP403 Best Practices: IBM Lotus Domino for Linux -- Daniel Nashed

Remote Server Administration

 No matter what tool you use, always use encryption  Many tasks you might think you need remote control software for, can be done with the Web Administration Tool and the Lotus Domino Administration Client
 Editing the NOTES.INI on the server  Starting and Stopping Windows Services

 Use the Server Controller and Java Console

 These can restart even crashed servers remotely  Start the server with "jc"  Stat the console from the Notes program directory "jconsole.exe"

Remote Control Software

 Make sure it is set to lock the console automatically if your connection drops  Make sure it requires encryption for connections  Keep up with the vendors patches and updates for the server side
 Security patches could be critical  These ports are scanned constantly

ADMINP is your best friend

 Properly configured, this will do a lot of the hardest and most tedious work for you
 Distribution of new databases to multiple servers  User move, add, or change requests

 This becomes more and more important with each new version of the IBM Lotus Domino server  Each server should have a replica of the "ADMIN4.NSF" from the administration server  For more information
 ID113 Maximize the Power of AdminP in IBM Lotus Domino -- Kathleen

McGivney, Susan Bulloch

Local Staff
 Nothing is better than local staff
 Before doing any kind of remote access work, compile a list of local contact

staff with phone numbers and availability

 Have someone check the cdrom trays you do not want to reboot to a

setup disk

 Nothing is worse than local staff

 Control access to the sever  More on this topic when we talk security

Monitoring and Event Handling

 Use Events Be the First to Know
 Easy to set up  Know about problems before your phone rings  Fix problems before the boss calls you
 Make sure to log them, so he knows what you do

 Event notices make great justification tools for new servers!

 For more information

 BP407 What are Your Servers Trying to Tell You Now: The (Even) Easier

Route to IBM Lotus Domino Reporting & Logging -- Gabriella Davis

Power-off Recycle Devices

 When all else fails, sometimes you need to power cycle a machine from 3000 miles away  Inexpensive power modules can be commanded to recycle power with a 5 second power down pause
 Controlled through serial port  Include "watchdog" software

 Many devices on the market

 Some include remote shell access  Some include Web browser control

Heres what I use http://www.cpscom.com/gprod/ipn.htm

Developer Management

Sir, please step away from that Designer Client.

Deployment Policies
 These are a good thing, and you should have some.  Questions to answer with your deployment policies:
 Who decides when a database has been tested enough?  Who will be called when a problem is reported?
 Do you have a contact number for this developer?

 How will you know when the database is no longer in use?

More Deployment Policy Questions

 How big is the database expected to get?  What servers does it need to be on?  Is external replication required?  How volatile is the access control going to be?  What kinds of agent code will be running at the server?
 Server side java agents? Agents that call COM objects?  File System Access? ODBC or Connector LSX Use?  API Calls?

Do Not Modify the Domino Directory

 Nothing impacts performance more than changes to the Domino Directory  There are two critical view indexes in the Domino Directory
 $ServerAccess  $Users

 If the indexer is busy doing other things in that database, these updates will take longer  If these indexes are not up to date, authentication and access rights may not be granted to users

ava Agents Must Be Tested at Full Scale

 Multi-threading is so powerful, you can shoot yourself in both feet at once  Very easy and common mistakes in Java agents can kill production servers easily
 Unlike LotusScript, when writing Java agents programmers must call

"recycle()" on every object you instantiate, or their parent document

 In test, it is frequently possible to get away with simply recycling the

"session" object when the agent terminates

 In production, this kills servers when the agent handles a large number of

documents in a loop, among other things

 Yes, I know this from bitter experience

Restricted vs. Unrestricted Agents

 Unrestricted agents can do to things outside the scope of the agent itself
 Access the operating system  Access files on the server important ones  Reboot or shutdown the server

 If someone needs to run an unrestricted agent, you need to understand why

Security Management

The Five Pillars of Security

 Physical Server Security  Operating System / File System Security  Lotus Domino Server Access
 Certificates & Cross Certification
 Public / Private Key Certification  Cross Certification

 Server Access Settings

 Database Access The ACL  Document Access Reader Names

Notes Client Side Security

 Guard Your Certifier
 Dealing with a compromised certifier

 Assume Users have Designer

 It's easy to get  Obscurity is not Security

 Encrypt Workstation Data

 Escrow ID Files

 Preventing Workstation Copies

 Third Party Tool: dotNSF Tools noCopy www.dotNSF.com

 Client to Server Communication Encryption

Browser Access Security

 Obscurity is not security!
 This is the #1 issue on Web sites
 URL Hacking  NoteID Crawling  Common Word Crawling

/database.nsf/knownViewName/<insert word here>

 SSL Preventing Man in the Middle Attacks

 Creating an SSL Key Ring  Obtaining an SSL Certificate
 An authority unto yourself Are you trusted?  Buying an SSL Certificate

 Deploying an SSL Key Ring to Domino

Securing the Other Protocols

 Understand your ports
 If your server faces the internet, put a firewall in front of it  Many of the server tasks listen on a port, understand them or dont load

them. Particularly, LDAP and SMTP can give away a lot of valuable information if improperly configured

 If you dont need a protocol, shut it down

 If nothing is listening on a port, that port is secure. Well, mostly.

Password Guessing isnt Just Browsers!

 User's "Internet" passwords are frequently less complex than their Notes ID Passwords Use the tools to enforce complexity  It is now very common for hackers to "Name Guess" via POP3, SMTP, and even "Harvest" names from Web sites, e-mail addresses, and open LDAP ports  Once a name is guessed or harvested, POP3 or other protocols are used to guess passwords  With a name and password, spammers can use your server using an authenticated username

Mail Management

This is probably why many of you came here in the first place.

Notes Mail Routing

 Servers on the same Notes Named Network
 Should be able to find each other "by name" without connection documents

with TCPIP, this would be DNS

 Servers on the same "named" network route mail automatically; no connection

document is needed
 This is a "least cost" indicator to Domino's routing cost matrix

 Use this to your advantage

 Set up your named networks to reflect your network's faster and slower links.

Put only servers that have excellent connectivity on the same "Named Network"

Connection Documents
 Connection documents tell servers which are not on the same "Notes Named Network" how to find each other
 They're also used for replication, but we'll get to that later

Internet Mail Routing -- Turning off SMTP inside the Network

 If you turn off the SMTP Inbound Listener, local Windows clients which have been infected with a virus, worm, Trojan horse, or spy-ware application cannot send mail through your servers.
 This also eliminates accidental or deliberate use of your internal servers for

spam routing.
 Even if you require password access for SMTP mail sending, password

guessing is now quite common.

 If you disable SMTP Outbound on your servers, it will force the mail to route through your single gateway. In many cases this is a more secure method and provides greater traffic control on your network.

Using a Single Internet Mail Gateway

 Server Documents (all but the server that will route smtp):
 Set "SMTP Listener" to Disabled  Set "Routing Tasks" to "Mail Routing" but not "SMTP Mail Routing"

 Create a "Foreign SMTP Domain" Domain Document

 Route *.* to "OurFakeName"

 Create a Connection Document

 Type: SMTP  Source Server: The domino server with smtp  Destination Server: MAKE UP a name  Destination Domain: "OurFakeName"  Routing Task: SMTP Mail Routing

 This method means you dont even need TCPIP as a protocol on your other Domino servers, because the routing all happens using Notes RPC protocols to the one server with SMTP capability.

Single Internet Mail Gateway -- What Really happens?

 All the servers where SMTP Mail Routing is not a task, look for a route to send the mail.  These servers see that *.* goes to the domain "OurFakeName"
 That's the SMTP Domain Document's Job

 The router task on the servers see that one Domino server has a connection to the "OurFakeName" domain so they route the messages to that server
 That's the connection document's job

 The server which is SMTP Mail Routing Enabled receives the mail in its INBOX and knows how to send SMTP mail directly, so it does.

Standardizing on a Mail Template

 Beware of Customized Templates
 Prevents Update & Bug Fix
 Look at the update lists in each point release and note how many related to

small fixes in the mail templates.

 Serious Performance Issues
 More views means more view indexing work for the server.

 Limiting Design Access to Mail Files

 People are most likely to make "quick" (untested) updates to the design of their

mail file, considering it their own problem if they cause a problem. These people can take down your server.

 If you want additional features, look for "Packaged" alternative mail templates which are properly supported.
 openNTF.org has a very popular one, for example.

Managing Mail File Size SCOS

 Single Copy Object Store has been a feature for many years.  It DRASTICALLY reduces disk usage by keeping one copy of each file no matter how many different people have it in their mail files.  It's significantly better than it was, and with "Transaction Logging" and Domino clustering can be much more reliable than ever before.  It's still a single point of failure if you do have a problem, everyone is affected by the problem.

Managing Mail File Size (continued)

 Take Advantage of Archiving
 Archiving can be easily set up and managed through policies  Put Archives on different server, they're less frequently accessed and

have different load characteristics

 Impose Realistic Limits with Quotas

Managing Unwanted Mail

 Don't be a Relay
 In the "Configuration" document for your server not the Server document,

on the "Router/SMTP:Restrictions And Controls:SMTP Inbound Controls" Tab

 Deny messages from the following internet hosts to be sent to external

internet domains:(* means all) Set to "*"

 This is the Default on all recent Domino versions

 Hold Undeliverable Mail

 Don't send bounce messages Frequently, the mail never even originated

on your site and you're only adding to the problem

 Fighting unwanted mail is much more complex than this

 BP405 Controlling Spam Mail In Your Organization  BOF509 Keeping Up with the Spammers with IBM Lotus Notes and

Domino

Don't Give Away Address Information

 Verify that local domain recipients exist in the Domino Directory:
 Pros:
 Stops inbound SMTP messages send with dictionary style drops and name

guesses from clogging your router  Can make your site less attractive to spammers who get credit for "delivered" messages accepted by your server
 Cons:
 Makes it easy for spammers to test for valid names on your server

 Consider using this if you have another tool that can detect multiple failed attempts from the same source and ban those sources at the firewall.

Other Message Filtering Considerations

 Using Black Lists (aka Real-time Black Hole or RBL)
 Many "black lists" exist that you can use
 (e.g. bl.spamcop.net; sbl-xbl.spamhaus.org)

 Not 100% accurate  Read the lists website to understand their criteria for listing

 Using White Lists (aka "Known Good" addresses)

 Most mail you get, is from people you've communicated with already  New to version 7 of Lotus Domino, but part of several 3rd party tools for

some time

Mail Filtering Tools

 Third Party Tools
 User-Interactive Products like spamJam can be excellent because each

user decides individually what's wanted and what's not

 Appliance Solutions can be inexpensive and effective, but less user-

specific

 My Recommendations
 spamJam because users really like being able to interact with it  Barracuda for simplicity and price, this device works very well  ASSP Open source proxy, good but scale is uncertain

Signed Mail
 Signed mail to Notes users
 Your Public Key
 Use "Files-Security-User Security" to get it or copy it from your Domino Directory

person document

 Signed Mail to Internet users

 X.509 Certificates The modern standard for authentication
 Self Certifying

If you create your own certificate authority, everyone will always have to decide accept it as trusted Excellent alternative for internal company use  Buying Certificates or Certification Rights  Free Certification Network

Importing Your X.509 Certificate

 If you obtain a personal x.509 certificate, you can import it into your person document in the Domino Directory
 Open your Person Document  Select "Actions Import Internet Certificates"

 Once this is done, you can "sign" mail to be sent to users with Internet addresses

Verifying Signed Mail

 From Notes Users
 The Lotus Notes Public Key
 You must have their public key in your address book

 Verifying Signed Mail from Internet Users

 Accepting a Cross Certificate
 Do this the first time you get signed mail from a user  Call the user, make sure its them sending the message

Adding a Sender's Public Key to Your Personal Address Book

 While viewing, use "Tools Add sender to address book"
 Advanced tab, check to add "x.509 certificate"

Mail Encryption
 The Recipients Public Key is required  The Public Key is used to create a one-way cipher that can only be read with the private key and only the user has the private key, it's in their Notes ID file (or other file if a non-Notes user)

Obtaining a Recipient's Public Key

 Notes Mail users in your domain already have it in their "Person" document in the Domino Directory.  Notes Mail users in other domains must send it to you. They can copy it from their record in their Domino directory, or use the options in "Files Security User Security" to get it.  Users can also simply send you a "Signed" document, and you can "Cross Certify" them when you receive the mail. (You'll be prompted.)

Adding a Sender's Public Key to Your Personal Address Book

 While viewing, use "Tools Add sender to address book"
 Advanced tab, check to add "x.509 certificate"

Database Management

Deployment Policies
 Limit Designer & Manager Access
 On the fly changes cause most problems

 Use Database Access Groups to Delegate Control

 Create Groups that a database owner can manage
 Example: "SalesTools.NSF Editors"  Set the database owner to be the owner of that group

The Connection Document for Replication

 A connection document is required for replication even on the same "Notes Named Network"  A common error on the connection document is not changing the schedule to work around the clock. Default is 8am-10pm.  Keep in mind that following replication, the indexer may be very busy. Consider having replication occur prior to the start of the normal business day.

Database Deployment Policies

 Track Database Usage & Ownership
 Every Database must have an Owner  Every Database must have a Review Date

 Remove Outdated or Unused Databases

 Even unused databases can load the server  Old data represents a security, accuracy, and legal risk

Replication Topologies
 Avoid "Everyone Replicates with Everyone"  Map Network Choke Points

Creating a Redundant Hub & Spoke

 Two distinct local area networks or well connected individual networks  One high bandwidth connection between the two clustered hubs  Reduces traffic across the expensive long haul network

Client Software Management

Common Policy Settings

 Use policies to define ECL (Execution Control List) settings  Use policies to make sure users have the right replicas on the local workstations  Policies in version 7 can be much more rigidly enforced

Client Version Update Rollout

 Excellent for ROI No more touching the desktop  Reduces support due to version/template incompatibility  BP404 Best Practices in IBM Lotus Notes Client Deployment -Steve Sterka, David Via  ID117 IBM Lotus Notes Deployment Made Easy -- Jeff Mitchell, John Paganetti

Handling User Support

Delegating Admin Roles Safely?

 Version 6.x added granularity to "Administrator" access  Allows you to delegate specific areas of responsibility without giving complete control to junior administrators.  Using the administrator task, you can allow area managers to register users without giving them a certifier.

Admin Roles in Version 6.x

 Full Access administrators
 Able to leap tall ACLs; impervious to Reader-Names

 Administrators
 Use all the power of the administrator tool, but subject to database and

document controls

 Database Administrators
 Manage databases, but not the server itself

 Full Remote Console Administrators / View-only Administrators  System Administrators

 No database controls, but plenty of server setup access

 Restricted System Administrators

 Restricted System Commands

Limit Use of Full Access Administration

 Full Access Administration should only be used rarely, when a need to override ACL or ReaderNames is required.  Grant this only to specific ID files. Make the administrator switch to this ID file when needed.  Create an "Event" notification to notify management any time this level of access is granted.  Use encryption on database you dont want full access administrators to read.

In summary
 It's no longer a question of whether or not something can be done, it's a question of which is the best way to do it and why.  This presentation serves as a guideline, not a bible.  This has been a high to medium high level look at the features you should be using, with pointers to where to find more detailed information.

Were all Lotus professionals here, please ask your questions so others can here the answers. You may also contact me directly if you like. Please fill out your evaluations The latest copy of this presentation will also be available at my website: http://www.thenorth.com

Thank you for playing!

For those playing the home game, direct questions & comments to: Andrew Pollack andrewp@thenorth.com http://www.thenorth.com