Indexes and Performance

Supercharging Your Progress

Applications
About
Paul Guggenheim & Associates
❂ Working in Progress since 1984 and training 
Progress programmers since 1986
❂ Designed 5 comprehensive Progress courses 
covering all levels of expertise
❂ Developers of the Sharp Menu System and 
the S.M.A.R.T.S. sales force automation 
package
http://www.pgasmarts.com 
Goals of this Presentation
❂ Educate the audience on how Progress 
treats indexes
❂ Suggest ways to take advantage of the 
index rules and ways to avoid the 
pitfalls.
Index Performance Impacts
Sample Table and Indexes
Stuchrg Table
RECID Charge-no Student-ID Charge-date Charge-amt
1234 1 1 2/3/97 $50.00
3211 3 3 3/5/97 $100.00
2456 2 2 3/3/97 $40.00
3769 4 2 3/12/97 $25.00

Charge-no Index Student-ID-Charge-date Index

Charge-no RECID Student-ID Charge-date RECID
1 1234 1 2/3/97 1234
2 2456 2 3/3/97 2456
3 3211 2 3/12/97 3769
4 3769 3 3/5/97 3211
Index Benefits
1. Rapid record retrieval

2. Specified record order

3. Enforced uniqueness
Index Costs
1. Index maintenance overhead occurs 
during an add or delete of a record or 
modification of an index component.
2. Additional disk space
General Indexing Properties
❂ Bracketing affects rapid record retrieval
❂ Sorting affects specified record order 
Bracketing
❂ Using  an  index  to  read  a  subset  of 
records  from  a  table  based  upon 
conditions specified in WHERE, OF and 
USING clauses.
Sorting
❂ Reading records in a specified order 
may be accomplished by using an 
index. However, if it cannot be 
accomplished by using an index, then a 
sort table is built.
When to Create an Index
1. Unique index

2. Foreign keys

3. Reducing record reads

A. Date fields

B. Logical fields

4. Composite names

5. Description fields
Unique Index
1. Relational database design requires 
that there should be at least one index 
per table that can uniquely identify each 
record in that table.
• Progress doesn’t require an index to be 
created, in which case it automatically 
creates a default index in dbkey order.
– This index may or may not be chronological, 
since dbkeys can be re­used.
Foreign Keys
2. Create indexes for all foreign keys of a 
table.
• A foreign key is a key in a table that is a 
unique key in another table. This key 
represents a parent record to an existing 
record.
• Indexing foreign keys gives parent records 
fast access to all child records.
Foreign Keys
/* Example 1 */
for each student,
each stuchrg of student:
display stuchrg.
end.

❂ 1000 students
❂ 40 charges per student
❂ 40,000 student/charge records
❂ Using index Student­ID­Charge­Date: 41,000 record reads
❂ Not using index Student­ID­Charge­Date: 40,001,000 record 
reads
Reducing Record Reads
3. Create indexes on fields used for record 
selection.
• Since the smaller the bracket the more 
efficient the index, you should apply 
bracketing concepts.
Bracketing with Date Fields
❂ Index date fields
• Users like to view records for certain time 
periods. 
– Without an index on charge­date, the search 
would have to read every record in the table.
Bracketing with Date Fields
/* Example 2 */
for each stuchrg where charge-date ge 9/1/96
and charge-date le 9/2/96:
display stuchrg.
end.
Bracketing with Logical Fields
❂ Indexing can be effective with logical 
fields if the split between the “yes” and 
“no” values will not be 50­50, often 
creating a smaller bracket.
• Avoid using NOT in front of a logical field, 
as in Example 4, since Progress won’t 
bracket. Use field = NO instead.
Bracketing with Logical Fields
/* Example 3 */
for each student where graduated = no:
display student.
end.

/* Example 4 */
for each student where not graduated:
display student.
end.
Composite Names
4. Consider building composite indexes for 
name fields (e.g. last­name +
first­name), since users like to see 
records in alphabetical order.
Description Fields
5. Description fields should be word 
indexed, so that records can be 
accessed using specific keywords.
• Examples:
– Part­description in an item table
– Order­comments in an order table
Bracketing with Word Indexes
/* word.p */
def var wordvar as char format "x(30)" label "Words".
repeat:
update wordvar.
for each stuchrg
where student-charge-description contains wordvar:
display stuchrg.
end. /* for each */
end. /* repeat */
No Bracketing with Matches
/* matches.p */
def var wordvar as char format "x(30)"
label "Word Match".
repeat:
update wordvar.
for each stuchrg
where student-charge-description matches wordvar:
display stuchrg.
end. /* for each */
end. /* repeat */
Index Rules Overview
Manual Index Selection
❂ If USE­INDEX is used in the 
Record Phrase, the requested index is 
always used, regardless of its 
efficiency.
❂ If a RECID or ROWID is used in the 
WHERE clause, the record is always 
retrieved using the RECID or ROWID, 
unless USE­INDEX is used.
Index Rules Overview
❂ Factors Progress considers when 
deciding which index(es) to use are:
• Unique indexes
• Equality matches
• Range matches
• Sort matches
• Primary indexes
• Word indexes
Equality Matches
❂ An  equality  match  occurs  whenever  a 
field is equal to (=, EQ) an expression in 
an OF, WHERE, or USING clause.
❂ If the expression includes a reference to 
any  fields  in  the  same  record  buffer, 
then the equality match is not active.
Equality Matches

Record-phrase AEM
offering.offering-id =5 Yes

offering.season-no = current-season + 1 Yes

offering.course-id = course.course-id Yes

offering.to-time =offering. from-time + 3600 No

Range Matches
❂ A range match occurs when a field is 
compared with an expression in a 
WHERE clause using either:
• Greater than (>, GT)
• Greater than or equal to (>=, GE)
• Less than (<, LT)
• Less than or equal to (<=, LE)
• Begins with (BEGINS)
Range Matches
❂ BEGINS counts as two range matches.
• Why?
– Because BEGINS is essentially a GE value  
AND LT next­value. For example, BEGINS 
“B” is treated as GE “B” AND LT “C”.
Range Matches
❂ Not equals (<>, NE) does not count as a 
range match. Even though x<> 4 is 
logically the same as x<4 OR x>4, 
Progress treats it differently.
❂ If the expression includes a reference to 
any fields in the same buffer, then the 
range match is not active.
Range Matches
❂ A CONTAINS is neither an equality nor 
a range match.  It is a word index 
operator.
Range Matches

Record-phrase ARM
offering.dow < 3 Yes

offering.season-no >= current-season + 1 Yes

offering.offering-id > curr-offering.offering-id Yes

course.course-name BEGINS "P" Yes

offering.course-id <> 5 No

offering.to-time <= offering.from-time + 5400 No

Active Equality & Range Matches
❂ An active match is one that can be used 
by Progress to select an index, and 
select a bracket on that index.
❂ AEM is an acronym for Active Equality 
Match.
❂ ARM is an acronym for Active Range 
Match.
Active Equality & Range Matches
❂ Beginning with Version 7, an equality or 
range match is considered active if no 
conditions exist that either standalone 
or connected with the OR operator use:
1. A non­leading component
2. Any field in the same record buffer
3. The NOT operator or not equal (<>, NE)
Bracketing Rules
❂ If a leading component of an index has 
an active equality match, then it can be 
bracketed and the next component of 
that index is examined for bracketing.
❂ If a leading component of an index has 
an active range match, then it can be 
bracketed but remaining components of 
that index cannot be bracketed.
Bracketing Rules
Bracketing on cust-order Index
file:///D:/idxprfrm/PwrPnt/Simple Bracketing.BMP

file:///D:/idxprfrm/PwrPnt/AND Bracketing.BMP
Sort Matches
❂ A sort match is present whenever a field 
and not an expression appears in a BY
phrase.
❂ You  can  have  multiple  BY  phrases; 
each  counts  as  one  sort  match  as  long 
as they are sorted in the same order as 
they are in the index.
Sort Matches

Sort-phrase Sort Match

BY offering.offering-id Yes

BY offering.dow No

BY offering.syear BY offering.season-no Yes

BY offering.from-time / 3600 No
Multiple Index Rules
Multiple Index Rules
❂ Since a word index is such a powerful 
search mechanism, it is always used if a 
CONTAINS is found in the WHERE 
clause.
❂ Next, Progress determines whether the 
AND or OR connectors are used and 
applies the appropriate set of rules, if 
any.
Multiple Index Rules –
Word Index
❂ When at least one criteria utilizes a 
word index, at least the word index will 
be used, and other indexes are used if 
Rule 2 and Rule 3 are met.
Multiple Index Rules – AND
❂ More than one index can be used with 
AND when:
• All components of each index are involved 
in an active equality match.
• None of the indexes involved in the active 
equality match are unique.
Multiple Index Rules – AND
❂ When multiple 
indexes are used 
with an AND, the 
final results list is an 
intersection of the 
results list for each 
index.
Multiple Indexes – AND
❂ Why such strict AND rules?
• The shaded area in the diagram is 
relatively small, normally reflecting a 
relatively small number of records 
returned.  Therefore, it is usually more 
efficient to use a single index, rather than 
pull records efficiently from multiple 
indexes only to discard them.
Multiple Index Rules – OR
❂ More than one index can be used with 
OR when both criteria utilize at least the 
leading component of an index. Unlike 
AND:
• Only the leading components of any index 
must be used.
• Range matches as well as equality 
matches may be used.
Multiple Index Rules – OR
❂ When multiple 
indexes are used 
with an OR, the final 
results list is a union 
of the results lists of 
the indexes, with 
duplicates 
discarded.
Multiple Index Rules – OR
❂ Why are OR rules not as strict as AND?
• Because OR is a union and not an 
intersection, fewer records will tend to be 
discarded.  Therefore, the efficiency of 
reading records based on leading 
components has a much greater positive 
affect on performance.
Single Index Rules

1. Unique index with fewest components where all 
components are used in active equality matches
2. More active equality matches
3. More active range matches
4. All components used in an active equality match
5. More sort matches
6. The primary index
7. The first index alphabetically
Single Index Rules – Rule 1
❂ A unique index with all of its 
components used in active equality 
matches (AEMs) will be chosen, even if 
another index has more active equality 
matches, since using all components of 
a unique index in AEMs means there’s 
a direct hit on a single record.
Single Index Rules – Rules 2 to 7
❂ Rules 1 to 4 are designed to choose the 
smallest bracket of records.
❂ After bracketing, Progress looks for the 
minimal amount of sorting needed, or 
arbitrarily selects the primary index, or 
first index alphabetically. (Single Index 
Rules 5, 6 and 7)
Counting AEMs and ARMs
❂ When counting AEMs and ARMs (active 
range matches), remember to apply 
bracketing rules for counting.
• If the comparison is an AEM, then the next 
component comparison may be counted 
for either an AEM or an ARM.
Counting AEMs and ARMs
❂ When counting AEMs and ARMs (active 
range matches), remember to apply 
bracketing rules for counting.
• If the comparison is an ARM, then no 
subsequent component comparison may 
be counted as either an AEM or ARM.
Single Index Rules – Rule 4
❂ Rule 4 tells Progress that if two possible 
indexes have the same number of 
AEMs, then look to see if all 
components are used in AEMs for any 
of the indexes.
Single Index Rules – Rule 4
❂ Progress assumes that an index with all 
components chosen will be a smaller 
bracket than an index where not all the 
components are chosen.
Single Index Rules – Rules 5 to 7
❂ Rule 5 – Sort Matches
• Progress tries to minimize sorting
❂ Rule 6 – Primary Index
• If nothing is better and it is still available, 
Progress will use the primary index
Single Index Rules – Rules 5 to 7
❂ Rule 7 – First Index Alphabetically
• If the primary index has been disqualified 
when compared to other indexes, Progress 
chooses the first index alphabetically of the 
indexes remaining.
Counting Records
❂ A quick and easy way to count records 
is to use the SQL SELECT statement 
with the COUNT(*) function.
Counting Records
/*cntsql.p */
form with frame a.

select count(*) column-label ‘Last Name!Begins C’

from student where slast-name begins ‘C’
with frame a.

select count(*) label ‘State = CA’

from student where st-code = ‘CA’
with frame a.
Compiling with XREF
❂ You can use the XREF option of the 
COMPILE statement to check which 
indexes you are using.
❂ Look for lines in the cross­reference file 
containing the word “SEARCH”.
Compiling with XREF
/* oneidx.p */

for each student where slast-name = "Kramer":

display student-id slast-name sfirst-name.
end.

XREF reference
...SEARCH school.student name
Compiling with XREF
/* mltidx.p */

for each student where slast-name = "Kramer"

or country-code = "USA":
display student-id slast-name sfirst-name.
end.

XREF reference
...SEARCH school.student name
...SEARCH school.student country-code
Compiling with XREF
/* whlidx.p */

for each student where slast-name = "Kramer"

or city = "Chicago":
display student-id slast-name sfirst-name.
end.

XREF reference
...SEARCH school.student student-id WHOLE-INDEX
Applying Indexing Rules
/* idx2.p */

for each student where postal-code = "67890"

and slast = "Smith":
display postal-code student-id sfirst-name
slast-name.

end.
Applying Indexing Rules

Index name postal-code

Component slast-name sfirst-name postal-code
WIC N/A N/A N/A
AEM 1 0 1
ARM 0 0 0
ACE N Y
SM 0 0 0
Applying Indexing Rules
…SEARCH school.student postal-code

❂ Only the postal­code index is used 
since all its components are utilized, but 
only the first component of the index 
name is used.
Applying Indexing Rules
/* idx7.p */

for each student where phone = "3125882300"

and student-id gt 7
and student-id le 12:
12
display postal-code student-id sfirst-name
slast-name.
end.
Applying Indexing Rules

Index phone student-id (pu)

Component phone student-id
WIC N/A N/A
AEM 1 0
ARM 0 2
ACE Y N
SM 0 0
Applying Indexing Rules
…SEARCH school.student phone

❂ Phone is the only index used since 
student­id is involved in a range match, 
not an equality match.
❂ Phone has more AEMs than student­
id.
Applying Indexing Rules
/* idx9.p */

for each student where graduated = yes

or slast-name gt "W":
"W"
display graduated sfirst-name slast-name.
end.
Applying Indexing Rules

Index graduated name

slast- sfirst- slast- sfirst-
Component graduated
name name name name
WIC N/A N/A N/A N/A N/A
AEM 1 0 0 0 0
ARM 0 1 0 1 0
ACE N N
SM 0 0 0 0 0
Applying Indexing Rules
…SEARCH school.student graduate-alpha-list
…SEARCH school.student name

❂ Graduate­alpha­list and name are 
both used since their leading 
components are used in the WHERE  
clause and are connected by the OR 
operator.
Applying Indexing Rules
/* idx11.p */

for each student where student-id = 7

or student-id = 846:
846
display student-id sfirst-name slast-name.
end.
Applying Indexing Rules

Index student-id (pu)

Component student-id
WIC N/A
AEM 2
ARM 0
ACE N
SM 0
Applying Indexing Rules
…SEARCH school.student student-id
…SEARCH school.student student-id

❂ The student­id index is used twice and 
will be efficient in the way it is used.
• Note  that  this  example  uses  two  brackets 
on the same index.
Applying Indexing Rules
/* idx12.p */

for each student where student-id = 7

or student-id = 846
or sfirst-name = "Bob":
"Bob"
display student-id sfirst-name slast-name.
end.
Applying Indexing Rules

Index student-id (pu)

Component student-id
WIC N/A
AEM 0
ARM 0
ACE N
SM 0
Applying Indexing Rules
…SEARCH school.student student-id
WHOLE-INDEX

❂ The student­id index is used but will 
not be efficient since Progress must 
search the entire table to see if there is 
a student whose first name is “Bob”.
Applying Indexing Rules
/* idx30.p */

for each stuchrg where charge-date ne ?:

?
display student-id charge-date charge-code
charge-amt.
end.
Applying Indexing Rules

Index charge-date-charge-code charge-no (pu)

Component charge-date charge-code charge-no
WIC N/A N/A N/A
AEM 0 0 0
ARM 0 0 0
ACE N N
SM 0 0 0
Applying Indexing Rules
…SEARCH school.stuchrg charge-no
WHOLE-INDEX

❂ Because Progress treats NE as an 
inactive range match, this is the 
inefficient way to retrieve records whose 
dates are not unknown.
Applying Indexing Rules
/* idx31.p */

for each stuchrg where charge-date lt ?:

?
display student-id charge-date charge-code
charge-amt.
end.
Applying Indexing Rules

Index charge-date-charge-code charge-no (pu)

Component charge-date charge-code charge-no
WIC N/A N/A N/A
AEM 0 0 0
ARM 1 0 0
ACE N N
SM 0 0 0
Applying Indexing Rules
…SEARCH school.stuchrg
charge-date-charge-code

❂ Since the unknown value for date fields 
sorts high, when reading records 
Progress interprets LT ? as dates that 
are not unknown. 
• The moral of the story: use LT ? instead of 
NE ? for reading all records whose dates 
are not unknown.
Joining Records
❂ The rule is:  Join from smallest bracket 
to largest.
❂ Should records always be joined from 
parent record to child?
• Most of the time, yes, but not always. It 
depends on the brackets.
Joining Records
❂ With no WHERE conditions, there are 
fewer parent records than child records, 
so start with the parent record first.
❂ In join1.p, a student registers for more 
than one course, so read the student 
records first, before the registration 
records.
Joining Records
/* join1.p - from parent to child */

for each student,

each registration of student,
offering of registration,
course of offering,
each grade of registration:
display student.student-id
sfirst-name + " " + slast-name label "Name”
format "x(30)"
course-name format "x(20)" label "Course"
grade-name.
end.
Joining Records
❂ What if the question was: “Give me 
students from zip codes 40000 to 79999 
who received a grade of ‘A­’ or better 
from ‘Bowling Made Easy’.”
Joining Records
❂ The following are two ways we could 
join the records, join2.p and join3.p.
• There about 400 records within that zip 
code range, and about a dozen records for 
students with an ‘A­’ or better in bowling.
• Join2.p starts with the student table.
• Join3.p starts with course, offering, and 
registration to narrow grades down to a 
specific course.
Joining Records
/* join2.p - from student to registration */
for each student where postal-code ge "40000"
and postal-code le "79999",
each registration of student where
registration.grade-point ge 3.67,
offering of registration,
each course of offering where
course-name = "Bowling made easy",
each grade of registration:
display student.student-id
sfirst-name + " " + slast-name label "Name"
format "x(30)"
course-name format "x(20)" label "Course"
grade-name .
end.
Joining Records
/* join3.p - from registration to student */
for each course where course-name = "Bowling made easy",
each offering of course,
each registration of offering
where registration.grade-point ge 3.67,
each grade of registration,
each student of registration where postal-code ge "40000"
and postal-code le "79999":
display student.student-id format "9999" label "Stu#"
postal-code label "Zip" format "x(5)"
sfirst-name + " " + slast-name label "Name”
format "x(20)"
course-name format "x(20)" label "Course”
grade-name label "Grade".
end.
Dynamic Queries
❂ The index rules mentioned previously also 
apply to dynamic queries.
❂ The Index­Information method returns the 
same index data that the xref option does on 
the compile statement.
❂ This method may be executed before the 
dynamic query is opened, thereby preventing 
the execution of an inefficient query.
Dynamic Queries
❂ The query­prepare method sets the 
conditions before the index­information 
method is executed.
qh:query-prepare("for each student
where slast-name begins “s” by postal-code”).
Dynamic Queries
❂ The index­information method returns a 
comma­delimited list of indexes selected for 
the nth record buffer specified.
idxseled:list-items = qh:index-information(1).

❂ The above statement loads the indexes for 
the first record buffer for query qh into the 
idxseled selection­list.
Dynamic Queries
❂ The WHOLE­INDEX keyword is returned if a 
particular record buffer is inefficient.
Index Xref Tool
❂ The following tool is provided by Johan 
Forssblad from G4 IT AB.
❂ It loads the xref information into a database to 
allow for easy analysis.
❂ The tools shows for each program, each line 
of the program that has a record reading 
statement.
Index Xref Tool
❂ On each line, it show the records read and 
the corresponding indexes selected.
❂ Within the indexes selected, it shows the 
components used and whether they are 
bracketed or not.
Summary
❂ Now that you understand how Progress 
uses indexes, you can analyze the 
index usage of your applications for 
optimal performance.
❂ You will also be better equipped to 
design indexes for new applications.
❂ If you are not sure what index is 
selected, check the XREF listing.